AI Is the Unit Test, People are Integration Tests

The claim

When you work with an AI, you are not responsible for the building. You are responsible for the conceptualization and for owning the product design. All the talk about taste, about needing to bring experience to the table, is really about one thing: determining whether the AI actually built what you wanted.

Without taste, genius is only sublime folly

That makes you the integration test. The AI is more like the unit tests.

Why a good spec isn't enough

The obvious objection: if the skill is knowing what you want, isn't this just "write a good spec?"

You can write what you believe is a good spec and still not know whether it resonated, whether it was actually what you wanted, until you see the thing work. How many times have you built something only to realize it wasn't what you wanted at all? The spec is a hypothesis. The running thing is the experiment. Without an experiment, you can't separate imagination from reality.

It doesn't matter how beautiful your theory is, it doesn't matter how smart you are. If it doesn't agree with experiment, it's wrong.

What AIs are good at

AIs are extremely good at the unit-test level. They build things that look like what you asked for. Even if you're vague, they'll produce something that satisfies your criteria. "Build me a game," and it will. Building the game you want is a very different proposition.

Evaluation becomes the bottleneck

If the human is the integration test, the constraint becomes how well a person can actually run that test. There are optimizations here. Years ago I read about the importance of modifiable testing harnesses: if you can change the input parameters, you can measure the edges of the output, and you get insight you wouldn't get otherwise.

The same applies to products. Like all testing, some value shows up at the edges. Unit testing works best when you think through the weird cases; evaluating a product works best when you think about users with different wants and needs and probe toward those edges. Figuring out how to explore the space you're developing in is now part of the requirement. It isn't the only thing, but proper evaluation requires every tool at your disposal, including tools that let you measure and probe the edges of your problem.

A concrete example

I'm reproducing a game I played as a kid. It has physics and controls, and making it feel like the original is part of the goal. I asked for what I thought I wanted. Gravity was wrong. Thrust was wrong. I started iterating with the AI on the parameters, but there were too many to evaluate one prompt at a time.

What worked was building a small user interface that let me tweak the values myself and feel the result immediately. Turn the knobs until it's right.

The recursive part

Here's what's interesting: the AI can build the harness too. So, you get a recursive loop where AI builds the thing, AI builds the instrument you use to judge the thing, and the human supplies the knowing when it feels right.

That shared experience of building harnesses together to decide whether the feel is correct is probably the more important connection between people and AI when it comes to designing things.

To understand recursion, one must first understand recursion.

This might not last forever

Can the AI run the integration test too? They can, and they're trying harder all the time. We treated code as the bottleneck and evaluation as the thing other departments did. Deciding whether code is correct, and especially whether a product is right, is an entire industry. Several, once you count product and marketing.

The deeper problem is the signal. Yes, you can look at all the code ever written and judge it by how it fared. You can look at products and whether they succeeded. But the noise is enormous: did it launch at the right moment, were people open to it, were the politics right? It's the stock market, but for how people value a thing. And because the system feeds back into itself, it's too dynamic to evaluate. Nobody can reliably say what a self-reinforcing system will do.

The obvious objection is that this applies to humans too. My "feels right" is just as contaminated by timing and fashion. If nobody can predict the market, why does the human get to be the judge?

That's the best argument I've heard that AI will take this job as well, and it's where people are going to aim over the next year. If a thing can be done by people given the same opportunities, it should be doable by AI, faster and with more nuance. So maybe this post is just the writing on the wall for today.

Humans are the integration test for now, not because taste is sacred, but because no one has yet figured out how to pull a training signal out of a market that reacts to itself.

Thanks to Steven Olson for his insights on this topic.