About 150 annotated traces before an agent earns autonomy

It takes about 150 annotated traces before an agent earns the right to work without a human checking every output.

Last week I wrote that Google paid $10 million for Spirit Airlines' internal traces, and that your agent logs are tomorrow's proprietary training data. A few people asked the question: fine, I'm logging everything. What do I do with it?

Here's the loop:

You put one agent in a harness. Isolated, observable, doing one job, say, drafting first-pass replies to support tickets.

At the start it's untrained. Every output gets reviewed by a human who knows what good looks like. That review is an annotation. You're building a dataset of your own expertise, one trace at a time.

Around 150 annotated traces, you split the data (not a magic number- Langfuse's eval guide puts a real regression set at 100+). A small cut, maybe 15 or 20 percent, becomes the examples that teach the agent. The rest becomes your golden dataset.

Then you train an LLM judge on what your experts marked as good, and you let it grade the agent's work. A popular bar is Cohen's kappa of 0.6 or better, on 30 to 50 labeled examples. You're holding three times that many labels. When it clears the bar, you have an eval built from your own annotated history instead of a vendor benchmark.

Once the judge is in place, the loop closes. Agent produces, judge grades, only the exceptions reach a human. And each correction becomes a new annotated trace, so the golden dataset keeps growing and the judge keeps getting sharper.

The harness kept getting called the moat. IMO it's commoditizing fast as OpenAI and Anthropic ship theirs as SDKs, and DeepSeek just open-sourced its harness under MIT and pulled 170K+ GitHub stars in its first week. When the machinery is free, the only scarce thing left in the loop is your annotated traces. No SDK will ever contain those.

The traces were always the moat. The judge is how you spend them.