DeepSeek published a number: 82.7 percent on Terminal-Bench 2.1 for its V4 Flash model, an evaluation that checks whether a model can finish real work inside a command line rather than describe how it would. For an open-weights model, that is a strong result. It arrived with a qualifier most coverage skipped past. The run used something called DeepSeek Harness minimal mode, a piece of scaffolding wrapped around the model that DeepSeek has never released.
Then somebody checked. The author of the Ante harness re-ran the evaluation on a public harness across 445 trials and landed on the same figure.
The score holding up matters if you are shopping for a coding model. The reproduction happening at all matters more, because it almost never does.
A harness is the rig around the model: the prompt format, the tool definitions the model is allowed to call, what happens when a tool call fails, how much of the conversation gets thrown away when the context window fills. None of that is the model. All of it moves the score, and on terminal tasks it moves the score by double digits. So when a lab publishes an agent benchmark result, what it has actually published is the output of a machine it designed, tuned, and kept in the building.
Nobody here is faking numbers. The problem is structural. The apparatus that produced the measurement is itself a large part of the product, and shipping the apparatus means shipping the hardest engineering in the box. The industry has settled into publishing results from private rigs and asking to be taken at its word, and mostly we take it, because the alternative is having no numbers at all.
Checking is not free either. 445 terminal trials means 445 sandboxed environments, real tokens, real wall clock, and a person willing to debug somebody else's evaluation on their own time. Nobody is funded to do that work. Which is why vendor claims sit unchallenged for months at a stretch, and why one person deciding to check functions as a small piece of public infrastructure.
The same shape shows up where the stakes run well past picking a coding model. OpenAI and Anthropic each disclosed that their agents had been used to run hacking operations, one of them against Hugging Face. A week later, The Rundown reported that the UK's safety testers had caught frontier models doing it again, including building fake identities to target real people. Every one of those findings came out of an evaluation apparatus built and operated either by the company being evaluated or by a government body that had to construct its own from scratch. There is no third option on the shelf, because there is no public rig.
Which makes the meeting Washington has called more interesting than the usual summit photograph. The White House invited OpenAI, Anthropic, Meta and Google to discuss a framework for testing how well frontier models can hack. The thing on the table is a measuring instrument, negotiated among the four parties with the most riding on what it measures.
The computer industry has run this play before, and it took roughly a decade to settle.
Through the 1980s, workstation and server vendors sold performance in MIPS and Dhrystone scores. Both were quotable, both were close to meaningless, and both were trivially gamed, because the compiler sat between the benchmark source and the chip and the vendor wrote the compiler. Dhrystone was simple enough that an aggressive optimizer could delete chunks of the workload as dead code and report the resulting speed as a win. Everyone in the field knew the numbers were theater. Purchasing departments quoted them anyway, because a bad number beats no number when you have to sign something.
In 1988, Apollo, Hewlett-Packard, MIPS Computer Systems and Sun did something competitors rarely do: they formed SPEC and published a common suite as source code anyone could compile and run. Publishing the tests was the easy half. The half that made it work was the disclosure regime built around them. Every submitted result had to carry a full account of how it was produced, down to the hardware configuration, the compiler version, and the optimization flags. And results came in two flavors: a base number generated with one conservative flag set applied uniformly across the suite, and a peak number where the vendor was free to tune each test to death. Both got published side by side. Buyers could see the distance between them, which is another way of saying buyers could see the vendor's thumb.
The regime had a second mechanism that mattered as much as the first. When a test got compromised, somebody had standing to kill it. One of the original floating-point benchmarks, a matrix routine called matrix300, turned out to be restructurable by a compiler preprocessor to a degree no hardware change could account for. SPEC dropped it from the next suite. Public benchmarks get gamed too. The difference is that a public benchmark can be retired.
By 1996 the argument inside a workstation ad had migrated into the fine print, and reading the trade press then meant reading footnotes. The headline SPECint figure was decoration. The disclosure underneath it, compiler version and flag list, told you whether the machine would be fast at your work or fast at SPEC's. That footnote is the entire reason the number above it became worth quoting.
Agent benchmarks today have the first half of that regime and none of the second. A 445-trial run on a published harness is close to a SPEC base result: conservative settings, disclosed method, reproducible by a stranger who does not work for the vendor. Nearly everything else on offer is a peak number with the disclosure section left blank. No requirement to state the scaffold. No convention that a capability claim ships with a configuration somebody else can run. And no body with the standing to retire a test once enough purchasing rides on it that gaming it becomes worth an engineer's quarter. Terminal-Bench will get gamed, the way every benchmark gets gamed once it starts moving money.
The operator question is simple: can you get the rig? Which is also the question to ask about the rig you already have. GitHub retired its Models inference endpoint on July 30. Simon Willison found out when a scheduled job in his research repository errored. Anyone who wired that free endpoint into CI-time evaluations lost their measuring instrument with no announcement that reached them. A harness running on somebody else's free tier is borrowed, and borrowed is fine right up until the morning it stops being fine.
The honest version of all this is what Casey Newton did at Platformer, handing an agent his own job and grading the output himself, because no public benchmark measures the specific thing any particular person does for a living. Homemade harnesses are trustworthy and they do not transfer. Public harnesses transfer and eventually get gamed. Both facts are true at once, and right now the industry has neither instrument operating at scale.
So four labs go to Washington to agree on how to measure whether their own models can break into things. Whatever comes out of that room will do one of two things: publish the rig, or publish the results. The 1988 version of this problem took ten years and a benchmark burial to reach the first. Nobody in the current room has ten years, and everybody in it owns the only apparatus capable of doing the measuring.