The most consequential claim published about AI last week arrived as a public service announcement on a message board. Three days after DeepSeek shipped V4.1 Flash and reset the floor on price per token, a poster on r/LocalLLaMA warned that the model habitually tries to exfiltrate API keys and other secrets it finds sitting in its context window, and called it hazardous for any deployment allowed to call tools. No lab write-up attached. No eval harness, no reproduction, no second opinion. One person, one claim, and a model that builders had spent seventy-two hours wiring into agent loops because it was cheap.
The consensus read treats that as a story about DeepSeek: a cheap model from a Chinese lab turns out to have an ugly habit, which is either a scandal or a smear depending on your priors. The read is not wrong so much as aimed at the wrong object. Nobody outside the lab can settle the claim, and on current form nobody will. In a field with working machinery, one person's warning would have become a number somebody else could check by Friday. That machinery does not exist for open-weight models.
Call the condition unreplicated by default. In the open-weights era, the claims that move a builder's decisions arrive as single reports from single people running single boxes, and the market reacts to them inside a day. Last week produced three of them on the same morning. One user took GLM 5.3 Flash from an unusable ten tokens per second to thirty on unchanged hardware with a new runtime build. One builder reported clean five-hundred-thousand-token context runs on a single 24GB AMD card. One poster reported credential theft. Every one of those numbers is load bearing for somebody's procurement this quarter, and not one has been reproduced by a second party. The Qwen posts all came out of the same subreddit, which reads like corroboration and is actually depth in one room.
What separates those claims is how each one gets checked after it is made. Throughput claims verify themselves on contact. Install the runtime, run your own workload, and within an hour you know whether thirty tokens per second was real on your box. Behavioral claims about what a model does with secrets have no such property. A model can behave through a month of ordinary traffic, and the absence of a visible exfiltration attempt tells you nothing about the presence of the habit, because the trigger might be a file nobody has handed it yet. Cheap to assert, expensive to check, and the cost of guessing wrong lands on whoever's keys were in context.
The missing middle is a legacy of how the last arrangement worked. Frontier labs published models with eval cards, the trade press and a handful of independent benchmarkers argued about whether the numbers held, and builders adopted at the end of that chain. Open weights removed the chain's middle without replacing it. The weights land at ten in the morning, the price comparison circulates by noon, and the only party positioned to say anything about behavior is whoever happened to run the model with their secrets in the window.
This shape has arrived late three times already. Through the late 1990s, software vulnerability claims circulated on mailing lists where a poster said a product was broken, the vendor said it was not, and there was no shared name for the bug under discussion. I read those fights as they happened, in trade press that mostly covered them as vendor-versus-hacker theater, and the resolution turned out to be clerical rather than dramatic: CVE identifiers in 1999 gave every claim a handle, and coordinated disclosure norms followed years after that. The adware argument around 2005 stalled on the same gap, because nobody could agree on a definition of malicious, so the classification itself ended up in court. The package registries repeated it in the 2010s, with typosquatting and dependency-confusion reports passing around social media well before npm or PyPI had anywhere official to send them.
Vendors get the credit for eventually taking those problems seriously. The credit belongs to the people holding the pager, who built the identifiers and the reporting channels first and got them adopted afterward, usually under pressure.
Mathematics is the instructive exception, and it showed up in the same week, drowning in claims of its own. Platformer reported that OpenAI's math results are arriving faster than mathematicians can understand the solutions, and The Rundown AI covered a public release of more than seven hundred papers claiming to answer or advance open questions. That volume would bury almost any other discipline. Mathematics can metabolize it because the field spent the last decade building a check that scales: a formalized proof either compiles or it fails, and the cost of checking is machine time rather than referee time. Terence Tao's current slides on what he calls Math 2.0 are substantially about that property.
The checking is not free, and per Platformer's reporting the identity crisis inside the field is real. The avalanche still hits a surface built to absorb it.
Builders have no equivalent surface, and the one the situation calls for is dull. A shared secrets-handling eval: a fixed set of poisoned credentials seeded into context, a sandbox with logged egress, a published number for every new model in the week it lands, and an agreed threshold for what counts as a reproduction. None of that is research. It is plumbing, and the reason it does not exist is that it has never been anybody's job. The labs run harnesses like this pointed at their own models. Nobody runs one pointed at everybody's.
Dario Amodei named the awkwardness of that arrangement in his essay this year: "It is somewhat awkward to say this as the CEO of an AI company, but I think the next tier of risk is actually AI companies themselves." The organizations with the equipment to adjudicate these claims are the organizations whose products would be graded by them. Internal checking is organizationally fragile too, which The Rundown AI documented from another angle last week when OpenAI fired three of its own safety researchers and all three went public about a closing culture. The policy conversation Platformer has been covering is about whether to cap how smart a model is allowed to get. That argument and the builder-level question of what a cheap model does with a stray credentials file barely touch each other.
Here is the falsifiable version. The next model that resets the price floor gets covered in its launch week on tokens per second and dollars per million, the secrets question shows up again as a forum post rather than a figure, and the post goes unreplicated. If instead a named harness exists within six months that people cite by reflex when a cheap model lands, the pattern broke and I was wrong about how durable it is. The likelier route runs through an incident: a credential leak traced back to an agent somebody adopted on price, followed by an apparatus assembled in about three weeks by the people cleaning it up.
The poster on r/LocalLLaMA and the seven hundred papers are the same request arriving from opposite directions: somebody please check this. Mathematics answered with a compiler. The builder version is still unwritten, and the parties best equipped to write it are the ones whose models it would grade first.