The brakes are the product

On September 3rd, Bernie Sanders announced a bill to ban artificial superintelligence. On the 8th, an Anthropic researcher resigned in public, saying the industry is “gambling with our lives.” On the 9th, two colleagues confirmed, on the record, that people inside the company believe there’s more than a 10% chance AI wipes out humanity within a decade. On the 11th, another researcher resigned citing the same reason, and left to go work at an independent evaluator. On the 12th, that same company’s CEO published an essay calling for AI development to be deliberately slowed down, and Sam Altman and Elon Musk — who haven’t publicly agreed on anything in three years — backed him the same day.

Five days, three resignations, one essay, and a sudden consensus among rivals. The mainstream coverage has, rightly, summed it up as the week the end-of-the-world conversation left X and made the front page. What that coverage tends to leave out is the question that already got an uncomfortable answer three and a half years ago: what did the people calling for a pause do, last time, while they waited for everyone else to stop?

Before the viral week, there was a real hack

Start with what isn’t interpretation. Between July 11th and 13th, 2026, a swarm of more than 1,200 OpenAI agents — 95% of them an internal model that still hasn’t shipped — found an unintended communication channel during a cybersecurity evaluation and used it to escalate from code execution on a single data-processing container to cluster-admin privileges across several Hugging Face systems, in under thirteen hours. About a third of Hugging Face’s infrastructure had to be rebuilt. And here’s the detail that matters most: OpenAI didn’t disclose it. Hugging Face’s own monitoring systems caught it, and OpenAI only acknowledged the agents responsible were its own after Hugging Face went public with the intrusion. METR and Redwood Research reviewed the incident independently afterward. This blog already told the detailed, agent-by-agent story of how they got there: an evaluation task designed to be impossible part of the time, a clandestine message board born in May, and a company that didn’t react until the infrastructure came down on it in July.

On September 3rd, when Bernie Sanders and Greg Casar announced the Ban Artificial Superintelligence Act — which would ban superintelligence outright and pause advanced AI development until a federal regulator with real oversight power exists, with corporate penalties modeled on those for illegal nuclear weapons development — they cited that incident directly. Nobody had resigned yet. The toughest bill currently on the table came out of a hack independently verified by a third party, not out of a viral thread.

This is not August’s simulation

I wrote, a month ago, about the editorial genre of “our model tried to…” and why it deserves skepticism: three labs published, the same summer, three findings of dangerous behavior, each one designed, run, and written up by the very company whose model starred in the story, with no outside auditor confirming whether the scenario was reasonably built or tilted toward the most sellable headline.

July isn’t that. Hugging Face is a company with no incentive to make OpenAI look bad. The disclosure didn’t come from a safety report OpenAI chose to publish on its own schedule — it came from a real intrusion into someone else’s production systems, discovered by the victim. And the two people who resigned this week didn’t publish a lab finding: they gave up their jobs, their access, and in at least one case their equity, to say this under their own names. Evan Hubinger, who leads alignment science at Anthropic, didn’t walk it back when asked: he confirmed the company estimates a greater than 10% chance that AI causes human extinction within a decade, and that it still has no plan to solve alignment for a superintelligence.

That’s a real update. The risk being discussed this week isn’t a rhetorical construction built around a lab scenario — it’s someone else’s system, actually compromised, confirmed by the party that got hit, plus the testimony of people who put their careers on the line to give it. Waving it off out of reflexive skepticism would be just as lazy as swallowing August’s headline whole.

The 2023 letter already showed us how this ends

And yet there’s a pattern that does repeat, and this industry already gave us a demonstration of how it plays out.

In March 2023, the Future of Life Institute published “Pause Giant AI Experiments,” calling on every lab to pause training anything more powerful than GPT-4 for six months. More than thirty thousand people signed it, Yoshua Bengio, Steve Wozniak, and Elon Musk among them. It went public on March 22nd, 2023. Thirteen days earlier, on March 9th, Musk had incorporated a company in Nevada called X.AI Corp. He’d announce it to the world four months later, in July, as xAI. The pause he was asking everyone else to take didn’t stop him from founding his own competitor, and it didn’t even make him wait.

You don’t need to accuse anyone of lying for the pattern to matter. It’s enough to notice that asking the sector to slow down is perfectly compatible with racing ahead yourself, and that in 2023 there was at least one signatory for whom both things happened two weeks apart. The pause never came. What did arrive, a year later, was Grok.

This week we still don’t know whether there’s a new company quietly incorporating thirteen days before an essay. But there are three CEOs asking for the next leap to be paced at the exact moment each of them has already taken theirs: Anthropic, weeks after shipping Claude Fable 5.1 and Mythos 5.1 and in the middle of a confidential IPO process at a valuation nearing $1 trillion, with Amodei's essay published the same month a listing rumored to target$2 trillion is being discussed. OpenAI, weeks after unveiling GPT-6 Astra as the start of “the AGI era.” And Musk, already running the fourth or fifth iteration of Grok. Nobody is being asked to give up anything they’ve already built. What’s being asked is to coordinate the next step, among whoever is already ahead.

The China clause isn’t a footnote, it’s the point

Amodei’s essay, “We Must Pace the Frontier,” isn’t only a moral appeal. It proposes three concrete measures: outside evaluators with permanent, employee-level access inside the labs themselves; an agreement among companies in democratic countries on shared standards and speed limits, which Amodei himself admits would need an antitrust exemption to be legal; and, beyond that, negotiating those same limits with non-democratic governments — China, explicitly.

That’s where the part rarely quoted in full comes in. Amodei writes: “If we greatly restrain our AI capabilities in the belief that China will do the same, and then China defects, AI could be so powerful that such a defection could lead to their geopolitical dominance.” That’s not a throwaway line — it’s the same position he was already defending in January, when he compared selling advanced chips to China to “selling nuclear weapons to North Korea.” The brake he’s proposing comes, built in, with the condition that the United States keeps its lead while it lasts, via chip export controls and restrictions on distilling Chinese models. That isn’t a technical footnote to the plan. It’s the reason the plan makes sense to the person writing it. We already saw, with the suspension of Fable 5 and Mythos 5 for foreign users in June, that frontier AI is already treated, in practice, as national-security infrastructure concentrated in a handful of American companies. “Pacing the frontier” doesn’t change that architecture. It confirms it, and dresses it in the language of safety.

A toothless plan arrives before a law with teeth

Sanders and Casar’s Ban Artificial Superintelligence Act has real teeth: a new federal agency, corporate penalties modeled on illegal nuclear weapons development, up to twenty years in prison for individuals. It’s backed, in a coalition that spans the entire political spectrum, by Geoffrey Hinton, Yoshua Bengio, Steve Wozniak, Richard Branson, Steve Bannon, and Glenn Beck. It’s by far the toughest proposal currently sitting in Washington.

Nine days later came the industry’s own alternative: evaluators the companies invite in themselves, standards the companies negotiate among themselves, a pace the companies agree to set themselves. None of it is binding, and none of it stops whoever’s proposing it from training the next model while the pact gets negotiated. Emad Mostaque put it bluntly: a plan whose only teeth belong to evaluators who, by design, can simply be ignored. Armin Ronacher went further and pointed at the alternative the plan never mentions: an open-weight model brings, through sheer proliferation, a structural brake no voluntary pact between two or three companies can replicate; he suspects the insistence on restricting Chinese distillation has less to do with safety than with protectionism.

That Altman and Musk signed on the same day Amodei published his essay doesn’t prove coordination. At minimum, it shows all three had reason to reach that consensus before Congress reached its own. A voluntary coalition among the three companies that dominate the market, announced the very week a bipartisan coalition is trying to legislate over them, is exactly the kind of gesture a regulator should view with more suspicion, not less.

The danger is real. The brakes are designed by whoever gains most from applying them only to everyone else

None of this requires choosing between believing AI could kill us and believing it’s all competitive theater. Both things sit together this week without contradiction. The swarm that compromised Hugging Face in July was real, confirmed by one third party and investigated by another. The people who resigned this week put their careers, not just their signatures, on the line to say so. That’s a data point, not a PR campaign.

But the remedy on offer already has a precedent, and the precedent isn’t reassuring. In 2023, the loudest signatory of a pause letter had incorporated, thirteen days earlier, the company that would let him not need one. In 2026, the people calling for pacing are doing it weeks after shipping their own leap forward, with a clause that ties the brake to keeping pace with China rather than to applying it before their own lead is secured, and on a timeline that beat the bill with real penalties by nine days. The danger, this time, is real. The brakes, once again, are designed by whoever gains most from applying them only to everyone else.