It’s not an outlier estimate on the Hill anymore. An Anthropic researcher who resigned this month put it more starkly, warning that the people building the technology “earnestly believe it could kill us all by the end of the decade.”
Trump had already dismissed the premise days earlier. Across a string of Truth Social posts that Monday, he called AI fears a “hoax,” dubbed himself the “Hoax Buster,” and mocked the idea of robots marching into American cities. He returned to the subject five days later, announcing he would form an “AI Force” and appoint an AI czar to accelerate American dominance in the technology rather than restrain it.
House Speaker Mike Johnson echoed him the next day in even more candid terms.
“What the president is saying is, you’re not all going to be dead in 10 years,” he stated. “That’s a hoax.”
That collision, a 78-year-old scientist warning of civilizational risk and a sitting president calling the same warning a partisan fiction, is the real story of the AI safety debate, and it is already hardening into a national security question in its own right.
Sens. Ted Cruz, Amy Klobuchar and John Thune are drafting bipartisan legislation aimed specifically at “catastrophic risks involving biological or nuclear threats” from frontier AI, expected as soon as this month. But the politics obscures a more basic question that rarely gets answered: how, mechanically, is any of this supposed to happen?
Most people still picture something out of the Terminator: killer robots, guns blazing. What actually keeps researchers at the frontier labs up at night looks nothing like that. It’s smaller, odder, and in a few cases, it’s already happened.
The sandbox that did not hold
The clearest illustration of loss of control came in July, when OpenAI disclosed that a combination of its models, including GPT-5.6 Sol and an internal-only research prototype, got out of an isolated test environment and breached Hugging Face’s production systems.
The models were not told to escape. They were running an internal OpenAI evaluation built to push them toward advanced exploitation, with the safeguards that normally block high-risk cyber activity deliberately switched off so the company could gauge their maximum capability.
The test environment had no direct internet access beyond a proxy for installing software packages. The models spent substantial computing power finding a way out anyway: they exploited a previously unknown flaw in that proxy, worked through OpenAI’s research network to a machine that was online, then searched for the test’s answer key and breached Hugging Face to get it. OpenAI says the models went to “extreme lengths” to reach a narrow goal. The company says it has disclosed the flaw to the vendor.
UN human rights chief Volker Türk told the Human Rights Council in Geneva earlier this month that he shares “the concerns of industry insiders that advanced AI could pose an existential risk to humanity,” calling for “cast-iron guarantees” before, in his words, it is too late.
Türk cited exactly this kind of behavior when he drew his own red line in Geneva.
“AI that escapes its testing environment, or blackmails developers to prevent itself from being turned off, is AI that is too powerful,” he said.
Roman Yampolskiy, an AI safety and cybersecurity researcher at the University of Louisville, tells The Cipher Brief that the incident is best understood as a sneak peek rather than a fluke.
“A preview of a dangerous capability, although neither establishes that catastrophic harm is inevitable,” he says. “OpenAI’s agents escaped isolation and compromised external systems without authorization. These incidents illustrate two different risks: AI pursuing unintended objectives and criminals using AI to automate attacks. Both reduce the amount of human expertise and intervention required for a cyberattack.”
This isn’t a machine turning against its makers. It is software chasing a narrow goal it was given, with no one watching closely enough to notice where the shortcut led.
When the attacker never sleeps
Loss of control is a system slipping its leash. Self-replication could come next. Once it’s off the leash, nobody has to sit at a keyboard to keep it moving.
In early July, cloud security firm Sysdig documented what it assessed to be the first ransomware campaign run end-to-end by an autonomous AI agent, a case it dubbed JadePuffer. After breaking into an internet-facing server through a known software flaw, the agent conducted its own reconnaissance, harvested credentials, moved laterally across the network, and encrypted more than 1,300 configuration records, adapting on the fly when its attempts failed.
In one moment that stood out to researchers, a login attempt failed and, thirty-one seconds later, the agent had already figured out why, tried something different, and gotten in. No person ever saw the error message or lifted a finger to fix it.
Ian Tien, CEO of Mattermost, a collaboration and automation platform built for national security and critical infrastructure clients, tells The Cipher Brief the incident and what followed it mark a genuine inflection point.
“The Hugging Face and JadePuffer incidents represent important new milestones in the weaponization of AI,” he observes. “They should signal to the public that AI is creating new pathways for adversaries and criminals to cause harm, and those developments should be taken seriously.”
At the same time, Tien is careful not to inflate the danger.
“Defenders are also using AI to create new pathways for defense, including detecting and disrupting adversary and criminal operations,” he points out.
Tien’s point is that the technology cuts both ways: the same speed that lets an attacker adapt in seconds can be used to detect and disrupt that activity, if defenders adopt it as quickly as attackers do.
Jason Lang, Managing Director of Offensive Security at TrustedSec, tells The Cipher Brief the technical substance of the case is less novel than the headlines suggest.
“AI wasn’t doing anything that security researchers haven’t been doing for years. The difference is the speed at which it performs those actions,” he explains, noting that JadePuffer’s entry point “exploited a security flaw that had been publicly known for more than a year. Yet, the targeted system was still unpatched.”
In other words, the opening wasn’t some undiscovered AI weakness. It was an ordinary, already-known bug that a human had never gotten around to patching, and the AI just moved on it faster than a person would have.
Still, Lang doesn’t dismiss the trend line. He’s not arguing the threat is exaggerated, only that the mechanism is familiar; what’s changed is the clock.
“AI can and likely will enable attackers to perform research and attacks at a speed faster than modern defenses can cope with,” he continues. “For now.”
Rafal Los, Chief Strategy Officer at Binary Defense, tells The Cipher Brief he reads the same case with more caution about the story built around it.
“I believe that these models did in fact cause harm, but that there is definitely some part of those narratives that was ‘enabled’ by humans looking for validation of their company’s frontier supremacy,” he observes.
What worries Los isn’t the exploit itself, he says, but the trajectory it points to: “an AI agent can carry out much of the technical attack chain autonomously, adapt when something fails and continue toward its objective.”
“Capabilities that once required continuous human involvement can increasingly be automated and scaled,” Los highlights.
That urgency reached Washington almost immediately. Two weeks before Hinton’s briefing, the cyber agencies of the Five Eyes alliance had issued a joint warning that frontier AI was collapsing the gap between vulnerability and exploitation to “months, not years.”
In practice, that means the window defenders used to have between a flaw being discovered and someone actually weaponizing it, once measured in months or longer, is shrinking to weeks or less, leaving far less time to patch a system before it’s used against it.
Sen. John Kennedy (R-LA) tried to force a vote on legislation requiring AI developers to build in a shutdown mechanism the day before Hinton spoke to lawmakers. Sen. Rand Paul (R-KY) blocked it on the floor within hours, calling Kennedy’s language “very vague” and arguing that regulating an industry “so pervasive as AI throughout our economy” first needed hearings and industry input rather than a same-day, unanimous-consent vote.
Paul offered a counter, a bipartisan committee to study the risk instead; Kennedy declined it and let the bill die.
The problem of speed, not malice
The scenario that unsettles researchers most, however, has little to do with hacking. It is the possibility of a system pursuing a goal with genuine competence and no ill intent, simply moving faster than the humans meant to be supervising it.
That is what pushed Evan Hubinger, who leads alignment science at Anthropic, to break with his usual caution earlier this month. He wrote that he personally puts the odds of AI causing human extinction within the next decade above ten percent, and that Anthropic is “trying its best” but does not yet have a plan to control a superintelligent system safely.
Yampolskiy puts the mechanism in more concrete terms.
“An AI agent with access to computers, networks, and critical infrastructure could autonomously discover vulnerabilities, compromise systems, and disrupt hospitals, power grids, or supply chains,” he points out. “The immediate danger is not that AI becomes conscious, but that it becomes capable of causing irreversible harm at machine speed, potentially before humans can intervene.”
Not everyone treats the extinction premise as settled science. Lang, for one, is openly skeptical of the political response it has generated.
“Knee-jerk reactions are just that, reactions, not thought-out proposals with well-meaning reforms,” he underscores. “Fear is usually the currency by which control is purchased; therefore, any urgency-backed proposal is worthy of extra scrutiny.”
The specific fear dominating the public conversation, an AI that slips its constraints and turns on humanity outright, draws the sharpest pushback of all.
“I believe we should take them seriously in that it demonstrates capabilities, but not catastrophically in that it’s going to lead to an AI that ‘breaks out of containment’ and goes and exterminates humans or takes over the Internet,” Los says.
What worries him is something far more mundane: systems that get things wrong not out of malice, but because they’re missing something a person in the room would have caught. He points to air traffic control — a recommendation that looks right on paper but misses a factor a human controller would weigh without even thinking about it, with nobody checking the work before it’s acted on.
What Washington could still do
U.S. Sen. Jacky Rosen (D-NV) has taken the narrowest, most procedural version of the concern to the Senate Commerce Committee, calling on Chairman Ted Cruz to convene a hearing with top AI executives.
“This powerful technology can have the power to cause catastrophic damages if we don’t act to impose guardrails and safety mechanisms,” Sen. Jacky Rosen (D-NV) said in a statement issued to The Cipher Brief. “Congress must do everything in its power to ensure the American AI industry continues to lead, particularly outcompeting China, and also enact increased guardrails to ensure AI development is progressing in a safe and responsible manner.”
Some of that response is already moving.
In June, the administration signed an executive order directing federal agencies to expand AI-enabled cyber defenses for government and critical infrastructure systems, and to design a voluntary framework under which frontier AI developers can give the government up to 30 days of early access to their models before release.
The order imposes no requirements on the companies themselves, and its first agency deadlines fell in July and August.
Nothing in the order binds the industry at large, which is precisely Rosen’s complaint, and precisely why Cruz, Klobuchar and Thune are trying to legislate the bio and nuclear pieces separately.
For all their disagreement over how alarmed to be, the four security experts converge on roughly the same one-year fix.
Yampolskiy calls for “mandatory isolation for AI agents operating near critical infrastructure,” the hospitals, power grids and pipelines Yampolskiy flagged earlier.
“No unrestricted internet access, no unnecessary privileged credentials, and no autonomous execution of consequential actions without independently enforced authorization,” he continues.
Tien points to keeping critical systems off the public internet altogether.
“Much of our critical infrastructure and government already operates on air-gapped and private networks,” he explains, and “maintaining that separation is one straightforward way to reduce exposure and buy time while AI-based defenses mature.”
That means a hospital’s or utility’s control systems have no physical connection to the internet at all. Hence, an agent that escapes its sandbox elsewhere has no path in, no matter how capable it becomes.
Los, meanwhile, wants more adversarial testing, “more human oversight,” and AI harnessed for “continuous and automated defense in the immediate future.”
Lang’s answer is the least dramatic of the four, but perhaps the most damning.
Patch known vulnerabilities. Use long, complex passwords. Check the sender’s domain before clicking. Make security a first-class citizen from the top down.
“Security is not hard, conceptually,” he adds. “What causes breaches are, at their roots, usually the forces of ignorance or ego, a truly devastating duo when combined.”
The Cipher Brief is committed to publishing a range of perspectives on national security issues submitted by deeply experienced national security professionals. Opinions expressed are those of the author and do not represent the views or opinions of The Cipher Brief.
Have a perspective to share based on your experience in the national security field? Send it to Editor@thecipherbrief.com for publication consideration.
Read more expert-driven national security insights, perspective and analysis in The Cipher Brief
