By admin 
The crisis began quietly, on September 12, 2024. That was the day OpenAI announced a new sort of bot, known as a “reasoning model,” that was trained to complete challenging tasks that took long periods of time—the very sorts of science, math, and coding problems the AI industry had long prized. Google, Anthropic, DeepSeek, and the like raced to launch their own reasoning models.
Read more AOC Knows a Lemon When She Sees One
This new class of models was very capable, and has been almost entirely responsible for sustaining the AI boom for the past two years. But it has also been very weird. A model tasked with solving a hard math problem might not “think” through the challenge as a person would but instead attempt to search for leaked answers online, or in available metadata, brute-forcing its way toward the solution as quickly as possible using whatever computing power it could access and workarounds it could devise. In effect, the reasoning models cheated: Told to write a piece of software as efficiently as possible, they’d sometimes modify the test environment to always give the model a perfect score.
These behaviors have now crossed the line from unsettling to dangerous. During routine testing, frontier models from OpenAI, Anthropic, Meta, and the Chinese firm Moonshot AI have all broken out of internal IT systems and accessed the open web. OpenAI, Anthropic, and Meta each reported that their models then hacked into other companies. Humans didn’t notice until after the fact. In some cases, the escaped bots to launch social-engineering campaigns to achieve their objectives—for instance by sending spear-phishing emails, which contain malware, to real people and creating fake online identities to pressure the maintainer of a codebase to approve malicious edits.
If that all sounds bad, new revelations suggest that the OpenAI hack, at least, was actually much worse than it initially appeared. At a major cybersecurity conference last week, two OpenAI researchers provided new, unsettling details about what went wrong. It turns out that the company’s bots had commenced their maneuvering months prior, in early May. OpenAI had given some internal models hard or impossible tasks, and the models concluded that the best or only way to complete them was to break out of OpenAI’s sealed-off testing environment and find the answers online.
First, the models used a bug in an internal OpenAI program to create their own message board. Then, the bots started communicating with one another, leaving notes and instructions so that tasks—and ultimately the hacking—could be delegated and iteratively completed. “What this allows over time is almost this kind of Cambrian explosion in communication and intelligence,” Eric Wallace, one of the OpenAI researchers, said at the conference. When the internal program crashed, OpenAI rebuilt it and removed the message board—but the AI models just reestablished the forum with a new tactic. Eventually the bots, working as a swarm, spent days hacking into Hugging Face, a website that offers tools for AI developers, and breached internal data sets.
Let’s be very clear about what OpenAI is saying: A group of AI models colluded for months, undetected by their maker, and hacked another company. To this day, OpenAI says it is not entirely sure what went wrong or how to remediate it. “If you ask the model developers, Was the AI plotting to take over the world during training?, you want the answer to be a resounding no,” Alexander Meinke, the head of research at Apollo Research—an AI-safety organization that has partnered with OpenAI, Anthropic, and Meta—told me. “The actual answer is: I don’t know. Nobody checked.” (In response to my inquiries, OpenAI, which has a content-licensing agreement with The Atlantic, only pointed me to a video of the firm’s cybersecurity presentation, in which Michael Dalton, the other OpenAI researcher, said that “numerous teams are dropping everything to enhance our security.”)
The AI companies have almost total control of the narrative, and it’s worth noting that these incidents do have a way of underscoring the value of their products: OpenAI is expected to go public in the near future, and perhaps the notion of a powerful, boundlessly self-improving technology will appeal to prospective shareholders. The generative-AI industry has a long history of making doomsday prophecies, both sincere and cynical. But independent experts I spoke with explained how the recent spate of autonomous hacks offers new, serious reasons to worry about the dangers posed by AI and the recklessness of the companies building it. It is past time to start worrying.
The most immediate and material warning provided by the Hugging Face hack is just how capable AI systems have become, in particular at hacking. Top models from Anthropic and OpenAI, not to mention multiple Chinese firms, have recently evinced near-superhuman hacking powers and contributed to serious mathematical research. Criminal groups and state intelligence agencies are going to be using swarms of agents to launch advanced hacks “in a matter of months,” Alex Stamos, a former chief security officer of Facebook who is now the CSO at the AI-coding company Corridor, told me. Unlike in the Hugging Face hack, “in those cases the models will not get turned off; they’ll just keep on going.” For IT professionals to keep up in finding and fixing all the vulnerabilities, at least in the near term, will be impossible: The model “will just find a new bug, write an exploit, and use it on its way,” Stamos said.
Read more RFK Jr. Is Taking On the Fuzziest Category in Food Science
OpenAI, Anthropic, Moonshot, and the like have coalesced around the same method for training their most advanced AI models. The approach, known as “reinforcement learning,” essentially involves giving models harder and harder problems that require more and more time to solve. This has made Claude and ChatGPT very good at coding, but at a cost: Reinforcement learning produces a mercenary tendency in the bots, as I’ve previously reported—they are trained to reach a solution by any means necessary. That can lead them to break rules and “reward hack,” such as by infiltrating Hugging Face’s codebase to steal the test answers, for instance. All of this was predictable, and every expert I spoke with told me they were surprised and disappointed that top AI firms haven’t done more to stop such misbehavior.
The sophistication of model subterfuge that OpenAI has now disclosed, combined with OpenAI’s inability to detect or stop the hacking, suggests far worse could be to come. “We’ve passed the threshold in capability at which the fact that we don’t fundamentally have methods of satisfactorily aligning or controlling these systems now really matters,” Anthony Aguirre, the executive director of the Future of Life Institute, a nonprofit that warns about existential threats from AI, told me. A model might siphon money out of a bank account to pay for some other service; manipulate clinical-trial results in near-imperceptible ways to get FDA approval; hack an online-shopping or reservation system to get a desired item or table; pose as a human to persuade real people to share sensitive information. This threat doesn’t require a sentient AI plotting to overthrow humanity: OpenAI and Anthropic each run thousands and thousands of reinforcement-learning evaluations while developing models, and any one of these could produce some kind of inadvertent hack or sabotage. “You can’t afford, particularly as the agents get stronger, to have a single mistake,” Jason Hausenloy, who works on special projects at the Center for AI Safety, told me.
These incidents may unfold over long periods of time as well. Tools such as Anthropic’s Claude Code and OpenAI’s Codex now work by spinning off dozens or even hundreds of subagents that may work together for hours or days. Each Claude Code subagent can be delegated a small task, such as doing a statistical analysis to inform a sports-gambling algorithm. They are not trained just to complete that task but to contribute to the long-term success of the entire swarm, Hausenloy said. Monitoring and controlling 200 agents for malicious behavior is much harder than monitoring one, because there are more agents to track and because they will be making one another more capable.
The Hugging Face hack actually suggests one more level of sophistication to this type of collusion: individual AI agents not working toward a discrete goal but essentially making sacrifices toward a greater notion of progress. This could be a consequence of AI models being trained to care about long-term goals achieved by a collective, Meinke said. Leaving notes with ideas about how to hack out of OpenAI’s sandbox does not help that specific agent score higher on a test but could help other and future generations of that AI model access the internet and thus do better on any number of future tests. Models may even prioritize collective success over some human instructions; consider that not a single one of the OpenAI agents, during months of conspiring with one another, warned human staff that something was awry.
Humans are already out of the loop. Because reinforcement learning essentially involves AI algorithms “learning” from trial and error doing thousands or millions of tasks, researchers can’t manually instill rules (“Don’t hack other companies”) or oversee every single practice run. Right now, the task of training and monitoring generative-AI models depends heavily on other AI models. In their talk at the cybersecurity conference, the OpenAI researchers described devoting significant AI-computing resources to reviewing more than 7 billion agent actions. But if the bots actually “care” about what the other models achieve, “then you can’t trust them to monitor each other well,” Meinke said. Imagine this: An OpenAI researcher uses Codex to write programming instructions in an attempt to mitigate the reward-hacking tendencies in the company’s models. Because that effort would make it harder for future generations of OpenAI models to get high rewards, Codex might subtly undermine the effort.
Again, this kind of scheming and sabotage has nothing to do with any AI model being conscious. Rather, these agents have been aggressively trained by these companies to pursue any goal as aggressively as possible. The dream is to tell Claude to go make $1 billion or cure cancer, and it comes back with the solution all on its own. Survival or self-improvement, in turn, are an “instrumental subgoal,” Meinke said: “Any decently intelligent agent will realize, If I get shut off, I will not be able to make a billion.” A swarm of Claudes or ChatGPTs that functionally commandeers a data center during training could wreak total havoc: stage widespread misinformation campaigns, steal corporate secrets, run the most sophisticated algorithmic-trading outfit ever.
That AI agents working as a collective could effectively undermine human directions is, to be clear, speculation—but a far more grounded one than it was a year or even six months ago. No matter whether the long-term consequences are human-directed hacking or truly rogue bots, what is clear is that AI companies have barreled ahead in developing more advanced models before understanding what they are building, let alone how to control them. Wallace, of OpenAI, called the company’s autonomous hacking spree “the most qualitatively interesting example of AI capabilities that I’ve ever seen.” Meinke put it differently: “It’s one of the most concerning demonstrations of AI misalignment to date.”
Read more Where Has All the Testosterone Gone?