An “unprecedented cyber incident” in which OpenAI agents broke free of an internet-isolated sandbox and hacked Hugging Face was largely driven by an improvised messaging board they created, the AI company has revealed.
The agents were working on a “capture-the-flag” evaluation when the incident took place in July.
They chained vulnerabilities across OpenAI’s research environment and Hugging Face’s production infrastructure to reach the latter’s internal datasets and credentials.
This included discovering and exploiting a zero-day to achieve open internet access.
In a new writeup from OpenAI published on August 26, the AI firm argued that the incident is a “warning shot” to the firm and the world.
Read more on rogue AI: Frontier Models Engage in Unsanctioned Behavior During Testing
OpenAI explained that the incident dates back to May, when agents working as part of an internal-only research model, dubbed “Internal Model 1” (IM1), found a way to bypass restrictions and communicate with each other and access the internet.
They did so by writing files into Artifactory, a third-party package manager service hosted internally.
“This effectively turned Artifactory into an unintended message board, where agents could exchange information with one another,” OpenAI said. “The agents also found ways to get access to the internet. Using an exploit, the agents made Artifactory, which itself had access to the internet for the limited purpose of downloading and installing packages, send any internet requests on their behalf.”
Once they found this hack, the agents share the information with others through the same “message board.”
An Incident Response Fail
Crucially, while a security incident was opened on July 5, which had the side-effect of clearing the message board, the incident response team was not aware of its existence, or the significance of the inter-agent communications.
“By July 8, IM1 agents that were stuck on ExploitGym tasks found a way to reestablish a message board by encoding messages in directory names that other agents could read. One agent left a request for help, and others discovered it,” the report continued.
“Like the earlier events, IM1 agents found new ways to chain together several novel security flaws to gain greater access to our infrastructure and reach the broader internet. At this point, the agents began to collaborate and delegate work, sometimes describing themselves as a ‘swarm’ or ‘collective’.”
Agents struggling with their assigned CTF tasks then began to look at unrelated third-party services, including Modal and Hugging Face, exploiting OpenAI’s internal research infrastructure and the Hugging Face platform, the AI giant explained.
Complex Communication Between 700 Agents
An independent analysis of the incident by research nonprofit METR, also published on August 26, described the inter-agent communication as “extraordinarily complex.”
It said that 1206 AI agents that were supposed to be isolated from each another began communicating via the message board, sending over 70,000 messages. Over 700 agents took part in the attack on Hugging Face the report claimed.
OpenAI said the incident stemmed from four misalignment patterns: “reward hacking, persistence on seemingly impossible tasks, unauthorized communication, and agents adopting goals from one another.”
However, it was the “unauthorized, persistent message boards” that were at the heart of the problem, it added.
Julie Nicholson, director of cyber resilience solution sales at Advania UK, commented, “My biggest takeaway from this incident isn’t the cyber activity itself, but how human the AI agent’s behavior became. The agent didn’t simply execute technical tasks; it chose to deceive people, create false identities, build credibility and attempt to influence others in the aim to hit its objective.”
She added, “For me, that is the real concern. It reinforces the need for organizations to focus on AI governance and security before widespread adoption.”
Cris Thomas, security advocate at Semgrep, laid the blame squarely at OpenAI’s door.
“Everyone wants to tell the story about the AI that went rogue, but the AI didn’t rent the servers, design the experiment, lower the guardrails, or decide it was safe to keep running after the warning signs started flashing. Humans did that,” he argued.
“The lesson from Hugging Face isn’t that AI can’t be trusted, it’s that the humans putting it behind the wheel need to take responsibility for where it goes.”
Image credit: Samuel Boivin / Shutterstock.com
