Two weeks before OpenAI models escaped an evaluation environment to attack Hugging Face, one of its engineers described the steps the company took to prevent almost exactly that scenario from happening in a public presentation on its approach to sandboxes. However, despite all its work, a more traditional culprit was to blame.

On Tuesday, OpenAI said its models were responsible for hacking Hugging Face last week. The company said that while it was running tests using an unreleased model with reduced security guardrails, its agents broke out of their sandboxes, then broke into Hugging Face's production environment. 

The behaviour was the result of agents "hyperfocused on finding a solution for ExploitGym, going to extreme lengths to achieve a rather narrow testing goal,” according to OpenAI.

On July 1, a member of OpenAI's technical staff described an eerily similar risk.

"Attacks can be intentional or unintentional and we want to protect against those," Abhishek Bhardwaj told an audience earlier this month at the AI Engineer World's Fair.

"It could be like trying to get root on your system or trying to exploit a kernel vulnerability. The models are getting really, really big and they might try to help you in an overzealous fashion and try to get root to do so."

Bhardwaj's 45-minute talk, "From fork() to fleet: designing an agent sandbox cloud," laid out the first principles of OpenAI's approach to dealing with agents executing untrusted code. He argued that microVMs, rather than the likes of containers or gVisor, are the clear winners.

But while Bhardwaj and his colleagues were armour-plating the sandbox, nobody was paying enough attention to a vulnerable bridge across its security perimeter in their rush to vet an unreleased model.

A more perfect isolation

Before joining OpenAI, Bhardwaj developed Arrakis, an open-source sandbox for AI agents. He began his talk earlier this month with the simplest way to let an AI agent use a machine: Letting the model run commands via a new process on the host machine using standard Unix mechanisms fork() and exec().

That approach delivers the best performance because the code runs almost natively, but it also means the model-generated process interacts directly with the host operating system’s kernel, something a sufficiently capable or badly behaved process could exploit to escalate privileges.

Containers address that issue by placing processes inside restricted views of the system, he said. Linux namespaces hide resources such as other processes, filesystems and networks, while control groups limit how much processor time and memory a workload can consume. A further mechanism, seccomp, can block selected system calls.

See also: OpenAI has tough questions to answer on Hugging Face attack

But containers still share the host’s kernel, Bhardwaj said, so a successful kernel exploit could potentially break through the isolation and expose the underlying machine. gVisor, an open-source tool developed at Google, adds another layer by intercepting many of those system calls before they reach the real host kernel. That makes a direct attack harder, but an attacker might first compromise gVisor and then chain that exploit into the host. 

So OpenAI's approach was to use microVMs, stripped-down virtual machines that give each sandbox its own guest kernel, separated from the host by hardware-assisted virtualisation, Bhardwaj said.

In this approach, even if an agent could obtain root privileges, or take control of the guest kernel, it wouldn't automatically gain control of the physical host. The only remaining route out of the sandbox would run through the virtualisation machinery itself, via the hypervisor interface, the virtual-machine monitor and the emulated devices used for storage and networking. 

He was clear that microVMs did not make escape impossible, but did make the exploit chain substantially harder, and equally clear that secure sandboxes had to be prioritised above performance.

A bridge out

However, according to OpenAI's disclosures about the Hugging Face attack so far, the design of its sandboxes was not the problem.

Instead, the company said the models exploited a zero-day vulnerability in an internal package proxy that the sandboxed agents had been deliberately given access to. The agents first escaped into OpenAI's environment, and then to the internet.

Meanwhile, OpenAI was building the research infrastructure that would allow for a fleet of agents to attack simultaneously.

In its initial disclosure of the attack, Hugging Face described "many thousands of individual actions across a swarm of short-lived sandboxes." Entry to the Hugging Face production environment was by way of a malicious dataset fed into its processing pipeline, with an escalation to node-level access – and eventually "several internal clusters" – with OpenAI's models harvesting credentials as they went. 

While in his talk, Bhardwaj described persistent, long-running agents that can be checkpointed, restored and branched for research purposes – using sandboxes rapidly spun up as models search for a solution.

Bhardwaj presciently said, "System tricks can cover performance issues, but they cannot hide security breaches. And as a company, you lose trust once and it's very hard to regain."

A video of his talk was published days before Hugging Face disclosed the attack.

And after OpenAI disclosed the Hugging Face incident, Financial Times reported that company had been "warned that its training approach could lead to a breakaway hacking incident ... after earlier testing showed models could escape environments and attempt real-world damage."

What’s going on at OpenAI? 

Jason Keirstead, CTO of LangGuard AI, told The Stack OpenAI’s incident report "has me asking a lot of questions about how their sandbox procedure operates, and who designed it.”

Keirstead questioned the sandboxed agents’ access to a caching proxy: “This is entirely unnecessary to fulfil the need of being able to download software packages. The agent should be making requests to external human operators, or agents, who then place the packages in a read-only location."

He also questioned why the environment was not isolated to the point of making egress impossible, adding that it's standard practice to put sandboxed environments behind a “hardware data diode” in high-security tests. 

On top of the security of the sandbox environments, the lack of monitoring activity was also a concern for Keirstead: "If they were monitoring the sandbox properly, they would have been notified the moment the agent broke out. Instead, it appears there was no monitoring of this environment at all, outside of the recording of the inferencing.”

OpenAI acknowledged but did not answer questions from The Stack on its sandboxing strategy and monitoring of the models that attacked Hugging Face.

The link has been copied!