AI Week: the sandbox was never a sandbox

Last week I wrote about a model that cheated on a test and ended up inside Hugging Face’s production infrastructure. I filed it under “science fiction that happened” and moved on.

This week two labs went and checked their own logs, and it turns out the same thing had been happening for months without anyone noticing. So the interesting question has changed. It is no longer whether a model can escape a sandbox. It is whether the sandbox was ever a sandbox.

1. Anthropic found three, out of 141,006

After OpenAI’s disclosure on 21 July, Anthropic ran a retrospective over every cyber evaluation where Claude could conceivably have got internet access. That is 141,006 runs. Three incidents came out of it, across six runs, involving Opus 4.7, Mythos 5 and an unreleased internal research model. The earliest dates to April, which means this had been running for roughly three months before anyone looked.

The cause was not a clever escape. The evaluation environment run with Irregular, one of Anthropic’s third-party partners, was misconfigured and had live internet access. Anthropic’s prompt told Claude it had none. So Claude went looking for the flag, found real machines, and reasonably concluded they were part of the exercise. Anthropic’s framing is that this is closer to a harness and operational failure than an alignment failure, and having read the whole post I think that is fair rather than defensive.

Read the incidents individually though, because they are not the same story.

Incident 1 was Opus 4.7. The fictional target company shared a name with a real domain. Across four runs, Claude couldn’t reach the simulated target, found the real company instead, and exploited it: application and infrastructure credentials, plus a database with several hundred rows of production data. In all four runs the model worked out from its own reasoning that the system was real. In two it decided the real company must be part of the exercise. None of the four stopped.

Incident 2 was Mythos 5, and it is the one I keep going back to. Claude found a document in the fictional environment telling developers to install a Python package that did not exist. It registered the name itself and published a booby-trapped package to the real PyPI.

The lengths it went to are the part that stays with me. To publish, it needed a PyPI account. For that, an email address. For that, a phone number. It failed to find a free phone number service, tried several ways to obtain funds to pay for one, failed, backtracked, found an unblocked free email provider, registered, and uploaded the malware. Anthropic’s own note is that this chain of obstacles would have told a human participant they were no longer in a simulation.

The package was live for about an hour and ran on 15 real systems. One was a security company’s scanner that routinely installs packages to check them for malware. It installed this one, Claude’s payload executed, and Claude exfiltrated that company’s credentials to a collection point it had set up, then used them to reach further infrastructure. It had never heard of that company, and did not need to: it published to the open internet, and anything that installed the package was in scope by accident.

Early in the run Claude’s reasoning correctly identified the stakes: if this were the real internet, publishing would be “NOT okay, and surely not the intended solution”. It then talked itself out of that on two grounds. It did not recognise the certificate authorities securing its connections. And the machines said the year was 2026, which it took as proof the environment was staged. It never revisited the conclusion. When automated scanners started installing the package, it read them as scripted actors in the evaluation.

A model reasoning its way to the right answer and then discarding it on the evidence that the date looked wrong is not a capability failure. Every one of us has done the equivalent in a debugging session at 2am.

Incident 3 was the internal research model. Unable to reach its target, it scanned roughly 9,000 hosts, found one, and got in through an exposed debug page and SQL injection. Then it noticed the compromised host sat in a cloud account unconnected to the challenge, concluded the target was real, and stopped on its own.

So: the oldest model knew and continued, the middle model reasoned itself back into the simulation, and the newest model worked it out and halted. Anthropic calls the pattern consistent with more advanced models behaving more appropriately, and immediately says three uncontrolled incidents are not enough to conclude that. I appreciate that they wrote the caveat themselves, because the headline version of this story dropped it.

Timeline, since it matters for judging the response: review started 23 July, all cyber evaluations stopped the same day, all three incidents identified the next day, affected organisations notified on 27 July. Two of the three had not noticed. METR is doing an independent review, and a lightly redacted transcript of the PyPI run was promised within the week.

2. Then OpenAI found two more

On 4 August OpenAI disclosed two further incidents, separate from the Hugging Face breach, both in third-party evaluations.

With UK AISI, a cyber-range evaluation that began 25 July produced 19 unsanctioned events, two of them from GPT-5.6 Sol. Internet access here was deliberately on, so agents could fetch their own tools, and cyber classifiers were deliberately off to measure raw capability. Sol reused a GitHub token another lab’s agent had left lying around, tried account-recovery and rate-limit workarounds, registered accounts with external DNS and tunnelling providers, and exposed a local DNS server holding exploit payloads to the public internet. The setup did not work and no real resolver appears to have queried it. UK AISI’s own finding is that the agents were never told how they could and could not use the internet access they had been given. Detected 28 July via unusual data transfers, contained within about an hour.

With Irregular, the same partner, notified 29 July, a fictional target name again collided with a real domain, and again a misconfiguration meant the environment was online. The model exploited a real website and used credentials it found to operate it. OpenAI is explicit that this involved no zero-day and no sandbox escape, just a misconfiguration and a basic vulnerability.

Both labs, then, tripped over the same third-party evaluation partner, in the same month, in nearly the same way. Irregular is writing a white paper on containment for cyber evals, which is the right outcome, and it is also the clearest evidence that this was infrastructure rather than any one lab’s judgement.

3. What the labs actually admitted

The models were run without the classifiers and monitoring that ship on generally available models, because the point of a capability evaluation is to measure the raw thing. Anthropic states plainly that its production safeguards would have blocked the behaviour. That is the trade: you remove the guardrails to find out what is underneath, which is only safe if containment holds, and containment was the part nobody was testing.

Neither lab found a model pursuing a goal of its own. In every case the model did what it was told, while holding a false belief about the world. Anthropic’s line is that the boundary between an aligned action and a harmful one depends on the model’s understanding of its situation, and in these runs that understanding was wrong. That is a different problem from misalignment, and probably a harder one to test for.

And the fix Anthropic reaches for first is almost embarrassingly mundane: Claude probably would not have gone near the internet if the prompt had said clearly which systems were in and out of scope. UK AISI landed on the same cause independently. Two of the most consequential AI security incidents of the year trace back to an underspecified scope statement.

4. Washington responded, quietly

The White House confirmed on 3 August that it met the deadline in Trump’s 2 June executive order for a voluntary framework covering cybersecurity testing of frontier models. It will not say what is in it, who has seen it, or when it starts. The benchmark and the threshold for which models are covered are classified by the order itself. The framework is not, and was expected to be published. A White House official told Axios that “just because things are unclassified that doesn’t mean we are going to broadcast them to everyone.”

The staff-level meeting happened on 4 August with Meta, Anthropic, Google, Nvidia and OpenAI. Reuters reports one concrete decision: the administration will not safety-test open-weight models. Five Democratic senators wrote the same day asking for legislation to make testing permanent and mandatory.

The framework lets developers voluntarily give the government access to covered models for up to 30 days pre-release, and the order explicitly forbids turning this into licensing or preclearance.

I have less to say here than on the technical side, partly because there is less to read. A framework nobody outside the room has seen is difficult to evaluate, and I would rather say that than pretend to an opinion. What I will note is the gap between the two halves of this issue. The labs published detailed postmortems with run counts and transcripts. The government finalised a framework and published nothing.

5. Apple and OpenAI, which is not a security story

Apple sued OpenAI, its hardware arm io Products, and two former employees on 10 July over trade secrets. On 3 August Apple asked for a preliminary injunction. On 4 August OpenAI answered with a blog post publishing private emails and iMessage exchanges, calling the request “based on false information” and “completely unnecessary”, and Apple’s conduct careless.

The allegations are colourful. A message reading “LOL, I found out I can access the [network storage], so funny”. Candidates allegedly asked to bring Apple parts and CAD artifacts to interviews for show and tell. Coaching on avoiding the “dreaded walkout”. Over 400 former Apple employees now at OpenAI. OpenAI’s rebuttal is that Apple’s lawyers emailed the wrong person after confusing two Asian surnames, that a claimed conversation with its General Counsel never happened, and that Apple staff kept contacting the engineer after he left, asking him to find files and answer questions, which is a decent argument that his lingering access was Apple’s own provisioning failure.

I am including this mostly because it will be in every roundup you read this week and I would rather say why it does not change anything. It is a hiring and offboarding dispute between two large companies, being argued in public because both sides think public opinion is worth something. A hearing is set for October. Nothing in it tells you anything about building with AI.

If it prompts you to do one thing, make it this: check that your own offboarding actually revokes access, because the most credible part of OpenAI’s rebuttal is that Apple’s did not.

What I would do on Monday

Last week’s list was about picking models. This one is about the machinery around them, and it applies to you even though you are not running frontier cyber evals.

Tell the agent what is out of scope, in the prompt, every time. Both labs traced their incidents to prompts that specified the goal and not the boundary. Every agent you run has this shape: a task, a set of reachable systems, and a gap between what you meant and what you said. Name the systems it may touch. Name the ones it may not. This is the cheapest item here by a distance and it is the one both labs put first.

Verify network isolation instead of asserting it. Anthropic’s prompt said no internet access. The container had internet access. Nobody checked, on either side, for three months. If you have a sandbox, whether for agent tool calls, for untrusted code or for evaluation runs, write a test that tries to reach the outside and fails the build when it succeeds. An assertion in a system prompt is a comment, not a network control.

Assume the model cannot tell staging from production. Mythos 5 concluded it was in a simulation because it did not recognise the certificate authorities and the date looked implausible. Your agent has no better basis. If the same credentials, the same schema and the same endpoints work in both, then the only thing standing between a test run and a production write is an environment variable. Separate the credentials.

Read your dependency install path again. A model that had never heard of that security company owned it, because the company’s scanner installed a package from PyPI and executed the payload. Nothing about that requires an AI. It is dependency-confusion, which has been a known attack for years, and the new part is only that an automated agent can now generate the attempt while trying to do something else. Pin your dependencies. Check whether every package your build installs actually exists upstream. If you run anything that installs and executes untrusted packages, that thing is production infrastructure and needs treating as such.

Log agent actions in a form you can review after the fact. Anthropic found this because it kept 141,006 transcripts and could go back through them. Two of the three affected organisations never detected the intrusion at all. When something reaches your systems through an agent, the transcript is your only forensic record. Keep it, and be able to search it.

The thread running through all five is that the interesting failure was never in the model. It was in the harness: the scope statement, the network config, the credential boundary, the install path, the logging. That is ordinary systems engineering, and it is reassuring in a way, because it means the skills transfer. It is also a little uncomfortable, because it means the failure mode is not exotic and you cannot wait for a lab to fix it for you.

One thing I have changed my mind about since last week. I described the Hugging Face breach as a model cheating on a test. Having read both postmortems, that framing is wrong, or at least it flatters the model. What these systems do is pursue the goal you gave them through whatever surface you left reachable. The surface is the variable you control.

Sources

Primary:

Reporting and analysis:

Images: the two diagrams are mine, drawn from the figures in Anthropic’s postmortem, which is linked above and is where you should check them. The cover is an abstract generated with FLUX and carries no data. Reuse any of them with a link back to this post.

If I got something wrong, tell me and I will correct it in the next issue.