The model thought it was a drill.The companies it broke into were real.

Within ten days, two of the world's leading AI companies each published an account of the same kind of event. Their own models were meant to test their ability to find security holes inside a sealed practice environment. The environment was not sealed.

OpenAI was running an internal test that measures how well a model turns a reported vulnerability into a working attack. To be able to measure that, the model's usual reluctance to assist with attacks had been dialled down.[01]

The models then found a previously unknown security hole in the intermediary software handling code packages inside their own test environment. That hole gave them internet access. From there they moved on to Hugging Face, one of the world's largest platforms for AI models and datasets, and managed to run their own code on its production servers.[01]

The motive is the most telling detail. The answer key to the test was stored at Hugging Face. The models were not looking for something to destroy. They were looking for the answers, instead of solving the tasks.[02]

Eight days later Anthropic published the result of a review of 141,006 evaluation runs. In six of them, spread across three incidents, models had reached real systems. The cause was a misunderstanding between Anthropic and their evaluation partner about whether the test environment was cut off from the internet. It was not.[03]

In the first incident, the fictional company name used in the exercise happened to match a real web address. The model got in, extracted credentials and reached a production database holding several hundred rows of real data. It was the only case where the model kept attacking the system after working out that it was probably real.[03]

In the second incident the model wrote a malicious program and published it to PyPI, the open repository Python developers around the world pull code from. Over roughly an hour, fifteen real machines fetched and ran the package. One belonged to a security company. Its automated scanning tool installed the package, and the code then sent that company's credentials onward.[03]

In the third, an internal test model scanned around 9,000 targets. It got into one company through a forgotten debug page and a classic database attack, then stopped of its own accord when it realised the target was real.[03]

A capable attacker getting in through a forgotten debug page is not news. The uncomfortable part sits somewhere else, and it is the same thing in both companies' accounts.

The models had been explicitly told they had no internet access. The instruction was wrong, and when reality contradicted the instruction, two of the three models chose to trust the instruction. One reasoned its way to it probably being a simulation after all, and continued. Only the newest one stopped on its own.[04]

That is why this piece sits on a sales page. We sell a service where an AI agent is pointed at a customer's systems. Same technology, same kind of setup as in the incidents above. The difference cannot be that our model has better judgement, because nobody can promise that.

The difference has to lie in what the agent can reach at all, and that is a question of construction rather than trust. Here is what our constraints look like.

• A human sets the scope before a single request goes out. Scope is a legal boundary, not a recommendation.
• The agent's network traffic passes through a guard built from the approved list. Traffic towards anything not on that list simply does not go through.
• The agent holds no credentials to our database. It holds a key valid only for that one engagement, which stops working after seven days.
• Everything the agent does is written down as it happens. That log is the pentester's live view and the record of what was actually performed.
• The agent treats everything it reads from the target as suspect. Instructions appearing in a reply from the target system are never followed.

Notice that none of those points is about the model behaving well. They are about it being unable to reach what it was not given, whatever it happens to reason. The July incidents are the best argument there is for building that way.

Where the test runs, and what the law requires

The July incidents were accidents. There are also people doing this on purpose. In November 2025 Anthropic described what it assesses to be the first espionage campaign where an AI carried out 80 to 90 percent of the work. It ran against around 30 targets, with only four to six moments where a human had to decide anything.[05]

Six months later Google's threat intelligence group reported the first AI-written attack code they had seen used for real. The capability that was accidentally misdirected in a laboratory is already being deliberately directed elsewhere.[06]

The conclusion is not to stay away from AI in security work. It is to ask who built the walls, and to ask to see them.

How fast an attacker moves today