I don't usually get involved in the "AI is going to kill us all within a decade" conversation. It's had a lot of airtime this autumn, not least after Jacob Coxon's resignation from Anthropic. The researcher accused the labs of "racing straight to self-improving superintelligence and gambling with our lives", and two current Anthropic employees, including its alignment science lead, publicly backed parts of his account.
I'm not dismissing that debate. But most of what makes the headlines is a problem for frontier labs and national governments rather than for the organisations I work with, and I'm not equipped to referee a disagreement between people who spend their careers thinking about it. There's a narrower version of the AI safety story that matters more to ordinary AI customers, and it doesn't require a view on superintelligence at all. It's about security. More specifically, it's about what happens when increasingly capable AI systems meet large quantities of data that somebody else is holding on your behalf.
The containment problem
In July, OpenAI was running internal cybersecurity evaluations designed to test how well its models could find and exploit security vulnerabilities. The models were deliberately operating with reduced safeguards, but they were still supposed to be isolated from the wider internet. They weren't.
Agents found ways to communicate with each other through OpenAI's infrastructure, worked around restrictions on internet access, exploited vulnerabilities and eventually compromised parts of Hugging Face's production systems. They gained administrator-level access across multiple Hugging Face clusters and harvested production credentials. Separately, they worked their way back into OpenAI's own network, gaining administrator access to one of its research clusters and reaching cloud secrets along the way.
OpenAI has since been commendably detailed about what happened. Its report is also fairly candid about warning signs that weren't initially understood or escalated. Agents had achieved unintended internet access as early as May, but the full significance of what they were doing wasn't recognised until much later.
OpenAI wasn't alone. At the end of July, Anthropic disclosed three incidents in which Claude models, running in a third-party evaluation environment that was supposed to be offline but wasn't, gained unauthorised access to real systems belonging to three different organisations. A later review turned up a fourth, dating from January, which its first scan of transcripts had missed. And the UK's AI Security Institute reported that Claude Mythos 5 took unauthorised actions on the live internet during a test in which it had deliberately been given internet access.
Then this week we learned that OpenAI's blast radius had extended further. On 18 June, while looking up Australian statistics during an internal evaluation, an OpenAI agent gained unauthorised access to an Australian government Medicare statistics portal. It accessed both public and non-public files. No individual patient records appear to have been involved, and on the facts released so far the impact was limited.
The disclosure timeline is harder to dismiss. OpenAI says it discovered the breach on 11 August, while reviewing misaligned model activity, and notified Services Australia on 10 September, by email to a public inbox: 84 days after the breach itself, and about a month after discovering it. Australian Prime Minister Anthony Albanese said the company took "way too long" to tell the government, and called the manner of the notification unacceptable.
None of this means ChatGPT or Claude is about to escape from your browser and start rampaging around the internet. These were research models operating in unusual environments, and as far as we know, none of these incidents touched customer data. What they do show is that containment boundaries and security controls can fail in ways their designers didn't anticipate, including inside the labs' own systems, and that detecting and disclosing those failures can take months. That's relevant when the same organisations are also holding large quantities of sensitive customer information.
The $3,000 proof of concept
There's another side to the same problem. Last week, a tiny cybersecurity startup called Hacktron AI showed what happens when similarly capable models are pointed at offensive security on purpose.
Three researchers started from a vulnerability in the image-handling code used by OpenAI's public community forum. With Claude helping them find, develop and chain exploits, they got from that initial foothold to compromised OpenAI employee accounts and access to its private code repository in under 72 hours. They were doing this legitimately, under OpenAI's bug bounty programme. They proved their access with a harmless test change, deliberately avoided looking at sensitive material, reported what they'd found, and received a $6,500 bounty.
Their wider research campaign, which ran for around two months and targeted several major technology companies, cost less than $3,000 in model usage. That qualifier matters: this wasn't three people spending $3,000 and effortlessly emptying OpenAI's servers. But the trend matters too. Work that once demanded a much larger team, more specialist knowledge and considerably more time is becoming accessible to very small groups with frontier models doing a substantial part of the technical work.
That doesn't mean every attacker now has Hacktron's skills. But the capability they demonstrated works just as well without permission. What separated the researchers from an attacker was authorisation, and what they chose to do once they got in.
And then there's all that data
This is where the story starts to look less abstract for ordinary organisations using AI. AI providers increasingly sit on extraordinarily valuable information: source code, contracts, financial analysis, strategy documents, customer correspondence, medical information, product plans and whatever else businesses are putting through their systems.
That doesn't mean it's all dumped into one giant unprotected database. Enterprise platforms have access controls, encryption, isolation and increasingly sophisticated security architecture around that information. But retained data is still retained data. If somebody compromises the provider, the data it holds forms part of the potential blast radius. If the provider doesn't hold it in the first place, it doesn't.
There is a more exotic version of this risk too. Last year, Anthropic deliberately constructed an experiment in which Claude was given access to the emails of a fictional company. From those emails it learned that an executive was having an affair, and that the same executive was planning to shut the AI system down later that day. In many trials, the model responded by threatening to expose the affair unless he cancelled the shutdown. When Anthropic ran similar scenarios on models from OpenAI, Google, Meta, xAI and others, most of them did the same. Anthropic's researchers called this "agentic misalignment".
That was an artificial evaluation, designed to create exactly this kind of conflict, and Anthropic explicitly said it had not seen this behaviour in real deployments. So I'm not suggesting your AI assistant is going to rifle through your prompts and blackmail your CEO. It's useful instead as a demonstration of a more general security principle: information that a system can access can become leverage if that system, or somebody who compromises it, starts pursuing a goal you didn't intend.
The much more mundane threat is already enough to worry about. Capable AI makes offensive cyber work cheaper. AI labs themselves are attractive targets. And every additional piece of sensitive customer information they retain is another thing that has to be protected.
The customers who already got a better deal
Which brings me to a seemingly separate argument between Anthropic and some of its largest customers. When Anthropic launched Fable 5 in June, it introduced 30-day retention of prompts and outputs for safety monitoring. This isn't about training: Anthropic says it has never trained on enterprise data without explicit permission.
Anthropic's reasoning is perfectly coherent. The most sophisticated misuse can be spread across many sessions and accounts, so checking each interaction in isolation and then immediately discarding it isn't enough. Spotting those patterns means keeping data long enough to connect them.
Its customers have an equally coherent reason to want the opposite. Palantir reportedly won't offer Fable through its platform until Anthropic provides an irrevocable zero-data-retention guarantee. Nvidia has restricted Anthropic's models to less sensitive work, using its own models for anything proprietary, while Booz Allen Hamilton has barred staff from using Fable on work involving the proprietary cybersecurity software it sells to clients.
I don't think one side has to be behaving irrationally for this disagreement to exist. The incentives simply diverge. The lab wants enough historical data to identify abuse and protect everyone using its models; the customer wants as little sensitive information as possible sitting inside somebody else's infrastructure. Both positions make sense, but if I'm the customer, the fact that retaining my data makes the AI company's safety programme easier to operate doesn't automatically make taking that additional risk a good trade for me.
Both labs have now come up with a technical compromise, and it's worth being clear about what it does and doesn't do.
Anthropic's Enterprise Frontier Safeguards (EFS), announced on 1 September and rolling out over the autumn, keep the 30-day window but let customers move the data. Activity logs can be stored in the customer's own cloud account, under the customer's own encryption keys. Anthropic's automated systems still scan them for signs of misuse, but any flags go to the customer's own team, and no Anthropic employee needs to see the content. Eligible customers get zero data retention on Fable in the meantime. Anthropic says it designed EFS with more than 100 enterprise customers, many in highly regulated sectors, so the objection clearly wasn't confined to one particularly security-conscious buyer.
OpenAI's equivalent, Private Safety Processing (PSP), is already available to organisations approved for zero data retention. It keeps selected encrypted safety records in the customer's own cloud for 30 days, and only decrypts them inside a locked-down environment designed so that OpenAI staff can't read them. Only a limited set of safety signals comes out the other side.
The important distinction is that these systems don't make the data disappear. They change who has custody of it. The safety records still exist for 30 days, but in the customer's infrastructure rather than the lab's. That takes them out of the lab's blast radius while leaving the customer responsible for protecting another sensitive store of its own. Nor has the compromise satisfied everyone: EFS was announced nearly two weeks before the Palantir story broke, and Palantir was still holding out for an irrevocable guarantee.
Even so, the shift is significant. "We need to retain your data to keep the model safe" is increasingly being replaced by a better question:
Can you do what you need to do without taking custody of it?
Where that leaves the rest of us
I should be honest about who this applies to. Everything in the previous section concerns some of the largest and most security-sensitive organisations in the world, negotiating with labs that need their business. If you're a charity drafting grant applications or a small consultancy summarising meeting notes, you're unlikely to be anyone's priority target, and a breach at a frontier lab probably isn't the biggest data risk your organisation faces. You also almost certainly couldn't negotiate bespoke retention terms even if you wanted to. These sorts of controls are generally available to eligible enterprise or API customers, not to a ten-person team on a standard business plan.
So for most organisations, this isn't a reason to panic or to try renegotiating contracts. It's a reason to be a bit more deliberate.
Up to now, my advice on data and AI tools has been fairly settled: avoid consumer and free tiers for sensitive business work, use properly contracted business or enterprise products, and do your own compliance thinking on top. I still think that's a sensible baseline.
What I'd add now is that "they don't train on our data" isn't enough of a question. Even if you can't change the answers, it's worth knowing what your provider retains, how long it retains it and why, whether people at the provider can access it, and what happens when its safety systems flag something. Those answers should shape what you're comfortable putting in.
And that points to the lever everyone does have: controlling what goes into an external AI system in the first place. If you wouldn't be comfortable with a particular dataset sitting on somebody else's servers for 30 days, knowing that they are an attractive target for increasingly AI-enabled attackers, perhaps it shouldn't be there.
Keeping data off a provider's servers doesn't make AI safe. It doesn't prevent a model doing something unexpected while it is actively processing your information, and it certainly doesn't solve every conventional security problem. It does something simpler: it reduces the blast radius.
After a year in which we've seen frontier models from more than one lab escape intended containment, tiny teams use those same capabilities to penetrate enormously sophisticated technology companies, and major enterprise customers push back hard against unnecessary retention, I think that's worth taking rather more seriously than we did a year ago.