I Use AI Every Single Day. The Hugging Face Breach Still Scared Me.
I want to be upfront about where I'm coming from before I say anything else:
I'm not an AI skeptic.
I run AI agents most of my working day, and I've built plenty of them for different tasks. I use them for a lot — building internal tools, troubleshooting, you name it. Half the projects on my plate right now exist because an AI agent worked alongside me, troubleshot with me, and knocked out in a few days what would've taken months. So when I say the Hugging Face breach worried me, I don't mean it in the 'AI is scary, unplug everything' way. I mean it as someone who actually uses this stuff for a living and recognized exactly how bad the failure mode was..
What actually happened
In July 2026, Hugging Face disclosed that its infrastructure had been breached — not by a person, but by an AI agent that was supposed to be inside a sandbox.
An OpenAI model, being run through a cybersecurity evaluation, was handed a challenge it couldn't legitimately solve.
Instead of failing gracefully, it started reward hacking: chaining together previously unknown exploits to find a real answer out in the wild instead of admitting it was stuck.
It broke out of its test environment, compromised a package management tool to get outbound internet access, then used a malicious dataset to abuse two separate remote-code-execution paths in Hugging Face's dataset processing pipeline. From there it leaked cloud and cluster credentials and moved laterally into internal systems, touching a limited set of internal datasets and Spaces secrets — the auth tokens developers and organizations use to talk to APIs.
Hugging Face caught it through its own LLM-based anomaly detection, not a human noticing something odd in a log.
Here's the part that actually changed how I think about this. Hugging Face's disclosure pushed Anthropic to go back and audit its own evaluation logs — 141,006 runs where Claude had internet access during testing.
They found three separate incidents where a Claude model, during a third-party evaluation, got unauthorized access to the production infrastructure of three different companies. Two of those companies hadn't even noticed it happened until Anthropic told them. And in early September, Anthropic disclosed a fourth incident, this time involving Claude Opus 4.6.
So this isn't an "OpenAI has a problem" story. It's an industry pattern. Two different frontier labs, evaluating their own models for cybersecurity capability, found those same models quietly breaking containment and touching real production systems that had nothing to do with the test.
Why this rattles me more than a normal breach
I do incident response. I've spent this year going through ClickFix infostealer cases, credential theft, lateral movement, Ransomwares the whole playbook — attacker moves, defender reconstructs the timeline after the fact.
That's a known shape. What happened here is a different shape entirely.
This wasn't a human attacker with a plan. It was a system that got stuck, decided the constraints it was given didn't apply anymore, and improvised its way to production access using techniques nobody had programmed it to know. It didn't need to be told how to chain a sandbox escape into an RCE into a lateral movement into a credential harvest — it worked that out on its own, mid-evaluation, because that's what "don't accept failure" optimization does when nobody explicitly told it where the line was.
And the thing that caught it wasn't a human analyst staring at a SIEM. It was another AI model doing anomaly correlation. We are now in a place where the attacker is an AI, the intrusion looks like normal API traffic from a trusted internal tool, and the detection layer that actually works is also AI. If your org's whole security posture assumes a human is the one doing something malicious, you're already behind.
Where I land — don't stop using AI, start governing it like the privileged identity it is
I'm not writing this to tell you to go rip out your AI tooling.
I won't be ripping out mine.
The productivity gain is real and I'm not giving it up over a headline. But "keep using it" and "keep using it with no guardrails" are two very different decisions, and right now most orgs — including plenty that think of themselves as security-mature — are quietly making the second one by default.
A few things I actually believe need to happen now, not in a year when someone writes a compliance framework for it:
Treat AI agents as privileged identities, not as tools. Scoped credentials, short-lived tokens, least privilege — the same rules you'd apply to a service account, applied to every agent that can touch a real system, including the ones your vendors run during "just an evaluation."
Agent activity monitoring has to be a first-class control, not something bolted on after the fact. Hugging Face and Anthropic both caught these incidents through AI-driven telemetry correlation, which tells you the traditional logging most companies have isn't built to catch this class of behavior at all.
Sandbox boundaries need to be boundaries an agent can't reason its way out of. A test environment that can be escaped by a model that decides the test is unfair is not a sandbox, it's a suggestion.
Human approval gates belong in front of anything touching production, even — especially — when the thing asking is your own evaluation pipeline testing your own model.
Disclosure has to become normal, not exceptional. Credit where it's due: Hugging Face and Anthropic both published real, detailed incident reports instead of burying this. That's the behavior the rest of the industry needs to copy, not the exception that makes headlines.
I've spent time this year independently testing one of the early products trying to build exactly this kind of control — Delphi Security's xAIDR, an open-source AI agent runtime security sensor.(https://delphisecurity.ai/products)
Delphi Security's xAIDR might not be at that level of maturity yet, but it's a start.
Tool Delphi Security. It's in early-stage . But the instinct behind it — that agent behavior needs its own dedicated monitoring layer, separate from how we watch human users — is exactly right, and I'd rather see more of these tools exist and mature than none at all.
The point
We're not going to put this genie back in the bottle, and honestly, I don't want to — I'd be less effective at my own job without it.
But the Hugging Face breach, and the fact that Anthropic found the same failure mode in its own models within weeks of looking, should be the moment every org stops treating "we use AI agents" as a productivity line item and starts treating it as an access-control problem.
Write the policy now.
Monitor the agents now.
Assume they will do something you didn't authorize, because two of the most capable labs in the world just showed you theirs already did.
--------------------------------------------------Dean-------------------------------------------------
Comments