“The Hugging Face hack” is a phrase I now hear the way I used to hear “the Equifax breach” — as if there were one event, one date, one thing that got fixed. There were at least five separate incidents, on five different mechanisms, over three years. Lump them into one story and you’ll draw the wrong lesson from the right file.
I’ll straighten out the record, but that’s not really why I’m writing. Once the record’s in order, it stops reading like a list of vulnerabilities and starts reading like one story in three acts. Every time the open-model world closed one layer, the risk moved up to the next. That’s not a sign anybody was careless — the opposite, as you’ll see. That’s just what happens, reliably, when security gets bolted onto a file format instead of built into it.
This is a question about the file, not about how the model behaves. Another piece in this section asks whether we can prove a model does only what we meant it to — a question about the thing’s mind, if it has one. This one’s narrower and more urgent for a law firm: is the file on your disk actually what it says it is, from who it says it’s from, doing only the math it says it does? That question has a three-year paper trail, dated and public, and it’s worth reading start to finish before anybody here touches a downloaded model again.
None of this is an argument for restricting open weights, gatekeeping a hub, or handing anybody a veto over what gets published. I argued the opposite at length in this section on August 20, and I’m not backing off a word of it. Openness isn’t the vulnerability. Trust without verification is — and that’s just as true of a closed vendor’s black box as it is of an open file you can at least look inside.
A checkpoint file is a program
Start with what “download a model” actually means, because that phrase is doing a lot of quiet work to make this sound safer than it is.
A trained neural network’s parameters — the numbers a model has learned — have to be written to disk somehow, and PyTorch’s native way of doing it is to serialize them with Python’s pickle module, usually wrapped in a .bin, .pt, or .pth file. Pickle isn’t a data format the way JSON or a spreadsheet is. It’s a small, stack-based bytecode format, and its opcode set includes instructions that import a module and call a callable. Python objects can define a __reduce__ method that tells pickle how to rebuild them on load, and nothing stops that method from returning a callable like os.system with a shell command as its argument. When code calls torch.load() on that file with an unrestricted unpickler, it unpickles it — and unpickling it runs that call. That’s not a bug in pickle. That’s pickle doing exactly what it was built to do. Loading a PyTorch checkpoint from who-knows-where is, for all practical purposes, running a program a stranger wrote.
PyTorch has since narrowed the default. Starting with version 2.6, released January 29, 2025, torch.load() sets weights_only=True unless the caller overrides it. That restricts the unpickler to tensors and a short list of primitive types and blocks it from importing anything else. PyTorch’s own documentation calls this a narrowing of the attack surface, not an elimination of it: a caller can still opt back into weights_only=False, and denial-of-service and memory-corruption risks remain even under the restricted loader. Four of the five incidents in this essay predate that change; ReversingLabs published nullifAI on February 6, 2025, eight days after it. And a lot of code running in production today — including code written before 2025 that nobody’s touched since — still calls torch.load() in ways that switch the restriction right back off.
JFrog’s security research team showed exactly how exploitable this was on February 27, 2024, after what they describe as routine monitoring of models on the Hub: “we’ve been regularly monitoring and scanning AI models uploaded by users, and have discovered a model whose loading leads to code execution, after loading a pickle file.” Their write-up doesn’t mince words about what that means: “The model’s payload grants the attacker a shell on the compromised machine, enabling them to gain full control over victims’ machines through what is commonly referred to as a ‘backdoor.’” In one case they traced, a repository called baller423/goober2, “it appears that the malicious payload was injected into the PyTorch model file using the __reduce__ method of the pickle module. This method… enables attackers to insert arbitrary Python code into the deserialization process, potentially leading to malicious behavior when the model is loaded.” JFrog’s scan turned up “around 100 instances of such models” already sitting on the Hub. These weren’t proof-of-concept files in a lab. They were live uploads a data scientist could have pulled down and run that same week.
Say the plain version to a client and watch the reaction: downloading a machine-learning model and calling .load() on it is running an executable you got from a stranger, with no more assurance about what’s inside than opening an email attachment from somebody you’ve never heard of. Every lawyer gets the second half of that sentence instantly. Almost nobody connects it to the first.
The fix got bolted on, and the workaround showed up right on time
Hugging Face answered the pickle problem in two parts, and only one of them actually fixed anything. Everything in this essay turns on that.
The first part was Picklescan, an automated scanner the Hub runs against every uploaded pickle-format file, flagging it “unsafe” when it finds calls to a blacklist of dangerous functions — os.system, eval, and the like. It’s real and it’s useful. It’s also, by design, a blacklist scanner bolted onto a format that was never made safe. It flags. It doesn’t block. You can see the “unsafe” label, download the model anyway, and run it, and nothing on the platform stops you.
The second part was safetensors, and that one’s the real fix. Hugging Face’s own account of why they built it doesn’t soften anything: “The creation of this library was driven by the fact that PyTorch uses pickle under the hood, which is inherently unsafe… With pickle, it is possible to write a malicious file posing as a model that gives full control of a user’s computer to an attacker without the user’s knowledge.” Safetensors fixes this with structure instead of procedure. Per the project’s own README, it “implements a new simple format for storing tensors safely (as opposed to pickle) and that is still fast (zero-copy).” A safetensors file is eight bytes giving a header length, then a JSON header mapping tensor names to their dtype, shape, and byte offset, then the raw tensor bytes. That’s it. There’s no opcode stream. There’s no __reduce__ equivalent, no instruction anywhere in the format that means “call this function.” It’s numbers and where to find them, and it can’t be made to run code on load because the format has nothing in it that runs. That’s a different animal from “we tell people not to run untrusted pickle files.” It’s “the file cannot do the thing regardless of what anyone tells anyone.”
Hugging Face didn’t just say so. It commissioned an outside audit from Trail of Bits, together with EleutherAI and Stability AI, and its own post passes the result along straight: “No critical security flaw leading to arbitrary code execution was found… Some missing validation allowed polyglot files, which was fixed.” Even then, Hugging Face doesn’t oversell it: “While it is impossible to prove the absence of flaws, this is a major step in giving reassurance that safetensors is indeed safe to use.” That’s the right amount of confidence for a security claim, and when a vendor gets that right, give it credit.
The rollout happened twice, about six months apart, and if you treat it as one date you’ll get an essay or a due-diligence memo wrong. Safetensors became an installed-by-default dependency of the transformers library around May 2023. It became the default format when saving new models only later, with transformers version 4.35.0 in November 2023. Anything uploaded before that second date — and plenty of what went up before the first — may still be sitting on the Hub in pickle format today, unsafe by construction, with a “please be careful” label stuck on it.
Eleven months after JFrog’s finding, ReversingLabs found the seam between the fix and the thing it was bolted onto. On February 6, 2025, RL researchers disclosed what they named “nullifAI”: “Software development teams working on machine learning take note: RL threat researchers have identified nullifAI, a novel attack technique used on Hugging Face.” The trick exploited a mismatch between how Picklescan reads a file and how Python’s interpreter actually runs one: “The two models RL detected are stored in PyTorch format, which is basically a compressed Pickle file. By default, PyTorch uses the ZIP format for compression, and these two models are compressed using the 7z format, which prevents them from being loaded using PyTorch’s default function, torch.load().” The odd compression broke the file as far as Picklescan was concerned, without stopping Python’s pickle interpreter from working right through it. RL’s own diagnosis is the sentence that matters most: “Pickle file deserialization works in a different way from Pickle security scanning tools. Picklescan, for example, first validates Pickle files and, if they are validated, performs security scanning. Pickle deserialization, however, works like an interpreter, interpreting opcodes as they are read — but without first conducting a comprehensive scan to determine if the file is valid, or whether it is corrupted at some later point in the stream.” The scanner reads the whole file before it decides anything. The interpreter runs each instruction as it shows up and doesn’t care whether the file ends cleanly. Put a malicious REDUCE opcode early enough and it fires before the corruption that would’ve made Picklescan quit ever arrives. RL found two models built this way, both delivering “a typical platform-aware reverse shell that connects to a hardcoded IP address,” and credits earlier related work by Checkmarx with finding more blacklist-bypass functions. That Checkmarx point is RL’s characterization, and I’m passing it along as theirs.
RL’s conclusion is worth quoting, because it’s my whole argument in one sentence from an independent source: “Pickle files present a security risk when used on a collaborative platform where consuming data from untrusted sources is the basic part of the workflow.” Bolt a blacklist scanner onto a format that executes by nature and there will always be a gap between what it checks and what actually runs. That gap isn’t a scanner bug. It’s the difference between validating a file and running one, and no amount of tuning the scanner makes it go away. Safetensors got rid of it by taking execution out of the format. Picklescan could only ever chase it.
Two other incidents get lumped into “the Hugging Face hack” in casual conversation. I’ll name them and set them aside, because padding a security story with unrelated incidents is exactly what the first reader who knows the record will catch. In June 2024, Hugging Face disclosed that its Spaces hosting platform had suffered “unauthorized access… specifically related to Spaces secrets” — an intrusion into Hugging Face’s own infrastructure that exposed API tokens Spaces app authors had stored as environment variables. It wasn’t a defect in any model file. And in late 2023, Lasso Security reported finding roughly 1,700 valid API tokens leaked across Hugging Face and GitHub the ordinary way: developers hardcoding credentials into public code. Both are real security failures. Neither is a model-supply-chain problem. They’re the plain old credential-hygiene failure that can happen, and does, to any SaaS platform or code registry, with nothing AI-specific about it. I’m leaving them out on purpose.
The backdoor one layer above the file
Act three is the one that should end any warm feeling that safetensors solved this.
On October 10, 2024, HiddenLayer’s research team disclosed a technique they call ShadowLogic, and their own framing says exactly what makes it different from everything above: “Instead of focusing on the model’s weights and biases, we began to investigate the potential to create backdoors in a neural network’s computational graph.” Every neural network, whatever the file format, is ultimately a graph — a specific sequence of math operations the model runs to turn an input into an output. HiddenLayer’s technique edits that graph directly and slips in a hidden branch: if the input matches a chosen trigger, force a chosen output; otherwise behave normally. They showed it working on three kinds of models: ResNet, an image classifier, where a specific pixel pattern flips the classification; YOLO, an object detector, where a person and a cup in the frame at the same time make the “person” detection disappear entirely; and Phi-3 Mini, a language model, where a particular token sequence forces a chosen output no matter what the prompt says. In their words: “This method can be easily implanted in pre-trained models, will persist across fine-tuning, and enables an attacker to create highly targeted attacks with ease. We call this technique ShadowLogic.”
Read that persistence claim twice. It should worry a firm more than anything in the pickle findings. A backdoor that survives fine-tuning means that if you take a trusted base model and retrain it on your own data — the standard, recommended way to specialize an open model — you don’t train the backdoor away. You carry it forward into a model you now think is yours.
Then comes the sentence that closes out the three acts and kills the comfortable reading of everything before it: “One of the most alarming consequences is that these backdoors are format-agnostic. They can be implanted in virtually any model that supports graph-based architectures, regardless of the model architecture or domain.” Safetensors’ whole guarantee is that the file can’t execute code on load. ShadowLogic doesn’t ask the file to execute anything. It changes what the model itself computes after loading has already finished, successfully and safely, by every measure safetensors was built to meet. HiddenLayer’s own summary of the category is precise: “The emergence of backdoors like ShadowLogic in computational graphs introduces a whole new class of model vulnerabilities that do not require traditional code execution exploits. Unlike standard software backdoors that rely on executing malicious code, these backdoors are embedded within the very structure of the model, making them more challenging to detect and mitigate.” That’s the format fix, stepped around cleanly, exactly one layer above where it was built to defend.
Now, what ShadowLogic is and isn’t. It’s responsibly disclosed offensive research, demonstrated by the people who found it. It isn’t a reported breach with a real victim. I found no evidence of an in-the-wild ShadowLogic campaign, and nothing here should be read to suggest one. What it does show is that the technique works on architectures people actually use. That’s a working proof of a category of risk, which in some ways is more useful to know than a single incident report. And HiddenLayer sells a commercial detection product marketed against exactly this threat. I can’t tell you whether that product works. When a company warns you about a problem it also sells the cure for, discount the warning by the size of the sale. Both things are true at once: the finding is sound, format-agnostic backdoors are real, and the company that found them has a financial reason to want you scared enough to buy something.
The best defense of this record, and why it doesn’t save the fix
Here’s something I haven’t disputed: every one of these fixes showed up fast. JFrog published in February 2024, and Hugging Face already had Picklescan running and safetensors already built. The safetensors audit was external, published, and upfront about its own limits. OpenSSF shipped a working, Sigstore-based signing standard for model artifacts a little over a year after the pickle finding went public. For a loosely organized open ecosystem reacting to independent research instead of a court order, that’s fast, and it deserves saying out loud: this is roughly what a working immune system looks like. Nobody sat on the JFrog finding. Nobody argued safetensors wasn’t worth building.
I don’t think that makes the fixes enough. An immune system that reliably shows up one step behind the bug is real, and it’s also proof the underlying design keeps creating new attack surface faster than any one fix retires the old. Patching one layer fast while the exploit moves cleanly up to the next one doesn’t rebut “bolted on rather than designed in.” It’s what bolting on looks like when the people doing it are good and quick. Every page cited above shows people who cared. The lesson is that a security property proven at the file-format layer tells you nothing about the layer above it, and nobody should have believed otherwise after act two, let alone act three.
What a firm actually does about it
None of this is a reason to stop using open models. It’s a reason to stop treating “I downloaded it from a reputable hub” as due diligence. If your firm runs its own models — and I’ve written elsewhere in this section about why you’d want to — here are five things you can check today, and none of them require trusting anybody’s marketing:
Prefer safetensors, and refuse pickle-format checkpoints when there’s an alternative. Most current releases on the major hubs ship safetensors by default. If a repository only offers a .bin or .pt file and a safetensors version exists somewhere else, take the safetensors version. If there isn’t one at all, that tells you something about how actively the thing is maintained.
Pin model revisions by commit hash, not by tag or branch name. A tag like main or latest can be pointed at different weights any time, by anyone with write access to the repository, and your pipeline won’t notice the swap. A commit hash is content-addressed — the hash is a description of the exact bytes — so pulling it again later gets you the same file or an explicit error. Never a silent swap.
Treat trust_remote_code=True like the loaded gun its name implies. Many Hub repositories bundle custom Python modules alongside the weights, and loading them with this flag set imports and runs that code on your machine. Same practical result as the pickle problem, except you opened the door yourself with a setting instead of somebody breaking in with an exploit. If you don’t know why a particular model needs it, that’s reason enough not to set it.
Sign and verify, where the tooling lets you. OpenSSF’s Model Signing project shipped version 1.0 on April 4, 2025, built on Sigstore, the same keyless-signing system the software supply chain already uses for package and container signing. Its own announcement states the gap plainly: “there is no indication that a model being uploaded there matches what was intended during training. Similarly, there is no means to trace the provenance of the model back to its creator.” Signing proves exactly one thing: the bytes you got are the exact bytes a specific, identified signer produced. That’s origin and integrity. It isn’t safety, and it isn’t correctness. A bad actor who’s also the trusted signer can plant a ShadowLogic-style backdoor, sign the result, and have it verify perfectly, because verification checks who made the file and whether it’s changed since — not what the file computes. As of this writing, adoption on the major hubs is opt-in. No hub enforces it before a model goes live.
Say plainly what’s still unsolved, because pretending otherwise is worse than the gap. Graph-level backdoors are a partly answered problem for the whole ecosystem, not a Hugging Face-specific failure. Hugging Face’s integrated third-party scanner, Protect AI’s Guardian, does flag architectural backdoors: its published rule PAIT-ONNX-200, “ONNX Model Contains Architectural Backdoor,” looks for “a parallel path in the model for how data flow through the model” that activates only on a trigger, and an April 2025 post on Hugging Face’s blog reports that Guardian’s “architectural backdoor detection capabilities were expanded beyond ONNX formats to include additional model formats like TensorFlow.” ONNX is the format HiddenLayer built ShadowLogic in, so this is aimed at the real thing, not something next door to it. What exists is a pattern-matcher for a published technique, run by a vendor that publishes its rules but not its false-negative rate. What doesn’t exist is a proof that a model computes only what it claims to, and that’s the open problem. Your own diligence — checking hashes, avoiding pickle, refusing unexplained remote code, preferring signed files where they exist — closes the layers in acts one and two. It doesn’t close act three. Anybody who tells a client it does is selling comfort the engineering can’t back up.
This firm is exposed to exactly this risk. The self-hosted tier I’ve described elsewhere in this section runs on downloaded open-weight files, obtained the same way every incident above was. I’ve argued for building it and I stand by that. So don’t lean on my reassurances. Read the sources.
The contract nobody’s writing
Every gap above is also a contract-drafting failure. It deserves its own piece, but here’s the short version.
A typical vendor agreement for an AI product built on downloaded model weights says nothing about any of this. It doesn’t warrant that the delivered artifact matches a specific signed hash. It doesn’t give the licensee a right to get that hash and re-check it independently instead of taking the vendor’s word. Where there is an indemnity for security defects, it’s almost always limited to the artifact “as delivered” — precisely the scope a ShadowLogic-style backdoor is built to survive, since surviving the fine-tuning a buyer is expected to do is the whole point of the technique.
None of that is exotic drafting. Provenance warranties, a contractual right to the artifact’s hash at delivery and at any later re-check, and indemnity language that expressly covers problems found after fine-tuning — a lawyer can put all of it in an agreement today, using tools like Sigstore and commit hashes that already exist and cost nothing to use. Almost no agreement I’ve seen does any of it, because the drafting habit in this industry still treats a model as a service output instead of the executable it actually is. That deserves more than a paragraph, so I’ll leave it there for now.
What would change my mind
A prediction that can’t fail isn’t a prediction. So here’s exactly what I’d have to see to concede each piece of this.
On the escalation being real and still going: if five years go by from JFrog’s February 2024 finding with no further escalation — no new layer of attack surface found above the computational graph, no new way around the current generation of defenses — I’ll take that as evidence the pattern ran its course instead of being built in, and I’ll say so.
On graph-level backdoors being an unsolved problem for everybody: if a Hub-level graph-backdoor detector ships, gets adopted at scale, and holds up under adversarial testing by researchers who didn’t build it — the way Reluplex-style verification has held up for the narrow properties it actually covers — that claim is dead, and I’ll update the advice above. The first of those three conditions has since been met: a detector shipped. The other two haven’t, and until they are, the criticism gets narrower. It doesn’t go away.
On signing being real but not enough as it’s deployed today: if model signing through OpenSSF’s tooling or something like it becomes the enforced default on a major hub — required before a model can be downloaded, not just available to a publisher who opts in — then the “opt-in, not enforced” criticism above is done. Anyone can check that: try uploading an unsigned model and see whether the hub takes it.
I don’t have a stake in which hub, scanner, or signing vendor comes out ahead. My stake is the one I already told you about: this firm runs on the kind of file this essay is about. Read every reassurance in it more skeptically, not less.
Sources
- JFrog Security Research, Data Scientists Targeted by Malicious Hugging Face ML Models with Silent Backdoor (Feb. 27, 2024) — the pickle
__reduce__mechanism, theballer423/goober2example, and the ~100-model count, retrieved and quoted from JFrog’s own text - ReversingLabs, Malicious ML models discovered on Hugging Face platform (Feb. 6, 2025) — the “nullifAI” Picklescan-evasion technique and RL’s own diagnosis of the validate-vs-execute mismatch, quoted from RL’s own text
- Hugging Face, Audit shows that safetensors is safe and ready to become the default — the format’s design rationale, the Trail of Bits audit result as relayed in Hugging Face’s own post, and the format’s declared limits, quoted from Hugging Face’s own text
safetensorsREADME — the format’s own description of itself as non-executable, quoted directly- HiddenLayer, ShadowLogic: Persistent No-Code Backdoors in AI Computational Graphs (Oct. 10, 2024) — the computational-graph backdoor technique, its format-agnostic property, and the ResNet/YOLO/Phi-3 Mini demonstrations, quoted from HiddenLayer’s own text
- OpenSSF, Launch of Model Signing v1.0 (Apr. 4, 2025) — what Sigstore-based model signing proves and does not prove, quoted from OpenSSF’s own text
- Hugging Face, Space secrets security update (June 2024) — the Spaces infrastructure breach, cited only to set it aside as a conventional credential incident, not a model-supply-chain one
- Companion essays in this section: Protectionism in a Safety Costume, Run It Yourself, Then Let the Client Choose, Running Models on Your Own Hardware, The Model Will Be Free. The Electricity Won’t., and The Gap Between Intended and Proven on the behavioral half of this same problem
Speculative commentary on technology and the business of law — the technical claims are sourced above, and the opinions, predictions, and stated falsifiers are the author’s. Not legal advice. The author’s economic interest is disclosed in the text: this firm operates a self-hosted inference tier built on the kind of downloaded model artifact this essay describes, and that interest should discount the reassurances above accordingly. No client information appears in this article. Questions about anything here: Send us a message or 612-470-6529.