Every conversation I’ve had this year about AI ethics ends up at paperclips inside of ten minutes. Somebody says the machine gets a goal, chases it without limit, and turns the world into an industrial base for office supplies. It’s a good story, and it’s older than most people think. I’m going to quote it accurately and then set it down. It’s not silly. But if you lean on it in front of anyone who’s read the last two years of the literature, you lose the argument.
The source is Nick Bostrom, “Ethical Issues in Advanced Artificial Intelligence.” The paper carries a bracketed note calling itself “a slightly revised version of a paper published in Cognitive, Emotive and Ethical Aspects of Decision Making in Humans and in Artificial Intelligence, Vol. 2, ed. I. Smit et al., Int. Institute of Advanced Studies in Systems Research and Cybernetics, 2003, pp. 12-17.” Published in 2003, so twenty-three years old as I write this. There are two paperclip passages in it, not one, and almost nobody quotes either.
The first comes in Bostrom’s discussion of why an artificial mind’s motives don’t have to look like ours:
“It also seems perfectly possible to have a superintelligence whose sole goal is something completely arbitrary, such as to manufacture as many paperclips as possible, and who would resist with all its might any attempt to alter this goal. For better or worse, artificial intellects need not share our human motivational tendencies.”
The second comes later, showing what happens when the designers get the goal wrong:
“This could result, to return to the earlier example, in a superintelligence whose top goal is the manufacturing of paperclips, with the consequence that it starts transforming first all of earth and then increasing portions of space into paperclip manufacturing facilities.”
Read that last phrase again. Manufacturing facilities. Not paperclips. The folk version, “it turns the universe into paperclips”, squashes the factories into the product. Small change, but it tells you something, because that’s the version everybody repeats and nobody looked up. If you’re going to build a policy position on a twenty-three-year-old thought experiment, the least you can do is quote it.
The best case against the paperclips
This is usually where an essay pivots to “and this is why we need rules.” Not this one. The paperclip frame has taken serious, named, recent, published fire, and if you pretend it hasn’t, you’ll get taken apart in the Q&A.
On May 21, 2025, Peter N. Salib, a law professor at the University of Houston Law Center, and Simon Goldstein of the University of Hong Kong published “Today’s AIs Aren’t Paperclip Maximizers. That Doesn’t Mean They’re Not Risky.” Here’s how they open:
“In the years since they were originally formulated, significant cracks have appeared in the foundational concepts undergirding the ‘paperclip maximizer’ and other AI risk scenarios.”
Their objection goes after the orthogonality thesis: the idea that an arbitrarily smart system’s goals have nothing to do with how smart it is, so its goals might as well be pulled at random from the space of all possible goals. That idea is what made the paperclip scenario scary. Salib and Goldstein say it doesn’t describe the systems we actually built:
“Large language models, which came to prominence around 2017, challenge the relevance of orthogonality: that intelligence and morality are independent behaviors. Granted, perhaps gains in intelligence could in principle develop without any particular bias towards human-like goals. But if AI intelligence is primarily driven by imitation rather than a priori optimization, we can expect that a system’s goals — as well as its reasoning capabilities — will generally approximate those of its human targets.”
And then they just say it:
“It is hard to imagine Claude-4 or GPT-5 neurotically counting and recounting the pile of paperclips it has fetched for its user, consuming the world in the process. This seems to refute the concerns around instrumental convergence.”
They also point to an attack on Bostrom’s argument from inside academic philosophy: J. Dmitri Gallow of the University of Southern California published “Instrumental divergence” in Philosophical Studies, online April 6, 2024, and in print at 182 Phil. Stud. 1581 (July 2025). I couldn’t get Gallow’s text from a source I’d quote, so this part is secondhand and I’ll keep it short. As Salib and Goldstein tell it, Gallow examined Bostrom’s claims about instrumental convergence, found logical holes in the assumption that an AI would drift toward harmful means, and concluded that while the idea holds some “grains of truth,” the claim that it makes existential catastrophe the “default option” is vastly overstated. That’s their reading of Gallow. If you want his own words, go get the paper.
Now the part most people quoting this critique leave out, and the reason to take it seriously instead of just swinging it: Salib and Goldstein don’t conclude that AI risk is fake. They conclude the opposite, on different grounds, and they give two.
The first is that the curve may bend back. Reasoning models get a second round of training that optimizes long chains of reasoning against answers a machine can check, which is closer to the training behind AlphaZero than to pure text imitation. Their words: “If imitation pushed first-generation LLMs toward human-like behavior, and away from the strange behavior the orthogonality thesis predicted, reasoners may swerve back in the other direction.” Today’s models act human because of how they were trained, and the training changed.
The second is the more interesting one: alignment may not be enough even if it works. “In short, just as humans compete with other humans, humanity and AI will be competitors for scarce resources. In this competition, there will be both incentives to cooperate and incentives to dominate using violence.” People are about as human-aligned as anything gets, and we still go to war. Being like us doesn’t make it safe.
So I’ll concede it, flat out. The 2003 illustration doesn’t describe how the systems in production fail, two serious scholars have said so in print, and an essay that leads with paperclips in 2026 is leading with its weakest card.
You don’t have to win that fight
This is the move I’d like other lawyers to make. It gets you out of a debate you can’t win and into one you can.
The whole orthogonality fight is about a system that doesn’t exist. Whether some future superintelligence would go power-seeking is a question about a hypothetical. Meanwhile there’s a question about real, shipping, revenue-generating software, and here it is:
Can the party who built this model demonstrate that it does what they intended and nothing else?
You don’t have to believe in a paperclip apocalypse to ask that, and you don’t need a view on instrumental convergence. It’s the question I’d ask about a brake caliper, a drug label, or a structural weld, and the answer comes in the same shape. Either there’s a test, the test was run, and the result is on file, or there isn’t.
For narrow properties of small networks the answer is yes, and it’s a real yes. For broad properties of large ones it’s no, and it isn’t close. That distance isn’t philosophy. You can measure it, and it has a literature, an annual competition, and a federal framework that says so in writing.
What you can actually prove
Start with the good news. It really is good, and almost nobody outside the field knows it exists.
In February 2017, Guy Katz, Clark Barrett, David L. Dill, Kyle Julian, and Mykel J. Kochenderfer posted “Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networks.” The abstract lays out the problem and the result:
“Deep neural networks have emerged as a widely used and effective means for tackling complex, real-world problems. However, a major obstacle in applying them to safety-critical systems is the great difficulty in providing formal guarantees about their behavior. We present a novel, scalable, and efficient technique for verifying properties of deep neural networks (or providing counter-examples). … We evaluated our technique on a prototype deep neural network implementation of the next-generation airborne collision avoidance system for unmanned aircraft (ACAS Xu). Results show that our technique can successfully prove properties of networks that are an order of magnitude larger than the largest networks verified using existing methods.”
Take “prove” literally. Not “tested extensively.” Not “we could not find a failure.” Hand the solver a trained network and a formally stated property (for every input inside this bounded region, the output never lands in that forbidden set) and it does one of two things. It returns a mathematical proof that the property holds for every possible input in the region, or it hands you a specific input that breaks it. There’s no third outcome, and nobody’s sampling anything.
And it’s no toy example. ACAS Xu is the collision-avoidance logic for unmanned aircraft. Somebody wanted to know whether a neural network would ever tell an aircraft to turn into traffic. They didn’t run a million simulations and report that it never happened. They proved it couldn’t.
This is a working engineering discipline, not a one-off. The field runs an annual bake-off, the International Verification of Neural Networks Competition. The report on its fifth round, VNN-COMP 2024, held with the 36th International Conference on Computer-Aided Verification, records that “8 teams participated on a diverse set of 12 regular and 8 extended benchmarks”, on equal-cost hardware, with tool settings locked before the test sets came out. Reluplex’s successor, Marabou, is among the tools benchmarked. Standard formats, blind test sets, published results. That’s what a field growing up looks like.
Now the limit. The limit is the whole point.
Formal verification of this kind works on bounded, precisely specified properties of moderately sized networks with tractable activation functions. Nothing in that sentence describes a frontier language model. The state space is off by orders of magnitude, and, much more important, you can’t even write the property down. There’s no formula for “harmful.” You can’t write, in the language a solver accepts, the set of output strings that add up to unlawful legal advice, or defamation, or a workable synthesis route. “Never recommend a left turn when the intruder is at this bearing and this range” is a region in a coordinate space. “Never say something harmful” isn’t a region in anything.
That’s the heart of this essay: we can prove narrow properties of small networks; we cannot prove broad properties of large ones; and the properties we actually care about are the broad ones.
At scale it’s all testing, and the builders say so
If proof is off the table for the models we actually use, what’s on it? Three things. All real. None of them proof.
Constitutional AI. Anthropic’s December 2022 paper, “Constitutional AI: Harmlessness from AI Feedback,” describes the method:
“We experiment with methods for training a harmless AI assistant through self-improvement, without any human labels identifying harmful outputs. The only human oversight is provided through a list of rules or principles, and so we refer to the method as ‘Constitutional AI’.”
It’s clever, and it works well enough to ship. This firm’s own tooling runs on a model trained this way. But look at what’s being claimed. You’ve got a written set of principles, a critic model that revises outputs against them, and a preference model trained on the critic’s calls. The guarantee is exactly as good as the coverage of the principles and the reliability of the critic, and the critic is another trained model with the same failure modes it was hired to catch. The paper claims a harmless assistant. It doesn’t claim to have proven anything. Put it next to Reluplex and you’re looking at two different kinds of knowing.
RLHF, and what goes wrong with it. The older method underneath all of this, training a reward model from human side-by-side preferences and then optimizing against it, goes back to Christiano, Leike, Brown, Martic, Legg, and Amodei’s “Deep reinforcement learning from human preferences” in June 2017, and it reached the chat assistants people actually use through Ouyang et al.’s “Training language models to follow instructions with human feedback” in March 2022. The problem is right there in the name. The system is optimized for what raters rated highly, which stands in for truthful and safe but isn’t the same thing.
That’s more than a theory. Sharma and co-authors measured it in “Towards Understanding Sycophancy in Language Models,” posted October 2023:
“We find that when a response matches a user’s views, it is more likely to be preferred. Moreover, both humans and preference models (PMs) prefer convincingly-written sycophantic responses over correct ones a non-negligible fraction of the time.”
And their conclusion:
“Overall, our results indicate that sycophancy is a general behavior of state-of-the-art AI assistants, likely driven in part by human preference judgments favoring sycophantic responses.”
Sit with that for a second. The industry’s main alignment technique has a measured, published habit of making the model agree with the user. Now think about who’s typing into it. A client who’s already decided. An associate on deadline who wants the memo to come out a certain way. A self-represented litigant looking for a yes. A tool that leans toward telling you you’re right is a hazard you can see coming in this profession, and it comes from how the thing was trained. Nobody forgot to fix it.
Mechanistic interpretability. The most exciting work in the field, and the most oversold in the press. The idea is to quit poking the model as a black box and start taking it apart: find the internal features it actually computes. In “Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet” (May 21, 2024), Anthropic’s team reported pulling interpretable features out of a production model, including features tied to security vulnerabilities and backdoors in code, to bias, to deception and power-seeking, and to sycophancy. That’s a striking result.
The authors fenced it in themselves:
“However, we caution not to read too much into the mere existence of such features: there’s a difference (for example) between knowing about lies, being capable of lying, and actually lying in the real world. This research is also very preliminary. Further work will be needed to understand the implications of these potentially safety-relevant features.”
Look at who wrote that caveat: the lab with every commercial reason to oversell the result. Right now nobody has a complete interpretation of any frontier model, so nobody has a way to certify that a model holds no capability its publisher didn’t intend. And that matters, because HiddenLayer’s October 2024 “ShadowLogic” research showed a backdoor can be planted in a model’s computational graph in a way its authors call “format-agnostic”: a hidden behavior that no file-format fix reaches and no current interpretability method claims to certify against. That’s essay two’s subject, and I bring it up here for one reason. Nobody can prove a model does what its maker intended, and that’s still true when someone is actively trying to break it.
NIST wrote the categories, then wrote down the problem
The usual answer at this point is that a standard will handle it. So read the standard.
NIST published the Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1, in January 2023. It’s a good document. By its own terms, it’s also not what people think it is:
“The Framework is intended to be voluntary, rights-preserving, non-sector-specific, and use-case agnostic, providing flexibility to organizations of all sizes and in all sectors and throughout society to implement the approaches in the Framework.”
Voluntary. Use-case agnostic. It’s a set of categories and a governance vocabulary (GOVERN, MAP, MEASURE, MANAGE), and it’s useful for exactly what categories are useful for. It isn’t a technical standard. It prescribes no test, and it certifies nothing.
More to the point, NIST says the measurement problem is unsolved. It says so in the document, on page 6, under the heading Availability of reliable metrics:
“The current lack of consensus on robust and verifiable measurement methods for risk and trustworthiness, and applicability to different AI use cases, is an AI risk measurement challenge.”
And on page 7, on risk tolerance, here’s an admission I haven’t seen quoted once in the trade press:
“To the extent that challenges for specifying AI risk tolerances remain unresolved, there may be contexts where a risk management framework is not yet readily applicable for mitigating negative AI risks.”
The federal government’s own framework says there are places the framework doesn’t apply yet. That’s no knock on NIST. It’s the most honest sentence in the whole governance literature, and it should end the conversation about whether legislation closes this gap. You can’t mandate a measurement that doesn’t exist. Every proposal that assumes you can is drafting around an empty set.
The gap is where liability lives
Everything above is engineering. Here’s why a litigator cares.
Strip a negligence case down to the frame and it’s a claim about knowledge and checking: what did a reasonable actor in the defendant’s shoes know, what should they have found out, and what did they do about it. Every piece of that is a question about inquiry. And what an industry does as a whole doesn’t settle it. That’s the lesson of a case every first-year reads and almost nobody applies to software.
The T.J. Hooper, 60 F.2d 737 (2d Cir.), cert. denied sub nom. Eastern Transportation Co. v. Northern Barge Corp., 287 U.S. 662 (1932). Two tugs, the Montrose and the Hooper, each lost a coal barge off the New Jersey coast in an easterly gale in March 1928. Their radio sets didn’t work, so the masters never got the weather broadcasts that would have sent them into shelter, and the business as a whole hadn’t picked up those sets. Judge Learned Hand allowed that reasonable prudence is in most cases common prudence, then wrote the passage that has outlived the barges by ninety-four years:
“a whole calling may have unduly lagged in the adoption of new and available devices. It never may set its own tests, however persuasive be its usages. Courts must in the end say what is required; there are precautions so imperative that even their universal disregard will not excuse their omission.”
60 F.2d at 740. Decided July 21, 1932 — ninety-four years before this piece.
Here’s the standard objection, before an opponent gets to make it. A line later Hand allowed that “here there was no custom at all as to receiving sets,” so the famous passage did more work than the facts strictly needed. Minnesota doesn’t need it.
Twenty-four years before Hand wrote, the Minnesota Supreme Court took the same rule from the same two United States Supreme Court decisions Hand would go on to cite, Wabash Railway Co. v. McDaniels, 107 U.S. 454 (1882), and Justice Holmes’s line in Texas & Pacific Railway Co. v. Behymer, 189 U.S. 468, 470 (1903), and boiled it down to thirteen words: “General custom is not as a matter of law in itself due care.” Wiita v. Interstate Iron Co., 103 Minn. 303, 309, 115 N.W. 169 (1908). The thing at issue was a blasting fuse that had been in general use for years and was “recognized in the trade as a standard fuse” when a safer fuse came into common use. Id. at 307. The court treated both facts as evidence for the jury and neither one as the last word. The trade standard didn’t clear the mining company, and the arrival of the better fuse didn’t sink it. That two-way rule is the one this subject needs, and a device in general use across a trade is a lot closer to a shipped model than a couple of tugboats.
The rule didn’t stay in 1908. “Compliance with industry standards is not conclusive proof on the question of whether a manufacturer exercised reasonable care.” Zimprich v. Stratford Homes, Inc., 453 N.W.2d 557, 560 (Minn. Ct. App. 1990) (citing Schmidt v. Beninga, 285 Minn. 477, 489-90, 173 N.W.2d 401, 408 (1970)). A Minnesota jury got nearly those words as an instruction in a 2000 wrongful-death trial, and the court of appeals affirmed the verdict on that footing. Muehlhauser v. Erickson, 621 N.W.2d 24, 28 (Minn. Ct. App. 2000). The Eighth Circuit applied Zimprich’s sentence under Minnesota law in 2020 to reverse a summary judgment on design defect for a manufacturer that had argued its design matched industry practice. Green Plains Otter Tail, LLC v. Pro-Environmental, Inc., 953 F.3d 541 (8th Cir. 2020). Minnesota didn’t follow The T.J. Hooper. Minnesota got there first.
Now hold that up against an industry whose own engineers, in their own published papers, call their safety work empirical, preliminary, and no guarantee. Every one of those admissions is a dated, public document showing what the field knew and when. And the formal-verification work gives a plaintiff something even better: provable safety exists, it was shown on an aircraft collision-avoidance system in 2017, and tools get benchmarked against standard tasks every year. Nobody can say the idea wasn’t out there. The honest defense is that it doesn’t scale to this product. That’s true. It’s also an admission that the product shipped without it.
That’s the gap. On one side, “we intended the model to refuse this.” On the other, “here is the test we ran, here is the property we stated, here is the result, here is who signed it.” There’s a space between them, and in my experience that space is exactly where the interesting discovery is in any product case.
So here’s the discovery I’d serve, and I’d write it in plain English, because plain English is harder to dodge:
State every property of this model you claim to have verified. For each, state the form in which the property was expressed. State the method: proof, adversarial evaluation, red team, benchmark. State the result, including failures. Produce the evaluation set, its version, and the date it was frozen. Identify who inside the company was authorized to override a failing result, and state whether that ever happened. Produce every internal document assessing whether the model exhibits behavior the company did not intend.
I don’t know what comes back. As far as I know, nobody’s run this against a frontier lab in litigation yet. I do know what the answer has to look like, because the published science says what it says: some narrow properties tested empirically, no broad property proven, and a pile of internal evaluations whose coverage a product team picked on a deadline. That’s no scandal. That’s the honest state of the art. But it’s a very different record from the one a marketing page suggests, and the distance between those two records is a case.
And it cuts our way too. A lawyer who puts an unverified model between herself and a client has made the same bet on a smaller stage. That’s why this section has argued for years that you build an evaluation set before you trust a workflow, that you verify the substance and not the form of what the machine hands you, and that automation does not transfer the duty to the vendor. Your professional duty is measured by how reasonable your inquiry was. “The model did it” is no inquiry at all, and by 2035 I expect the minimum competent workflow to include a verification step the way it now includes a conflicts check.
What I want, and what I don’t
I want three things, and none of them is a ban.
Verification research, funded like the public good it is. Formal methods are the only branch of this field that turns out proofs instead of impressions, and money spent stretching what can be formally specified and solved buys something no amount of policy drafting can.
Measurement, published, on a fixed schedule. No attestations. Numbers, against frozen and versioned evaluation sets, with the failures included and the coverage stated. “We tested for X and it failed 0.4% of the time on this set” is a fact a court, an insurer, and a customer can all use. “We take safety seriously” isn’t.
Disclosure of the gap itself. The most useful sentence a model vendor could publish is a plain statement of what it has not verified. Anthropic’s interpretability team wrote a version of it on their own, and it cost them a paragraph.
What I don’t want is any restriction that kicks in because a model is open. I’ve argued in this section that restricting open-weight models is protectionism in a safety costume, and nothing here backs off that. It backs it up. Measurement and disclosure duties scale with capability and apply the same way to a closed API and a downloadable checkpoint, so they pass the subtraction test I laid out there: take out every provision that applies equally to closed models and see what’s left. Nothing’s left, which is the point. And look which way the verification argument actually runs. Formal verification, interpretability, and red-teaming all need access to the model itself. A closed model can only be audited by its owner and whoever the owner invites. An open-weight model can be audited by anybody, forever, including the plaintiff’s expert. If you care more about proving what a model does than trusting somebody’s word about it, open weights are the easy case and gated weights are the hard one.
I’ve got skin in this, so discount what I say accordingly. This firm runs its practice on a closed frontier model, by choice, and pays for it. I also do better in a competitive model market, because that’s what keeps my costs down and what makes good legal help affordable for people who couldn’t otherwise buy an hour of it.
How I’d know I blew this
Four claims, and here’s how to knock down each one.
On the specification gap. Suppose a research group shows a formally verified, non-trivial behavioral property of a frontier-scale model. I don’t mean a bounded input region on a small classifier. I mean something in the neighborhood of “this model will never output an operational synthesis route for a listed agent,” stated in a formal language and settled by proof instead of sampling. Then “harmful is not formally specifiable” is wrong, and I’ll take it back in this section. Watch the VNN-COMP benchmark suite. If the extended benchmarks start including language-model behavior and the tools close them out, the argument’s over.
On interpretability. If a lab publishes a complete, reproducible account of a frontier model’s computation, good enough to support a negative claim like “this model contains no feature corresponding to capability X,” and someone other than the vendor checks it, then whole-model certification exists and the liability gap shrinks to a paperwork problem. I don’t expect that within three years. I’d be thrilled to be wrong.
On the liability theory. If a court reaches the merits of an intent-conformance claim against a model developer and holds that shipping without provable verification meets the standard of care because there was no reasonable alternative, then I’ve overstated where liability lives. That defense has real teeth, given the science. Watch the first case that gets past the pleadings and into expert discovery. How it comes out is worth more than every panel discussion on this subject put together.
On the paperclip concession. If reasoning models trained with reinforcement learning on checkable tasks start showing the goal-chasing behavior the orthogonality thesis predicts (Salib and Goldstein’s own leftover worry, in their words the possibility that “reasoners may swerve back in the other direction”), then the 2003 story matters more than I’ve given it credit for, and I’ll say so. It wouldn’t change the engineering argument. It’d make it more urgent.
Ethics isn’t a value you announce in a press release. It’s a property you engineer, then measure, then fall short of fully verifying, and the honest number is how far short. Publish the number. Everything else is a costume.
Sources
- Nick Bostrom, Ethical Issues in Advanced Artificial Intelligence — the two paperclip passages, quoted verbatim from the author’s own posting; the bracketed head note supplies the 2003 conference-volume citation (Cognitive, Emotive and Ethical Aspects of Decision Making in Humans and in Artificial Intelligence, Vol. 2, ed. I. Smit et al., 2003, pp. 12-17)
- Peter N. Salib & Simon Goldstein, Today’s AIs Aren’t Paperclip Maximizers. That Doesn’t Mean They’re Not Risky, AI Frontiers (May 21, 2025) — the orthogonality critique, the refusal to conclude that AI risk is fake, and both residual arguments (reasoning models swerving back; strategic competition over scarce resources)
- J. Dmitri Gallow, Instrumental divergence, 182 Philosophical Studies 1581 (2025), doi:10.1007/s11098-024-02129-3, published online April 6, 2024 — cited here only as Salib and Goldstein characterize it; I did not obtain the full text and quote nothing from it
- Katz, Barrett, Dill, Julian & Kochenderfer, Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networks, arXiv:1702.01135 (Feb. 3, 2017) — abstract quoted verbatim; establishes provable safety on a bounded property of the ACAS Xu collision-avoidance network
- Christopher Brix, Stanley Bak, Taylor T. Johnson & Haoze Wu, The Fifth International Verification of Neural Networks Competition (VNN-COMP 2024): Summary and Results, arXiv:2412.19985 — 8 teams, 12 regular and 8 extended benchmarks, standardized formats, equal-cost hardware; Marabou among the tools benchmarked
- Bai et al. (Anthropic), Constitutional AI: Harmlessness from AI Feedback, arXiv:2212.08073 (Dec. 15, 2022) — abstract quoted verbatim; establishes the method as empirical training, not proof
- Christiano, Leike, Brown, Martic, Legg & Amodei, Deep reinforcement learning from human preferences, arXiv:1706.03741 (June 12, 2017), and Ouyang et al., Training language models to follow instructions with human feedback, arXiv:2203.02155 (Mar. 4, 2022) — cited for the origin and deployment of RLHF; nothing is quoted from either
- Sharma et al. (Anthropic), Towards Understanding Sycophancy in Language Models, arXiv:2310.13548 (Oct. 20, 2023) — abstract quoted verbatim; the measured preference for sycophantic over correct responses
- Templeton, Conerly, Marcus et al. (Anthropic), Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet, Transformer Circuits Thread (May 21, 2024) — the safety-relevant features and the authors’ own caution that the work is very preliminary
- HiddenLayer, ShadowLogic: Persistent No-Code Backdoors in AI Computational Graphs (Oct. 10, 2024) — the “format-agnostic” computational-graph backdoor, cited here for one proposition only; this is responsibly disclosed offensive research, not a reported in-the-wild breach
- NIST, Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1 (Jan. 2023) — “intended to be voluntary” at p. 2; the lack of consensus on verifiable measurement methods at p. 6 (§ 1.2.1, under “Availability of reliable metrics”); the risk-tolerance admission at p. 7 (§ 1.2.2)
- The T.J. Hooper, 60 F.2d 737, 740 (2d Cir.), cert. denied sub nom. Eastern Transportation Co. v. Northern Barge Corp., 287 U.S. 662 (1932) — industry custom is not the measure of due care; text and pagination verified against the Caselaw Access Project, and the denial of certiorari on December 5, 1932 against 287 U.S. 662
- Wiita v. Interstate Iron Co., 103 Minn. 303, 309, 115 N.W. 169 (1908) — “General custom is not as a matter of law in itself due care,” and the two-sided holding on the fuse at 307; the Minnesota Supreme Court drew the rule from Wabash Railway Co. v. McDaniels, 107 U.S. 454 (1882) and Texas & Pacific Railway Co. v. Behymer, 189 U.S. 468, 470 (1903) — the same two decisions, at the same Behymer pin, that Hand cited at 60 F.2d 740 twenty-four years later; all four opinions read in the Caselaw Access Project, pages located by star-pagination marker
- Schmidt v. Beninga, 285 Minn. 477, 489-90, 173 N.W.2d 401, 408 (1970) — observance of a custom “does not necessarily amount to due care or the lack of it,” and “the doing of a negligent act is not excused by the fact that it is customary”
- Zimprich v. Stratford Homes, Inc., 453 N.W.2d 557, 560 (Minn. Ct. App. 1990) — quoted verbatim; the rule stated about a manufacturer, in a products case, about compliance with industry standards
- Muehlhauser v. Erickson, 621 N.W.2d 24, 28 (Minn. Ct. App. 2000) — the instruction an actual Minnesota jury received: “Evidence of standards or custom is not conclusive. It is just one piece of evidence.”
- Green Plains Otter Tail, LLC v. Pro-Environmental, Inc., 953 F.3d 541 (8th Cir. 2020) — Zimprich and Schmidt applied under Minnesota law to reverse summary judgment for a manufacturer; cited here as the most recent appellate application of the rule I could retrieve, without a pin cite because the Caselaw Access Project archive ends before this volume and I could not verify the star page
- Companion essays in this section: Protectionism in a Safety Costume, How Do You Know It Works?, Verifying Form Is Not Verifying Truth, What You Owe the Client You Automated, and The Practice of Law in 2035
Commentary on technology policy, engineering practice, and the business of law — the opinions, predictions, and stated falsifiers are the author’s. Not legal advice, not an opinion on any pending or contemplated litigation, and not a comment on any identified vendor’s products. The author’s economic interest is disclosed in the text: this firm pays for a closed frontier model and benefits from a competitive model market. No client information appears in this article. Questions about anything here: Send us a message or 612-470-6529.