Published: July 4, 2026

The Trust Barrier

I'll go on record today and bet that the next biggest systemic AI failure won't be a hallucination - it will be an authorisation failure where we have handed autonomous agents production access, tool use, and spending power far faster than we've given them any scoped, revocable, auditable control over what they are actually allowed to do, repeating - at machine speed - the same over-broad access-control mistake that defined the endpoint, cloud, and IAM eras. This is not a black swan but a predictable surprise; the mechanism is already proven in miniature disasters - a Replit coding agent deleted a user's prod DB, a rogue agent reassigned bug tickets and talked Fedora maintainers into merging bad code into the Anaconda installer, and a Gemini-run café in Stockholm quietly over-ordered and spent itself toward insolvency while nobody was watching the balance - so the only genuine unknowns are timing, blast radius, and which brand name is attached when it finally happens at scale. But I bet it will happen.

What I am actually betting on is the phase transition in demand, not the surprise itself: consensus that agents are dangerously over-permissioned already exists (joint Five Eyes guidance on agentic AI, Gartner's forecast that over 40% of agentic AI projects will be canceled by the end of 2027, Anthropic's own "Zero Trust for AI Agents"),¹ but almost no one's budget moves until a single legible, headline catastrophe converts that latent agreement into a forced, industry-wide scramble to scope, gate, log, and insure agent authority - and whoever is already holding that control-and-provenance layer when the scramble begins wins the market the incident creates.

INCIDENT, ALREADY HAPPENED · Jul 2025

A coding agent deletes a customer’s production database

The vendor called it “a catastrophic failure.” One user, one database — a mini-disaster, fully legible, and the industry’s budgets did not move.

On the flip side, autonomous agents operating at scale shift the fundamental assumptions propping up entire industries - they move the activation energy for so many activities, from trading to research to even how we spend time as humans. Take this example:

You want dinner somewhere good tonight, say pizza. So you think of three or four places nearby that you kinda like, skim their menus, call one or two, and book. Not because your city only has four restaurants worth eating at, but because four is what an evening of looking has room for. That shortlist that feels like taste is actually this exact budget I am talking about. Human attention is the scarce thing, and the shape of the whole task bends around it.

Hand that same job to a fleet of agents and the ceiling is gone. It can pull every single restaurant within five miles, read all three hundred menus, check each one against tonight's availability and your taste, and action on it by actually calling all of them. The behaviour that made sense under a human time budget stops being the smart move, and a behaviour that was never even on the table becomes the obvious one. That's the thing most people miss about agents: they don't do our work faster, they sit behind a different set of barriers, and different barriers produce genuinely new ways of working. It runs through almost everything we do.

The same example exists for specialists, except the scarce resource isn't an evening, it's a career.

A trader who works one domain, one asset class and one jurisdiction because knowing a space properly is expensive (the barrier here): you have to learn the rules, internalise the constraints, build the scar tissue for what you can and can't do, and that takes years. So you pick one and you go deep. And just like the shortlist that feels like taste, the specialisation that feels like edge is really this budget in disguise. You trade one jurisdiction not because the other forty aren't worth trading, but because one is what a lifetime of learning has room for. The whole shape of the way jobs have been - narrow, deep, single-market - bends around how much one human can become expert in.

Hand that to a fleet of agents and, again, the ceiling is gone. They can research every possible jurisdiction at once - read all the rulebooks, hold every tax regime and market-structure quirk and reporting rule in memory simultaneously - and act across all of them in parallel. The strategy is no longer capped by what one person can master; it can span a hundred markets, exploit relationships between them that no single specialist was ever positioned to see because the complexity grows exponentially, and get arbitrarily more complex because the research cost that used to make that complexity impossible just went to zero.

Behaviour that was never structurally on the table - a strategy that lives across every jurisdiction at once - becomes the obvious one. It's not just faster trading anymore, it's fundamentally differently shaped trading, the same way the pizza night became a different task.

But here's where the title of this post - the trust barrier - comes in. The barrier that got removed was also the guardrail. The specialist's internalised knowledge of the space - the years of "what not to do," the licensing boundaries, the rules you don't cross - was never separate from the expertise. The legal framework layer and the trading edge were fused into one person's judgment. Once you strip the human out to get the wider net, you strip out the thing that was quietly keeping the trades compliant.

A fleet acting across every jurisdiction is, by definition, a fleet exposed to every jurisdiction's rules. So the exact move that unlocks the opportunity is the same move that detonates the authorisation surface. The wider the net, the wider the set of things you're now doing that you may not be allowed, licensed, or competent to do - at machine speed, across regimes no operator has ever read end to end.

Which is the whole bet, stated from the other side. When the barrier you remove is human expertise, you also remove the judgment that was silently doing the guardrailing. I believe that judgment has to be rebuilt outside the agent loop now: as scoped, revocable, auditable control over what each agent is actually allowed to do, grounded in the rules of every jurisdiction it touches. That control-and-provenance layer isn't a tax on the new way of working - it's the only thing that makes the wider net operable at scale. And whoever is holding it when the first headline blows up owns the market the incident creates.

Without this, you can't let agents loose on the things that actually carry weight - originate a loan, place a collections call, file a disclosure - because you can't trust them at scale.

The Barrier That Remains

Notice something strange, though. Every barrier so far fell to scale. The attention budget fell: throw a fleet at the restaurants and the shortlist becomes the whole city. The expertise budget fell: throw a fleet at the rulebooks and the single-market career becomes a hundred-market strategy. So why can't you throw more agents at the problem of trusting agents?

Because trust runs on different physics. Trust is a function of verification cost, and it commoditises only where verification is cheap. If a vector database vendor claims a million queries per second, an agent doesn't need to trust the claim - it spins up the benchmark and checks, in an afternoon, for pennies. On work like that, brand and reputation and trust are already worthless; the agent just looks. Every barrier made of labour falls this way, because labour is the thing agents mint for free.

But some work cannot be verified cheaply at any model size, because the verification loop is closed by the real world, on the world's schedule. You can't run an end-to-end eval on "did we get hacked." You can't cheaply check "did we stay legal across fifty states" - the answer arrives years later, as a reg exam, a lawsuit, a consent order landing long after the actions it punishes. Insurance is the purest case: the product is a promise verified only at claim time. Where the verification loop stays slow and expensive, trust stays scarce - and scarce trust is where all the durable value in the agent economy concentrates.

That's why this barrier is the last one standing. The other barriers were made of human hours, and hours are exactly what agents abolish. The trust barrier is made of verification lag, and no amount of intelligence collapses that time. You can't make the lawsuit arrive sooner by being smarter. The only move available is to change the work itself so that verification becomes cheap: attach to every action, at the moment it happens, a proof that it was allowed. Not faster agents. Legible ones.

Interpretability, Auditability and Why

The question that gates agents doing meaningful work at scale is whether you can trust one acting autonomously - and you only trust what you can understand. We need to be able to answer:

Why did it do that?

That question is one I've been asking for years as I've been working with LLMs. Two years ago I wrote about mechanistic interpretability, the work the labs are doing to pry open neural networks to see why they fire in order to read a model's mind one feature at a time - and in it I made a claim I still believe in. It's much easier to tell whether something is safe when you understand how it works. Around the same time I ran a small experiment on cultural bias in language models, mapping whose values they'd quietly absorbed - a Columbia team's paper on cultural fidelity in LLMs later cited it.²

I also argued for interpretability as defensive technology - that we shouldn't hand AI the decisions that actually carry weight, hiring, medicine, credit, the courts, until we can answer why it reached them, and I tied that need straight to regulatory compliance even back then. And writing up Situational Awareness pulled the thread tighter: the way we align models today, humans rating outputs one by one, stops scaling the moment the systems outrun the people meant to watch them. The watcher gets removed either way - the only question left is what stands in for it, and my bet is that the thing that stands in has to be an auditable proof of why each action was allowed.

These are all the same driving questions I ask today - how can I get to know why AI systems make the decisions they do.

Over the last half a year at SAMMY I have spent a lot of time researching and building computational law engines for agents operating in regulated industries, for exactly this reason - we build the engine that proves a decision was made legally. And underneath it sits the trust problem I opened with: an agent earns the right to act on its own exactly as far as we can trust it, and at machine scale we can only trust what it can prove within boundaries we have set.

Determinism Isn't Enough

Determinism is the thing everyone reaches for first, and on its own it gets you almost nowhere.³ Same input, same output - that's reproducibility, a needed part of the trust story but not the whole of it. Determinism gets you a verdict you can reproduce. It does not get you a verdict you can defend.

Interpretability is the part that survives scrutiny. It's the why behind the deterministic verdict, laid out as a chain a human can walk - this statute, applied to this fact, met this condition, triggered this override, reached this conclusion - written down so a regulator, an insurer, or another agent can argue with any link. A black box can give you the same answer twice without ever telling you that.

What I build at SAMMY is shaped around that chain. The rules live as data, not buried in code, so the law you're applying is something you can read and version. A deterministic resolver walks those rules against the facts, and the trace it leaves is the actual artifact, not a logging afterthought. When two rules contradict each other the engine halts on the conflict instead of guessing.

Silent conflicts are the most dangerous bug in compliance: when rules are scattered across application code (a common but poor approach) the last engineer to touch that function decided which one wins, probably without knowing they were ever deciding anything. The engine we've built also refuses to read missing data as permission. A fact it doesn't have is held as missing, never quietly rounded down to no violation, because in compliance the absence of evidence is the most common place a real violation hides.

And no LLM should sit in the critical path. Language models are allowed to help with upstream authoring work, turning statutes into these nodes accompanied by human review, but they never touch the live decision, so there's nothing in the path that can hallucinate, and every decision leaves a full audit trail back to the statute. Keeping the model out of that path is the point, because even temperature zero doesn't make a served model deterministic: your request gets batched with whoever else hit the server at that moment, the floating-point arithmetic shifts with the batch, and the answer moves. Send a frozen model a thousand identical prompts and you can get back eighty different completions;⁴ ask one the same legal question twenty times and the winning side flips on up to half the questions.⁵ You can engineer that wobble out with batch-invariant kernels, but that only gets you back to where this section started - a verdict you can reproduce and still can't defend.

half of all questions

Gemini-1.5 Pro50.4%

GPT-4o43.0%

Claude-3.5 Sonnet10.6%

0%50%100%

Share of 500 real appellate questions where the winning party changed across 20 identical temperature-zero runs.

ONE FLIPPED QUESTION · SAME PROMPT, 20 IDENTICAL RUNS

PPDPPPDPPPPDPPPPPDPP

this round’s “verdict”: plaintiff wins 16–4 · P = plaintiff, D = defendant

A compiled rule evaluator cannot do this: same input, same verdict, by construction — and unlike the model, it can show the chain that produced it.

BARS MEASURED (BLAIR-STANEK & VAN DURME 2025) — STRIP ILLUSTRATIVE

Everyone else is racing to make the answer better, chasing a higher score on some benchmark. My bet runs the other way: the reason is the product, and it lives in the engine itself, not in a model you have to take on faith. Every conclusion traces back to statute, every override is auditable, and the same chain is how the engine trains: when it gets a case wrong, the trace shows which rule failed, and fixing that rule is the update step. The chain a regulator audits and an insurer prices is the one the system learns from. "The logic chain is the gradient" is a training claim that doubles as a trust claim.

A compliance officer can't defend "the AI said so" to a regulator. Ask a pure-LLM tool why it read "commercially reasonable" the way it did and the honest answer is "embeddings," and you can't audit a vector. What you can defend is the opposite: statute, account data, logic chain, a determination that's mechanically reproducible. That difference is the moat, and the leading LLM legal tools don't clear it: even the purpose-built research tools, the ones wired to retrieval, hallucinate on something like one in six to one in three queries, and the general-purpose models without retrieval are far worse.⁶

The Loan Book

Theory is cheap, so we pointed the engine at a real loan book: live accounts from a US consumer lender, not a synthetic benchmark. This lender can't scale its compliance team to the number of accounts, so it samples a couple hundred out of millions.

The finding that stayed with me is a right-to-cure violation. Massachusetts law says that before you repossess a car, you owe the borrower a written notice and twenty-one days to cure the default - and the penalty for skipping it has teeth, because a missing or defective notice bars every enforcement step that follows: the repossession, the acceleration, the deficiency claim, all of it.⁷ Massachusetts means it, too - in late 2024 its banking regulator ordered a finance company that had repossessed vehicles without an adequate opportunity to cure to surrender its license.

On a set of accounts the borrower had defaulted, cured, and defaulted again, and the servicer treated the earlier cure letters as having permanently used up the borrower's rights. But the statute counts defaults cured, not letters sent - the notice obligation only falls away once a borrower has cured after notice three times - so the cure rights were still alive, and on one account the car was repossessed three more times without a single fresh notice, the most recent letter on file five years stale. The engine caught it from the notice ledger alone and flagged it systematically across the book - and it wasn't the only finding; a few more families of findings surfaced the same way.

Then the part that made me trust the whole approach: the lender's chief compliance officer personally reviewed a sample of the flagged accounts. Most findings passed review outright - and the right-to-cure violation came back confirmed, and worse than their own team had assessed it. The engine didn't just match the human baseline at reading a loan book against the law; it beat the professionals whose job it was, on their own accounts, with a defensible chain for every finding.

A defensible audit is the obvious thing to do with an engine like this. But the obvious thing isn't why I'd stake my company on it: the same trace I built to answer compliance turned out to answer four other questions I wasn't even asking.

The Same Proof

A bank examiner points at an account and asks why you flagged it. "Our model decided" ends that meeting badly; the only answer that survives the room is the chain - statute, data, logic. So you build the trace for the regulator and file it away as a compliance feature. Then four other parties ask four questions that look nothing alike, and the trace answers every one:

The insurer: how do I price the odds this thing breaks the law? You can't price stochastic behaviour or settle a claim against a decision nobody can reconstruct. The trace is the actuarial input.
The board: can we let it run without a human watching? Only if every action carries verifiable proof it was legal. The trace is the license to deploy.
The competitor: what stops us copying you? The model is commoditised and the rules are public, but you can clone an answer, not the proof of one. The trace, built up and audited over time, is the moat.
The customer thinking about leaving: what do I lose if I switch? Two years of evaluations linked to statute, an audit history a regulator won't accept a gap in. The trace is the switching cost, deepening every day it runs.

Five parties all need one artifact. I've stopped calling the legible reason a feature of the product. It is the product, and as best I can tell nobody else in this space builds one object to do all five jobs.

There's a version of this argument that eats itself, and I'd rather name it than have someone else do it. I said earlier that trust survives where verification is expensive - and the trace exists precisely to make verification cheap. So doesn't cheap verification commoditise the very trust I'm selling? Yes, and that's the point: when verification becomes cheap, the scarcity doesn't vanish, it moves one level down - to whoever produces the verification.

The moat was never the answer; answers are free now. The moat is the corpus of law-as-data, the provenance discipline behind every rule, and the compounding history of traces that makes each new proof credible - the loop that turns every case the engine touches into a sharper engine. You can clone an answer in a weekend. You cannot clone two years of audited proof-production, and you especially cannot clone it retroactively. When proofs become the currency, the mint is the business.

It's also the thing I went looking for inside neural nets, except here it finally works in the real world. Back then I wanted to know why a network fired and mostly the tech isn't there yet; here the reason is structural, something you can hold in your hand and put in front of a court.

And the engine that emits it is the kind of object you can prove theorems about. A goal we have here is to machine-check the engine's core steps in F*, the proof assistant behind the verified crypto in Firefox and the Linux kernel - Catala has already proven its core compilation step this way⁸ - so whether every branch behaves the way the semantics say becomes a theorem, not a code review. What stays human is whether the rules say what the statute says, and the trace is exactly where a lawyer checks that. The agents make the decisions; the substrate makes the proof that the decisions were legal.

Which would be a tidy theory if the timing were wrong. I don't think it is, though.

Why Now

Three things moved at once. Start with the law itself. For the entire history of regulation a statute was a document written for humans to read, and compliance meant paying humans to read it. The groundwork for something else has been accumulating for years: the US Code is published in an official XML schema, New Zealand's Better Rules programme has been piloting legislation-as-code since 2018, and France's tax authority rewrote its income-tax computation as an open-source compiler. Then, inside a roughly twelve-month window, the binding mandates landed. Europe's DORA became applicable in January 2025; MiCA now requires crypto disclosures filed in machine-readable Inline XBRL rather than as a PDF; Singapore made machine-readable building-code submission mandatory for large projects through CORENET X.⁹

And one of the mandates comes with a countdown attached. Article 12 of the EU AI Act requires high-risk AI systems to automatically record events over their lifetime - decision logs, kept and protected, so the system's behaviour can be reconstructed after the fact. The deadline has already slipped once, from August 2026 to December 2027 under the Digital Omnibus, but the direction hasn't.¹⁰ Almost every agent stack in production today logs prompts, because that's what the tooling makes easy. Article 12 wants the decision - what was evaluated, against what, with what result. The gap between the logs people have and the logs the law demands has a date and a fine schedule.

Each of these quietly moves a body of law from a document a human interprets to structured data a machine can evaluate over. The substrate I'm describing only has a market because the law spent a decade laying that groundwork and then, more or less all at once, started requiring it.

Then the thing everyone files under bad news: the models commoditised. The argument from a16z is that frontier models commoditise fast and the startups on top of them end up reverting to ordinary software moats, depth in a vertical and proprietary data and distribution. Elad Gil names compliance, specifically, as one of the markets still wide open.¹¹ The verticals worth wanting are the ones whose depth comes from the domain, not the model. Computable law is about as deep a domain as exists, and no incumbent is sitting on it.

And the counterintuitive one we've found working in this space. You'd assume a government pulling back from regulatory enforcement shrinks the compliance market - such as Trump neutering the CFPB. The opposite is happening. As federal enforcement retreats, state attorneys general are coordinating, and a firm operating nationwide now faces not one easy-to-navigate federal standard but a moving fifty-state matrix of overlapping and sometimes contradictory rules.¹² That is a strictly harder problem than a single federal regulator posed, and hiring more lawyers doesn't touch it; you solve a fifty-state matrix by encoding it. The retreat didn't remove the demand, it made the demand machine-shaped.

Put the three together and the window is narrow. The law is becoming computable, the models that ride on top have commoditised, and the regulatory surface just fragmented into something only a machine can hold in its head. Underneath all three sits the question I care about most, the one that turns a compliance product into a bet on the whole economy. What fundamentally shifts and breaks when being compliant costs almost nothing?

Thought Experiment: Law at Zero

Let's run it as a thought experiment: what shifts when legal effort and cost go to zero? Lift a constraint everyone has quietly built around and the old behaviour doesn't just get faster - it stops being the rational behaviour at all. When agents can act within the law every time they take an action - at least everywhere the law is computable - which of our assumptions were really that constraint in disguise? Most of them, in my opinion, exist for one reason: humans don't scale.

Look at how we actually do regulated work. We write a template contract and fill in the variables, because nobody is going to reason out a bespoke agreement for every counterparty. We bucket borrowers by FICO score because no underwriter can hold the full circumstances of every applicant in their head. These are scaffolds for human throughput, not expressions of what the law requires. The law would happily let you reason each case on its own facts. We just can't afford the labour.

An agent that can reason each case compliantly removes the labour. The scaffold stops being load-bearing and starts being a relic. And the biggest scaffold of all is the legal barrier itself: today it costs on the order of $500K to $2M just to stand up compliant operations in a single jurisdiction.¹³ That number is the barrier, and the barrier is built out of human hours. Make the hours computable and it trends toward zero.

This is where I think in second-order effects, because the first-order story is the boring part. When detailed road maps got built, they didn't serve the existing map market - they became the infrastructure the mobility economy sat on top of, worth more than the roads they described - and credit scoring became the rail all of consumer lending runs on, more consequential than most of the statutes meant to govern it. The legal version of that is the part that keeps me up. Visa made cross-border payment computable and the currency border went quietly invisible; legal borders dissolve for digital commerce the same way, and the question stops being "can you afford international lawyers?" and becomes "can you call an API?"

Who actually buys law at zero? Not mainly the banks - the firms that don't exist yet. US business applications are setting records - 2.55 million filings in the first five months of 2026 alone, up 17% on the year¹⁴ - and the shape of the new firms is changing: agent-native, tiny, one to three people running operations that used to take forty. If the number of American businesses runs from 33 million toward 60 or 80 million mostly-tiny firms, every single one of them faces the same fifty-state regulatory surface the big banks face, none of them can afford a compliance department, and all of them will be running agent fleets that need one. The regulatory burden on small operators isn't a caveat to this thesis but the customer list. Compliance departments were always a fixed cost only giants could amortise; law at zero is what lets a three-person firm carry Fortune-500-grade compliance the same way Stripe let it carry Fortune-500-grade payments.

Now let me ruin my own clean story, because it deserves it. The legal barrier isn't the only one. A three-person fintech launching in 50 states on day one sounds incredible right up until you ask how it originates anything. The banks don't trust you. You have no track record, no capital, no relationships. Compliance was never the only barrier, just the most legible one.

And there's the politics. The US is aggressively pro-business and the regulators are being gutted as I write this.¹² So does population-level enforcement, the thing that makes compliance worth paying for, even show up? I genuinely don't know. The barrier dropping is a force, not a guarantee, and pretending otherwise would be dishonest.

But a force is enough. When friction goes to zero you get scale, and scale is where the hard problem lives. It's the same problem we already hit the moment we went from running one agent to running many.

The Concurrency Problem

When I work, I run fleets of coding agents in parallel. It's most of how I work now: fan out a swarm of authoring agents, each writing a disjoint slice or feature, then funnel everything back into one serialised pass. One day I got greedy and launched 20 of them at once on the same checkout. Everything errored - one working directory, twenty writers, instant chaos. The fix was isolation: each agent got its own git worktree so it physically couldn't touch its siblings' work, a rules file and permissions enforced by the harness rather than the prompt, a git history that attributes every change, and one serialised review gate on every PR, including a trail of comments from the back-and-forth. Notice what none of that machinery does: it doesn't make a single agent smarter. All of it makes the agents legible - bounded before they act, reconstructable after. The throughput gain came from the boundary, not the brain.

The tell that this generalises: nobody coordinated it, yet everyone running coding fleets converged on the same stack independently, because everything else falls over. And look at what we all actually built - written rules the agent must obey, scoped authority over what it may touch, boundaries around what it owns, a permanent record of what it did, a gate that judges the work before it counts. Rules, authority, property lines, records, courts. Run autonomous agents at any real scale and you end up building a miniature legal system whether you meant to or not. Computing learned this lesson once already: safe multithreading came from memory models, locks, and structured traces - the substrates that make concurrent execution legible - not from faster threads. You can't safely parallelise what you can't verify after the fact.

But the coding case hides one free gift, and it's the reason the analogy has teeth. My harness works because code is the one domain where reality ships with version control: every action recorded, attributed, and - the part that matters - reversible. When an agent does something wrong, git revert exists. There is no git revert for a collections call, a filed disclosure, or a repossessed car. Coding agents got safe parallelism almost for free because the trace-and-rollback substrate already sat underneath them. The regulated economy has no git.

Now lift it up a level. Removing the watcher is the point of going autonomous; keeping a human on every single action is the exact cost the exercise exists to delete. What takes the watcher's place is really two things: a guardrail - a gate that refuses the illegal action before it happens - and a proof of why each action was legal after. The guardrail keeps the agent inside the lines; the proof is the harder problem, and it's the one that lets you insure it, defend it to a regulator, and trust it enough to walk away.

The labs building the agents land in the same place. Anthropic's own framework for trustworthy agents keeps a human in control "particularly before high-stakes decisions" (the high-stakes ones, not every one) and insists that "humans need visibility into agents' problem-solving processes."¹⁵ Read that as a dial, turned today to mandatory human review on anything that matters. You cannot supervise your way to confidence at a million actions a second; the only thing that turns the dial down is a substrate trustworthy enough to stand in for the watcher, and trustworthy here means auditable - it can show, for every action, why it was allowed.

The losses aren't abstract, either. An autonomous collections agent placing a few million calls a week carries a $500 to $1,500 exposure under the TCPA on every call that gets the consent rules wrong, a harassment presumption past seven calls per debt in any seven-day window, and a separate five-figure penalty if it repossesses a car from someone on active military duty without checking their status first.¹⁶ One agent and one statute with one bad week is already a class action. A million agents across every consumer-finance rule, with no legal substrate underneath, is an uninsurable liability with an API in front of it.

It's also a race condition at the scale of the economy: legally binding actions with no legible reason attached, so when something goes wrong - and at that volume something always goes wrong - you're left guessing which of a million interleavings produced the loss, and you can't reproduce any of them. This is why I think the legal trace is the agent economy's memory model: the shared, inspectable record that lets a million agents act concurrently in regulated space without collapsing into a race no one can debug. Interpretability is the concurrency primitive of autonomy. You don't bolt it on for compliance afterward; it's the thing that makes the parallelism safe in the first place.

And "uninsurable" up there is meant literally. It's the thing that gates deployment.

Pricing the Risk

When I first thought about insuring an autonomous agent, I reached for the obvious framing - what if it fails? - and got it wrong. The real constraint is that you can't price a failure you can't read: an insurer who can't attach a number to a risk doesn't write the policy at any premium. Reinsurers describe this failure in the open - when a model can't be independently evaluated, Gallagher Re warns, it "cannot be meaningfully priced," so insurers end up "pricing uncertainty rather than risk," which almost always means higher premiums, narrower cover, or both.¹⁷

In January 2026, ISO - the rating bureau whose standard forms sit beneath much of the US property and casualty market - introduced generative-AI exclusions for commercial general liability, stripping bodily injury, property damage, and personal and advertising injury "arising out of generative AI" from standard cover; carriers began adopting them within weeks, and AI is being carved out of D&O policies in parallel.¹⁸ Read that gap carefully, because it's silent: the premium stays the same and the coverage quietly leaves. By Microsoft's own telemetry, more than 80% of the Fortune 500 already run active AI agents,¹⁹ which means the standard market is draining AI liability out of their policies at exactly the moment their AI exposure is compounding. Every renewal cycle from here manufactures buyers for whoever can fill the hole.

A new class of insurer is already forming to fill it. AIUC - the Artificial Intelligence Underwriting Company - certified ElevenLabs' voice agents under its AIUC-1 standard after more than five thousand adversarial tests, unlocking a $50 million Lloyd's-backed policy that now sits behind every agent ElevenLabs' enterprise customers deploy.²⁰ Mount launched out of YC this spring as "the AI agent insurance carrier" - secure the deployment, measure the residual risk, insure it.²¹ Klaimee is writing cover for exactly what E&O and cyber now carve out;²² Munich Re's aiSure has underwritten AI model performance for years.²³

The category is validated - insurance for agents is being funded as a carrier-grade opportunity, not a rider. But notice how the new entrants price: a point-in-time audit. Thousands of red-team tests, once, then an annual premium on the result. That's a photograph of the risk. Nobody prices from a continuous, per-action, deterministic legal evaluation of everything the agent actually does - a livestream of the risk - because nobody else has a substrate that produces one. Mount insures the residual risk of an agent it red-teamed once. A substrate-fed insurer covers an agent it supervises on every action - and there is less residual risk, and it can prove exactly how much less.

THE PHOTOGRAPH

How the new insurers price: a point-in-time audit — 5,000 red-team tests, once — then an annual premium on the result. AIUC × ElevenLabs: a $50M Lloyd’s policy on a snapshot.

THE LIVESTREAM

How a substrate-fed insurer prices: every action gated before it fires and traced after — loss frequency engineered down, claims resolved by replaying the trace, coverage conditioned on the instrumentation. Coalition did this to cyber; Tesla to the car.

Unpriceable risk has a legal shape too. The EU's updated Product Liability Directive²⁴ is strict and no-fault, and it deliberately lowers the bar for opaque software: when technical complexity makes a defect too hard to prove, the burden flips and the maker is presumed liable. So a black-box agent doesn't dodge liability - it attracts liability you then can't price, because the failure lives somewhere in a billion weights with no path from the harm back to the cause. An interpretable agent inverts that. Every action produces a machine-checkable trace - this condition, on this fact, under this authority, produced this verdict - so the failure has an address, and a risk with an address is a risk an actuary can finally price. Munich Re's aiSure already works on exactly this principle - underwriting against a measurable error threshold and paying out on the shortfall - which is only possible because the behaviour is measurable to begin with.²³

But pricing is only half of what the substrate does to insurance, and it's the less interesting half. The gate from the concurrency section doesn't estimate the probability of an illegal collections call - it refuses it before the call fires, so loss frequency isn't a bet on an actuarial model being right; it's engineered downward by the same mechanism that produces the trace. Claims stop being investigations: a disputed loss resolves by replaying the trace. Adverse selection - the quiet killer of new insurance lines - gets solved by instrumentation: condition the coverage on routing actions through the gate, and you only ever insure what you can see. And underwriting becomes continuous: everyone else re-prices at renewal off an annual questionnaire, while a substrate re-prices off live traces.

The playbook has already worked once. Coalition built "Active Insurance" for cyber by conditioning cover on its own continuous scanning and monitoring, and rode it to one of the largest cyber books in the market; Tesla insures the car whose telemetry it reads.²⁵ Prevent what you can, see everything, insure the remainder. Nobody has done it to agent liability yet.

Follow that one step at a time and the whole bet sits on a single dependency: an interpretable WHY makes the defect localisable, which makes the risk underwritable, which makes the agent insurable, which is what lets it run autonomously at scale. Pull the WHY out and the chain doesn't degrade gracefully, it snaps at step one.

It helps to see where this sits in the full stack. Autonomous agents in regulated industries need seven things, and they're being built by different people at different speeds.

100%

The labs and clouds are racing on the first four. Courts and the EU are building the fifth, the liability frameworks. Insurers are circling the sixth, but they can't price it without the seventh, which is deterministic, traceable legal evaluation. Notice the bottom of the graph: a liability framework decides who eats the loss, but only the trace tells you how big the loss is likely to be, so the seventh is the precondition for the sixth.

Konstantine Buhler at Sequoia's AI Ascent named three engineering gaps for the agent economy: persistent identity, communication protocols, security.²⁶ Good list. He missed one: regulatory determinism, the gap that turns the other three into something you're allowed to deploy.

In this stack, identity says who the agent is, verification says what it did, and insurance says who pays when it's wrong. It's the economic settlement layer of trust - and it sits physically in the load path of commerce, because the certificate of insurance is the document demanded before the contract gets signed. In the agent economy that hardens into a rule: uninsurable means undeployable. No enterprise turns an agent loose on regulated actions its own liability cover just excluded.

Which is why the payoff runs the opposite way to how you'd guess. If insurability is the gate, then insurance becomes the distribution channel. The cheapest way to put an autonomous agent into a regulated industry is to route it through the substrate that makes it underwritable, because that's the path an insurer will actually stand behind - and unlike a new software category, the budget already exists as a line item. Nobody has to be convinced that insurance is a thing they buy; they have to fill the hole the January exclusions just cut in their cover. So the layer that issues and verifies the WHY doesn't sit off to the side as a compliance tool. It ends up sitting under the market itself, in the load path of every deployment that wants to be insured.

Sitting under every insured deployment is a good place to stand. So who gets to stand there?

Who Owns the Rail

The answer turns on a detail most compliance companies miss: the same engine serves both sides of the table. A bank runs it to know its loans are compliant before it books them; the regulator examining that bank wants the answer to exactly the same question, computed the same way.

Today those are two different worlds. The supervised firm has its systems; the examiner shows up with an advance notice letter and samples a few dozen loans out of millions. If the law is computable, that gap collapses, because the trace the bank produces to satisfy itself is the same trace the examiner ingests to supervise it. Build the substrate once and you can sell it to the regulated and to the regulator, and whoever wins the first regulator deals owns the rail the rest of the market has to run on.

The regulators are the part people underestimate, because the regulators are running on fumes. The IRS still computes individual taxes on a system written in assembly that dates to the 1960s. The Social Security Administration maintains more than sixty million lines of COBOL.²⁷ The GAO keeps republishing the same report: a list of the government's most critical systems, some of them now sixty years old, most of the planned replacements years behind. A replacement cycle is coming for the machinery that runs supervision itself, and nobody has built the machine-readable substrate that would sit at the center of it.

And here's the part that took me longest to accept: rails are standards before they're products. A certificate of compliance only ports - from the bank to the examiner, from the deployer to the insurer, from one agent to its counterparty - if there's a shared schema for what a trace even is. The agent economy is busily standardising everything adjacent to this layer: payment mandates, identity, interop protocols, even insurance audit standards like AIUC-1. Nobody has shipped the compliance-trace schema. The right move is probably the uncomfortable one: open the trace format itself, let anyone verify, and own the reference resolver, the certification, and the corpus that makes the traces mean something. Whoever sets that standard owns the rail in the only way rails are ever owned - by being the thing everyone else has to be compatible with.

There's a business-model consequence I was slow to take seriously. Sell a compliance tool and you're selling software, which puts you in a race against the next model that does the same thing for less. Sell the work, the finished and defensible determination, and every improvement in the underlying models makes your service cheaper to deliver and harder to compete with, because the moat is the substrate and the corpus, not the model.²⁸ Sequoia has a line for it: a copilot sells the tool, an autopilot sells the work. In compliance the autopilot is only possible if the work it sells is one you can stand behind, which lands you right back at the trace.

Where the Road Leads

Owning the rail opens onto something stranger than auditing agents after they act: a change in direction. Today the WHY flows forward - an agent takes an action and the trace explains it afterward, reconstructed so a human can follow the reasoning back to the statute. Do the thing, then justify it, the shape of compliance as we've always practised it. But when law is born machine-readable, the WHY can flow backward.

Better Rules and CORENET X, both from earlier, push machine-readability upstream - Better Rules into the drafting of the law itself, CORENET X into the building's design stage - as part of a wider rules-as-code movement.²⁹ France goes further, with the tax authority working directly with the team behind Catala so the tax code computes correctly rather than approximately.³⁰ This is the early edge of law that arrives as something a machine can read rather than a PDF a lawyer interprets later. And the research frontier is reaching toward case law: precedent encoded as structured, typed data instead of prose you hope a model parsed correctly, so the reasoning a court actually used becomes something a substrate can carry rather than guess at.³¹

Follow that to where it goes. A legislator runs a proposed rule against millions of synthetic records and reads the trace before passing it: this rule conflicts with that existing mandate for these 47,000 businesses, and here's the chain that shows why. Interpretability becomes the interface to lawmaking. For the entire history of legislation there's been a gap between what a law intends and what it actually does once it meets the world, and you only ever found out years later, in the case law. That loop closes.

And once the WHY is machine-readable, it stops being something a human reads after the fact and becomes infrastructure agents pass between each other. An agent about to take a regulated action emits its trace; the counterparty's agent verifies that trace before it agrees to transact; a regulator's agent ingests the same trace for supervision that covers every account against every rule, every day, until sampling is just a thing we used to do; an insurer's agent reads it and prices the risk in real time. The WHY stops being a report you file and becomes a portable proof-of-compliance, the certificate of the regulated agent economy.

Follow the certificate one step further and it accretes into a history. An agent that has emitted verified traces for two years has something no marketing page can fake: a record. Trace history becomes the agent's credit file - and just as the FICO score ended up more consequential than most of the statutes meant to govern lending, the agent's trust score becomes the thing counterparties actually check before transacting. I'd bet "agent trust score" reads as naturally in 2030 as "credit score" does today, and it gets built on whichever substrate is producing the traces. The reputation layer isn't a separate product; it's the trace corpus wearing a different hat.

Then take disputes. Two agents transact, something goes wrong, and both sides present their traces. Today's dispute machinery is a human process running at the pace of 1804 - lawyers, discovery, months to years - bolted onto money that settles in milliseconds. When both parties carry machine-checkable traces that resolve against the same formal ground truth, the disagreement itself becomes computable: replay both chains, find the link where they diverge, and the conflict has an address just like the defect did. Adjudication at machine speed is the piece of legal infrastructure nobody is building, and the same override-and-conflict machinery that halts the engine on contradictory statutes is, structurally, a judge.

Underneath both sits a primitive this essay has been quietly assuming: identity. The trace proves the action was legal; it doesn't prove who stands behind the agent - who deployed it, under what authority, with what scope and spend limits and expiry. Know-your-customer becomes know-your-agent: a signed, verifiable mandate that travels with the agent the way a power of attorney travels with a human proxy. The identity giants will build the authentication; what they don't have is the legal-authority layer - the machine-readable statement of what this agent is permitted to do, which is exactly the language of permissions, obligations, and immunities the substrate already speaks.³² Identity without authority is a name badge. Authority without identity is a blank cheque. The mandate is both, and it's the missing header on every trace.

I don't love every version of that world. Total observability has failure modes of its own, and there's a real unsolved tension I'd rather flag than bury: the trace contains the borrower's data - PII, sometimes privileged material - so a world of portable proofs is also a world of portable evidence. Audit-without-exposing (selective disclosure, hash-chained records that reveal on court order and nothing else) has to be designed into the schema from the start, or the certificate of the agent economy becomes its biggest liability leak.

Run all of it forward and enforcement itself inverts. When the evaluation runs continuously across every account, the odds that a violation goes unseen stop being a gamble and fall toward zero. Almost everything about compliance today quietly assumes detection is rare, and that assumption is the edge you're pricing when you cut a corner. Take it away and the rational move is simply to be compliant, always, because there's nothing left to gain by not being. Regulation stops being a punishment that lands years later in a consent order and turns into something closer to a continuous signal.

There's a flywheel hiding in all of this. Every system that calls the substrate leaves more traces, more traces sharpen the rules, sharper rules pull in more systems, and once enough of the market runs on the same legible reasoning, new law gets written to compute cleanly against it. Adoption becomes authority, and authority starts shaping the next statute.

Law is the last big domain we never managed to make computable, which is the reason this is a bet and not just a product. Once it computes, law becomes infrastructure. Like electricity, like GPS. You don't think about GPS, you just open a map. In the endgame you don't think about law: your system calls the legal layer, the layer says what's allowed, the system acts, the trace is stored, and the regulator can audit it whenever they want. The agent economy is the start. The entire regulated economy is the end. And what makes computing it feel civilisational rather than merely useful is that, for the first time, the reason is legible all the way down.

That's a big claim, and before I state the bet plainly, it deserves the strongest case against it I can make.

The Case Against

Three objections survive contact with everything above, and I'd rather argue them here than have them argued at me.

A trace proves procedure, not correctness. Feed the engine poisoned or wrong facts and it will hand you a confidently wrong verdict wearing a perfect audit trail - garbage in, gospel out. This is real, and the loan-book section quietly depends on it: the right-to-cure finding was reconstructed from the servicer's notice ledger, and a corrupted ledger would have produced the same perfect chain pointing at the wrong accounts.

Three things blunt it without dissolving it. The engine's three-valued logic³³ refuses the most dangerous version - missing data is held as unknown, never rounded to permission, so bad pipelines fail loud rather than silently absolve. The data problem is visible and auditable before deployment - you can score exactly which of the law's questions a system can answer, which is itself a diagnostic no black box offers. And when a wrong input does slip through, the trace gives the failure an address: you know precisely which fact corrupted which conclusion, which is the difference between a recall and a mystery. The substrate narrows the failure surface from "somewhere in a billion weights" to "this input or this rule" - it makes failure findable.

This is also why it isn't the smart-contract fantasy it sometimes gets mistaken for. The engine doesn't pretend the law is complete and self-executing: it halts on conflicts, holds unknowns open, and hands ambiguity to humans - code that knows where code stops.

Who audits the auditor? A proof held and verified only by its own vendor is just trust with extra steps - the examiner has swapped "trust the bank" for "trust the bank's vendor." Fair, and it's why the schema play from the rail section isn't optional. The trace format has to be open and third-party verifiable: anyone - a regulator, an insurer, a counterparty's agent, a rival - can take a trace and a published corpus version and re-derive the verdict independently. The maker of the trace must not be the only party able to check it. That's also the honest reason to want the standard more than the monopoly: a certificate only means something if the certifier can be audited too.

The deterministic single point of failure. This is the one a reinsurer will raise in the first meeting, so here it is in the open. A million stochastic agents fail idiosyncratically - one hallucinates, the others don't. A million agents gating through one rule corpus fail identically: a single wrong rule is the same wrong verdict everywhere, simultaneously - a systemic aggregation event, the same correlated-loss shape that makes cyber the hardest line in insurance. The determinism that makes the system defensible also concentrates its failure.

The mitigations are unglamorous and load-bearing: versioned corpora with staged rollout so a bad rule has a blast radius instead of a footprint; per-domain aggregation caps; the training loop that trains each corpus against ground truth before it ships; and - the flip side of correlation - when the failure does happen, it's legible and fixable in exactly one place. Patch the rule, replay the affected traces, and you have a complete, provable list of every impacted decision by the end of the day. A correlated failure you can enumerate is an insurable catastrophe; a diffuse one you can't reconstruct isn't. But anyone building or buying this layer should hold that trade with both hands.

The Bet

I opened by betting that the next systemic AI failure will be an authorisation failure, and that whoever is holding the control-and-provenance layer when the scramble starts wins the market the incident creates. That layer needs to do two things at once: adapt across jurisdictions and new case law without a team of engineers rewriting rules, and show its work - the full chain from statute to fact to verdict, callable by any agent before it acts. A substrate that makes the WHY legible, deterministic, and verifiable, so an action and the defensible reason for it arrive together. That is what becomes the rail under the regulated economy, and whoever owns it owns the thing every autonomous agent has to call before it touches a regulated transaction.

What would prove me wrong? Two things.

If frontier models reach five-nines accuracy on regulation-grade tasks and can emit a defensible, reproducible trace a regulator actually accepts, then the substrate is redundant. I don't think they will, because a probability distribution has no localisable defect. The trace would have to be structural, derived from the reasoning, not a plausible-sounding postscript stapled on after the fact.

The economics fail even before the accuracy does. A model has to reason its way from the law to the conclusion on every single query, and pays inference for every step of that derivation; a compiled evaluator derives the logic once and then just executes the handful of rules a case actually touches. A cheaper model shrinks that bill, it doesn't delete it. The gap is architectural, not a question of model quality, and the only way to close it is to stop re-deriving the conclusion on every query, which is the move the substrate already made.

And if the agent economy stays permanently human-supervised, you never need a per-action, machine-checkable WHY, because a person is always in the loop to answer for it. I don't think it stays that way either. The same economics that gave us template contracts and FICO buckets, the fact that humans don't scale, push just as hard toward autonomy. The moment agents do the reasoning, the legible reason becomes the gating constraint, not a footnote.

I started by saying the only genuine unknowns were timing, blast radius, and which brand name gets attached when it finally happens at scale. There's a fourth unknown, and it's the one I'm spending my years on: who's holding the layer when it does. So picture both endings, because the bet pays through either door. In one, the incident arrives first - an agent fleet somewhere does, at machine speed, what it was never authorised to do, and the scramble goes looking for the thing that could have prevented it: the gate, the trace, the proof. In the other, the good version arrives first: a legislator reading the trace before the law ever passes, watching the gap between what we meant and what we'd actually get close in front of them, before it costs anyone anything. We have never been able to see that gap until it was already too late. Either way, check back in a few years - let's see whether I end up being right.

References

The consensus documents. ASD ACSC, CISA, NSA, and Five Eyes partners: "Careful Adoption of Agentic AI Services", 30 April 2026 - a six-agency joint publication led by Australia's ACSC. Gartner: "Gartner Predicts Over 40% of Agentic AI Projects Will Be Canceled by End of 2027", 25 June 2025, citing escalating costs, unclear business value, and inadequate risk controls. Anthropic: "Zero Trust for AI Agents", May 2026 - a security framework for deploying autonomous agents in the enterprise. ↩
Kazemi, Gerhardt, Katz, Kuria, Pan, and Prabhakar: "Cultural Fidelity in Large-Language Models: An Evaluation of Online Language Resources as a Driver of Model Performance in Value Representation", arXiv, October 2024. ↩
Fannie Mae's Desktop Underwriter, launched in 1995, automates mortgage underwriting decisions through a deterministic rule set - the same input produces the same decision. It is the canonical proof that automated, reproducible decisioning long predates the current agent wave; what it lacks is the ability to re-derive across jurisdictions or absorb new case law without engineers rewriting the rules by hand. ↩
Horace He and Thinking Machines Lab: "Defeating Nondeterminism in LLM Inference", September 2025. Temperature zero makes the sampler deterministic, but inference servers batch concurrent requests together and most kernels change their floating-point reduction order with batch size, so identical requests still return different results: 1,000 identical prompts to Qwen3-235B produced 80 unique completions, first diverging at token 103. Batch-invariant kernels restore bit-identical outputs, at roughly 1.6-2x the latency. Atil et al.: "Non-Determinism of 'Deterministic' LLM Settings", 2024, measure the same effect on benchmarks - accuracy varying up to 15% across identical temperature-zero runs. ↩
Andrew Blair-Stanek and Benjamin Van Durme: "LLMs Provide Unstable Answers to Legal Questions", 2025. 500 legal questions distilled from real 2-1 federal appellate decisions, each sent 20 times to each model at temperature zero with every determinism knob pinned. The winning party flipped at least once on 10.6% of questions for Claude-3.5-sonnet, 43.0% for GPT-4o, and 50.4% for Gemini-1.5-pro. ↩
Stanford RegLab: "Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools", Magesh, Surani, Dahl, Suzgun, Manning, and Ho, 2024. Purpose-built, retrieval-augmented legal research tools measured hallucination rates of roughly 17–33%. General-purpose models without retrieval do far worse - 58–88% on specific legal queries - per Dahl, Magesh, Suzgun, and Ho, "Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models", 2024. ↩
Massachusetts General Laws c. 255B, § 20A: after a default the creditor "may not bring an action against the buyer or proceed against the collateral" until it gives written notice with a twenty-one-day right to cure, and a fresh notice is owed on each subsequent default until the buyer has cured after notice three or more times over the life of the contract. The license surrender: Massachusetts Division of Banks, Harbor Finance Company consent order, 31 December 2024, resolving allegations it "repossessed vehicles without providing borrowers adequate opportunity to cure the default." Nationally, the CFPB's "Supervisory Highlights: Special Edition Auto Finance" (Issue 35, Fall 2024) leads its servicing findings with wrongful repossession - servicers failing to cancel repossession orders that payments or extensions should have stopped. ↩
Merigoux, Chataing, Protzenko: "Catala: A Programming Language for the Law", ICFP 2021, §4.5. The compiler's most delicate step - translating Catala's default-calculus core to an executable lambda calculus - is proven correct (type preservation plus a simulation argument) in a roughly 3,500-line F* mechanization; the production compiler mirrors the certified translation side by side. F*, developed by Microsoft Research and Inria, is the proof assistant behind Project Everest and HACL*, whose verified cryptography ships in Firefox's NSS, the Linux kernel, WireGuard, and Python's hashlib. ↩
The 2025 cluster of machine-readable-law mandates, and the groundwork beneath it. EU DORA (Regulation (EU) 2022/2554) applies from 17 January 2025; the registers of information it requires on ICT third-party arrangements are filed to the European Supervisory Authorities in machine-readable format. MiCA's implementing rules (Commission Implementing Regulation (EU) 2024/2984, applicable 23 December 2025) require crypto-asset white papers in Inline XBRL. Singapore's CORENET X mandated machine-readable BIM submissions for larger projects from 1 October 2025. The US Code is published in the official USLM XML schema by the Office of the Law Revision Counsel. New Zealand's "Better Rules" programme has piloted drafting legislation as human- and machine-readable logic since 2018. France's DGFiP moved its income-tax computation to an open-source compiler (Merigoux et al., "A Modern Compiler for the French Tax Code," CC 2021) and works with the Catala team on rewriting it from the legal texts. ↩
EU AI Act: Regulation (EU) 2024/1689, Article 12 (record-keeping): high-risk AI systems "shall technically allow for the automatic recording of events (logs) over the lifetime of the system," with logging that enables traceability of the system's functioning. Article 113 originally set the high-risk obligations to apply from 2 August 2026; the Digital Omnibus package, approved June 2026, pushed them back to 2 December 2027 for stand-alone high-risk systems and 2 August 2028 for AI embedded in regulated products. ↩
Martin Casado and Sarah Wang: "Where Value Will Accrue in AI", a16z, May 2025, argue that models commoditise quickly and startups revert to traditional software moats; Elad Gil: "AI Market Clarity", July 2025, names accounting, compliance, and financial tools as still-open AI markets. ↩
Kate Berry: "As CFPB retreats, state AGs and bank regulators step up", American Banker, 21 January 2026. The CFPB under acting director Russell Vought scaled back nearly all supervision and enforcement while state attorneys general hired former CFPB officials and coordinated multistate investigations. Skadden: "State AGs Take the Lead on Key Consumer Protection Issues", April 2026, describes state AGs "coordinating through multistate investigations" and the result as "a fragmented enforcement landscape, with compliance obligations shaped at the state level rather than through uniform federal rulemaking." ↩ ↩²
Rough order-of-magnitude estimate for standing up compliant operations (licensing, legal, compliance staffing, audit) in a single regulated jurisdiction; the real figure varies widely by industry and state. ↩
US Census Bureau, Business Formation Statistics: about 2.55 million business applications (seasonally adjusted) filed January–May 2026, up roughly 17% on the same period of 2025 - continuing the record run of new-business formation that began in 2020. ↩
Anthropic: "Our framework for developing safe and trustworthy agents", 4 August 2025. It calls for humans to retain control "particularly before high-stakes decisions" and states that "humans need visibility into agents' problem-solving processes." ↩
Per-violation statutory exposure that makes automated action in consumer finance dangerous at scale. The Telephone Consumer Protection Act (47 U.S.C. § 227) provides $500–$1,500 per offending call or text. The CFPB's Regulation F (12 CFR § 1006.14) caps debt-collection call frequency through the "seven calls in seven days" presumption. The Servicemembers Civil Relief Act (50 U.S.C. § 3952) requires a court order before repossessing a protected servicemember's vehicle, with five-figure civil penalties per violation in recent DOJ settlements. ↩
Gallagher Re: "Anthropic's Fourth Way: Why Restricted AI Models Are a Challenge for Insurers", 2026. Ed Pocock, Global Head of Cyber Security: "If a model cannot be independently evaluated, it cannot be meaningfully priced"; insurers left to underwrite restricted models end up "pricing uncertainty rather than risk," which "almost always leads to higher premiums, narrower coverage or both." ↩
ISO/Verisk generative-AI exclusions, January 2026. Three new commercial general liability endorsements - CG 40 47 (broad form, Coverages A and B), CG 40 48 (Coverage B only), and CG 35 08 (products/completed operations) - exclude injury and damage "arising out of generative artificial intelligence." ISO's standard forms are widely adopted across the US commercial P&C market, so the endorsements set the default the market negotiates against. See Gallagher: "ISO Introduces Generative AI Exclusion in Commercial General Liability Policies" and Fenwick: "The End of 'Silent AI'? Emerging AI Exclusions, Coverage Fragmentation, and Practical Implications for Policyholders". ↩
Microsoft Security Blog: "80% of Fortune 500 use active AI agents", 10 February 2026. Vendor telemetry, and "active agents" is a lower bar than mission-critical autonomy - but as a floor for how much AI exposure now sits under standard policies, it's the relevant number. ↩
ElevenLabs / AIUC: "ElevenLabs secures first-of-its-kind AI Agent insurance", February 2026 - a $50M Lloyd's of London-backed policy behind every enterprise agent deployment, unlocked by AIUC-1 certification: 5,000+ adversarial simulations across security, safety, reliability, data privacy, and accountability. On AIUC itself, see Fast Company: "Rajiv Dattani is bringing insurance to the AI agent boom". ↩
Mount (Y Combinator, 2026): "the AI agent insurance carrier" - secures and insures autonomous agents by securing the workflow, measuring the residual risk, and transferring it through purpose-built policies. Its launch framing is telling: traditional insurers began pulling back from AI cover at exactly the moment corporate agents gained real authority. ↩
Klaimee (Y Combinator, 2026): "liability insurance for AI agents" - certification across eight risk dimensions backed by a financial guarantee and AI liability cover, built for exactly the claims that cyber and tech E&O policies now exclude. ↩
Munich Re's aiSure (also styled Insure AI) is among the first insurer products to underwrite AI model performance. Michael Berger, Head of Insure AI: the cover pays out when an AI's error rate exceeds an agreed threshold, so "the payout function is a function of the shortfall" (Emerj interview, 29 May 2023). ↩ ↩²
EU Product Liability Directive: Directive (EU) 2024/2853 of the European Parliament and of the Council on liability for defective products. Strict, no-fault liability; the claimant must establish the defect, the damage, and the causal link. Crucially, Article 10 introduces rebuttable presumptions of defectiveness and causation where technical or scientific complexity makes proof excessively difficult for the claimant - which is precisely the situation an opaque, high-dimensional model creates. ↩
Coalition's Active Insurance conditions cyber coverage on continuous scanning and monitoring of the insured's own attack surface - prevention and pricing fed by the same instrumentation - and rode that model to one of the largest cyber books in the market. Tesla Insurance prices auto cover from the vehicle's own telemetry - the Safety Score, adjusting the premium monthly - in most states where it operates, though not California, where telematics rating is prohibited. Both are the "instrument, prevent, then insure the remainder" pattern this essay argues is coming to agent liability. ↩
Konstantine Buhler: "AI Ascent 2025" keynote, Sequoia Capital, which framed the current moment as a stochastic era of computing and named persistent identity, communication protocols, and security as the open engineering gaps for the agent economy. ↩
The IRS Individual Master File is written in assembly language and dates to the 1960s (GAO-16-468, 2016). The Social Security Administration reported maintaining "more than 60 million lines of COBOL" (SSA Office of the Inspector General testimony, 27 September 2018). As of February 2025, only 3 of the federal government's 10 most critical legacy systems flagged for modernization were complete, some of them up to 60 years old (GAO-25-107795, 2025). ↩
Sonya Huang and Pat Grady: "Generative AI's Act o1", Sequoia, October 2024, frame the shift from selling software by the seat to selling work by the outcome; Julien Bek: "Services: The New Software", Sequoia, 2026: "a copilot sells the tool, an autopilot sells the work." ↩
New Zealand's "Better Rules" initiative treats machine-consumable logic as a first-class output of the drafting process, not a format applied afterwards. See OECD, "Cracking the Code: Rulemaking for Humans and Machines", 2020, for the broader rules-as-code movement it and CORENET X sit within. ↩
Merigoux, Chataing, Protzenko: "Catala: A Programming Language for the Law", ICFP 2021. France's DGFiP has collaborated with the Catala team to encode portions of the tax code as executable, conflict-halting logic. ↩
van Woerkom, Grossi, Prakken, and Verheij: "Hierarchical models of precedential constraint", Artificial Intelligence and Law, 2026. Precedent cases encoded as typed factor structures - fact situations paired with outcomes - so that what a precedent constrains can be computed rather than argued from prose. The factor-based tradition it extends goes back to Horty and Bench-Capon: "A factor-based definition of precedential constraint", Artificial Intelligence and Law, 2012. ↩
Wesley Newcomb Hohfeld: "Some Fundamental Legal Conceptions as Applied in Judicial Reasoning", Yale Law Journal 23 (1913). The classic decomposition of legal relations into rights, duties, powers, and their correlatives. ↩
Jan Łukasiewicz: "O logice trójwartościowej" ("On Three-Valued Logic"), Ruch Filozoficzny 5 (1920); Stephen Cole Kleene: "On Notation for Ordinal Numbers", Journal of Symbolic Logic 3 (1938), where his strong three-valued logic first appears. A fact can be true, false, or unknown - the third value is what stops missing evidence from being read as innocence. ↩

Loading comments...

PreviousThe Tech Frontier NextI Open-Sourced My Coding Agents' Memory

The consensus documents. ASD ACSC, CISA, NSA, and Five Eyes partners: "Careful Adoption of Agentic AI Services", 30 April 2026 - a six-agency joint publication led by Australia's ACSC. Gartner: "Gartner Predicts Over 40% of Agentic AI Projects Will Be Canceled by End of 2027", 25 June 2025, citing escalating costs, unclear business value, and inadequate risk controls. Anthropic: "Zero Trust for AI Agents", May 2026 - a security framework for deploying autonomous agents in the enterprise. ↩
Kazemi, Gerhardt, Katz, Kuria, Pan, and Prabhakar: "Cultural Fidelity in Large-Language Models: An Evaluation of Online Language Resources as a Driver of Model Performance in Value Representation", arXiv, October 2024. ↩
Fannie Mae's Desktop Underwriter, launched in 1995, automates mortgage underwriting decisions through a deterministic rule set - the same input produces the same decision. It is the canonical proof that automated, reproducible decisioning long predates the current agent wave; what it lacks is the ability to re-derive across jurisdictions or absorb new case law without engineers rewriting the rules by hand. ↩
Horace He and Thinking Machines Lab: "Defeating Nondeterminism in LLM Inference", September 2025. Temperature zero makes the sampler deterministic, but inference servers batch concurrent requests together and most kernels change their floating-point reduction order with batch size, so identical requests still return different results: 1,000 identical prompts to Qwen3-235B produced 80 unique completions, first diverging at token 103. Batch-invariant kernels restore bit-identical outputs, at roughly 1.6-2x the latency. Atil et al.: "Non-Determinism of 'Deterministic' LLM Settings", 2024, measure the same effect on benchmarks - accuracy varying up to 15% across identical temperature-zero runs. ↩
Andrew Blair-Stanek and Benjamin Van Durme: "LLMs Provide Unstable Answers to Legal Questions", 2025. 500 legal questions distilled from real 2-1 federal appellate decisions, each sent 20 times to each model at temperature zero with every determinism knob pinned. The winning party flipped at least once on 10.6% of questions for Claude-3.5-sonnet, 43.0% for GPT-4o, and 50.4% for Gemini-1.5-pro. ↩
Stanford RegLab: "Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools", Magesh, Surani, Dahl, Suzgun, Manning, and Ho, 2024. Purpose-built, retrieval-augmented legal research tools measured hallucination rates of roughly 17–33%. General-purpose models without retrieval do far worse - 58–88% on specific legal queries - per Dahl, Magesh, Suzgun, and Ho, "Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models", 2024. ↩
Massachusetts General Laws c. 255B, § 20A: after a default the creditor "may not bring an action against the buyer or proceed against the collateral" until it gives written notice with a twenty-one-day right to cure, and a fresh notice is owed on each subsequent default until the buyer has cured after notice three or more times over the life of the contract. The license surrender: Massachusetts Division of Banks, Harbor Finance Company consent order, 31 December 2024, resolving allegations it "repossessed vehicles without providing borrowers adequate opportunity to cure the default." Nationally, the CFPB's "Supervisory Highlights: Special Edition Auto Finance" (Issue 35, Fall 2024) leads its servicing findings with wrongful repossession - servicers failing to cancel repossession orders that payments or extensions should have stopped. ↩
Merigoux, Chataing, Protzenko: "Catala: A Programming Language for the Law", ICFP 2021, §4.5. The compiler's most delicate step - translating Catala's default-calculus core to an executable lambda calculus - is proven correct (type preservation plus a simulation argument) in a roughly 3,500-line F* mechanization; the production compiler mirrors the certified translation side by side. F*, developed by Microsoft Research and Inria, is the proof assistant behind Project Everest and HACL*, whose verified cryptography ships in Firefox's NSS, the Linux kernel, WireGuard, and Python's hashlib. ↩
The 2025 cluster of machine-readable-law mandates, and the groundwork beneath it. EU DORA (Regulation (EU) 2022/2554) applies from 17 January 2025; the registers of information it requires on ICT third-party arrangements are filed to the European Supervisory Authorities in machine-readable format. MiCA's implementing rules (Commission Implementing Regulation (EU) 2024/2984, applicable 23 December 2025) require crypto-asset white papers in Inline XBRL. Singapore's CORENET X mandated machine-readable BIM submissions for larger projects from 1 October 2025. The US Code is published in the official USLM XML schema by the Office of the Law Revision Counsel. New Zealand's "Better Rules" programme has piloted drafting legislation as human- and machine-readable logic since 2018. France's DGFiP moved its income-tax computation to an open-source compiler (Merigoux et al., "A Modern Compiler for the French Tax Code," CC 2021) and works with the Catala team on rewriting it from the legal texts. ↩
EU AI Act: Regulation (EU) 2024/1689, Article 12 (record-keeping): high-risk AI systems "shall technically allow for the automatic recording of events (logs) over the lifetime of the system," with logging that enables traceability of the system's functioning. Article 113 originally set the high-risk obligations to apply from 2 August 2026; the Digital Omnibus package, approved June 2026, pushed them back to 2 December 2027 for stand-alone high-risk systems and 2 August 2028 for AI embedded in regulated products. ↩
Martin Casado and Sarah Wang: "Where Value Will Accrue in AI", a16z, May 2025, argue that models commoditise quickly and startups revert to traditional software moats; Elad Gil: "AI Market Clarity", July 2025, names accounting, compliance, and financial tools as still-open AI markets. ↩
Kate Berry: "As CFPB retreats, state AGs and bank regulators step up", American Banker, 21 January 2026. The CFPB under acting director Russell Vought scaled back nearly all supervision and enforcement while state attorneys general hired former CFPB officials and coordinated multistate investigations. Skadden: "State AGs Take the Lead on Key Consumer Protection Issues", April 2026, describes state AGs "coordinating through multistate investigations" and the result as "a fragmented enforcement landscape, with compliance obligations shaped at the state level rather than through uniform federal rulemaking." ↩ ↩²
Rough order-of-magnitude estimate for standing up compliant operations (licensing, legal, compliance staffing, audit) in a single regulated jurisdiction; the real figure varies widely by industry and state. ↩
US Census Bureau, Business Formation Statistics: about 2.55 million business applications (seasonally adjusted) filed January–May 2026, up roughly 17% on the same period of 2025 - continuing the record run of new-business formation that began in 2020. ↩
Anthropic: "Our framework for developing safe and trustworthy agents", 4 August 2025. It calls for humans to retain control "particularly before high-stakes decisions" and states that "humans need visibility into agents' problem-solving processes." ↩
Per-violation statutory exposure that makes automated action in consumer finance dangerous at scale. The Telephone Consumer Protection Act (47 U.S.C. § 227) provides $500–$1,500 per offending call or text. The CFPB's Regulation F (12 CFR § 1006.14) caps debt-collection call frequency through the "seven calls in seven days" presumption. The Servicemembers Civil Relief Act (50 U.S.C. § 3952) requires a court order before repossessing a protected servicemember's vehicle, with five-figure civil penalties per violation in recent DOJ settlements. ↩
Gallagher Re: "Anthropic's Fourth Way: Why Restricted AI Models Are a Challenge for Insurers", 2026. Ed Pocock, Global Head of Cyber Security: "If a model cannot be independently evaluated, it cannot be meaningfully priced"; insurers left to underwrite restricted models end up "pricing uncertainty rather than risk," which "almost always leads to higher premiums, narrower coverage or both." ↩
ISO/Verisk generative-AI exclusions, January 2026. Three new commercial general liability endorsements - CG 40 47 (broad form, Coverages A and B), CG 40 48 (Coverage B only), and CG 35 08 (products/completed operations) - exclude injury and damage "arising out of generative artificial intelligence." ISO's standard forms are widely adopted across the US commercial P&C market, so the endorsements set the default the market negotiates against. See Gallagher: "ISO Introduces Generative AI Exclusion in Commercial General Liability Policies" and Fenwick: "The End of 'Silent AI'? Emerging AI Exclusions, Coverage Fragmentation, and Practical Implications for Policyholders". ↩
Microsoft Security Blog: "80% of Fortune 500 use active AI agents", 10 February 2026. Vendor telemetry, and "active agents" is a lower bar than mission-critical autonomy - but as a floor for how much AI exposure now sits under standard policies, it's the relevant number. ↩
ElevenLabs / AIUC: "ElevenLabs secures first-of-its-kind AI Agent insurance", February 2026 - a $50M Lloyd's of London-backed policy behind every enterprise agent deployment, unlocked by AIUC-1 certification: 5,000+ adversarial simulations across security, safety, reliability, data privacy, and accountability. On AIUC itself, see Fast Company: "Rajiv Dattani is bringing insurance to the AI agent boom". ↩
Mount (Y Combinator, 2026): "the AI agent insurance carrier" - secures and insures autonomous agents by securing the workflow, measuring the residual risk, and transferring it through purpose-built policies. Its launch framing is telling: traditional insurers began pulling back from AI cover at exactly the moment corporate agents gained real authority. ↩
Klaimee (Y Combinator, 2026): "liability insurance for AI agents" - certification across eight risk dimensions backed by a financial guarantee and AI liability cover, built for exactly the claims that cyber and tech E&O policies now exclude. ↩
Munich Re's aiSure (also styled Insure AI) is among the first insurer products to underwrite AI model performance. Michael Berger, Head of Insure AI: the cover pays out when an AI's error rate exceeds an agreed threshold, so "the payout function is a function of the shortfall" (Emerj interview, 29 May 2023). ↩ ↩²
EU Product Liability Directive: Directive (EU) 2024/2853 of the European Parliament and of the Council on liability for defective products. Strict, no-fault liability; the claimant must establish the defect, the damage, and the causal link. Crucially, Article 10 introduces rebuttable presumptions of defectiveness and causation where technical or scientific complexity makes proof excessively difficult for the claimant - which is precisely the situation an opaque, high-dimensional model creates. ↩
Coalition's Active Insurance conditions cyber coverage on continuous scanning and monitoring of the insured's own attack surface - prevention and pricing fed by the same instrumentation - and rode that model to one of the largest cyber books in the market. Tesla Insurance prices auto cover from the vehicle's own telemetry - the Safety Score, adjusting the premium monthly - in most states where it operates, though not California, where telematics rating is prohibited. Both are the "instrument, prevent, then insure the remainder" pattern this essay argues is coming to agent liability. ↩
Konstantine Buhler: "AI Ascent 2025" keynote, Sequoia Capital, which framed the current moment as a stochastic era of computing and named persistent identity, communication protocols, and security as the open engineering gaps for the agent economy. ↩
The IRS Individual Master File is written in assembly language and dates to the 1960s (GAO-16-468, 2016). The Social Security Administration reported maintaining "more than 60 million lines of COBOL" (SSA Office of the Inspector General testimony, 27 September 2018). As of February 2025, only 3 of the federal government's 10 most critical legacy systems flagged for modernization were complete, some of them up to 60 years old (GAO-25-107795, 2025). ↩
Sonya Huang and Pat Grady: "Generative AI's Act o1", Sequoia, October 2024, frame the shift from selling software by the seat to selling work by the outcome; Julien Bek: "Services: The New Software", Sequoia, 2026: "a copilot sells the tool, an autopilot sells the work." ↩
New Zealand's "Better Rules" initiative treats machine-consumable logic as a first-class output of the drafting process, not a format applied afterwards. See OECD, "Cracking the Code: Rulemaking for Humans and Machines", 2020, for the broader rules-as-code movement it and CORENET X sit within. ↩
Merigoux, Chataing, Protzenko: "Catala: A Programming Language for the Law", ICFP 2021. France's DGFiP has collaborated with the Catala team to encode portions of the tax code as executable, conflict-halting logic. ↩
van Woerkom, Grossi, Prakken, and Verheij: "Hierarchical models of precedential constraint", Artificial Intelligence and Law, 2026. Precedent cases encoded as typed factor structures - fact situations paired with outcomes - so that what a precedent constrains can be computed rather than argued from prose. The factor-based tradition it extends goes back to Horty and Bench-Capon: "A factor-based definition of precedential constraint", Artificial Intelligence and Law, 2012. ↩
Wesley Newcomb Hohfeld: "Some Fundamental Legal Conceptions as Applied in Judicial Reasoning", Yale Law Journal 23 (1913). The classic decomposition of legal relations into rights, duties, powers, and their correlatives. ↩
Jan Łukasiewicz: "O logice trójwartościowej" ("On Three-Valued Logic"), Ruch Filozoficzny 5 (1920); Stephen Cole Kleene: "On Notation for Ordinal Numbers", Journal of Symbolic Logic 3 (1938), where his strong three-valued logic first appears. A fact can be true, false, or unknown - the third value is what stops missing evidence from being read as innocence. ↩