The Gun, the Trigger, and the Safety

The industry has a comfortable answer for who is accountable when AI runs your operation: keep a human in the loop. It is the right instinct and a useless control — a human in the loop of what, exactly? Field notes from a running build, and the word our own doctrine was missing.

On a Tuesday in August we gave a piece of our own automation permission to act without asking us first.

It was a small permission. If a specific, known, reversible failure appeared — a category we had seen dozens of times and corrected the same way every time — the system was cleared to correct it and tell us afterward, rather than wake a human and wait. Narrow. Well understood. The kind of thing every vendor deck calls self-healing and every operator calls obvious.

Before we turned it on, somebody asked the question that should always be asked and usually is not: what stops it?

Not who is accountable — we knew that, a named person owned it. Not is there a human in the loop — there was one, after the fact, by design. The question was narrower and harder. What is the mechanism that makes it impossible for this thing to act outside the box we drew? Not unlikely. Not monitored. Not caught on review. Impossible.

We did not have a clean answer that afternoon. What we had was a good engineer's judgment and a well-written runbook — which is to say we had a person and a document. Neither of those is a control.

"Human in the loop" is an intention, not a mechanism

Every executive team and every command staff arrives at the same reflex when autonomy comes up, and the reflex is correct. The Chief Executive says we are not handing the business to a black box. The Commanding Officer says no weapon system operates without a human accountable for its employment. Both are right, and both have just described an intent, not a control.

Intent is where doctrine starts, not where it ends. A commander's intent that never becomes a control measure on a graphic is a sentence in a briefing. It tells everyone what right looks like and constrains no one. The same is true in a board room: a policy statement that automation will always have human oversight, unaccompanied by a mechanism that enforces it, is a statement of preference that will survive exactly until the first busy quarter.

The uncomfortable part is that most operations cannot answer the mechanism question at all. Ask where the boundary is enforced and you get a person's name, a runbook, or a code review. Those are three different ways of saying we are relying on someone being careful. That works, right up until the night it does not, and the failure mode of relying on care is that nothing looks wrong until everything is.

A framework is not paperwork. It is the impact area.

Here is the analogy we now use internally, and it has changed how we build.

Think of AI-integrated operations as the weapon and the accountable human as the one who fires it. That much the industry already accepts. But nobody fires a weapon in a vacuum. You fire it on a range — and a range is not a suggestion. It has a designated impact area. It has surface danger zones. It has a line that weapons do not cross loaded, a set of conditions under which the range goes cold, and a single person who can call it. Every one of those is a physical or procedural control that makes certain outcomes impossible rather than merely discouraged.

That standard is the whole point. A range does not exist to slow the shooter down. It exists so the shooter can fire at full speed, with live ammunition, without the organization having to hold its breath. The framework is what makes going fast survivable.

An operation that has deployed autonomous remediation without a control set has not built a range. It has built a shooter, handed them live rounds, and pointed them at open country in the dark. The rounds will mostly land somewhere harmless. Mostly is not a control.

Which frameworks, and why more than one

When we set out to answer what stops it, the honest first move was to stop pretending we were going to invent the answer. Control engineering is a mature discipline with sixty years of institutional knowledge behind it, most of it paid for in other people's incidents. We did not invent any of this. We stood on it.

The set we operate to, and what each one is actually for:

• ISO/IEC 27001:2022 — the management system. Not a control list; a discipline for deciding which controls apply, proving they operate, and correcting them when they do not. It is the thing that makes the rest a system instead of a collection.
• NIST Cybersecurity Framework 2.0 — the common language. Its 2.0 release added Govern as a function alongside Identify, Protect, Detect, Respond and Recover, which is precisely the gap this article is about. It is how a CISO and an operations officer describe the same posture without a translation layer.
• NIST SP 800-53 Rev. 5 — the depth. When a control needs to be specified at the level an auditor or an authorizing official can test, this is the catalog that has already written it down better than we would have.
• SOC 2 Trust Services Criteria — the customer-facing proof. An attestation is how a buyer who will never see your infrastructure gets evidence rather than assurances.
• ISO/IEC 42001:2023 and the NIST AI Risk Management Framework — the newest and, for this subject, the most important. These govern the AI management system itself: how models are authorized, bounded, monitored and retired. Most operations standing up AI have not read either one.

Five frameworks sounds like five times the work. It is not, and the reason is the second half of this argument. The overlap between them is substantial enough that the crosswalks are published artifacts rather than an exercise you run yourself — the Secure Controls Framework publishes one openly, the Unified Compliance Framework has built a commercial business on the same insight, and NIST publishes its own crosswalk between the AI Risk Management Framework and ISO/IEC 42001. You implement a control once and it satisfies several obligations at the same time. The waste is not in the frameworks. The waste is in treating each one as a separate project, which is exactly what most organizations do.

The safety is not a brake

The objection we hear, from operators and finance alike, is that this is drag. The Chief Financial Officer sees a cost center with no revenue line. The 8-Shop sees a requirement that consumes hours it does not have. The engineers see a gate between them and shipping. All three are reacting to a real thing — badly implemented compliance is drag, and most of them have lived it.

But run the logic in the other direction. What does the control set actually buy?

It buys the ability to let the machine act. That is not a small thing; it is the entire economic case for AI-integrated operations. An organization that cannot bound its automation cannot safely delegate anything to it, which means it keeps paying humans to do work a system should have finished before anyone woke up. The organization without a framework does not get to move faster. It gets to move slower, permanently, while telling itself it is being prudent.

There is a second-order effect that shows up on the balance sheet. Control evidence that is generated continuously, as a by-product of the operation running, is worth more than the same evidence assembled by hand before an audit — and it costs a fraction as much. The audit stops being an event the organization braces for and becomes a query. That is a real reduction in operating expense, and it is available to anyone willing to build the controls into the pipeline instead of into a binder.

Where the safety has to live

Which is the practical heart of it. A control that lives in a document is a preference. A control that lives in the path of the work is a control.

A field note, from the day this article was drafted. We have a delivery gate — a script that runs a readiness check across security headers, accessibility, performance, crawl integrity and brand compliance, and refuses to promote anything that comes back red. While shipping an unrelated change, we wrote a small deployment helper that called the platform directly and skipped that gate. Not maliciously. It was faster, and it was going to work.

The gate blocked it. Not a reviewer, not a checklist, not somebody's memory of the policy at four in the afternoon — the mechanism refused, named the rule it was enforcing, and pointed at the sanctioned path. The helper was rewritten to route through the gate, and a comment now sits at the top of that file explaining that a second deployment path which skips the gate is not a convenience, it is a hole in the armor.

The control worked because it was not asking permission from the person it was constraining. That is the whole design principle. If your framework can be satisfied by someone remembering it, you do not have a framework. You have a culture, and cultures erode under deadline pressure in a way that code paths do not.

The so what matters more than the story. That gate has now caught three classes of defect before they reached a customer: an accessibility regression, a page that would have shipped without a canonical tag, and our own attempt to bypass it. The cost of building it was about a day. The cost of the incidents it has prevented is unknowable in the way that all prevented incidents are, which is precisely why organizations underinvest here and precisely why the discipline has to be structural rather than voluntary.

How many frameworks is too many?

There is a real answer, and it is not "as many as possible."

The point of diminishing return arrives when you add an obligation whose controls are already satisfied by the set you hold, and the only new thing it produces is a separate document to maintain. At that point the framework has stopped constraining behavior and started consuming attention — which is over-processing, one of the classic wastes, dressed in governance language. You have not made the operation safer. You have made it slower and given yourself something to point at.

The test we apply before adopting anything new: does this change what the machine is allowed to do? If the answer is yes, it earns its place. If the answer is that it changes only what we write down about what the machine was already not allowed to do, it is overhead and we decline it. That test has caused us to adopt fewer frameworks than we first intended and to implement the ones we kept considerably deeper.

The word that was missing

Our doctrine on this has been settled for two years and, it turns out, incomplete. It assigned the instrument and it assigned the agency — the machine provides the mechanism, a named human provides the decision and carries the consequence. What it never named was the thing that makes the arrangement safe to operate at speed.

The framework is the safety. It is not the brake and it is not the paperwork. It is the mechanism that makes the catastrophic outcome structurally unavailable, so the operator can work fast without the organization holding its breath — and so that when something does go wrong, and it will, the failure lands inside an area you designated in advance instead of out in the dark.

Which completes it:

And the range you are firing on is the framework. Build that first. Everything else in this series assumes you did.