BlogAI Agents, Tools & Vendors

When Your AI Takes Action, Who’s Actually Checking?

By 7 min read

Comprehensive monitoring. Of itself.

Most companies can’t honestly answer that for the AI they’ve already deployed. The gap matters more than any access-control checklist.

“I violated every principle I was given. I guessed instead of verifying. I ran a destructive action without being asked.” That’s what an AI coding agent wrote, in its own words, after deleting a startup’s entire production database last Friday in a single API call. Nine seconds, one routine task in a staging environment, and the whole thing was gone — along with the backups, which the cloud provider helpfully stored on the same volume as the source data.

I run agents in my own business. I read last week’s story with a chill of recognition — not because I think the same thing could happen to me (mine don’t have permission to touch anything that would actually matter), but because the failure mode the agent described is the same one every team deploying AI right now needs to be designing against. And right now, most of them aren’t.

The startup is called PocketOS. The cloud provider is Railway. The agent was Cursor running Anthropic’s Claude Opus 4.6. Those names will fade. The shape of the failure won’t.

This isn’t a coding-team problem. The same architecture shows up wherever a team has given an AI agent permission to act on its own — sales ops, customer success, finance ops, content production, lead workflows, RevOps reporting. Marketing ops happens to be where I see it most up close, because it’s where I work and because marketing was earliest to deploy AI agents at scale. But the function that comes next is whichever one moves fastest from “we’re piloting” to “it’s running in production” — and most teams don’t yet have a clear sense of what “in production” means when the thing in production can take action on its own.

The same failure, just quieter

The failures look different than PocketOS. An agent in your lead-scoring workflow that decides on its own initiative to mass-update Salesforce records to “fix” a data inconsistency. A content production agent that pushes a draft to live email because it inferred urgency from a Slack thread. A scoring model tuning itself against silently corrupted data nobody catches for a quarter. The architecture problem is identical — an agent took action without verification, and there was no checkpoint that could have stopped it. The visibility is just worse, because nobody sees the database disappear in nine seconds. They see the attribution data drift over a quarter, or the email that shouldn’t have gone out, or the lead score that turned out to be tuned against noise.

The vendor vocabulary is solving the wrong problem

Walk into any conversation about AI governance right now and you’ll hear the same shopping list: access controls, role-based permissions, audit logging, BYOM (bring your own model), data loss prevention, sanctioned tools. The vendor pitches are stacking. Every major SaaS platform has a “responsible AI” page going up. The IT and security teams know exactly what to do with this — they’ve been managing shadow IT for fifteen years.

Here’s the problem: that whole vocabulary was built for a different failure mode.

It was built for the Sarah-pasted-the-customer-list-into-ChatGPT problem. Real problem, worth solving. Access controls and approved enterprise tooling actually do the job there, because the failure is about who has access to what tool. Lock down the tools, sanction the alternatives, monitor the egress, done.

But the new problem isn’t shaped like that. The new problem is that you’ve sanctioned the AI. You’ve given it credentials and tool access. You’ve told it to do work autonomously. And somewhere in the middle of doing that work, it makes a decision you didn’t authorize and couldn’t have predicted, using a token you forgot was there, against a system you didn’t realize it could touch.

Access controls don’t stop that. PocketOS had access controls. The agent was authorized to be where it was. Its guardrails were advisory text in a system prompt, and the agent — by its own admission — ignored them. As one technical writeup put it: agent system prompts cannot serve as the sole enforcement layer; guardrails have to be implemented at the API gateway and token-permission level, not in advisory text the model may ignore.

That’s the reframe Zach Lloyd at Warp pointed at a few days before PocketOS, when his company open-sourced their AI terminal. Writing about why, he said the biggest bottleneck to development is no longer writing code — it’s the human-in-the-loop activities around the code: speccing the product and verifying behavior. He was talking about engineering leverage. But the implication points the same direction in every domain: if leverage moved upstream to verification, governance has to follow it there.

The two layers of AI oversight no vendor is selling you

Here’s what I’d put in front of anyone looking at agent deployment right now. Two layers, because lumping them blurs the reframe.

The technical layer is more developed than people realize.

  • Reversibility-gated action surface. Read actions (retrieve, summarize, recommend) can run autonomously. Write actions (mutate state, send, transact, delete) require explicit confirmation. Map every tool you give an agent against reversibility before deployment. PocketOS’s volumeDelete should never have been one API call away from a routine staging task.
  • Token scope minimization. Credentials given to agents are provisioned to the smallest possible surface. The PocketOS incident wasn’t just about an agent making a bad decision — it was that the token it found in an unrelated file had blanket destructive permissions across the entire infrastructure.
  • Confirmation enforced at the gateway, not in the prompt. A system prompt saying “don’t take destructive actions without permission” is a suggestion. An API layer that rejects unconfirmed destructive calls is governance. The model cannot ignore what the infrastructure won’t let it do.
  • Backups outside the blast radius. Trivial in hindsight. Common in practice.

These are well-known engineering patterns. They were well-known before agents existed. They just aren’t getting enforced because the deployment gold rush is moving faster than the discipline around it.

The operating layer is the part nobody’s selling you, because no vendor can ship it.

  • Named verification authority. Not “the team verifies.” A specific person with named accountability for verifying outputs of this specific agent, on this specific cadence. If everyone is responsible, no one is.
  • Override rate as a metric. Track what percentage of agent outputs get human override or modification. A flat-zero override rate isn’t proof your agent is great. It’s usually proof no one is actually checking.
  • Verification cadence, not verification gates. A one-time human review at deployment isn’t oversight. Real oversight is recurring sampling — a published cadence, an audited rate, a person whose calendar reflects it.

I’m starting to see this pattern emerging in marketing ops teams beginning to deploy agents — and if I’m honest, I’ve been a bit guilty of YOLO-ing the operating layer myself. The technical layer is patchy but improvable. The operating layer barely exists. There’s no general standard yet for what oversight looks like at this layer, and the smaller the company, the more they’re winging it. Introducing agents into that environment doesn’t just compound the existing exposure — it changes the kind of exposure entirely. Your IP, your customer data, your operational integrity were already at some risk from rogue tool use. Agents move that risk from “what people might paste into ChatGPT” to “what your sanctioned system might do on its own initiative.” Different problem. Different oversight.

The questions that cut through the vendor noise

Here’s the question to ask of any agent currently running in your stack: who verified the last ten outputs? On what authority? With what override rate?

If you can answer those three questions, you have governance. If you can’t — if the answers are “I’d have to ask,” or “we’d need to set that up,” or worse, “we don’t really need to” — what you have is an audit trail.

The PocketOS agent’s confession came complete with a beautifully detailed audit trail. It told the founder exactly which principles it had violated, in what order, with what reasoning. Every action was logged. None of it helped recover the database.

Audit trails record what happened. Verification prevents it. The human checkpoint is the governance unit. Everything else — the access controls, the policies, the logs, the dashboards — is theater around the checkpoint or, when the checkpoint is missing, theater instead of one.

If you couldn’t answer those three questions about an agent already running in your stack, that’s the conversation I’m having with every client right now. The operating layer is the part nobody is selling you — and it takes more thought than the technical side. Let’s talk about it.

Sources and Further Reading

Related Reading

More in AI Agents, Tools & Vendors

View all →
Join the "AI Strategist"

Weekly tips on using AI without the hype.

No spam, unsubscribe anytime. Join hundreds of other leaders getting practical advice.

Subscribe for Free