Three ways an ops agent fails. One of them happened to us.
None of the three is a model failure. Agent incidents in operations look like ordinary operations incidents — credentials, schedulers, retries — which is both reassuring and the reason they get reviewed by the wrong people.
Failure mode one: the model executes
The most common design mistake we see is handing the language model the verbs. The model reads the customer's email, decides the right remedy, and calls the tool that issues the refund or amends the order. It demos beautifully, because the demo case is a case where the model is right.
Butler's support desk is split the other way. The model drafts the reply. Orders are executed deterministically by Butler, never by the model, and a human approves the send. The reason is not distrust of the model in the abstract; it is that the two halves have different failure surfaces and different people catch them.
Is caught by the reviewer, before anything leaves the building. Cost: a few seconds of somebody's attention. The failure is legible — you can read the wrong sentence.
Is caught by the customer, after inventory moved and an order record changed. Cost: a correction in the ERP, and the trust of the person it happened to.
The split has a second, quieter benefit. When the acting parts of the system are ordinary code rather than prompt-shaped behaviour, they get tested like ordinary code, with assertions that either pass or fail rather than evaluations you have to read statistically. Butler carries roughly 4,300 backend tests — a count of the test suite in the project's internal capability document, not a quality claim. That number would mean nothing if the model were the thing placing orders, because the interesting behaviour would sit outside the tests.
Failure mode two: the retry that becomes a second order
Agents retry. Schedulers retry. Partner systems retry. They should: from the caller's side a timeout is indistinguishable from a rejection, and resending is the only safe move available to something that cannot tell which one it got.
Which means the safe behaviour of the caller creates duplicate financial records unless the receiver is built to absorb it. Butler's merchant API — where partners push sales orders that create and confirm real ERP order records and deduct real inventory — is idempotent on the order reference, so a resubmission resolves to the same order rather than a new one, and it fails closed: when the write into the system of record cannot complete, the call errors rather than half-succeeding or parking the order somewhere the ERP will never see it.
The general form of the rule: if an agent can call it twice, and calling it twice creates something with money or stock attached, it needs a key supplied by the caller. “The agent will not retry” is not a property of an agent; it is a hope about one.
Failure mode three: environment bleed — the one that happened to us
A staging copy of the stack was still holding production Slack credentials. It ran its scheduled end-of-day report exactly as designed, and every day for nine days it posted a duplicate EOD draft into the real channel, alongside the real one.
Nothing in the agent went wrong. The scheduler fired on time. The code was the code we meant to run. The model was not involved in any part of it. The staging stack was simply configured with a token whose scope was production, and a token does not know which environment is holding it.
Three things we took from it, in the order they matter:
- An agent's blast radius is its credentials, not its prompt. Reading the system prompt and the tool list tells you what the agent intends to touch. The token tells you what it is permitted to touch. Only the second one is enforced, and only the second one still applies when the code is running somewhere you forgot about.
- “Staging is a copy of production” is exactly how they come to share secrets. A copy inherits the configuration, including the parts that should not have been copied, unless somebody deliberately breaks them. The default outcome of cloning an environment is a second environment with production's keys.
- We found it because it was loud. A duplicate message in a channel people read all day reports itself. The same misconfiguration on a write path — a staging stack confirming staging orders into the production ERP — would have been silent for the same nine days, and we would have found it in a stock count. Loudness is not a control. It is luck. The control is scoping the credential.
It is worth being precise about the severity too, since it would be easy to dress this up as a near-miss with a catastrophic tail. The credential in question could post messages. It duplicated drafts into a channel; it did not move money or stock. The lesson is not that we narrowly avoided disaster — it is that the property which kept this cheap (the token's scope) was the one nobody had checked, and it could as easily have been scoped wider.
The specific remediation is recorded in our internal incident record. We do not publish configuration details of live systems, and that includes the fix. The rule that came out of it is the one above: scope the credential, and never let a non-production stack hold a production one.
What this adds up to
Two of these three failure modes have nothing to do with language models, and the third is about keeping the model away from the verbs. That is the honest shape of running agents in operations so far: the model is rarely the dangerous component, and attention spent on prompt-level safety is attention not spent on the token, the scheduler and the retry path, which is where the incidents actually came from.
The practical consequence is about who reviews what. An agent deployment should go through the same review as any other scheduled process with credentials — what is it allowed to touch, in which environment, and what happens when it runs twice — and that review belongs to the people who already run that review for everything else.