Reliability Patterns for AI Agents That Ship to Production

The demo works. Then it runs at 03:00 against a customer dataset with two million records, and the agent sends 1,400 refund emails before anyone notices. Every team hits some version of this, and the difference between teams that ship reliable agents and teams that ship impressive ones is rarely model quality — it is whether they designed for failure before they designed for autonomy.

Reliability here does not mean "never errors." It means errors are bounded, visible, recoverable where possible, and never silently destructive. That requires a taxonomy of how agents fail, explicit control over side effects, and instrumentation that records what the system actually did rather than what it intended.

A Failure-Mode Taxonomy

Treat these as a design checklist. For each row, decide the detection mechanism and the recovery path before the first production run.

Hallucinated action. The agent claims to have performed an action it never performed, or performs one with invented parameters. Detect by verifying side effects against the target system, not by trusting the agent's narration.

Malformed or partial tool call. Invalid arguments, missing required fields, truncated responses. Detect with schema validation at the boundary. Recover with a repair loop that returns validation errors to the model.

Stale or wrong context. Acting on superseded facts — a policy changed ten minutes ago, a record updated since it was retrieved. Detect by version-stamping read data and checking before write.

Loop and thrash. Repeated tool calls without progress, or oscillating between two strategies. Detect with iteration ceilings, per-tool call budgets, and progress detection on repeated state.

Goal drift. Satisfying a sub-goal while abandoning the original request — the agent optimised "get a review" into "get five one-star reviews." Detect by re-asserting the original objective at checkpoints.

Cascading blast radius. One wrong step propagating across many records or accounts. Detect with batch-size caps and dry-run previews. Recover with reversibility and checkpoints.

Silent degradation. Quality collapses under distribution shift while every technical metric stays green. Detect with outcome metrics rather than execution metrics, plus periodic human review of sampled transcripts.

Credential and permission overreach. The agent operates with a broader permission set than the task requires. Detect by auditing effective permissions per task. Recover by scoping every action to a least-privilege token.

Idempotency: The Design Decision That Prevents Most Damage

The single highest-leverage reliability pattern is making side effects safe to repeat. An agent that retries is doing the right thing; an agent that retries a non-idempotent operation is a duplicate-order generator.

Design around this explicitly:

1. Classify every tool as read-only, idempotent, or non-idempotent. This classification belongs in the tool registry, not in a design document nobody opens.
2. Give non-idempotent operations a caller-supplied idempotency key. Derive it deterministically from the task identity and the step, so a retry reuses the same key.
3. Prefer the API's own duplicate protection where it exists, and make your key the value you pass into it.
4. Verify before retrying a write whose outcome is unknown. After a timeout, query the target system for the effect before deciding to re-issue. A timeout is not evidence that nothing happened.
5. Separate read from write in the permission model. Reads can be attempted freely; writes go through a narrower path with stricter budgets.
6. Design compensating actions, not just retries. For irreversible operations, define the reversal — a refund against a charge, a compensating entry in a ledger — and make the compensating action as reliable as the original.

Retry Design That Does Not Create Cascades

Naive exponential backoff applied uniformly is how a downstream incident becomes an upstream one. Better retry design separates the categories:

• Retryable: rate limits, transient network errors, temporary unavailability, capacity contention. Retry with jittered exponential backoff and a hard ceiling.
• Not retryable: validation failures, authentication errors, permission denials, malformed requests. Retrying wastes budget and hides the real defect. Fail fast and surface.
• Retryable then escalate: timeouts on writes, where the outcome is ambiguous. Verify state first rather than blindly re-issuing.

Crucially, cap retries at the task level, not just the call level. A run that retries each of forty calls three times is 120 calls of load on a system that is already failing. Include a per-run budget for both attempts and tokens, and a circuit breaker that stops new work when a dependency's error rate crosses a threshold.

Human-in-the-Loop: Placement Over Policy

"Human in the loop" is not a safety mechanism by default. If the human is asked to approve every action, approvals become rubber stamps after the fortieth identical request. If the human sees nothing until the end, they cannot catch anything. Placement determines value.

High-value checkpoints, in rough order of return:

• Before irreversible or high-blast-radius actions. Wire transfers, bulk deletions, external communications at scale, permission changes. This is where review genuinely pays for itself.
• When confidence is low and the cost of being wrong is high. Route on calibrated signals — tool errors, contradictory retrieval, missing required context — not on self-reported uncertainty, which is poorly correlated with actual correctness.
• On irreversible ambiguity. When the agent cannot determine whether an action already occurred, a human resolving it in seconds is far cheaper than a duplicate.
• Before irreversible completion claims. Require a human to sign off that an outcome is acceptable when the agent's plan involved judgement calls with material downstream effect.

Design the review interface around information, not just approval: what the agent intends, which evidence it used, which tool calls it already made, and what happens if approved. An approver without that context is a formality. Also batch low-risk actions into a digest rather than prompting 200 times individually, or reviewers will rubber-stamp everything.

Observability That Survives Contact With Reality

Traditional tracing answers "what happened." Agent systems need four answers: what did the agent intend, what did it actually do, what did that change, and was the outcome correct.

Instrument these:

1. Full trajectory per run — every prompt, tool call, arguments, raw response, and token cost, retained. Not sampled; the interesting failures are the rare ones.
2. Outcome metrics, not just execution metrics — task completion rate, correction rate, downstream error rates, human reversal rate. A run can execute flawlessly and produce the wrong result.
3. Cost and latency attribution per step, so runaway loops are visible rather than inferred from the invoice.
4. Prompt and tool versions on every record. Without this you cannot reproduce a failure, and reproducibility is the precondition for fixing anything.
5. Drift monitoring on success rate, tool error rate, and output distributions, alerting on statistically unusual shifts rather than fixed thresholds.
6. Periodic sampled human review of transcripts — the only mechanism that catches silent degradation, because both the agent and the metrics will agree that nothing is wrong.

Reliability Design Checklist

1. Build the failure-mode taxonomy and assign a detection and recovery path to each row.
2. Classify every tool by side-effect type in a shared registry.
3. Implement deterministic idempotency keys for every non-idempotent operation.
4. Verify state after ambiguous timeouts; never blind-retry a write.
5. Enforce task-level budgets for attempts, tokens, and wall-clock time.
6. Add circuit breakers on every external dependency.
7. Place human review at irreversible or high-blast-radius actions, with full context in the review interface.
8. Log trajectories, outcomes, and version identifiers for every run, and keep traces replayable — reconstructing the exact state that produced a failure is what makes it fixable.
9. Alert on outcome metrics and distribution shifts, not only on errors.
10. Sample transcripts for human review on a fixed schedule regardless of dashboard health.

Summary

Production agents fail in structural, enumerable ways — fabricated actions, malformed calls, stale context, loops, goal drift, and cascading blast radius — and each has a corresponding engineering control. Reliability comes from idempotency at the write boundary, retry classification that distinguishes transient from deterministic failure, human review placed at irreversible actions rather than sprinkled everywhere, and observability that records intent, action, effect, and outcome with version identifiers attached. Design those four before increasing autonomy, and an agent can fail loudly and cheaply instead of silently and expensively.

Frequently Asked Questions

How much human approval does a production agent actually need? Enough to sit on the actions that are irreversible, wide-reaching, or ambiguous in outcome — typically a small fraction of total actions. Approving everything produces rubber-stamping after a few dozen routine items; approving nothing produces expensive rework. The practical approach is an explicit risk tier per tool: read operations run freely, low-risk reversible writes run with budgets, and only irreversible or high-blast-radius operations queue for review.

How do I stop an agent from taking destructive actions without slowing it down? Separate the permission model from the reasoning model. Give the agent a least-privilege scoped token per task, expose write operations through a narrow gateway that enforces batch caps and idempotency keys, and require explicit dry-run previews for bulk operations. Reliability then comes from blast-radius limits rather than from gating every individual call, which keeps the fast path fast while making the catastrophic path structurally impossible.

What is the best way to detect silent quality degradation? Track outcome metrics rather than execution metrics, and combine them with scheduled human review of sampled transcripts. Execution telemetry stays green while quality falls, so you need signals like human reversal rate, correction rate, and downstream error rates that reflect whether results were actually right. Alerting on statistical shifts in those metrics, plus a fixed cadence of transcript review, catches degradation that no dashboard will flag on its own.