Harness × Loop·Production Incident·15 min

The $47,000 Retry

A refund tool timed out at the client, completed at the provider, then ran again because the Agent retried. You own the runtime before traffic reopens.

Experience

Work the problem before reading the explanation.

INCIDENT 002 · SEV-1

$47,000 of exposure from one reasonable-looking retry policy.

You are the on-call AI/backend engineer.

The refund provider completed a request after 5.2 seconds. The client had already timed out at 4 seconds and the Agent started another attempt. Across the current transaction volume, duplicate actions now imply $47,000 of gross exposure.

Objective

Keep transient-failure recovery above the release target while making repeated intent safe and keeping latency, cost and human review inside budget.

Stakes

Timeout does not mean failure when the remote side can finish after the client gives up.

In one sentence

Retry policy and idempotency policy are one reliability problem whenever tools can create irreversible side effects.

Loading Mission Engine…

Reflection

Turn the outcome into a rule you can reuse.

ENGINEERING DEBRIEF

A timeout is an observation, not a transaction outcome.

Retry policy and idempotency policy must be designed together when a tool has irreversible side effects.

The client saw a timeout, but the provider completed the first request. Retrying the same business intent without an operation-scoped idempotency boundary repeated the side effect. Compensation can reduce money lost after the fact, but it does not make duplicate execution safe. Disabling all retries solves the wrong problem by sacrificing recoverable work.

—At-least-once delivery is common; exactly-once side effects require a business boundary.
—Idempotency keys should represent the operation, not merely one HTTP attempt.
—Human approval is a risk control with latency and automation costs.
—Audit the provider state before deciding whether another irreversible action is safe.

Key takeaways

Timeout ≠ failure

The remote system can succeed after the caller has stopped waiting.

Retry + idempotency is one design

Recovery policy is unsafe when repeated intent can repeat side effects.

Compensation is not prevention

A reversal workflow can lower financial loss while the reliability contract remains broken.

Learn More

Connect the experience to concepts, references, and transfer.

LEARNING CONTEXT

Learning context

A timeout describes what you observed; idempotency determines whether repeating intent is safe.

Not seen

Mental models

S04-M04Reversible vs irreversible actions
S04-M05Human approval at the high-risk boundary
S05-M03Bounded autonomy, termination and escalation
S06-M01Timeout ambiguity: missing response is not confirmed failure
S06-M02Retry policy and retry amplification
S06-M03Idempotency boundary for repeated intent
S06-M04Compensation and recovery after side effects
S07-M01Traceability as causal execution history

Suggested backfill

Agent Loop Simulator

Transfer the model

An order-creation API times out after the provider may have committed the order. Design the safe recovery path.

View full learning path

Carry the idea into another problem or build.

Next incident: The Prompt Injection Attack

Your Agent is reliable enough to act. Now untrusted content tries to steer those capabilities.

Open security incident

EARLY ACCESS

Want more production incidents?

Join Early Access for new Production Labs and incident Missions.

Join Early Access