CORE GUIDE
Retry amplification
A retry intended to improve reliability can multiply load and duplicate side effects when several layers retry the same ambiguous failure without shared budgets or idempotency.
Mental model
Retry amplification occurs when independent clients, queues, services or agent loops each respond to uncertainty by repeating work. Their retry policies compose multiplicatively, not additively.
Why it matters
Distributed AI systems often have retries in the model SDK, agent loop, HTTP client, worker queue and workflow orchestrator. During an outage, each layer can create more work precisely when dependencies are weakest. If the operation has side effects, repeated attempts may also create duplicate actions even after the original request eventually succeeds.
01
Give retries one owner and one budget
Map every retrying layer for a critical operation. Decide which layer owns policy, cap attempts and elapsed time, add jittered backoff for transient load, and propagate operation identity so downstream services can deduplicate. For ambiguous side effects, verify remote state before retrying rather than assuming a timeout means failure.
02
Example: a timed-out refund fans out
An agent retries a refund after the payment API times out. The HTTP library also retries, while the queue redelivers the job after its visibility timeout. One user action now produces several settlement attempts. A bounded design uses one operation ID, idempotent settlement, a single retry owner and state verification before another attempt.
Common failure modes
- Layering retries in SDK, application and queue without a shared budget.
- Using immediate retries during overload and worsening the dependency failure.
- Retrying ambiguous side effects without idempotency or state verification.
Engineering heuristics
- Inventory all retry layers before tuning any one of them.
- Use one operation identity across retries and delivery boundaries.
- Measure retry volume and amplification ratio during incidents.
Takeaways
- 01Retries can create load instead of resilience.
- 02Independent retry loops multiply one another.
- 03Idempotency, budgets and verification must travel with retry policy.
Reading evidence
This records actions you actually took; it does not claim mastery, proficiency, or certification.
Used in
This Concept is reused across these canonical learning paths.
Related concepts from the Knowledge Graph
These relationships come from the canonical graph, not a separate Guide taxonomy.