Agent Runtime·Advanced·17 min

Long-Running Agent Recovery Mission

The worker died at step 613. Restarting from zero repeats 147 external side effects.

RECOVERY INCIDENT

Resume without replaying the world.

You own a 2,000-document Agent workflow.

The process disappears after hundreds of completed actions. No durable checkpoint exists, cancellation is ignored, and external tools are not transactional.

Objective

Recover from worker loss without losing state, duplicating side effects, or letting obsolete jobs run forever.

Stakes

Long-running autonomy without recovery semantics converts routine infrastructure failure into silent data corruption and spend.

In one sentence

Long-running Agents need durable workflow state and replay semantics; process memory and prompt history are not a recovery architecture.

Loading Mission Engine…

Key takeaways

Memory ≠ durability

Prompt history cannot replace workflow state.

Replay is a contract

Every resumed step needs a side-effect policy.

Cancel is a guarantee

Obsolete work must stop predictably.

MENTAL MODEL

Durability is a workflow property, not a model capability.

Resume is safe only when completed state and side effects can be independently reconstructed or verified.

A production Agent needs durable execution records, idempotent side-effect boundaries, a resume policy, and bounded cancellation. These guarantees live in the runtime around the model.

✓Persist enough state to reconstruct what actually completed.
✓Make replay safe at the same granularity you resume.
✓Cancellation must have a bounded operational meaning.

LEARNING CONTEXT

Reusable mental model

Resume is safe only when completed state and side effects can be independently reconstructed or verified.

Not seen

CONCEPTS PRACTICED

concept-async-long-runningAsync and long-running task lifecycle
concept-checkpoint-resumeCheckpoint, resumable state and recovery
concept-interrupt-cancelInterrupt and cancellation semantics
concept-bounded-autonomyBounded autonomy, termination and escalation

Suggested backfill

No shipped prerequisite is required before entering this incident.

Transfer the model

A tool reports success, but your worker crashes before checkpointing it. What evidence can make resume safe?

View full learning path

Apply the method to an outcome

Use the same source, context, verification, and workflow ideas to produce a book-length artifact.

Build a Book Workflow