Checkpointing is an easy feature to sell:
If the agent crashes, it resumes where it left off.
That sentence is reassuring and incomplete.
A workflow engine can restore its own state. It cannot atomically restore the external world. Between “call the refund API” and “save the successful checkpoint,” the refund may succeed, the acknowledgement may disappear, and the worker may die.
On resume, the workflow sees no recorded success. It calls the API again.
Congratulations: the agent is durable enough to duplicate damage.
The first production requirement for a resumable workflow is not persistence. It is safe repetition.
The Effect Gap
Consider this node:
def refund_node(state):
receipt = payments.refund(state.order_id, state.amount)
return {"refund_receipt": receipt}
There are at least four outcomes:
- the refund fails and the node records failure;
- the refund succeeds and the checkpoint records the receipt;
- the refund never reaches the service;
- the refund succeeds, but the worker dies before the checkpoint.
Outcomes three and four can look identical to the workflow: no receipt exists.
No local checkpoint algorithm can distinguish them after the fact. The ambiguity crosses a system boundary.
This is the classic effect gap:
external effect committed | local success not committed
Exactly-once processing is usually a combination of at-least-once delivery and idempotent effects, not a magical transport guarantee.
Resume Re-Executes Code
Durable agent frameworks make this visible. LangGraph’s interrupt documentation explains that when execution resumes after an interrupt, the node starts again from the beginning rather than continuing at the exact source line. Its execution guidance warns that side effects before an interrupt must be idempotent.
The same risk appears without human interrupts:
- a worker lease expires;
- a process crashes;
- a network response times out;
- an operator replays a task;
- a retry policy runs the node again;
- a new worker recovers the checkpoint.
Therefore, design every node under this assumption:
Any code since the last durable boundary may run more than once.
If that assumption is unsafe, the node boundary is wrong or the effect protocol is incomplete.
Give Every Logical Effect an Identity
An idempotency key should identify the business action, not the network attempt.
Bad:
random UUID generated for each retry
Good:
hash(workflow_run + logical_step + business_object + intended_effect)
For example:
refund/run-781/order-42/full-damaged-item
Every retry uses the same key. The payment service stores the first result and returns that receipt for later duplicates. If the same key arrives with a different amount or order, the service rejects it as a conflict.
The model should never generate this identity. A model may change wording, reorder steps, or hallucinate a new identifier. The orchestrator derives the key from durable state.
Record an Effect Ledger
Store effect intent separately from conversational state:
{
"effect_id": "eff_...",
"run_id": "run_...",
"type": "refund",
"request_hash": "sha256:...",
"idempotency_key": "refund/run-781/order-42/full-damaged-item",
"state": "REQUESTED",
"attempts": 1,
"external_receipt": null,
"last_error": null
}
Use explicit states:
PROPOSED → AUTHORIZED → REQUESTED → CONFIRMED
↘ UNKNOWN
↘ FAILED
UNKNOWN matters. A timeout after sending the request is not the same as a confirmed failure. The recovery action is reconciliation: query the external system by idempotency key or business reference before trying again.
Without UNKNOWN, teams turn ambiguity into a retry storm.
Use Outbox and Inbox Patterns
When you own both systems, reduce gaps with transactional messaging.
The workflow writes its state change and an outbox record in the same database transaction. A relay publishes the command. The receiving service stores the message ID in an inbox table while applying the effect in its own transaction. Duplicate deliveries see the inbox record and return the previous outcome.
This does not create one global transaction. It creates recoverable local transactions with deduplication:
workflow DB: state + outbox
message transport: at least once
tool DB: inbox + effect
For third-party APIs, you may only have provider idempotency keys and status lookup. If neither exists, classify the action as non-idempotent and require a stronger reconciliation or human-review path.
Do not pretend every tool supports safe automatic retries.
Put the Checkpoint After the Receipt, but Expect the Gap
The normal happy path is:
- persist authorized effect intent;
- call the tool with stable idempotency identity;
- receive and validate the external receipt;
- persist receipt and new workflow state;
- advance to the next node.
There is still a crash window between steps two and four. The idempotency key and reconciliation path make that window survivable.
Store receipts as immutable records containing:
- canonical request hash;
- external transaction or object ID;
- provider result;
- policy and approval IDs;
- timestamps;
- status and version;
- integrity hash.
A model-facing summary can say “refund completed.” The runtime retains the receipt that proves what that means.
Human Pauses Make Repetition More Likely
An approval node may:
- create an approval request;
- notify an approver;
- interrupt the workflow.
On resume, the node can begin again. Without idempotency, it creates another request and sends another notification.
Treat “create approval” as an effect with a stable key. If the approval already exists, return it. Store the approval separately from the node’s in-memory variables. When the workflow resumes, reload and validate the approval rather than assuming it is fresh.
The same applies to webhooks and callbacks. Providers may deliver them multiple times and out of order. Record event IDs, compare versions, and make state transitions conditional.
Compensation Is Not Rollback
Some effects cannot be made naturally idempotent, and some workflows fail after several valid effects.
A compensation is a new business action:
- cancel a reservation;
- issue a corrective credit;
- close a duplicate ticket;
- revoke a temporary permission.
It is not a time machine. The original effect may have triggered emails, user decisions, taxes, or downstream work. Compensation needs its own authorization, idempotency key, receipt, and failure policy.
Model the saga explicitly:
reserve inventory
→ authorize payment
→ create shipment
if shipment fails:
void authorization
release inventory
Do not ask the model to improvise compensation from a transcript. The allowed compensating action belongs to the workflow definition and domain service.
Version the Resume Contract
A run can pause for hours while code changes. If a node’s effect key derivation, schema, or policy changes, resume may no longer be compatible.
Record:
- graph and node version;
- state schema version;
- tool schema version;
- effect-key algorithm version;
- policy version.
Then choose a documented migration or pinning strategy. Test old checkpoints against new code. A resumed workflow must not reinterpret an old REQUESTED effect as safe to recreate.
Test by Crashing at Every Boundary
Happy-path tests barely exercise durability. Inject failure:
- before writing effect intent;
- after intent but before the tool call;
- after the tool commits but before it replies;
- after the reply but before checkpointing;
- after checkpointing but before acknowledgement;
- during human interrupt;
- after lease expiry;
- on duplicate and out-of-order callbacks.
Then resume twice.
The invariant is not “the node completed once.” It is:
- the business effect happened at most once where required;
- the runtime can determine confirmed, failed, or unknown;
- unknown outcomes reconcile safely;
- every transition has a receipt;
- no retry invents a new logical action.
The Order of Operations
Teams often build in this order:
checkpointing → retries → tools → production
The safer order is:
effect identities
→ idempotent tool contracts
→ receipts and reconciliation
→ explicit unknown outcomes
→ compensation
→ checkpointing and retries
Persistence is still essential. It makes long-running agents recoverable and enables human-in-the-loop work. But it also makes repetition routine.
If a workflow cannot safely execute a node twice, it is not ready to resume once.