Recovering Agent Workflows Without Repeating Side Effects

A checkpoint is evidence of what your workflow saved. It is not a receipt from the CRM.

That distinction becomes expensive when an agent changes a record, waits for a response and gets a timeout. Restarting from the last checkpoint can repeat an action that already succeeded. A fluent explanation from the model does not resolve the uncertainty. Neither does a green tick beside the previous node.

I’d put recovery semantics in the tool boundary before adding more autonomous steps. The model can propose the next action; durable code should decide whether an earlier action is confirmed, safe to retry or still unresolved. The design below is a hypothetical integration.

The failure sits between two commits

Consider an agent closing a customer onboarding case. It updates the CRM and then emails the account owner. The CRM commits the update, but the response disappears. The worker crashes before saving the result. On restart, the checkpoint still says the update is pending.

Now add another failure window: the update result is saved, the email provider accepts the message, and the worker dies before recording its message ID. The CRM step and notification step need separate recovery decisions. Replaying the whole node bundles two independent uncertainties into one convenient mistake.

LangGraph’s current checkpointer documentation describes persisted graph state and resumption, while replay from an earlier checkpoint re-executes downstream API calls. Synchronous checkpoint durability reduces lost local progress; it does not create a shared transaction with a remote CRM. Treat orchestration recovery and external-effect recovery as separate contracts.

Persist intent before crossing the boundary

Create a durable operation record before dispatch. Give each intended effect a stable, opaque operation ID. Keep its tenant, target, action version, canonical payload digest, authorization reference, provider and status. Store sensitive payloads only where access and retention are appropriate; identifiers should not contain customer details.

operation_id: op-7d4c...
workflow_id: onboarding-42
action: crm.close_case.v1
target: case-318
payload_digest: sha256(...)
status: prepared | dispatching | confirmed | rejected | unknown
provider_request_id: ...
provider_result_id: ...
evidence_checked_at: ...
record_version: ...

These are proposed application fields, not a framework schema. A retry reuses the original operation and frozen payload. A changed recipient, amount or business intent creates a separately authorized operation. Generating a fresh operation ID on every attempt defeats deduplication. Using only a payload hash can incorrectly merge two intentionally identical actions.

AWS’s idempotent API guidance makes the same intent distinction with caller-provided request identifiers. It also places an atomicity requirement on the service implementing deduplication: recording the token and applying the mutation must belong to one reliable commit boundary. A client-side ledger cannot supply that guarantee for an arbitrary third-party endpoint.

Make the retry decision explicit

For the hypothetical CRM, the adapter should prefer a provider operation-status API or an idempotency contract scoped to the correct account. After a timeout, it reads authoritative evidence for the same operation. A matching receipt confirms the result; a definitive rejection records failure; an inconclusive lookup leaves the operation unknown.

A missing result in an eventually consistent search is inconclusive. So is finding a case whose final status happens to match the requested value: another user could have changed it. Stronger evidence binds the operation identifier, target, payload and resulting version. If the original request might still be running, do not let a negative lookup justify a concurrent second attempt.

Where the provider explicitly supports safe retries, reuse the same key and payload within its retention window. Check the actual endpoint and API version. Stripe is a useful concrete warning: its API v1 stores the first executed response, including a 500 response, and keys may be removed after at least 24 hours. Reusing a pruned key can create a new request. Those details are provider-specific, not a universal retry rule.

When the provider offers neither reliable deduplication nor decisive reconciliation, there is no clever checkpoint setting that removes the ambiguity. Hold the affected action for review. Show the operator what was attempted, the evidence collected and the possible consequences of retrying. Do not label an unknown result failed simply to keep the queue moving.

Separate business state from notification delivery

If you own the database, commit the business change and an outbox event in the same local transaction. A dispatcher can then attempt delivery independently. This proposed pattern closes the gap between that local change and the intent to notify; it does not make the email provider part of the database transaction.

Give the notification its own operation ID and persist any provider receipt immediately. An accepted message is not proof of delivery to an inbox. If the provider has no idempotent send contract, a deterministic marker and Sent search help investigation but leave concurrency and search-lag gaps. An uncertain send should stay uncertain until evidence resolves it.

Avoid compensating automatically just because confirmation is missing. Reopening the CRM case is another side effect, and it cannot unsend an email. Compensation needs its own business rules, permissions and recorded outcome.

Keep recovery workers from racing

Use conditional updates on the operation record to decide which worker may advance it. For expiring leases, add a monotonically increasing fencing value and require every controlled write path to reject stale values. Checking a lease once at startup does not prevent a stalled worker from waking after its replacement has begun.

A third-party API may not enforce your fencing value. That boundary still needs provider deduplication or serialized dispatch with explicit handling of requests already in flight. Lease expiry alone is not proof that the previous call stopped. On recovery, recheck current authorization before a new write; revoked authority can stop further work without erasing evidence of completed actions.

Test the crash windows

Anthropic’s agent-evaluation guidance distinguishes what an agent says from the final state of its environment. Apply that distinction to recovery. The following are proposed fault-injection cases, not benchmark results:

• Crash before dispatch: resume the prepared operation without inventing a second business intent.
• Commit the CRM change, drop its response, then restart: reconcile the original operation and assert that the mutation was applied once.
• Accept the notification, lose its receipt: retain unknown status unless authoritative evidence or the provider’s retry contract resolves it.
• Resume two workers together: verify that stale ownership cannot advance controlled writes, including after a lease expires.
• Change the payload, expire the provider key or delay search visibility: reject unsafe retries and route unresolved cases for review.

Measure duplicate effects and unresolved operations as well as completion rate. An implementation that finishes fewer cases automatically while preserving uncertain ones may be behaving correctly. Otherwise the easiest way to improve the success chart is to redefine uncertainty as success. The customers tend to notice.

For agents integrated into business infrastructure, recovery belongs in the integration design. Start with one consequential action, document its retry and reconciliation contract, and break the worker at every boundary. Expand autonomy only after the evidence shows that resumption preserves both business intent and confirmed outcomes.

Key takeaways

• A persisted checkpoint does not prove that an external action failed or succeeded.
• Freeze each intended effect under a stable operation ID and require a provider-specific retry contract.
• Keep uncertain results explicit; search absence and matching final state may not identify the original operation.
• Use separate operation records for business updates and notifications, with conditional ownership and prompt receipt checkpoints.
• Test timeouts after remote commits, concurrent recovery and expired deduplication windows before increasing autonomy.