Incident Recovery

Agents will fail. This runbook is the sequence you follow when a production agent goes wrong — the same discipline incident responders use, applied to LLM systems.

The Runbook

1. Stop the bleeding

Freeze the affected session so the agent can’t keep mutating state while you investigate:

sb.sessions.update_state(
    session_id=session_id,
    state={"incident": True, "frozen": True},
    reasoning="INCIDENT: pausing session for investigation"
)

Redirect new traffic to a safe version or the human queue.

2. Reconstruct what happened

Do not trust the agent’s summary. Replay the actual decision trace:

turns = sb.sessions.list_turns(session_id=session_id, limit=100)
traces = sb.traces.list(session_id=session_id)

for t in turns:
    print(t.reasoning)     # what the agent said it was doing
for tr in traces:
    print(tr.action, tr.details)  # what it actually did

3. Find the divergence point

Identify the exact turn where state stopped matching reality. This is your rollback target:

# Show the state at every checkpoint
state_versions = sb.sessions.list_versions(session_id=session_id)
for v in state_versions:
    print(v.version, v.reasoning, v.created_at)

4. Roll back to the last good state

sb.sessions.rollback(
    session_id=session_id,
    version=last_good_version,  # e.g. -3 (three checkpoints back)
    reason="INCIDENT-2314: rollback to last verified state",
)

Rollback is versioned and auditable — the rollback itself becomes a new version, so you can inspect or undo it.

5. Prevent recurrence

Check what enabled the failure:

Missing checkpoint? Add a checkpoint before the risky operation.
No validation? Add a schema check on tool output.
No permission check? Enforce an allowlist (see HITL).
State drift? Sync with the source of truth (see Failure Modes).

Patch the agent, test the scenario in staging with injected failures, then resume.

6. Resume

sb.sessions.update_state(
    session_id=session_id,
    state={"incident": False, "frozen": False},
    reasoning="INCIDENT-2314 closed, agent resumed",
)

Severity Matrix

Severity	Example	Action
S1	Agent deleted or corrupted critical data	Immediate freeze + rollback
S2	Agent produced wrong financial/medical output	Freeze, rollback, review
S3	Bad recommendations, no data loss	Monitor, correct, review
S4	Cosmetic errors	Log, fix in next deploy

Post-Incident Review

What failure mode was it? (hallucination, drift, tool failure, loop…)
Where was the missing checkpoint?
Add the scenario to your failure-injection tests (see Templates).

Next Steps

Failure Modes: the 7 ways agents fail
Rollback Strategies: technique comparison
Replay & Audit: debugging production runs

Getting Started

Core Concepts

Live Demos

API Reference

Agent Patterns

Integrations

SDKs

Production Playbook

Security & Compliance

Templates & Examples

Incident Recovery

Incident Recovery

The Runbook

1. Stop the bleeding

2. Reconstruct what happened

3. Find the divergence point

4. Roll back to the last good state

5. Prevent recurrence

6. Resume

Severity Matrix

Post-Incident Review

Next Steps

​Incident Recovery

​The Runbook

​1. Stop the bleeding

​2. Reconstruct what happened

​3. Find the divergence point

​4. Roll back to the last good state

​5. Prevent recurrence

​6. Resume

​Severity Matrix

​Post-Incident Review

​Next Steps

Incident Recovery

The Runbook

1. Stop the bleeding

2. Reconstruct what happened

3. Find the divergence point

4. Roll back to the last good state

5. Prevent recurrence

6. Resume

Severity Matrix

Post-Incident Review

Next Steps