# Crash resume and high availability

A run whose app process dies goes on from its last finished step on the other process, and writes are never blindly repeated: checkpoints, leases, step keys.

Production serves from two app processes behind the edge (see [Zero-downtime deploys](/docs/operate/zero-downtime-deploys)). If one is killed or runs out of memory, visitors are retried on the other. This page is about the work that was in flight on the dead process: an agent run halfway through its tool calls.

## Which runs survive a crash

| Run | How it is taken back |
| --- | --- |
| Managed runs that may wait (an agent that opts into suspend, started from the playground, A2A or a resume) | Resumed on the surviving process from the last written step; the answer lands on its transcript and A2A task |
| Cron trigger runs | The cron run is handed back and runs again, adopting the checkpoint, so the agent continues from its last step and the run's outcome is recorded once |
| Teams turns | The [Teams inbox](/docs/operate/zero-downtime-deploys) row is handed back; the turn adopts the checkpoint and the reply, with its cards, is posted once |
| Ticket-event and webhook trigger runs, co-pilot turns | Resumed as any run; the caller has hung up, so the answer lands on the transcript |
| Source syncs and fine-tuning jobs | Resumable background jobs, queued again for any process's job worker |

`TICKETIQ_CRASH_RESUME_ENTRIES` (default `true`) covers cron, triggers, the co-pilot and Teams turns. Set it to `false` to write ahead only the runs that may wait, as before.

An entry-point run that completed keeps its answer, sealed, for a while. A process that dies after the run's last step but before it handed the answer on is replaced by one that hands on that answer, never running the turn or its writes again.

## How it works

### Write-ahead checkpoints

After a model step answers with tool calls, and before any of them is made, the run writes a sealed checkpoint in one statement: the conversation so far, the calls the step asked for, the run's counters, history and delegation keys.

One more write marks a call that changes something (a write, a delegation, a remote agent, a sub-agent with tools) as in doubt just before it is made. A read is never marked: making it again is harmless, so a read-heavy step costs one write.

### Run leases

The run holds a lease on `checkpoint:<run id>` for as long as it executes, renewed by its process. The checkpoint lease is short (8 s) so a dead process is noticed within seconds, and the leader looks for lapsed leases every 3 s. Each checkpoint write matches only while the row still belongs to the writing attempt, so a process that only stalled and was taken over can write nothing more: it stops at its next step. It is never doubled.

### Resumed once per loss

The leader claims each running checkpoint whose lease ran out exactly once: the lease is taken in the database first, so of two sweeps one wins. The run continues from its checkpoint:

- The calls the last step left are routed again, through every gate as it stands now.
- The model is asked only for steps after the last one written. A step already answered is not asked again.
- The checkpoint is verified against its seal first; one that does not verify is failed. The owner must still be allowed to run it, and the resumed run passes containment, a Stop, the channel and its run-as authority again.

A run lost more than twice is failed with the reason, and its transcript and A2A task are told. Every resume is audited as `agent.run.crash_resumed`, with the step, how many times it was lost, and how each call it left was settled.

### Writes in flight

A call that was in doubt when the process died may or may not have reached its server. It is never blindly sent again:

| The call in doubt | What the resume does |
| --- | --- |
| A read | Makes it again |
| A write held for approval | Routes it again; the approval queue coalesces it with the request already waiting. If a person decided that request meanwhile, the resume reads the outcome instead of filing a second one |
| A write whose vendor dedupes on the step key for longer than a resume can take (incident.io incident creates and updates) | Sends it again with the same key |
| A delegation to a sub-agent | Asks again by its delegation key, which finds the child's answer, or takes over the child's lapsed lease once |
| Any other write | Reads the outcome back where a reader exists; otherwise tells the model the outcome is unknown and that it was not sent again, and audits the settlement |

A call once marked as a write stays a write, however its tool is classified by the time of the resume. Writes in flight are at most once, except a keyed write, which its vendor dedupes.

### Step idempotency keys

Every execution of one step sends one idempotency key, derived from what identifies the step and never from the clock or a random number:

| Scope | Key derived from |
| --- | --- |
| An agent run's tool call | The run's correlation id, its agent loop, how many model steps the run had taken, and the provider's tool-call id, all restored from the checkpoint |
| A workflow step | The workflow run id, the step id and how many steps the run had executed, so a retry sends the same keys and a loop back to the step sends new ones |
| An approved call | The approval id, however often it is sent |

A vendor that dedupes on a key then applies the call once. A call outside every scope, such as a person's ad hoc call from the inspector, gets a fresh key per call. A provider tool-call id that is only a position number (`call_0`) is not an identity, so no stable key is derived from it.

## What is not promised

- A carried run's final answer is delivered at most once. A process that dies after the last step and before its entry point delivered loses it (entry-point runs keep theirs, above).
- The model text of a step that was being generated when the process died is asked for again.
- Discord nonces and SQS or SNS FIFO deduplication ids dedupe for minutes only, so those writes are settled rather than sent again.
- Classifier training and red-team runs whose process stops are failed `interrupted` and must be started again.

## Kill trials

`tests/unit/test_kill_trials.py` runs two local app processes over one Postgres. Each round starts a fresh *victim* process that begins cron fires, Teams turns from the durable inbox and webhook calls, then SIGKILLs it at a random moment between 0.2 s and 4 s after it is up, anywhere from before a run's first step to after its answer. The *survivor* runs the app's own recovery jobs throughout. Every trial is judged from the database:

| Judged | Fails when |
| --- | --- |
| Lost | A trial's record never reaches its end, ends failed, or a Teams turn ends with no reply |
| Duplicate writes | More than one incident from one create, a Jira comment sent more than once, or a Teams reply posted more than once |
| Recovery time | Time from the kill to the trial's end, for every trial caught in flight |

The targets are no lost trials, no duplicate writes, and recovery within 15 s. The harness is slow (about 15 s a round), so it runs only when you ask. It uses the test suite's Postgres, like the other database tests, and `LIYAGENT_KILL_SEED` fixes the kill timings for a repeat run:

```bash
LIYAGENT_KILL_TRIALS=200 .venv/bin/python -m pytest tests/unit/test_kill_trials.py -q
```

Measured results: see the published performance evidence.

## Next steps

- [Zero-downtime deploys](/docs/operate/zero-downtime-deploys)
- [Capacity and scaling](/docs/operate/scaling)