Crash resume and high availability · Liyagent Documentation

Production serves from two app processes behind the edge (see Zero-downtime deploys). If one is killed or runs out of memory, visitors are retried on the other. This page is about the work that was in flight on the dead process: an agent run halfway through its tool calls.

Which runs survive a crash

Run	How it is taken back
Managed runs that may wait (an agent that opts into suspend, started from the playground, A2A or a resume)	Resumed on the surviving process from the last written step; the answer lands on its transcript and A2A task
Cron trigger runs	The cron run is handed back and runs again, adopting the checkpoint, so the agent continues from its last step and the run's outcome is recorded once
Teams turns	The Teams inbox row is handed back; the turn adopts the checkpoint and the reply, with its cards, is posted once
Ticket-event and webhook trigger runs, co-pilot turns	Resumed as any run; the caller has hung up, so the answer lands on the transcript
Source syncs and fine-tuning jobs	Resumable background jobs, queued again for any process's job worker

TICKETIQ_CRASH_RESUME_ENTRIES (default true) covers cron, triggers, the co-pilot and Teams turns. Set it to false to write ahead only the runs that may wait, as before.

An entry-point run that completed keeps its answer, sealed, for a while. A process that dies after the run's last step but before it handed the answer on is replaced by one that hands on that answer, never running the turn or its writes again.

How it works

Write-ahead checkpoints

After a model step answers with tool calls, and before any of them is made, the run writes a sealed checkpoint in one statement: the conversation so far, the calls the step asked for, the run's counters, history and delegation keys.

One more write marks a call that changes something (a write, a delegation, a remote agent, a sub-agent with tools) as in doubt just before it is made. A read is never marked: making it again is harmless, so a read-heavy step costs one write.

Run leases

The run holds a lease on checkpoint:<run id> for as long as it executes, renewed by its process. The checkpoint lease is short (8 s) so a dead process is noticed within seconds, and the leader looks for lapsed leases every 3 s. Each checkpoint write matches only while the row still belongs to the writing attempt, so a process that only stalled and was taken over can write nothing more: it stops at its next step. It is never doubled.

Resumed once per loss

The leader claims each running checkpoint whose lease ran out exactly once: the lease is taken in the database first, so of two sweeps one wins. The run continues from its checkpoint:

The calls the last step left are routed again, through every gate as it stands now.
The model is asked only for steps after the last one written. A step already answered is not asked again.
The checkpoint is verified against its seal first; one that does not verify is failed. The owner must still be allowed to run it, and the resumed run passes containment, a Stop, the channel and its run-as authority again.

A run lost more than twice is failed with the reason, and its transcript and A2A task are told. Every resume is audited as agent.run.crash_resumed, with the step, how many times it was lost, and how each call it left was settled.

Writes in flight

A call that was in doubt when the process died may or may not have reached its server. It is never blindly sent again:

The call in doubt	What the resume does
A read	Makes it again
A write held for approval	Routes it again; the approval queue coalesces it with the request already waiting. If a person decided that request meanwhile, the resume reads the outcome instead of filing a second one
A write whose vendor dedupes on the step key for longer than a resume can take (incident.io incident creates and updates)	Sends it again with the same key
A delegation to a sub-agent	Asks again by its delegation key, which finds the child's answer, or takes over the child's lapsed lease once
Any other write	Reads the outcome back where a reader exists; otherwise tells the model the outcome is unknown and that it was not sent again, and audits the settlement

A call once marked as a write stays a write, however its tool is classified by the time of the resume. Writes in flight are at most once, except a keyed write, which its vendor dedupes.

Step idempotency keys

Every execution of one step sends one idempotency key, derived from what identifies the step and never from the clock or a random number:

Scope	Key derived from
An agent run's tool call	The run's correlation id, its agent loop, how many model steps the run had taken, and the provider's tool-call id, all restored from the checkpoint
A workflow step	The workflow run id, the step id and how many steps the run had executed, so a retry sends the same keys and a loop back to the step sends new ones
An approved call	The approval id, however often it is sent

A vendor that dedupes on a key then applies the call once. A call outside every scope, such as a person's ad hoc call from the inspector, gets a fresh key per call. A provider tool-call id that is only a position number (call_0) is not an identity, so no stable key is derived from it.

What is not promised

A carried run's final answer is delivered at most once. A process that dies after the last step and before its entry point delivered loses it (entry-point runs keep theirs, above).
The model text of a step that was being generated when the process died is asked for again.
Discord nonces and SQS or SNS FIFO deduplication ids dedupe for minutes only, so those writes are settled rather than sent again.
Classifier training and red-team runs whose process stops are failed interrupted and must be started again.

Kill trials

tests/unit/test_kill_trials.py runs two local app processes over one Postgres. Each round starts a fresh victim process that begins cron fires, Teams turns from the durable inbox and webhook calls, then SIGKILLs it at a random moment between 0.2 s and 4 s after it is up, anywhere from before a run's first step to after its answer. The survivor runs the app's own recovery jobs throughout. Every trial is judged from the database:

Judged	Fails when
Lost	A trial's record never reaches its end, ends failed, or a Teams turn ends with no reply
Duplicate writes	More than one incident from one create, a Jira comment sent more than once, or a Teams reply posted more than once
Recovery time	Time from the kill to the trial's end, for every trial caught in flight

The targets are no lost trials, no duplicate writes, and recovery within 15 s. The harness is slow (about 15 s a round), so it runs only when you ask. It uses the test suite's Postgres, like the other database tests, and LIYAGENT_KILL_SEED fixes the kill timings for a repeat run:

LIYAGENT_KILL_TRIALS=200 .venv/bin/python -m pytest tests/unit/test_kill_trials.py -q

Measured results: see the published performance evidence.

Which runs survive a crash#

How it works#

Write-ahead checkpoints#

Run leases#

Resumed once per loss#

Writes in flight#

Step idempotency keys#

What is not promised#

Kill trials#

Next steps#