Run durability, history and replay

Crash resume keeps a run going when its process dies. This page covers what happens when the run itself fails, and the history that every run leaves behind.

Saga compensation#

A run that creates a Jira issue, then a ServiceNow change, then fails to page the on-call leaves two records describing work that will never happen. Saga compensation undoes them.

Declare an undo#

In a workflow, give a tool step a compensate block. Its arguments can read the step's own output:

{"id": "issue", "type": "tool", "server": "jira", "tool": "create_issue",
 "args": {"summary": "{{ input.title }}"},
 "compensate": {"tool": "delete_issue",
                "args": {"issue_key": {"$expr": "steps.issue.output.key"}}}}

For a managed agent's own tool calls, save a compensation map on the agent (PUT /api/agents/{key}/compensations). Each entry maps a server/tool call to its undo, with expressions over the call's args and its result:

{"compensations": {"jira/create_issue": {"tool": "delete_issue",
                                         "args": {"issue_key": "{{ result.key }}"}}}}

An agent or workflow with no undo declared behaves exactly as before. Liya in Teams, the co-pilot, the HR and SAP agents, and scheduled runs are unchanged until an operator adds a map.

What happens on failure#

1. When a write succeeds, its undo is resolved at once and registered against the run. The arguments are fixed at that point, so a later step cannot change what gets undone.
2. Retries run first. A failure an on_error handler or fallback takes is not a failure, and nothing is undone.
3. Once the run fails past its retries, the undos run newest first. Each goes through the same gates as the agent's own calls: the tool allow-list, Cedar, run-as, data policies and the write posture.
4. Each undo is marked as sending before its call and settled after it. It is sent under a step key derived from the run and the step, so a crash between the two never undoes twice at a vendor that honours the key. Running compensation again only runs what is still owed.
5. An undo that fails is retried up to three times, then left for the person. An undo the server holds for approval is recorded as held, with its approval id, and is not waited on.

Hand-off to a person#

A failed run that registered an undo is handed to a person automatically, once, with its full context: the failed step and its error, every attempt, the run's input and variables, and each undo with its outcome.

A workflow can set "on_failure": {"handoff": "ticket"} to be handed off even without undos, or "on_failure": {"compensate": false} to keep its writes and only hand off.

See and retry#

• GET /api/agents/{key}/sagas lists an agent's sagas and their status: open, compensating, compensated, partial, handed_off or closed.
• GET /api/sagas/{run_id} shows one, with every undo and the hand-off.
• POST /api/sagas/{run_id}/compensate runs whatever is still owed now.

Every step is in the audit log: saga.registered, saga.compensating, saga.compensation (one per undo, with its outcome), saga.compensated, saga.handoff and saga.closed.

The run event log#

A run's crash-resume checkpoint is one row, rewritten at every step, and its transcript is pruned by retention. Neither tells you what the run did, in what order, on whose authority, once it has moved on. The run event log does: every run is an ordered sequence of immutable events.

What it guarantees#

• Ordered and chained. Each event has the next number in its run and the hash of the event before it, so a dropped, edited or reordered event is detected. GET /api/run-events/{run_id} returns the events and whether the chain verifies.
• Immutable. The database refuses every update and every delete except the retention purge.
• State rebuilt from events. The same response includes the run's state folded from its events alone: status, steps, every tool call with its authority, decisions, approvals, retries, resumes and how it ended. It also names the crash-resume checkpoint of the run, so you can check the two agree.
• The same run across a crash. A run's events are keyed by its correlation id, which crash resume carries to the process that continues it, so a resumed run continues the same sequence.
• Words only where you keep them. Tool arguments and results, the task and the answer are kept only for agents recorded in full. Otherwise only their SHA-256 is kept.

Cost and settings#

Events are buffered in the process and written in one insert per model step, under a lock on that run alone, never a global one. A write that fails is logged and dropped; it never fails or holds up the run.

GET /api/agents/{key}/run-events lists an agent's runs in the log.

Cohort replay#

Before you change an agent's model, provider or instructions, replay its recent runs under the change and see what moves. The playground's replay asks one past question again with live tools. A cohort replay re-runs many past runs against the world they saw, and nothing is executed.

POST /api/agents/{key}/cohort-replays
{"override": {"model": "gpt-5-mini", "instructions": "Answer in one sentence."},
 "limit": 20}

• The cohort. The agent's most recent runs that ended and were recorded in full, or the run ids you name in runs (up to 50). A run recorded as metadata only kept no words to replay, so it is skipped and listed with the reason. A run whose event chain does not verify is skipped too.
• The replay. Each run's task goes through the agent as it runs today, with your override: the same guardrails, Cedar, caps and hooks. Every tool call is answered from the recording: the recorded call with the same tool and arguments, else the next unused call of that tool. A call the original run never made gets an error saying so, and is counted.
• Nothing for real. No server is called, no approval is filed, no undo is registered and no approval digest is touched. Calls to remote agents or the agent's own state are refused during a replay. Replays leave no runs of their own in the event log.
• The diff. For each run: whether the answer changed and how similar it is, whether the sequence of tool calls is the same (and which were added or dropped), whether the status changed, and the calls the recording could not serve. A run that answered before and fails now is a regression.

The cohort's verdict is same, changed or regressed. Results are kept: GET /api/agents/{key}/cohort-replays lists them and GET /api/cohort-replays/{id} shows one. A replay needs agents.operate and transcripts.content.read, because it re-sends what users said to a model, and every replay is audited as agent.cohort_replay with its override and verdict.

Routines from repeated runs#

An agent that resolves the same kind of request the same way every time is asking its model to rediscover that order on every run. Liyagent looks for those patterns in the run event log and offers each one as a routine: a workflow that makes the same calls with no model choosing them.

How a routine is compiled#

Once a day, and on demand (POST /api/agents/{key}/routines/mine), the agent's recent runs are read. A run counts only if it answered, every tool call succeeded, no call was held for approval or refused, and it was recorded in full. Runs that called the same tools in the same order are grouped, and a group seen in at least TICKETIQ_ROUTINE_MIN_SUPPORT runs (default 3) is compiled:

Every tool step falls back, on error, to the agent itself with the task it was given (input.task). The routine runs first, and the model takes over where the world no longer matches the pattern.

Adopting a routine#

A compiled routine is only a suggestion. GET /api/agents/{key}/routines lists them with their definition, parameters, the runs they came from and the definition's SHA-256 digest.

• POST /api/routines/{id}/adopt files an approval in the agent queue that names the digest. Nothing is saved until a person approves it.
• Approving it saves the workflow only if the definition still hashes to the digest the approver saw. A definition changed in between is refused.
• Rejecting it returns the routine to the suggestions. POST /api/routines/{id}/dismiss drops it for good: mining never suggests that pattern again.

Once adopted, the routine's tool calls go through the governed path like the agent's own: its writes are still held for approval, each bound to its own digest.

Suggestions, adoption requests, adoptions and dismissals are audited as agent.routine.*.

Immutable hand-offs#

What one agent hands another, or what a run hands a person, is kept as a snapshot that cannot change. "What did the planner actually tell the executor?" always has an exact answer.

• Content-addressed. A hand-off's content (who from, who to, what kind, the body) is hashed with SHA-256 and stored under that digest. Reading one hashes it again, so a row changed in the database shows as not verified.
• Versioned, never edited. The database refuses every update and delete. Changing what was handed over creates a new version whose parent is the digest of the one it replaces. Only the head can be revised (expected_version), so two people revising at once cannot both win.
• Recorded for you. A managed run's delegation to a sub-agent or a remote agent is a thread: version 1 is the task the parent gave, version 2 is what came back. A failed run handed to a person (see Saga compensation) is snapshotted, and the approval and the saga carry its digest. The words are kept only for agents recorded in full; otherwise the body keeps their SHA-256.

Each hand-off is a handoff event in the run event log and an agent.handoff audit row.

One mission id across everything#

A piece of work often spans more than one request: an operator asks Liya in Teams, the run waits for an approval, resumes on another process, delegates to a sub-agent and calls three MCP servers. The mission id ties all of it together, so "everything that happened for this incident" is one query.

Naming a mission#

Send X-Liyagent-Mission: <id> on any API call, or the W3C baggage member liyagent.mission=<id>. An MCP or A2A message can carry it in params._meta["io.liyagent/mission"]. The id is 1 to 128 letters, digits, ., _, : or -, starting with a letter or digit. Anything else is dropped, never repaired, and the work falls back to its own id.

A request that names no mission gets its correlation id (X-Request-ID) as its mission, so every row still has one.

Where it goes#

A detail key passed by code cannot set or change a row's mission: it is always taken from the work's context.