# Output drift

Find out nightly whether each agent still answers as it did last week: refusals, answer length, tool mix, escalations, confidence, grounding, rejected writes.

The same model, prompt and tools can start behaving differently: a provider updates the model behind an alias, someone edits a prompt, or a knowledge base loses a collection. Output drift shows that shift before it shows up as a complaint.

Two checks run side by side:

| Check | Asks | Compares |
| --- | --- | --- |
| **Input drift** | Do today's tickets and questions still look like what the system was built for? | The day's inputs, embedded and matched to the tenant's approved taxonomy, against the shares the categories were learned from |
| **Output drift** | Do the agents still answer the way they did? | Each agent's outputs in the last window against the days before it |

Input drift explains a change in what arrives. Output drift catches a change in what agents do, even when the inputs have not moved. Read them together: output drift with no input drift points at the agent, its model or its knowledge.

## What output drift measures

Per tenant and per agent, the last `TICKETIQ_OUTPUT_DRIFT_WINDOW_HOURS` (default `24`) are compared with the `TICKETIQ_OUTPUT_DRIFT_REFERENCE_DAYS` (default `7`) before them:

| Metric | Read from | Test |
| --- | --- | --- |
| `tool_mix` | Trace spans, tool calls by tool name | Chi-square |
| `answer_length` | Trace spans, output tokens per model call | Two-sample KS |
| `refusal_rate` | Answer records not answered | Chi-square |
| `escalation_rate` | Answer records escalated | Chi-square |
| `confidence` | Answer records by confidence band | Chi-square |
| `groundedness` | Answer records by grounding verdict | Chi-square |
| `approval_rejection` | The agent's write approvals that were rejected | Chi-square |

The score is the population stability index (PSI) of the window against the reference; answer lengths are binned at the reference's deciles. One set of bars reads across every metric:

| Level | When |
| --- | --- |
| `ok` | Below the warning bar, or the test says the shift could be chance |
| `warning` | PSI at least `TICKETIQ_OUTPUT_DRIFT_WARNING` (default `0.10`) and p < 0.01 |
| `critical` | PSI at least `TICKETIQ_OUTPUT_DRIFT_CRITICAL` (default `0.25`) and p < 0.01 |
| `insufficient` | Fewer than `TICKETIQ_OUTPUT_DRIFT_MIN_ITEMS` (default `30`) on either side |

A window with too little data is marked `insufficient`, never `ok`.

## When it runs

The nightly job (`TICKETIQ_OUTPUT_DRIFT_NIGHTLY`, default `true`) runs on the scheduler, at most once every 20 hours. A run stops starting new series after `TICKETIQ_OUTPUT_DRIFT_BUDGET_S` seconds (default `300`) and reports the rest as skipped.

Rows hold counts, shares and scores, never an answer's text. They are kept 180 days, per tenant, under row-level security.

Everything is computed from this instance's own tables: no model or embedding call, so nothing of a tenant's data leaves its region. Input drift, by contrast, embeds each input with the instance's embedding service, held to the tenant's residency.

## Read it

Open **Evaluations** › **Drift and judges**. Output drift is listed under input drift, worst first, with **Measure now** to run it for the tenants you can read.

| Question | Endpoint | Permission |
| --- | --- | --- |
| Each tenant's newest output drift rows per agent and metric, the bars, the last run | `GET /api/evals/output-drift` | `governance.read` |
| One series' history | `GET /api/evals/output-drift?tenant=<id>&agent=<key>&metric=refusal_rate&days=30` | `governance.read` |
| Measure output drift now | `POST /api/evals/output-drift/run` | `governance.write` |
| Input drift rows, or one tenant's history | `GET /api/evals/drift` (`?tenant=&source=&days=`) | `governance.read` |
| Measure input drift now | `POST /api/evals/drift/run` | `governance.write` |

Another tenant's rows are never shown. A manual run is audited as `eval.output_drift.run`.

## Alert on it

Create an alert rule on the `output_drift` metric (**Incidents** › **Rules**, or `POST /api/alerts/rules`). It reads the worst score in the rule's window, scoped to the fleet, a tenant or an agent. Input drift has its own metric, `input_drift`.

## Limits

- Evaluator scores over time are not part of output drift. Alert rules on `eval_score` partly cover them.
- The reference is a rolling window, so a slow drift over several weeks moves the reference with it.

## Next steps

- [Playground and evaluations](/docs/agents/playground-evals)
- [Cost per action](/docs/monitor/cost-per-action)