Output drift
Find out nightly whether each agent still answers as it did last week: refusals, answer length, tool mix, escalations, confidence, grounding, rejected writes.
- Read each agent's output drift and its history
- Measure drift on demand
- Alert when an agent's behaviour shifts
- Tell output drift from input drift
The same model, prompt and tools can start behaving differently: a provider updates the model behind an alias, someone edits a prompt, or a knowledge base loses a collection. Output drift shows that shift before it shows up as a complaint.
Two checks run side by side:
| Check | Asks | Compares |
|---|---|---|
| Input drift | Do today's tickets and questions still look like what the system was built for? | The day's inputs, embedded and matched to the tenant's approved taxonomy, against the shares the categories were learned from |
| Output drift | Do the agents still answer the way they did? | Each agent's outputs in the last window against the days before it |
Input drift explains a change in what arrives. Output drift catches a change in what agents do, even when the inputs have not moved. Read them together: output drift with no input drift points at the agent, its model or its knowledge.
What output drift measures
Per tenant and per agent, the last TICKETIQ_OUTPUT_DRIFT_WINDOW_HOURS (default 24) are compared with the TICKETIQ_OUTPUT_DRIFT_REFERENCE_DAYS (default 7) before them:
| Metric | Read from | Test |
|---|---|---|
tool_mix | Trace spans, tool calls by tool name | Chi-square |
answer_length | Trace spans, output tokens per model call | Two-sample KS |
refusal_rate | Answer records not answered | Chi-square |
escalation_rate | Answer records escalated | Chi-square |
confidence | Answer records by confidence band | Chi-square |
groundedness | Answer records by grounding verdict | Chi-square |
approval_rejection | The agent's write approvals that were rejected | Chi-square |
The score is the population stability index (PSI) of the window against the reference; answer lengths are binned at the reference's deciles. One set of bars reads across every metric:
| Level | When |
|---|---|
ok | Below the warning bar, or the test says the shift could be chance |
warning | PSI at least TICKETIQ_OUTPUT_DRIFT_WARNING (default 0.10) and p < 0.01 |
critical | PSI at least TICKETIQ_OUTPUT_DRIFT_CRITICAL (default 0.25) and p < 0.01 |
insufficient | Fewer than TICKETIQ_OUTPUT_DRIFT_MIN_ITEMS (default 30) on either side |
A window with too little data is marked insufficient, never ok.
When it runs
The nightly job (TICKETIQ_OUTPUT_DRIFT_NIGHTLY, default true) runs on the scheduler, at most once every 20 hours. A run stops starting new series after TICKETIQ_OUTPUT_DRIFT_BUDGET_S seconds (default 300) and reports the rest as skipped.
Rows hold counts, shares and scores, never an answer's text. They are kept 180 days, per tenant, under row-level security.
Everything is computed from this instance's own tables: no model or embedding call, so nothing of a tenant's data leaves its region. Input drift, by contrast, embeds each input with the instance's embedding service, held to the tenant's residency.
Read it
Open Evaluations › Drift and judges. Output drift is listed under input drift, worst first, with Measure now to run it for the tenants you can read.
| Question | Endpoint | Permission |
|---|---|---|
| Each tenant's newest output drift rows per agent and metric, the bars, the last run | GET /api/evals/output-drift | governance.read |
| One series' history | GET /api/evals/output-drift?tenant=<id>&agent=<key>&metric=refusal_rate&days=30 | governance.read |
| Measure output drift now | POST /api/evals/output-drift/run | governance.write |
| Input drift rows, or one tenant's history | GET /api/evals/drift (?tenant=&source=&days=) | governance.read |
| Measure input drift now | POST /api/evals/drift/run | governance.write |
Another tenant's rows are never shown. A manual run is audited as eval.output_drift.run.
Alert on it
Create an alert rule on the output_drift metric (Incidents › Rules, or POST /api/alerts/rules). It reads the worst score in the rule's window, scoped to the fleet, a tenant or an agent. Input drift has its own metric, input_drift.
Limits
- Evaluator scores over time are not part of output drift. Alert rules on
eval_scorepartly cover them. - The reference is a rolling window, so a slow drift over several weeks moves the reference with it.