Write an eval set for Triage

Prerequisites#

• A running Liyagent instance with resolved tickets that carry triage labels: from a source that brings its own category, team and priority, or tickets where a person corrected triage. Sync a source or run the Jira lab first.

Steps#

1. Create the dataset from your labelled tickets. Every resolved ticket with triage labels in scope becomes a case:
   Copy

   curl -b cookies.txt -X POST http://127.0.0.1:8787/api/evals/datasets/triage-core/from-tickets \
     -H "Content-Type: application/json" -d '{"limit": 500}'

2. Add edge cases as JSON Lines, one per line, with the expected category, team and priority:
   Copy

   {"id": "printer-3f", "title": "Cannot print to 3rd floor printer", "labels": {"category": "Hardware", "team": "Field Services", "priority": "P3"}}
   {"id": "ransomware", "title": "Ransomware note on finance share", "labels": {"category": "Security", "team": "Security Operations", "priority": "P1"}}

   Send them with POST /api/evals/datasets and {"name": "triage-core", "kind": "triage", "append": true, "jsonl": "<the lines>"}. Each write is a new dataset version.

3. Split it into calibrate and holdout sets: POST /api/evals/datasets/triage-core/split with {"seed": 7}.
4. Run it (POST /api/evals/runs with {"dataset": "triage-core"}) and open the run on Evaluations. Choose Pin as baseline.
5. Make a change that should hurt: route Hardware to Service Desk L1 with PUT /api/routing and {"table": {"Hardware": "Service Desk L1"}}.
6. Run again. The comparison shows the Hardware cases regressing against the baseline.
7. Put the gate in CI:
   Copy

   tiq eval run --dataset triage-core --fail-on-regression

Clean up#

DELETE /api/routing restores the default routing table.

Next steps#

• Playground and evals