# Computer use

Let an agent work a system that has no API, such as an SAP GUI transaction, by screenshots and clicks, with every write held for a person's approval and every step recorded.

Some systems have only a screen: an SAP GUI transaction, a green screen, a vendor's desktop app. A **computer-use** agent works them one action at a time: it takes a screenshot, the model chooses the next action, the platform checks it, and only then is it sent.

Computer use is **off by default**, and only agents you configure can use it.

## How a run works

1. **Screenshot.** The screen host (below) returns the current screen.
2. **Plan.** The model gets the goal, the steps so far and the screenshot, through the agent's governed model call: its guardrails, budget and transcript apply. It answers with one action: `click`, `double_click`, `type`, `key`, `scroll`, `wait`, `done` or `fail`.
3. **Check.** The platform validates the action: the coordinates must be on the screen, the text at most 500 characters, and only navigation and commit keys are sent. It then classifies the action as a read or a write by its own rules, not the model's word:
   - Typing into a field is a write.
   - `Enter`, `F8`, `F11`, `Ctrl+S`, `F5` and `Delete` are writes.
   - A click on a target labelled with a committing verb (Save, Post, Submit, Delete, Approve, Release, Book, Pay, Sichern, Buchen and similar) is a write.
4. **Act.** A read is sent. A write is held for approval.

The agent's kill switch is checked before every step. A run stops at its step cap (20 by default, at most 40) or its time cap (300 seconds by default), and after three invalid actions.

## Approving writes

A held write appears with the agent's other waiting actions for anyone with `agents.approve`. The run waits in `awaiting_approval`. After a person approves, resume the run with `POST /api/computer-use/runs/{run_id}/resume`: the approval is spent once, for exactly the action that was held. If the screen changed while the write waited, the platform does not click: the run stops, because what the approver saw is not what the click would hit. A rejected write ends the run without acting.

Turning `require_write_approval` off for an agent lets its writes act straight away. That loosens a control, so it needs `governance.weaken` and a reason.

## The screen host

The platform drives a small host agent that runs next to the UI, for example on the Windows VM where SAP GUI is signed in. It speaks two calls over HTTPS with a bearer token:

| Call | Does |
| --- | --- |
| `GET /screenshot` | Returns the screen as PNG, with `X-Screen-Width` and `X-Screen-Height` headers |
| `POST /action` | Performs one action, given as the JSON object the platform checked |

The platform never holds the target system's own credentials: the session on the host is already signed in.

## Set it up

```bash
curl -X PUT "$BASE/api/admin/computer-use" -H "Authorization: Bearer $TOKEN" -d '{
  "enabled": true,
  "agents": {"sap-orders": {
    "screen_url": "https://sapgui-host.internal:8443",
    "screen_token_env": "TICKETIQ_SAPGUI_HOST_TOKEN",
    "system": "SAP GUI", "max_steps": 20, "max_seconds": 300}}}'
```

`screen_token_env` names the environment variable that holds the host's token; the token itself is never stored in the configuration.

Start a run with `POST /api/computer-use/runs` and `{"agent": "sap-orders", "goal": "Show the delivery date of sales order 4711 in VA03"}`. It needs `agents.operate`.

## Recording and audit

Each step keeps the screenshot the agent saw (by its SHA-256), the action, how it was classified and why, and the outcome. Read a run with `GET /api/computer-use/runs/{run_id}` and replay a step's screen with `GET /api/computer-use/runs/{run_id}/steps/{n}/screen`. A run is visible only in its tenant.

The audit log has `computer_use.start`, `computer_use.write.held`, `computer_use.write`, `computer_use.write.refused`, `computer_use.write.stale`, `computer_use.action.refused`, `computer_use.refused` and `computer_use.end`.