Computer use

Some systems have only a screen: an SAP GUI transaction, a green screen, a vendor's desktop app. A computer-use agent works them one action at a time: it takes a screenshot, the model chooses the next action, the platform checks it, and only then is it sent.

Computer use is off by default, and only agents you configure can use it.

How a run works#

1. Screenshot. The screen host (below) returns the current screen.
2. Plan. The model gets the goal, the steps so far and the screenshot, through the agent's governed model call: its guardrails, budget and transcript apply. It answers with one action: click, double_click, type, key, scroll, wait, done or fail.
3. Check. The platform validates the action: the coordinates must be on the screen, the text at most 500 characters, and only navigation and commit keys are sent. It then classifies the action as a read or a write by its own rules, not the model's word:
   • Typing into a field is a write.
   • Enter, F8, F11, Ctrl+S, F5 and Delete are writes.
   • A click on a target labelled with a committing verb (Save, Post, Submit, Delete, Approve, Release, Book, Pay, Sichern, Buchen and similar) is a write.
4. Act. A read is sent. A write is held for approval.

The agent's kill switch is checked before every step. A run stops at its step cap (20 by default, at most 40) or its time cap (300 seconds by default), and after three invalid actions.

Approving writes#

A held write appears with the agent's other waiting actions for anyone with agents.approve. The run waits in awaiting_approval. After a person approves, resume the run with POST /api/computer-use/runs/{run_id}/resume: the approval is spent once, for exactly the action that was held. If the screen changed while the write waited, the platform does not click: the run stops, because what the approver saw is not what the click would hit. A rejected write ends the run without acting.

Turning require_write_approval off for an agent lets its writes act straight away. That loosens a control, so it needs governance.weaken and a reason.

The screen host#

The platform drives a small host agent that runs next to the UI, for example on the Windows VM where SAP GUI is signed in. It speaks two calls over HTTPS with a bearer token:

The platform never holds the target system's own credentials: the session on the host is already signed in.

Set it up#

curl -X PUT "$BASE/api/admin/computer-use" -H "Authorization: Bearer $TOKEN" -d '{
  "enabled": true,
  "agents": {"sap-orders": {
    "screen_url": "https://sapgui-host.internal:8443",
    "screen_token_env": "TICKETIQ_SAPGUI_HOST_TOKEN",
    "system": "SAP GUI", "max_steps": 20, "max_seconds": 300}}}'

screen_token_env names the environment variable that holds the host's token; the token itself is never stored in the configuration.

Start a run with POST /api/computer-use/runs and {"agent": "sap-orders", "goal": "Show the delivery date of sales order 4711 in VA03"}. It needs agents.operate.

Recording and audit#

Each step keeps the screenshot the agent saw (by its SHA-256), the action, how it was classified and why, and the outcome. Read a run with GET /api/computer-use/runs/{run_id} and replay a step's screen with GET /api/computer-use/runs/{run_id}/steps/{n}/screen. A run is visible only in its tenant.

The audit log has computer_use.start, computer_use.write.held, computer_use.write, computer_use.write.refused, computer_use.write.stale, computer_use.action.refused, computer_use.refused and computer_use.end.