AX Pilot: pixel-free computer use

How it works

The original is a macOS app. It reads the frontmost application's AXUIElement tree, prunes it to at most 60 actionable elements, and asks Jev five or six questions in one request per step: which element to activate, which key to press, whether the goal is reached, whether text must be typed, whether the action is destructive, and which text to type. It then presses the chosen element through the accessibility API and repeats.

This page keeps the loop and the pruning and replaces the Mac. TreeFlattener, TextCandidates, AnswerHandler and the metrics are ported from Swift to TypeScript line for line, and they run in your browser. The five simulated apps are state machines that draw a window; the reading step then walks that rendered window and takes each control's role, title, value, focus, selection and getBoundingClientRect() frame, which is the browser stand-in for the Swift AXUIElement walk. The second tab ships the five recorded macOS trees unedited, 39 KB to 119 KB of raw AXUIElement JSON, 105 to 293 nodes each, and the same flattener prunes them in the browser when the page loads — so one pruner walks both.

What goes to which route

• Which action next — /classify. One label per element, named e7 row 'Appearance' and described by a sentence built in code from its role, label, value and selected flag, plus the pseudo-actions type_text, press_key, scroll, done and stuck. Elements are capped at 55 so the label set stays inside the 64 label limit. A pseudo-action is only offered when it is possible: type_text needs a focused text input, stuck needs the deterministic heuristic to be out of moves too. The text of the call is the compact state: goal, app, window title, focused element, modal sheet, visible text, text candidates, already typed, arithmetic worked out in code, the last eight actions, and the element list.
• Goal reached, needs text, destructive — one /yes-no call with three statements against the same state text. The first statement carries the goal verbatim: The state above already shows this finished: …. The second is about text that is not there yet — Text from the text candidates still has to be typed into a field; it is not there yet — because a statement about the focused input reads true of a field that already holds the typed text, and would then veto every ending for the rest of the run. Shared when_true / when_false hints pulled all three probabilities towards a half, so each statement carries its own specifics and the batch goes out with no hints. Sending the three separately changes nothing: measured on these states, a statement scored alone returns the same probability to the percentage point as the same statement inside the batch, so the batch is one call saved for free.
• Which shortcut — a second /classify over the thirteen chords, sent only on the steps where the first call chose press_key. Which text — a third /classify over the candidates, sent only when the goal yields more than one. Most steps therefore cost two calls.

What the code decides, not the model

Arithmetic is evaluated in code and handed over as a fact. A label carrying a word like delete, erase, send or pay is blocked outright, and so is any action the destructive statement scores above 0.5. Text typed into a field is never treated as an outcome until something submits it, and a sum is not done until that number is on screen. The thresholds are the Swift ones unchanged, goal_reached at 0.8 included. That bar is not decoration: on these five goals the statement reads as high as 0.79 with work still outstanding, so a lower one ends the run on a start page and calls it finished. A high needs_text also vetoes an ending, but only while a text candidate is still untyped: once the text is in the window the veto is spent, or the model could never finish a goal that involved typing at all.

Both calls of a step go out together and the page queue spaces every request half a second apart, so the loop runs at about one step per second. The model owns the ending: a run stops when it returns done, stuck or blocked, or at twelve steps, sixty seconds, Escape or the Stop button. Each preset also carries a hard-coded check on the app state, and that check is a second opinion, not the referee — when it passes, the model still gets two more steps to notice by itself, and the HUD says which of the two ended the run. On three of the five presets the model calls it done; on Safari and System Settings the statement settles around 0.6 on the finished window, under the 0.8 bar, and the HUD says ended by the page check. A custom goal has no check at all, so there the model ends the run or nothing does. On an error the deterministic heuristic takes the step and the HUD marks it fallback; three failures in a row stop the run.