blogs by ramaa.

Computer-use agents

Six chapters · October 2026

A computer-use agent controls an application by observing its screen and sending input. A model chooses what to try. A separate program carries out the action and returns a new picture of the screen.

This series follows one task through that process. We open a fictional cricket scorecard, check the final result, and save a local note. Each chapter explains a different part of the work, including what can fail and how to check it.

The loop in four moves01 Observea screenshot plusthe running history02 Proposethe model returnsa tool call03 Executevalidate, scale,then inject input04 Checkread the effect,or the saved filea separate program, not the modelanother turn, until the model answers in text or a limit stops the runThe user's task sets the permissions. An independent check sets the verdict.
Each chapter takes one of these moves apart.

Chapters

  1. 1. How the agent loop works

    Follow a screenshot through the model, the executor, and the next observation.

  2. 2. The agent's computer

    Build a virtual display and control which files, permissions, and network routes the agent can use.

  3. 3. Executing a click

    Validate a proposed action, send input events, and check what the application changed.

  4. 4. Images and coordinates

    Map a point in a screenshot to the display, then account for crops, scale, and small targets.

  5. 5. History, caching, and cost

    Compare batches, approval checks, growing requests, and ways to keep useful history.

  6. 6. Safety, recovery, and evaluation

    Handle hostile page content, uncertain outcomes, and independent checks of the final result.

The match and run traces are invented examples. The calculations state their inputs. They do not report measured agent performance.