How the agent loop works

The last column holds the component that affects the computer. The model's reply stays data until that component accepts it. We call it the executor; it checks a proposed action, performs it, and returns a result.

Suppose the first screenshot shows a search box at the page center. The model proposes a click at horizontal position 640 and vertical 512. Pixels are the small colored points forming an image. The executor must know which screenshot they describe.

A tool call is a structured request for an operation the executor offers. It names the operation and its arguments. A click needs positions, typing needs text, and a capture returns a fresh image.

We use plain operation names such as click here. They explain the steps without promising one provider's exact format. Anthropic documents a client tool loop with ordered calls and matching results. Its computer-use documentation defines the current message format.

The executor checks whether the operation exists, then the argument types and permitted range. A position outside the current display must fail before input occurs. It also checks whether the task permits that operation now.

Validation and permission answer different questions. A valid click could press a button that deletes a match. Its positions can be correct while its effect remains unauthorized. Both checks matter because the model can misread a button or follow misleading content.

After the search box receives focus, the next proposal types BB-042. Focus means the destination for keyboard input. Without the earlier click, the same text could enter an unrelated field, which explains why actions on one desktop must stay in order.

A tool result tells the model what the executor observed. It should distinguish success, rejection, failure, and skipped execution. Saying "input sent" reports the executor's action, not the application's acceptance of that input.

Each tool call needs a stable identifier. The result uses it so the loop can match request and response. Consider two clicks followed by one error. Without identifiers, the model may attach the error to the wrong click.

The next request includes the assistant response and matching tool results. The client also resends earlier observations. That material is the history, the record for the model's next decision. It replaces any assumption that the model remembers a desktop independently.

This diagram compresses a longer task into four turns. Opening the scorecard and saving the note may take more actions. Its last text reply illustrates a possible exit, not proof that BB-042 passed. Completion needs a separate definition.

The five exits have different meanings and must stay separate in the run record. A final text response makes the run a completion candidate, a turn limit means the program stopped further decisions, and a user cancellation means authorization ended.

A turn is one model request and its response. One turn can propose several tool calls, so a 30-turn cap does not imply a 30-action cap. If actions matter for cost or risk, count them.

A token is a unit of model input or output. It can represent part of a word, punctuation, or encoded image content. Counts depend on the model and input format. The loop should read usage reports rather than estimate from visible words.

A token budget stops another request when too little allowance remains. The program must also reserve room for the next response, since an input that fits the allowance can still produce a larger one. Budget checks belong before requests, not after an expensive failure.

A timeout limits how long an operation may take. The overall timeout also includes model and application waiting. Suppose the fictional task allows 120 seconds. Spending 100 on one stalled page leaves 20 for everything else.

A user cancellation must reach the executor between actions. If the model proposes click, type, and Enter together, checking only before the sequence is insufficient; the cancellation may arrive after the click. The executor should skip the remaining actions and record why.

A transport error means the model connection failed. A refusal means the model declined the request. A cut-off response means generation ended early. None means "done", even if the response contains no complete tool call.

The executor must never run an incomplete streamed call. Streaming sends response fragments before the whole response arrives. A partial argument can change meaning as later fragments appear. Wait for a complete call, then validate its final arguments.

Timeouts add one problem when an effect may already have happened. Suppose saving the note succeeds, but its acknowledgment never reaches the loop. Repeating Save could overwrite later work. First inspect the saved state, then decide whether to retry.

A useful run record stores the stop reason and the task check. It might say "turn cap reached, note missing" or "model finished, note verified". This gives the user evidence and separates resource failures from saved-fact errors.

The final three attempts repeat the visible state and the proposed action. That pattern gives the loop a reason to pause. It does not prove the cause. A blocked permission, misplaced click, or slow response could produce a similar record.

A hash is a short value computed from data. Equal image files produce equal hashes under the same method. Different hashes only show that some image data changed. They do not show that the task progressed.

Exact screenshot comparison can fail because a text caret blinks. A clock or loading icon can change between captures too. A repetition check may ignore known moving regions before comparing, but should keep the original captures for diagnosis.

The action matters as much as the image. Repeated screenshots during a deliberate wait can be normal; repeated Save clicks during the same wait can create duplicate effects. The detector should weigh recent operations and targets alongside the visible state.

A reflection turn asks the model to describe the evidence before another action. For BB-042, it might identify the still-open cookie dialog and suggest a different next step. It remains a recovery proposal, not a task result.

The diagram offers Escape as one recovery action. Some dialogs close with Escape; others do not. The executor must observe the result rather than assume success. If the dialog requires a consequential choice, the task's permission rules apply.

Recovery also needs a limit. One new explanation followed by the same failure should not restart an unlimited sequence. A simple policy can permit one attempt before stopping for help; the useful value depends on the task and requires evaluation.

The report should name the unchanged condition and attach the recent evidence. "Save remained covered after three attempts" explains more than "the agent got stuck". A person can inspect the obstruction or change the environment. The next run begins with a fresh observation.

This is why repetition detection and hard limits belong together. The detector catches recognizable patterns early. The hard limit catches patterns the detector misses. Neither replaces checking the saved note.

Consider two runs with the same reply. Both models say "Done. Jaipur Kites won by 6 wickets." In the first, the note has the right scores; in the second, no file exists because a Save dialog stayed open. Only the first satisfies the request.

Now compare two files that both exist. One names BB-042, the other BB-024. Their scores might otherwise match. The second still fails because the user requested a particular match; a correct answer on the wrong record is an incorrect result.

The score notation needs careful reading. Pune's 168/6 means 168 runs with six wickets lost. Jaipur's 169/4 means 169 runs with four wickets lost. For this fixture, the result is a six-wicket win. The note must preserve the source's meaning, not treat the slash as division.

The agent should observe the page after its last meaningful action. If Save closes a dialog, a fresh screenshot can show that change. That screenshot supports a claim about the visible application; reading the file supports a stronger claim about the requested artifact.

An independent check should avoid reusing the same mistaken reasoning. Asking the model "Did you save the right note?" may repeat its error. A direct file read and comparison catches different failures. For synthetic evaluation, expected fixture values can stay outside the model's history.

A real task may have no hidden expected answer. The check can still verify the output path, required fields, and cited source and compare the note with a fresh view of the page. It must describe its confidence honestly, not claim certainty it cannot establish.

If the check fails, the loop should preserve the file and failure details when safe; erasing an imperfect note would remove useful evidence. A retry may repair the specific failure within the remaining limits, and should not silently replace a failed result with a success message.

The steps show a selection process, not measured cost differences. Start with a route that is documented and authorized for the task. Use screen control when the alternatives cannot complete the work. The best route depends on access, output, and permission scope.

Suppose the fictional website offers an approved read-only match endpoint. One request could return the identifier, scores, status, and result. The agent would not need to find the search box or wait for its animation; a note-writing operation could then save the requested file.

Now remove that endpoint while keeping a normal web page. The browser may expose the DOM, its structured description of page elements. A named search field can be selected without estimating a pixel location. An accessibility tree offers another structured description for assistive software.

Structured information can also be incomplete. A custom canvas may draw a scorecard as pixels without named text elements. A native application may offer no browser structure at all. In those cases, the screen route supplies information the other route lacks.

A shell is a program that runs text commands. It can write a note without opening a text editor. But a general shell can also read unrelated files or delete folders. Fewer actions do not automatically mean fewer harmful effects.

The executor should expose the narrowest operation that completes the job. A fixed note-writing operation can restrict writes to the task folder. A general shell requires broader checks because commands combine many effects. Tool choice changes what the executor must validate.

Mixing routes also requires clear evidence. If the browser supplies the result and a file tool saves it, record both steps. A screenshot cannot prove what the file tool wrote. A successful write cannot prove the browser displayed the correct match.

An illustrative input price of $2 per million tokens makes that input cost $0.024. The arithmetic is 12,000 divided by 1,000,000, multiplied by $2. If output totals 800 tokens at $8 per million, output costs $0.0064. The combined model charge is $0.0304.

These invented rates explain the calculation only. They exclude computer rental, storage, retries, and other charges. Real image token accounting and cache discounts depend on the service. A cache lets a service reuse earlier input processing under its documented rules.

Waiting time also adds across turns. Suppose four model responses each take 2 seconds. Suppose the computer operations add 4 seconds altogether. The task takes at least 12 seconds before other delays. Batching safe dependent actions may remove some round trips.

Longer batches reduce chances to inspect the screen between actions. Extra observations cost time but can prevent mistakes after a page change. The correct choice depends on what each action can affect. Later chapters examine these costs and checks in detail.

You now have a complete path from request to verified output. The model proposes operations, the executor checks and performs them, and the task check reads the result. Before that runs, the browser needs a controlled place to draw. That is the next chapter.