The agent's computer

The browser draws the scorecard into the virtual display. The capture tool reads it and produces the screenshot sent to the model. The input tool sends pointer or keyboard events to the display. No monitor is needed, because no step requires a person to see physical light.

Each X server has a display identifier such as :99, which identifies the server to local clients. It is an example value, not a display reserved for agents. The browser and executor must connect to the same display.

A process, a running instance of a program, can use the DISPLAY setting to choose its X server. If the browser draws on :99 while capture reads another display, their observations disagree, and a successful capture may show an unrelated empty desktop.

The input tool in the diagram is xdotool. X11 is the protocol, the shared communication rules, used by this display system, and the XTEST extension adds input generation to it. The upstream xdotool documentation describes this mechanism and its Wayland limits.

A virtual display size should stay fixed during a run. Suppose the setup uses 1280 by 800 pixels and each stored pixel uses four bytes, so one image occupies 4,096,000 bytes (1280 × 800 × four). That is raw image memory, before compression.

A larger display reveals more page content but gives capture and processing more pixels to handle. A smaller one can hide the Save button below the visible area. Neither size is universally best; choose one that keeps the controls the task needs.

A person can watch through remote viewing software. VNC sends a desktop view to a separate viewer, and noVNC provides a browser-based viewer for it. Viewing is optional, and interactive viewers add another source of input.

For BB-042, a viewer must not let someone move windows without the loop noticing. A person's click can invalidate the screenshot used for the next action. Pause execution during a supervised change, then capture again. Human input needs the same coordination as model input.

The managed arrangement makes the active window easier to identify, with its dialog over the parent window. Without suitable window management, placement and focus become harder to predict. Applications also influence their own placement, so a window manager does not remove every problem.

Suppose BB-042 opens a Save dialog while a second browser window sits behind it. The model proposes typing the note filename. If the second window has focus, the text enters that window instead, so a correct typing operation has the wrong effect.

Now compare a deliberate focus check. The executor activates the intended dialog and captures its state, then types only after confirming the destination. The text enters the filename field because the destination matches the proposal.

A modal dialog blocks interaction with another window until someone responds, changing which controls the agent can use. The Save button may stay visible under a page layer while rejecting clicks, the repeated failure from Chapter 1.

A consistent window arrangement reduces these surprises. Start only the applications the task needs, keep browser size, zoom, and placement stable, and still inspect new dialogs that the application can change during execution.

The model should not infer focus from cursor position alone. A pointer can rest over one window while another receives typing, and though some screenshots show a focus outline, others hide it. Where available, the executor can combine visible evidence with documented window state.

These steps explain why a small existing window manager is useful: a custom one adds unrelated placement and input behavior to maintain. The task needs a predictable desktop, not a new desktop implementation. That desktop still needs a boundary around its files and processes.

The stacks differ in how application requests reach the operating system. A kernel is the component that manages memory, processes, and devices. A system call asks the kernel to perform an operation, such as opening a file.

A container gives programs restricted views of resources while sharing the host kernel, and its configuration controls visible files, privileges, and other access. Docker explains these mechanisms in its Engine security documentation. A separate container name alone does not establish the required restrictions.

Sharing the kernel means its reachable behavior remains part of the boundary, so a flaw there can threaten isolation. Dropping unnecessary privileges reduces what a process may request, and updates address known flaws, but neither proves an escape impossible.

gVisor adds an application kernel called the Sentry between the workload and the host kernel, handling application system calls itself. gVisor's architecture guide explains how the Sentry uses limited host operations. Some behavior needs compatibility checks because gVisor implements its own operating system interface.

A virtual machine runs a separate guest operating system, and a microVM is one designed with a small set of virtual devices. Firecracker provides this design using KVM, the Linux facility for hardware virtualization. Its project documentation describes the guest kernel and virtual machine monitor.

A virtual machine monitor, labelled VMM, manages the virtual machine's access to hardware. The separate guest kernel changes the route an application crosses to reach the host, adding a boundary, but configuration and host maintenance still matter.

Consider two environments with the same browser and virtual display. The first mounts the user's entire home folder as writable; the second exposes only a task output folder. A mistaken file operation can damage personal files in the first, but cannot reach them through the absent mount in the second.

A mount makes a filesystem location available at a path. Broad host mounts defeat the purpose of isolation, and host control sockets are equally dangerous because they expose powerful services. BB-042 needs neither a home-folder mount nor a container-management socket.

Use a read-only base filesystem where the applications permit it, with separate writable locations for temporary browser state and the note. A read-only path rejects changes instead of relying on the model to avoid them. The output folder still needs a path rule to prevent overwriting unrelated artifacts.

Resource limits solve another class of failure. Bound processor use, memory, disk space, and total runtime, because an accidental download loop can otherwise fill the host's storage. These limits protect availability even when no program escapes its environment.

Choose an existing isolation system that fits the threat and operational requirements, and compare the browser's actual behavior under it before relying on it. The teaching stack establishes no universal ranking or deployment default; its purpose is to show which component enforces each boundary.

The allowed connection reaches the task website, while the blocked one represents a transfer to an unrelated destination. Exfiltration means sending information to a destination that should not receive it, and restricting connections reduces the destinations available for it.

An allowlist names permitted destinations. Start BB-042 with the smallest list that loads the approved page. A real site may need additional image or login hosts. Add them because the task needs them, not by allowing the entire internet after one blocked request.

The enforcement component must check each connection's destination. A permitted page can redirect the browser elsewhere, and a hostname can resolve to an unexpected address. A proxy is a service that forwards approved requests; it helps only if direct connections cannot bypass it.

Compare two policies. The first permits the website's hostname but allows any direct connection; the second requires connections through the enforcing proxy. In the first, another tool can avoid the hostname check; in the second, the network route itself enforces the restriction.

The clipboard stores copied text between applications. Sharing the host clipboard could reveal a password copied before the run, or let the environment replace text a person later pastes. Disable shared clipboard access unless the task requires it.

Downloads and uploads need separate rules. A browser download writes a file inside the environment; exporting it to the host is another operation. Keep downloaded content in a temporary folder, then export only the intended note after checking it.

A credential proves an identity or grants access. Passwords, login cookies, and access tokens are credentials. An access token can authorize requests without revealing the password, and its permissions and lifetime determine what a leaked token could do.

Keep the model provider credential in the coordinating process outside the browser environment; the browser does not need it to open BB-042. When login is necessary, give the environment only task-specific access. Prefer credentials that expire soon and permit only the required operations.

Do not place credentials inside a reusable environment image, the saved base used to start new environments, because a credential there persists across many runs and copies. A temporary credential supplied at startup has a clearer lifetime and removal point.

A person may handle login during a supervised pause. After login, the browser's session cookies still grant access, so treat the session as sensitive even though the password never entered the model's context. Stop capture during secret entry when required, then resume from a fresh observation.

After the note passes its check, export it and remove temporary state. A browser profile stores settings and session data, so reusing it can reveal one user's data to the next. Starting from a clean base avoids that carryover. Preserve necessary run evidence separately under its own access and retention rules.

On macOS, screen recording access controls capture of screen content, while Accessibility access lets an approved application control the Mac through accessibility features. Apple describes these separately in its screen recording guidance and Accessibility guidance. A person reviews them in Privacy & Security settings.

The diagram uses TCC, the name for Transparency, Consent, and Control, which groups macOS privacy permission checks under one label. Apple documents these settings in its deployment guide. An application must check the abilities it needs; permission to capture does not imply permission to send input.

On Windows, an integrity level identifies a process's trust level for certain checks, and a normal application commonly runs below an elevated administrator one. UIPI, User Interface Privilege Isolation, restricts input injection across those levels.

Microsoft states that SendInput can inject input only into equal or lower integrity applications, and a blocked call does not identify UIPI as the cause through its return value or error. See the SendInput reference. The executor must inspect the outcome instead of reporting "click sent" as success.

User Account Control requests approval for elevated operations, and its prompt appears on the secure desktop by default. That desktop separates the prompt from ordinary application input. Microsoft's UAC explanation describes this default and its policy settings.

For BB-042, opening a match should not require administrator approval. If an unrelated program requests elevation, pause and report it. Raising the executor's privileges would expand what every later action can affect, and would not explain why a read-only task required elevation.

Linux also has different display routes. The X11 tools described here must connect to their assigned X server, while Wayland uses different control and capture mechanisms. The xdotool project warns that many operations do not work there, so a Wayland session needs a separately supported route and preflight checks.

A permission failure can resemble an ordinary missed click, as can stale capture, an inactive window, or a disabled button. Check those causes before guessing. If the program cannot establish the cause, stop with an unknown-failure report and the observed evidence.

Next capture a known harmless test page. Confirm the image has the configured dimensions and recognizable content, because a file with valid dimensions can still hold only black pixels. Capture success means checking useful content, not just whether a file exists.

Then test control on a harmless target. Click a known control and capture its changed state, then type a short test value into a designated field and read it back. This checks keyboard delivery separately from pointer delivery.

Calibration checks that image positions and input positions refer to the same locations. A resized screenshot, for example, may show a button at half its actual display position, so a click sent without converting would miss. Test several separated targets, because one successful center click can hide scaling errors.

Compare two synthetic preflight outcomes. In the first, captures show the test page but its checkbox never changes, so capture works while input remains unproven. In the second, the checkbox changes under a person's click but automated input still fails. That narrows the problem without proving a permission caused it.

A successful preflight allows the task to start. A missing permission needs a supervised setup step and another check, and an unsupported display route should stop with a clear explanation. An unexplained failure should preserve its evidence rather than become another model prompt.

Permissions can change after startup: the display may lock, a viewer may disconnect, or a person may move the browser. Repeat the relevant checks after unexplained input or capture failures, and resume BB-042 only from a fresh observation.

The environment now has defined places for pixels, input, files, and network access, and each boundary answers a specific failure we could otherwise mistake for model reasoning. The next chapter follows one accepted proposal into the executor, explaining how that program turns an action into an observable result.