Executing a click

The new screenshot comes after the input and the wait. This order gives the next model call fresh evidence, but it does not certify that the intended button received the click. We still compare the new screen with the intended result.

An action contract defines the input an executor accepts. A click might require an action name, two coordinates, and an observation identifier, which names the screenshot used to choose those coordinates. Text insertion needs a string instead of a point.

The executor checks the action name against its supported actions, then checks each required value and refuses unexpected combinations. A click without a coordinate is incomplete, and a screenshot request carrying a shell command has no place in this contract.

Coordinates must be finite numbers inside the stated image. A value outside it may come from confusion about its size or from a malformed reply, and neither gives the executor a useful place to click.

Consider two proposed clicks for a 1280 × 800 image. The point (412, 230) passes the image bounds check. The point (1412, 230) fails because its horizontal value exceeds the image width. The executor should return that error before sending any input.

Do not repair the second point by forcing it onto the nearest edge; that creates an action the model never proposed, and the new point might close a window or select another control. An explicit error lets the model inspect the dimensions and choose again.

A point inside the image can still refer to an old screenshot. Suppose the proposal names obs-12, but another action already produced obs-13. The executor should detect the mismatch, or a valid number can select a control from the wrong screen state.

Validation answers whether the request has a usable shape, and permission checks answer whether the action is allowed. These are separate questions; a correctly formed click can still attempt an action outside the user's task.

Once both checks pass, the executor converts the point into the input system's coordinates. In this example, the capture has 2560 × 1600 pixels. The model saw an image with 1280 × 800 pixels. Therefore (412, 230) becomes (824, 460) before input.

The input program receives fixed command arguments rather than a constructed shell command. Model text belongs in a text argument and must never become shell syntax, especially when a note contains quotes, line breaks, or characters a shell would interpret.

For example, the note text can contain Jaipur Kites won by 6 wickets. as one argument. The executor does not need a shell to type that sentence. Keeping data separate from commands also makes failures easier to locate.

Finally, validation should finish before the first event. If the executor discovers an invalid destination halfway through a drag, the button may already be held. Checking the complete request first avoids a partial action from an avoidable input error.

A click contains a button press followed by a release. A drag keeps the button pressed while the pointer moves, and the release ends it. Without that release, the application can treat later movement as part of the same action.

The diagrams simplify movement and timing and show event order, not a required implementation for every desktop. Different input systems expose different controls, so the executor must translate its action names into the events its own system supports.

For a drag, move to the source and press the chosen button, move toward the destination while keeping it held, then release at the destination. Some applications need intermediate movement to recognize a drag, so a final position alone is insufficient.

Compare moving a note within a list with clicking its destination row. The drag tells the application which note started the movement; a click at the destination carries no such information. The final pointer location can match even though the application receives different instructions.

A key chord holds one key while another is pressed. For Ctrl+S, press Control, press S, release S, then release Control. The exact names depend on the input tool; what matters is that the modifier does not survive the action.

Text insertion has a different goal: it asks for characters in a field, not a physical sequence of key positions. A keyboard layout maps physical keys to characters, and the wrong layout can produce different punctuation even when every key event arrives.

An input method helps users enter characters that need more than one key press, and it can combine several keystrokes before committing text. A sequence that works for an English title may fail for another writing system, so the executor must check the text that appeared.

Clipboard paste can offer another way to insert text. The clipboard is shared storage that applications use when copying and pasting. Replacing it can erase the user's current copied content. A paste path therefore needs an explicit rule for clipboard access and cleanup.

Paste also changes the event sequence, since an application may process pasted text differently from keystrokes. Test the path against the target editor before relying on it, and do not assume paste repairs every character problem.

The executor should track which keys and pointer buttons it holds. This record supports cleanup, the release of input state after an action ends or fails. Cleanup belongs in the failure path as well as the success path, and cancellation must reach it too.

Suppose a drag presses the left button, then the input program stops responding. A timeout should stop the unfinished action and attempt a release, and the executor should report whether that release succeeded. It must not claim a clean desktop when the release failed.

Compare two failed drags. In the first, the executor releases the button before returning an error, so the next pointer move stays ordinary. In the second, it leaves the button held, so the next move may relocate a different item.

The original drag failed in both cases, but the second also changes the meaning of later actions. That is why release cleanup is part of correctness: it limits how far one incomplete action affects the session.

A restarted executor cannot assume its empty in-memory record describes the desktop; the previous process may have stopped after pressing a key. The new process needs a known input state before continuing, and if it cannot establish one, it should stop and report uncertainty.

Scrolling needs an equally clear target. A page can hold a main document and a smaller scrolling note list, and the pointer position may decide which receives wheel input. After scrolling, capture again, because every visible row may have moved.

These actions provide a shared vocabulary, but their names alone do not describe success. A successful typing action reports that text insertion ran; only reading the field confirms the intended sentence. That distinction matters more when an action starts work that takes time.

Two equal frames inside the quiet gap do not predict the later change, while three equal frames near the end look better in this example. They still provide a local observation, not a guarantee about future updates, and the curve is synthetic, not a measurement of a particular site.

A fixed sleep waits for a chosen duration without checking the application. Suppose we always wait three seconds after Save. If the save finishes after half a second, we waste two and a half seconds; if it takes four, the sleep ends too early.

For twenty actions, that difference accumulates: three-second waits consume sixty seconds, half-second waits ten. These made-up values show why waiting can dominate an otherwise short interaction.

Reducing every sleep to half a second creates another problem: slow responses now produce screenshots before the needed content appears, and the model may click again because it sees no change. A short wait can increase both errors and total work.

Polling means checking a condition repeatedly until it becomes true or a deadline arrives. A screenshot executor can compare successive captures, waiting until a chosen region changes very little across several checks. The comparison needs a tolerance, because small visual changes can continue.

A blinking cursor can prevent exact image equality, and a rotating advertisement can change many pixels while the editor is ready. Conversely, a disabled Save button can stay completely still. Pixel stability measures appearance, while readiness depends on the task.

A useful comparison isolates the relevant region. When waiting for a note to save, inspect the note status area and ignore an unrelated live scoreboard. This narrows the question from whether the screen stopped changing to whether the needed result appeared.

However, an unchanged region still cannot prove a server accepted a write, and the application may show an old saved indicator. Match the status to the current action and note; here, the evidence should concern BB-042 and the latest note text.

A browser executor can sometimes read a specific condition instead of comparing pixels, waiting for an enabled control or a visible confirmation. Playwright checks whether click targets are visible, stable, enabled, and able to receive events. These checks help before input; a separate result check helps afterward.

A postcondition is the state that should hold after an action: for opening a scorecard, the visible heading BB-042; for saving, a saved indicator associated with that note. Choosing the postcondition makes the wait meaningful.

Page loading alone answers a different question. A document can finish loading before its application data appears. A page can also keep making requests after its controls become usable. Playwright therefore discourages network-idle as a readiness test.

Compare two visits to the same scoreboard. One keeps polling the score every second, but the note editor is ready immediately; the other makes no requests while waiting for a local dialog to close. Network activity alone misclassifies both.

Every wait needs a deadline, or an absent indicator can block the whole agent indefinitely. At the deadline, return the last observation and the condition that failed. A timeout means the executor did not establish the expected state within its limit.

That does not mean the action never took effect; the server may have saved the note just before the connection stalled. The next step should inspect the note before repeating the save, preserving the distinction between a late response and a missing effect.

The first image places Save under the pointer, and the second places Accept all at the same location. The coordinate calculation did not fail; the application changed the meaning of that location before the input arrived.

This is a race, where the result depends on the order of independent events. The executor checks the page at one moment, the banner changes it at another, and the click reaches whichever state exists when input arrives.

A fresh screenshot just before acting shortens the gap but does not remove it, because an application can still change after that capture. A browser tool may reduce the risk by locating the target again just before a click.

Coordinate input cannot silently gain that extra knowledge. If the target moved, take another observation and choose again, rather than reuse the old point because its numbers remain in bounds. Bounds describe the image, not the identity of the control.

Focus creates a similar problem for typing. Focus is the application or control that receives keyboard input, and a visible editor does not necessarily have it. A dialog, browser address bar, or another window may receive the text instead.

Consider typing the BB-042 title after clicking its field. In the first case, the field keeps focus and receives the title; in the second, a dialog opens between the click and typing, and the text enters the dialog or disappears into a control that rejects it.

The executor should establish focus close to text insertion, then inspect the resulting field value. A visible caret can help in a screenshot but is not enough for every custom editor. Browser tools can sometimes check the focused element directly.

Long action batches increase the assumptions between observations. Clicking a field, typing a title, pressing Tab, typing a body, and saving assumes focus follows every step, so one unexpected dialog can invalidate all later input. Shorter batches make the first wrong assumption easier to find.

That does not require a screenshot after every character: group the events of one coherent action, such as entering a short title, and observe when the next step depends on a changed page or focus state. The useful boundary follows the application's behavior.

For BB-042, read the title after entering it. Read the note body after entering the scores. Then save and inspect the status. These checks distinguish a wrong field from a wrong sentence and a save failure.

The body should preserve the match facts independently of input success. Pune Lanterns made 168/6, while Jaipur Kites made 169/4. The winner was Jaipur Kites by six wickets. Correct clicking cannot repair an incorrect result copied into the note.