Images and coordinates

The unconverted click lands near the target's upper left. It reuses a small-image point from the larger capture. Turning conversion on restores the intended location while the model's point stays the same.

We must name three spaces before the arithmetic. Capture space holds the original screenshot's pixels, sent-image space the pixels actually submitted, and input space the positions the pointer tool accepts.

These spaces sometimes match. A 1280 × 800 capture sent unchanged to a pixel-based input tool needs no conversion. They differ when the application shrinks the capture before sending, and display scaling adds another between capture and input space.

Start with a 2560 × 1600 capture reduced to 1280 × 800. The width ratio is 2560 / 1280 = 2. The height ratio is 1600 / 800 = 2. A model point (412, 230) therefore becomes capture point (824, 460).

Now compare a 1920 × 1080 capture reduced to 1280 × 720. Both ratios are 1.5, so the same model point becomes (618, 345). Reusing the multiplier of two would overshoot by 206 horizontal and 115 vertical pixels.

The rule depends on this observation's dimensions, not the monitor's resolution or a remembered preset. Store the capture and sent dimensions together, and never reconstruct them from a later screenshot.

A transform maps a point between spaces. For a plain resize, multiply each sent coordinate by its capture-to-sent ratio. Horizontal input uses the width ratio, vertical the height ratio.

Use separate ratios even when they usually match. The 3840 × 2160 preset sends 1430 × 804 pixels. Its horizontal ratio is about 2.6853, its vertical about 2.6866. Integer dimensions create this difference without a deliberate stretch.

A large target may tolerate that difference; a narrow control near the screen bottom may not. Keeping the real ratios costs little and removes an assumption, and a stretched image needs separate ratios too, though it may distort the controls the model sees.

One more question precedes these. Some tools expect normalized coordinates, fractions of an image rather than pixel positions. Read the tool contract and define its unit. Do not apply a pixel formula to an unrelated convention.

The interactive example assumes top-left origins, pixel coordinates, and a resize with no crop or border. Those assumptions keep the arithmetic short. Real preparation can add margins or crop the capture, and we must undo those changes in order.

The stepped curve rises when another patch becomes necessary; its flat portion illustrates a square-image budget cap. It does not mean unlimited pixels cost the same or preserve unlimited detail, so image preparation must respect the selected model's limits.

A visual token represents image information for model processing. Claude's vision documentation describes 28 × 28 pixel patches. Its formula is ceil(width / 28) × ceil(height / 28), where ceil rounds up to the next integer.

For a 1092 × 1092 image, each axis spans 39 patches, so 39 × 39 = 1521 visual tokens. The standard tier allows a 1568-pixel long edge and 1568 tokens. The high-resolution tier allows a 2576-pixel long edge and 4784 tokens.

Ordinary oversized images are resized under the rules. Oversized computer-use or browser-use tool-result images instead need client resizing, or the API rejects them. These limits describe Claude's current tiers, not every vision provider; check the Claude vision documentation against the selected model before sending.

The arithmetic also explains why a small size change can change cost. A 280 × 280 image needs ten patches per axis; at 281 pixels it needs eleven horizontal patches, so 110 instead of 100, though its width grew by one pixel.

One extra pixel is not inherently harmful; dimensions and patch boundaries determine this cost. Text tokens from instructions add their own cost, so a complete request costs more than the visual-token count shown here.

Shrinking a full desktop reduces pixels per control. In a made-up example, a twelve-pixel-high label becomes six pixels high after a half-size resize, so several letters may look similar. No coordinate formula recovers information removed before the model saw the image.

Compare shrinking the whole screen with sending a relevant window unchanged; both may produce the same image dimensions. The window image gives its labels more pixels, since it excludes unrelated desktop area, but less context about other windows.

Choose the image around the task. A scorecard reading step needs legible team names, scores, and the match identifier. A window-switching step needs desktop context to identify the window. One image size does not serve both equally well.

The left square covers four pixels but one logical point. The right layout includes positions left of and above the primary. A point's sign and unit matter as much as its value. Read the platform's reported geometry before injecting input.

A logical point is a display unit covering several physical pixels. High-density displays use it to keep controls a useful apparent size. Apple's high-resolution explanation describes the points-to-pixels relationship. Do not assume every capture or input API shares that unit.

Suppose our earlier capture point is (824, 460). If the input API uses logical points at scale two, divide both values by two, so input point (412, 230) addresses the target. The two conversions cancel here.

That cancellation can hide a bug. An implementation may look correct after skipping both conversions on one display. Change the sent-image size or display scale, and the same code fails. Record each relationship even when the numbers look familiar.

Multiple monitors introduce origins as well as scales; Windows describes a virtual screen whose coordinates go negative outside the primary monitor. A captured monitor can still have its own top-left origin, and mapping between spaces requires its offset.

Put a 1920-pixel-wide monitor directly left of the primary. In a pixel-based global input space, its left edge can have coordinate -1920. Local point (400, 200) then becomes global point (-1520, 200). Rejecting negative global coordinates would exclude a valid target.

This assumes matching units and vertically aligned monitors. Mixed display scales need the platform's own conversion rules. A screenshot stitched from several monitors may not share one scale across its width, so store the monitor geometry rather than guess from the combined rectangle.

Round only when the input API requires an integer. Earlier rounding loses detail that later conversions may magnify. If a sent coordinate maps to 100.4 capture pixels, rounding gives 100. Multiplying that by two differs from rounding 200.8 at the end.

The difference is one pixel here. That may not matter for a large Save button. It can matter near a narrow border between controls. Choose an interior point with room for rounding error, then check the input bounds.

The enlarged view shows the chosen icon more clearly. The crop origin determines where that icon sits. A correct visual choice with a forgotten offset remains an incorrect click. Detail and geometry solve different parts of the problem.

A crop helps when the original capture has detail the sent image lost to shrinking. Take it from the original capture whenever possible. Enlarging a shrunken image adds pixels without recovering missing letters, making blurred edges larger but no more certain.

The eightfold crop is schematic. Suppose the crop starts at capture point (440, 20). A model point (160, 80) in an eightfold enlargement maps to (20, 10) inside the crop. Adding the origin gives capture point (460, 30).

A fresh crop may expose a changed interface. Suppose the toolbar moved after the earlier full capture, so the new crop and old transform describe different moments. Give the crop its own observation identifier and geometry rather than attaching it to the old image.

Compare a labeled Save button with an unlabeled disk icon. The button provides text naming the action. The icon requires interpreting its drawing and context. Hovering may reveal a tooltip, but that requires another input and observation.

A tooltip reduces uncertainty without proving the task. It may name Save while leaving the destination note unclear. Keep the BB-042 title or nearby editor context visible. Enlarging only the icon can remove the context needed to associate it with the document.

We have not measured how often a model succeeds on either control. The example explains why the evidence differs. Smaller controls leave less room for location error. Missing labels require more interpretation before the click begins.

Use a crop to answer one unresolved question. It might reveal whether an icon is Settings, whether the score reads 168/6, or whether a row says BB-042. Once answered, return to enough context to act safely.

All four methods point at the same button here. They give different reasons for choosing it. None removes the need to check the page state before input or proves that the saved note contains the correct match result.

With pixels alone, the model reads the label and estimates a location. This works for interfaces with no machine-readable structure, though it requires the screenshot to preserve enough detail. A painted control can look enabled even when the application ignores it.

The Document Object Model, or DOM, is a browser's structured page representation. The accessibility tree describes controls through properties such as role and accessible name. A role gives a control's kind, such as button; its accessible name identifies it to assistive software.

These structures can select a button named Save within the BB-042 editor. They also distinguish it from another Save button elsewhere. A rectangle alone cannot establish that context. The target's relationship to the right note matters.

Structure has limits. A page can include hidden controls or labels that do not match the screen. A canvas may draw controls with no separate DOM elements. Use the evidence the interface provides, and compare uncertain structure with a fresh image.

Set-of-Mark prompting places identifiers on image regions the model can refer to. The original Set-of-Mark paper describes the method. Here the model chooses mark 6 instead of estimating a point. The executor resolves that mark into a rectangle from the same observation.

The marks depend on finding useful regions. A missing button cannot become selectable merely because other regions have numbers. Old marks can identify the wrong places after scrolling or layout changes. Store the mark map with its image and discard it when the state changes.

A vision parser detects regions and attaches descriptions or labels. Microsoft's OmniParser is one implementation. It can supply candidate controls before the action model chooses among them, though a parser can still miss, split, or mislabel a region.

Its confidence score describes the parser's own output under its method. It does not combine target identity, permission, input delivery, and application success into one certainty, so a high score for detecting Save cannot replace checking which note receives the save.

For the note, combine the button's label with the editor title and match facts. The title should identify BB-042. The body should state Pune Lanterns 168/6 and Jaipur Kites 169/4. The result should name Jaipur Kites winners by six wickets.

The target evidence identifies the intended control; the transform maps its point into the input system. The executor needs both, then a fresh observation to establish the effect.