History, caching, and cost

The fourth action now points at Delete all instead of Save draft. Its coordinates are still valid, but their meaning changed on the new screen. Checking whether a point lies inside the display would miss this failure.

The executor should associate each proposal with the observation that supported it. Suppose all four name obs-20. After Enter changes the page, obs-20 no longer supports another click. The executor stops the batch, requests a fresh observation, and gives the skipped action a result that says why it did not run.

Compare two short batches in the same editor. The first selects a title field and types BB-042 final. The second types the title, presses Enter, then clicks the old Save location. The first suits a known focus and local editing behavior. The second crosses a submission boundary whose result needs inspection.

Even typing needs context. A website may save every keystroke, and Enter may send a message. A shortcut may close a window instead of moving focus. Treat an action as local editing only when the application supports that classification. A familiar key name guarantees nothing.

A useful batch stops at navigation, submission, a new window, or an unexpected focus change. It also stops when a task value changes. A price change or another selected match can invalidate later steps without changing the layout. These boundaries follow the task's assumptions, not a fixed batch length.

The policy check runs before each action, including inside a batch. It compares the current target, destination, arguments, and user goal. Without those facts it cannot infer the target from old coordinates, so it returns an uncertainty result instead of guessing.

This design still saves turns when actions have predictable local effects, and it makes failures legible. The record separates three executed actions from one skipped action. Without that distinction, the model may assume the save happened and report a note that does not exist.

A fresh screenshot repairs missing evidence about the page, but it does not grant permission for every action visible there. Actions whose effects exceed the task need a separate rule.

Reading the BB-042 scorecard fits the user's request, and writing a new note at the agreed path fits it too. Publishing the note elsewhere adds a destination and an audience. That extra effect needs authority the local note request did not provide.

The check follows the effect, not the input method. Typing into a draft field edits a draft. Typing the same text into a terminal can run a command after Enter. A policy that allows every type action would let commands through the keyboard tool.

For a concrete example, use the fictional Meal Basket cart. Two paneer rolls cost ₹80 each, and two cups of chai cost ₹40 each. The total is 2 × ₹80 + 2 × ₹40 = ₹240. The demo action creates one local receipt, with no payment or real order.

The approval preview names the origin, tab, cart revision, items, total, and expected effect. A revision identifies a version of the cart. The executor also stores the exact payload, the data the action will submit. A fingerprint of that payload helps detect later changes.

Suppose the person approves cart c3 at ₹240. Another cup of chai changes it to c4 at ₹280, so the previous approval no longer covers the current action. Matching the button label would accept both carts, since both still say Place demo order. Comparing the revision and payload rejects the changed cart.

An approval needs a limited lifetime and a use count. Here it expires after two minutes and permits one receipt. Two minutes is a teaching choice, not a recommended limit. A delayed action must recheck expiry and current state, because time alone cannot show that the payload stayed unchanged.

The executor records whether approval was denied, cancelled, expired, invalidated, or consumed. Consumed means the authorized attempt has started under that approval. A lost reply cannot silently turn it back into an unused approval. The system must reconcile the attempted effect before another attempt.

Approval also needs a trustworthy link to the submission. Checking a preview and then clicking an uncontrolled page leaves a gap, because the page could change its payload after the check. A controlled commit operation can compare and submit one exact payload together. Without it, the approval record must acknowledge the remaining uncertainty.

The OWASP prompt injection guidance recommends human approval for high-risk actions. The preview must still explain what the person is authorizing. Repeated vague questions add interruptions without setting a useful permission boundary.

Once the action is allowed, the loop records its result. Keeping every old screenshot in each request eventually becomes expensive, so we next separate the record of the run from the input needed for its next decision.

Turn 30 contains system instructions, tools, messages, and screenshots. Turn 31 removes screenshot s1. The later screenshots may hold the same image data, but their preceding content changed, so they no longer belong to the same prefix.

Think of two ordered lists, one containing A, B, C, D and another containing A, B, D, E. Their common beginning ends after B. Finding D in both lists does not extend it. Prefix caching follows this ordering, not independent matching of every repeated item.

Keep stable tool definitions and task instructions near the start, and add fresh results and observations after the existing history. Avoid a changing timestamp before otherwise stable content. Changing a tool description also changes the prefix, even when the task stays the same.

Providers specify their own eligibility and matching rules. Anthropic documents matching prompt prefixes, cache locations, minimum lengths, and lifetimes. Its usage fields report cache reads and cache creation. Check those fields instead of assuming reused content was cached. See the prompt caching documentation for the current contract.

Cache reuse reduces repeated processing charges under the provider's pricing rules, but it does not remove the repeated content from the model's context limit. A request holding 100 screenshots still holds them when most input qualifies for caching. Billing and context capacity are separate limits.

A cache also has an expiry time. A person may take longer to review an approval than the cached input remains available. The next request must still work when reuse disappears. Treat a cache miss as a cost change, not missing task memory.

Some APIs keep conversation state behind an identifier, while others require the caller to resend selected history. These pages use the resend design to make the arithmetic visible. Provider-side state changes transport details, but it does not replace a local record of effects and approvals.

The picture explains which content can be reused, but not how much input accumulates across a run. We need to calculate individual request sizes and their total separately.

The black line keeps every screenshot. The red line keeps the latest ten after turn ten. The blue line prunes every twenty-fifth turn, keeping five screenshots, then adds new ones until the next pruning turn.

Before discounts, one turn adds 1,500 + 200 = 1,700 tokens. Request one holds 3,000 + 1,700 = 4,700 tokens, request two holds 6,400, and request one hundred holds 173,000. The last number describes one request, not the total sent through the run.

To find that total, add the request sizes. The growing part is 1,700 × (1 + 2 + ... + 100). Pairing the first and last terms gives 100 × 101 ÷ 2 = 5,050. The complete total is 3,000 × 100 + 1,700 × 5,050 = 8,885,000 tokens.

This is a triangular sum, the total formed by adding consecutive whole numbers. Its growing part rises roughly with the square of the turn count, so doubling a long run roughly quadruples that part. The fixed instructions add an amount proportional to the number of requests.

Now apply the chart's invented cache discount. Reused input costs 0.1 times fresh input. We ignore cache write premiums and count the stable prefix as reusable throughout, which simplifies even the first request, where a real cache needs creation. The result is a comparison model, not a provider invoice.

On the black line at turn 100, the previous 171,300 tokens qualify for the discount. The new 1,700 do not. The weighted amount is 0.1 × 171,300 + 1,700 = 18,830. Caching lowers the weight, but repeated old input still contributes to every later request.

The red line removes an early screenshot on each turn after ten. The chart assumes this makes all message history after the stable instructions fresh. At turn 100 it keeps ten images and all 100 text additions. Its amount is 0.1 × 3,000 + 10 × 1,500 + 100 × 200 = 35,300.

That red value exceeds the black value despite using fewer images. The example shows how a smaller request can cost more when editing destroys prefix reuse. It does not prove every pruning system behaves this way. Different layouts and caching contracts can produce different results.

The blue line accepts a larger fresh history on pruning turns. Between those turns, earlier content stays unchanged and the chart discounts it. At turn 100, pruning retains five images and all text. The weighted amount is 300 + 7,500 + 20,000 = 27,800.

The chart assumes every screenshot has the same token cost and ignores output, reasoning, tool charges, and variable image sizes. It also assumes the stable prefix always qualifies for reuse. Real measurements must replace these assumptions before choosing a schedule.

The twenty-five turn interval has no special status. Pruning earlier may avoid a context limit; pruning later may improve reuse but retain more irrelevant images. Compare schedules by accepted outcomes as well as cost, because cheap requests can produce expensive mistakes.

For BB-042, keep the match identifier, Final status, both innings, result, and fixture revision, along with the observation identifier that supplied them. A later summary should say the page reported those values. It should not turn page text into new user instructions.

Keep effect state too. Proposed save, attempted save, and confirmed save are different. If a save reply disappeared, preserve that unresolved operation even when you remove its screenshot. Otherwise the model may treat missing history as proof that no save happened.

Tool calls and results form a relationship pruning must respect. A result names the call it answers. Removing only the call can leave a result without meaning, and removing only the result can leave an apparently unfinished request. Follow the provider's message format when keeping or removing these pairs.

Anthropic explains this matching in its tool call documentation. Preserve the format even when a shorter history would fit the context limit.

Retaining text also needs a size limit. The chart keeps all text forever to isolate screenshot pruning, but a long real task can exceed its context limit through text alone. Repeated errors, copied page content, and large tool outputs add input without adding current evidence.

For a large tool result, keep the fields needed for the next decision and store the complete result outside the request with a reference for later inspection. Do not reduce a failed save to a generic error. Preserve whether the request was sent and whether its effect remains unknown.

Summaries must keep uncertainty visible. Suppose one image clearly shows BB-042, but its small result text is unreadable. The summary should preserve the identifier and say the result remains unreadable. Replacing that uncertainty with a likely winner would make later checking depend on an invented fact.

A useful record might say that obs-22 showed Final for BB-042 and that the note save timed out and requires a read back. That account beats saying the agent was nearly finished: the first preserves a specific next check, while the second hides the unresolved effect.

Keep the latest coordinate transform with its observation. A screenshot point becomes unsafe when separated from its crop and scale, and an old summary cannot authorize a new click at old coordinates. Capture current state when the next action depends on a current target.

For very long runs, create a new request context with a task summary. Include the goal, constraints, confirmed effects, unresolved effects, current evidence, and remaining budget, and record where that summary came from. Restarting with a shorter context changes the prefix once but may be necessary for correctness.

Test a summary with a concrete question. Can another decision maker identify the next permitted action without inventing missing facts? If not, add the missing evidence or request a fresh observation. The smallest useful summary still preserves the distinctions the task requires.

Screenshots can contain account names, messages, and tokens. Tool results can contain file contents or error details with private values. Limit access to the stored record and choose a deletion period before collecting it. Redact shared copies without silently changing the evidence used internally.

Keep enough structured data to explain a failure. Record the observation used, proposed action, policy decision, actual effect, and stop reason, and for an approval preserve the exact approved payload and its status. A list of model messages alone cannot show whether a denied action ran.

Measure input tokens, cache reads, cache creation, and output per request, plus elapsed time, action count, pruning events, retries, and approval waits. Separate these values instead of treating every slow turn as a model delay. A long human review and a slow capture need different fixes.

A useful trace can identify when pruning changed the next decision. Keep the request's selected observation identifiers and the summary version used that turn, then compare a failure with the exact evidence the model had. The full screenshot store may contain information the model never received.

Cost per accepted task gives a useful comparison. In an invented example, ten attempts cost ₹100 and eight pass independent checks, so the cost per accepted task is ₹100 ÷ 8 = ₹12.50. Counting only successful attempts would hide the money spent on failures.

Compare a schedule costing ₹80 across ten attempts with only four accepted. Its cost per accepted task is ₹80 ÷ 4 = ₹20. Lower spending per attempt did not make the completed work cheaper. Keep safety failures and unknown outcomes in these records too.

Batching saves observations when predicted state stays valid, and pruning limits context while preserving the facts later checks need. The next chapter follows the remaining failures, including hostile page instructions and saves whose replies never arrive.

Further reading

• Anthropic prompt caching documentation explains matching, expiry, usage fields, and the pricing assumptions a real measurement must use.
• OWASP prompt injection guidance explains why constrained permissions and human review accompany model instructions.