A plausible final frame can still be a failure
The agent clicked Save, the dialog closed, and the final screenshot looks calm. That does not mean the file exists on disk. The application may have shown an error that disappeared, rendered an intermediate state, or saved into another profile.
Many computer-use evaluations rely on screenshot sequences and a final verdict from a vision-language model. That judge evaluates appearance. The actual task is usually a claim about state: a file exists, a setting is enabled, a record was created, or a message was sent.
Interactive Reward Agent first turns an instruction into explicit success conditions, then gathers evidence through system, application, and GUI tools. It can inspect a file, application setting, or environment state instead of judging the final frame alone.
On GUI-RewardBench's 321 trajectories, the system reached 86.9% accuracy. Using it as a reinforcement-learning reward produced 34% success on OSWorld. The sample is small, and the verifier is itself an agent that can fail. The durable contribution is the order of operations: state the postconditions, then query the state.
Visual judges have a leniency bias
OSReward collected human-verified computer-use trajectories across several platforms. Strong vision-language judges showed a systematic leniency bias, accepting failed executions as successful.
The project released OS-Shepherd-100K, 100,000 labeled judgments, along with 9B and 35B reward models. The authors report performance comparable to commercial judges at 30% to 60% lower cost. That comparison is author-reported, but the open data and checkpoints make external testing possible.
A related audit of five popular benchmarks manually inspected 150 trajectories labeled as failures. For 15.3%, the verdict was wrong: 10.7% were evaluator false negatives and another 4.7% were broken tasks.
Both directions damage training. A false success rewards a non-working action. A false failure discards a valid trajectory. One aggregate success rate hides the two mechanisms.
Screenshot order is not event order
Desktop rendering is asynchronous. After a click, one region updates immediately, another waits on the network, and capture may occur before rendering settles. When frames are presented in a sequence, models tend to accept that sequence as causal.
Desktop-Delta contains 2,013 verified transitions from Linux applications. A model must identify the change caused by an action, reject stale or unrelated frames, and reconstruct temporal order.
The best exact-match result was 65.1% to 65.7%. Models frequently trusted the proposed frame order. Clicks were recognized much more reliably than drags, with F1 of 0.96 versus 0.76.
This explains a common computer-use failure. The agent sees a frame resembling the target and continues even though the application has not completed the transition. Its next action lands in the wrong window or modifies the wrong object.
Verify the delta, not the picture
Each action should have a predeclared expected delta. Saving should create a file at a known path with expected content. Changing a setting should produce a value readable through the application, configuration, or system query. Sending a message should return a server-side identifier.
Screenshots remain useful evidence, particularly when no other interface exists. They should carry capture time and window identity, and the state should survive a delay, refresh, or reopening step.
When an application exposes an API or database, the verifier should read it directly. Ideally it uses a channel different from the one the agent used to act. Otherwise one application defect may both create a false state and confirm it.
Irreversible operations need an additional safeguard. The agent should prepare the expected state delta before execution. A controller performs the action once with an idempotency key and checks the service response. Retrying after a frozen screen must not create a second message, order, or payment. When no direct state channel exists, combine weaker signals: wait for stability, reopen the object, search by a unique identifier, and check for an error state.
Label evaluator failures separately
A manual audit needs more than PASS and FAIL. At minimum, distinguish agent failure, verifier error, broken task, and indeterminate outcome. Without those labels, improving the verifier may appear to make the agent worse, while repairing a task may look like a capability gain.
A small sample of both successful and failed trajectories should be reviewed by people after every substantial evaluator change. Delayed transitions, drag operations, multiple windows, and states invisible on screen deserve oversampling.
The evaluator itself should be versioned. Replay old traces through both the previous and new version so a score change can be attributed to agent behavior, verifier behavior, or a repaired task.
A screenshot is an observation. Task success is a claim about the system. A verifier has to connect the two.