More tasks can make an agent worse
Training a computer-use or terminal agent requires applications it can change, break, and reset. Production services are poor training grounds: they are login-gated, contain real data, and cannot safely absorb thousands of exploratory actions. Teams build synthetic copies instead.
The failure begins when a copy reproduces the surface but not the causal logic. An agent learns to satisfy the grader in the toy application and transfers the wrong strategy to the real service.
Echoverse measures this effect directly. It compiles specifications into stateful applications, generates tasks, and grades results against each application's database. Every rollout is used twice: as training signal for the model and as evidence for repairing the environment, task, and verifier.
A 9B model trained on 12 environments improved from 36.5% to 67.1% across 14 evaluation splits. Yet shallow environments reduced live-site accuracy from 80% to 75%. Deep environments raised it from 80% to 85% in one setting and from 48% to 65% in another. Repairing one defective environment lifted the model trained on it from 16.2% to 38.5%.
Environment count did not predict transfer in these experiments. Causal fidelity did.
Depth means hidden state, not a polished interface
A deep environment preserves relationships from the production workflow. A created order appears in the list, changes inventory, survives a refresh, and can be cancelled only through a valid transition. A failed operation changes the database correctly rather than merely displaying a prepared message.
A shallow copy often connects a button to a success banner. The agent learns the shortest route to the banner. On the real site, that route may not create an order, enforce authorization, or survive a page reload.
The verifier should read internal state independently of the interface. If a task asks for a user with a restricted role, it should inspect the record, role, scope, and absence of excess permissions. A final screenshot supports diagnosis but is not the source of truth.
The environment also needs negative branches: expired consent, insufficient privileges, version conflicts, duplicate requests, and partial external-service failure. Those branches distinguish process knowledge from memorizing the happy path.
Long tasks can be synthesized with verification
Recursive Synthesis for Long-Horizon Terminal Tasks tackles scale for terminal agents. Direct task generation by a language model often desynchronizes four components: instruction, environment, reference solution, and verifier.
RST starts from a verified seed. Each round extends the reference solution, then realigns the instruction and verifier with the new workflow. The candidate runs in a fresh sandbox. Only accepted tasks become seeds for the next round.
Across 15 rounds, the authors produced 37,484 tasks at roughly $0.05 each. Median reference-solution length grew from 67 to 374 lines and executed commands from 40 to 244. DeepSeek-V4-Pro pass@4 fell from 90% in round one to 2.5% in round fifteen.
Fine-tuning Qwen3.5 on rejection-sampled trajectories improved results by up to 10 points across three terminal benchmarks. These are author-reported results. A key external check is whether recursive generation leaves recognizable templates that models can exploit without solving the underlying task.
The verifier and environment must evolve together
Updating only the model is insufficient. A stronger agent reaches states that earlier versions never visited and exposes simulation defects. If every such trajectory is labeled as an agent failure, training rewards workarounds for a broken world.
Every unusual outcome needs attribution. Did the agent violate the condition? Did the verifier read the wrong state? Is a causal relationship missing from the application? Is the task impossible? After a repair, old trajectories should be replayed so evaluator changes are separated from capability changes.
For voice workflows, the same rule requires a business model behind the dialogue. Realistic customer utterances are not enough. The scenario needs orders, consent, limits, available time slots, and consequences for every tool call. Evaluation should inspect the resulting business state, not phrase similarity.
Test transfer before training at scale
A new environment should pass a few manual invariants first. Performing an operation through the interface and through a direct system request must yield the same state. Refreshing must not erase the result. Retrying must behave like the production service. A prohibited branch must actually be unreachable.
Then evaluate a base model on both synthetic and real holdouts. If training improves the artificial environment while degrading the live workflow, stop the release even when training reward continues to rise.
Synthetic data becomes cheap only after verification is reliable. Without that investment, scale teaches an agent patterns that do not exist in the world where it will act.