One model, four different results
Agent performance is usually attributed to the model. When a system fails, the first proposals are a stronger model, a larger reasoning budget, or another reflection pass. Yet a substantial software layer sits between the model and the task. It assembles context, exposes tools, compresses old observations, handles errors, and decides when execution should stop.
OpenForgeRL shows that this layer changes results too much to be treated as packaging. The authors ran the same base model on the same tasks inside ReAct, OpenClaw, ZeroClaw, and a Codex-like harness. Success varied materially. Training on trajectories from several harnesses transferred better than training inside one.
OpenForgeRL places a proxy between the production harness and the inference server. It intercepts model calls and records trajectories, tool outputs, and terminal rewards. Training therefore occurs in the runtime where the agent will operate, closing the artificial gap between model capability and orchestration behavior.
The full reproducible code and models were not public at release, so its claims still need independent reproduction. The underlying observation is easy to test: keep the model and tasks fixed, then change only context construction or retry policy.
The harness can be trained separately
Harness-R1 treats the runtime itself as the optimization target. A separate 9B model receives batches of failed trajectories and edits executable agent code, including context assembly, tool handling, action validation, and recovery. Reward comes from rerunning the task after the patch, not from a language-model judge's opinion of the code.
Across WebShop, ALFWorld, and DBBench, Qwen3.5-9B improved from 44.3% to 53.6% without fine-tuning the agent model. After direct agent fine-tuning, harness changes raised the result from 59.2% to 64.2%.
These environments are much smaller than a production repository or a multi-hour browser workflow. An automated harness engineer can also grant dangerous permissions, hide an error, or overfit a verifier. The useful contribution is the unit of change: weights and prompts are not the only trainable parts. Executable procedure can be optimized too, provided every patch is rerun on a holdout set and checked for prohibited actions.
A reasoning failure may be a transition failure
Read-Gate analyzed 12,000 search trajectories. Agents often called search, received snippets, and answered without opening the retrieved documents. Increasing the reasoning budget did not fix this behavior.
A simple search -> read -> answer invariant improved accuracy by 14.9 to 19.9 percentage points on trajectories where reading would otherwise be skipped. Across full experimental cells, the gain was 3.2 to 9.4 points.
The model did not necessarily misunderstand the source. The harness allowed completion before evidence was acquired. A stronger model could produce a more convincing answer from the same snippets while preserving the underlying control-flow bug.
HALT applies the same principle to search termination. It creates expected intermediate claims and stops retrieval only when the evidence covers them. If coverage cannot be verified, abstention may be the correct terminal state. More search is not automatically more evidence.
Performance lives between model calls
The harness affects systems performance as well as accuracy. Architectural Implications of Agentic AI studies Azure workloads that alternate among GPU generation, CPU orchestration, and external tools. Utilization is bursty: GPUs wait during tool execution, CPUs spike during parallel builds and tests, and latency accumulates across several unrelated queues.
The authors built Agora, a research scheduler that colocates phases and consumes otherwise idle resources. They report 30% higher CPU utilization with less than 3% agent slowdown. GPU consolidation freed one third of the devices, increased generation throughput by 82%, and reduced tail latency by 2.5 times.
Those figures come from one infrastructure and a research prototype. They still demonstrate why tokens per second is not task throughput. An agent may spend more time waiting for a database, build, network request, or process lock than generating text.
Measure the system, not the model label
An agent comparison should pin both the model version and the harness version. At minimum, record task success, model and tool calls, retries, total tokens, tool wait time, wall time, and full cost.
Termination reasons and context-compaction points matter too. If a critical tool result disappeared during history compression, a model upgrade may hide the failure by chance without repairing it.
Production harnesses need the same discipline as ordinary software: versioning, component tests, controlled rollout, and rollback. The model sets a range of possible capabilities. The harness determines how much of that range reaches the final state.