The agent read the rule and violated it anyway
A production agent can have a careful system prompt, a detailed operating manual, and precise tool descriptions. That is often enough for a demo. Over a long workflow, the arrangement breaks down. A recent user request overrides standing policy, the result of a required check is forgotten a few steps later, or the agent reports compliance that never occurred in the external system.
HANDBOOK.md tests this failure directly. Its 65 tasks span finance, insurance, medical billing, logistics, and HR. Agents receive 20 to 124 pages of operating procedures and work with files, email, calendars, issue trackers, and commerce services. The evaluator uses 824 deterministic criteria covering both required and prohibited actions.
The best of 30 configurations completed 36.2% of the tasks. Most frontier configurations stayed below 25%. Retrieval was not the only problem. Agents performed a required check, observed a negative result, and still took the prohibited branch. Some ended with a confident compliance report even though a mandatory tool call was absent.
Another paragraph in the prompt cannot reliably repair this. Text describes a rule. It does not execute one.
The model should not own every part of the workflow
Enterprise operations contain two different kinds of work. One requires semantic judgment: interpreting an email, matching a document to a request, or drafting a response. A language model is useful here.
The other kind is deterministic. Identity verification must precede a payout. A failed check must not lead to approval. Consent must be recorded before a customer record changes. These requirements look like a program: they have states, transition conditions, and forbidden actions.
COVENANT separates the roles. It compiles a natural-language procedure into a workflow tree and then a control-flow graph. The model sees the current step and proposes an action. Before any state change, a separate controller validates the branch condition, the permitted tool, and its arguments.
Across 120 cases from seven workflows, task success increased from 50.0% to 83.33%, while workflow misalignment fell from 42.5% to 15.83%. The sample is small, and compilation can introduce its own errors. The useful result is the boundary: the model handles ambiguous meaning; software enforces order and prohibitions.
Enforcement is needed before and after a tool call
A pre-action check answers one question: is this operation allowed from the current state? It does not prove that the tool succeeded or that the external system reached the expected state.
Real-Time Detection and Repair of Failures in AI Agents compares behavioral monitoring with deterministic verification. Its telemetry monitor detected 71% of failures at a 5% false-alarm budget, taking roughly 200 microseconds per step.
For structured workflows, direct checks were stronger. Recomputing results from actual tool outputs, confirming mandatory calls, and verifying postconditions detected up to 96% of failures with no false positives in the reported sample. The system then rolled back and reran the failed branch. Task success rose from 52% to 73% for roughly one extra model call.
Behavioral monitoring helps with failure modes that were not anticipated. A known policy condition does not need to be inferred statistically. It is cheaper and more reliable to encode it directly.
Memory is evidence, not ground truth
Even a correct workflow graph can receive a false input from agent memory: an expired consent, an outdated account state, or a result copied from a different customer. If the controller trusts memory as fact, a formally valid transition can still cause harm.
SafeCommit evaluates a side-effecting action against several plausible worlds. These worlds are assembled from memory, observations, tool outputs, provenance, and policy. The action is allowed only when it is safe in every retained world. Otherwise the agent may perform a low-risk probe that resolves a specific uncertainty, choose a safe fallback, or refuse.
This is still a formal design demonstrated in a small simulator, not a production runtime. Its architectural test is useful: uncertain information may remain available for reasoning, but it cannot silently become a payment, a CRM change, or a promise to a customer.
A minimum production design
A critical workflow needs four separate layers rather than one enormous prompt.
The operating procedure should be versioned. States, mandatory checks, permitted transitions, and explicit prohibitions should be extracted from it. The model receives the relevant step instead of rereading a hundred pages on every turn.
Before each side effect, a controller validates the current state, branch condition, permission, and tool arguments. After execution, another verifier reads the actual system state: whether the order exists, consent was recorded, or the final amount matches the tool outputs.
If verification fails, the system should not ask the model to explain the result more persuasively. It should roll back reversible changes, preserve the trace, and either retry a safe branch or escalate to a person.
The model remains inside the workflow, but it is no longer the sole executor and judge. This distinction is particularly visible in voice support: a fluent conversation says nothing about the status written to the CRM after the call.
Prompts are good at interpretation. The authority to change an external system belongs in verifiable software.