The first words may not survive the sentence
A caller begins an instruction, pauses, and corrects it: "Transfer five... no, fifty thousand." A streaming ASR system has already placed the first amount in the conversation history. If later words are only appended, the agent may act on a version the speaker explicitly abandoned.
Transcripts are usually treated as immutable logs. That is convenient for storage and unlike actual speech. False starts, repairs, and late qualification change the meaning of an earlier span. Production state therefore needs bounded revision while preserving the raw record for audit.
The problem is visible even in the standard error metric. Transcription Policy as a Latent Variable separates two tasks. A verbatim policy preserves repetitions, pauses, cutoffs, and fillers. An intended-speech policy records what the speaker ultimately meant.
When a model and its reference use different policies, the mismatch explains up to 60% of measured word error. Explicit task tokens raised zero-shot German disfluency F1 from 10% to 79%, despite English-only training. One reference transcript cannot simultaneously be an exact record and an edited prompt for a downstream model.
Revise a bounded active span
AgenticASR represents streaming transcription as revisable state. It keeps a bounded active segment and rewrites it when later audio shows that an earlier phrase was a false start, abandoned content, or a self-correction. Older history becomes immutable.
This avoids waiting for the end of a long recording while preventing every new word from triggering a full rewrite. The size of the editable window is a real design decision. Too short, and a repair cannot reach the wrong phrase. Too long, and the system may silently rewrite facts that have already influenced an action.
A production voice agent therefore needs more than the current text. It needs the original hypotheses, revisions, timestamps, and reasons for each change. At publication time, the authors had promised code and AASR-Bench but had not yet released a complete reproducible package. The architecture is more actionable than its reported scores until the release can be inspected.
Domain correction must be able to abstain
The next failure appears after acoustic recognition. Product names, surnames, and internal terms may be transcribed incorrectly in a repeatable way. A language-model corrector can repair them, but it can also overwrite a sentence that was already correct.
Voice Memory adds an auditable memory.md file on top of a frozen ASR system. It stores domain correction rules. For each utterance, the corrector either applies a rule or leaves the original 1-best hypothesis untouched. A separate asynchronous optimizer edits memory and accepts a change only after it improves a holdout score.
Weighted WER fell from 8.36% to 7.52% without regressing any individual dataset. The share of corrections that damaged correct tokens dropped from 64% to 35%. That second number exposes the cost of the task: even an improved corrector can cause harm, so abstention is part of the product behavior.
An explicit memory file also makes a rule versioned and removable. A team can attach provenance and scope to a customer name or product vocabulary, then roll it back without retraining the acoustic model.
Cleanup can destroy the missing evidence
Turning conversational speech into polished prose creates another failure mode. HIVE models distortions associated with spoken and typed input, then measures downstream instruction following.
The most damaging perturbations were not fillers or pauses. They were deletions and structural rewrites of the source tokens. More reasoning budget largely compensated for typos but did not recover spoken registers. Performance sometimes fell further on heavily compressed speech.
A model can infer a missing preposition. It cannot reliably reconstruct a deleted negation, amount, or consent condition. A cleaned transcript should never become the only representation of the call.
Store three linked versions
A practical voice pipeline should keep three related objects.
The first is the raw streaming transcript with timestamps and alternative hypotheses. It supports audits and later reprocessing.
The second is the revisable state of the active utterance. It records abandoned spans, self-corrections, and the currently effective intent. This is what the dialogue agent reads.
The third is a cautiously normalized version for retrieval and analytics. Every domain correction has a rule, provenance, scope, and rollback path. When confidence is insufficient, the original text survives.
WER alone cannot evaluate this design. Tests need explicit negations, numbers, names, cutoffs, and late repairs. The final verifier should inspect the business action: which amount was stored, which address was confirmed, and which condition governed the transaction.
The audio and revision history should be immutable. The current meaning of the conversation should not be.