Cross Utterance Context for Speech Recognition

63 concepts60 papers1 proposal

When humans listen to a conversation, we do not wipe our memories clean at the end of every sentence. If someone says, "Meet me at the bank," and their previous sentence mentioned cashing a check, we instantly resolve the homophone. We track the topic, remember who is speaking, and anticipate what might be said next based on the flow of dialogue.

Yet, traditional Automatic Speech Recognition (ASR) and Spoken Language Understanding (SLU) systems have long operated under a strict independence assumption. As noted by Anthony Kum Hoe Tung et al., standard frameworks process audio segments in isolation. Every time a speaker pauses or a new turn begins, the internal state resets. This context-agnostic approach creates severe vulnerabilities: when acoustic signals are degraded by noise or phonetic ambiguity, an isolated model has no way to utilize historical data to inform current interpretations.

To build truly robust speech systems, researchers are moving beyond isolated utterance processing, developing architectures that preserve and exploit cross-utterance context.

The Cost of the Independence Assumption

Isolating utterances discards long-term patterns that persist across a conversation or lecture. As Chiori Hori et al. point out, this isolation prevents models from utilizing speaker characteristics, ongoing topics, or specialized vocabulary.

The consequences of this blind spot cascade through every layer of speech processing:

  • Acoustic and Semantic Ambiguity: Without context, acoustic models struggle to distinguish between similar-sounding words or phrases.
  • Stateless Downstream Failures: As Xingkun Liu et al. observe, stateless natural language understanding services fail to resolve contextual ambiguities or follow multi-turn conversations because they treat each user utterance as an independent unit.
  • Discourse Disconnection: In translation and semantic parsing, processing sentences in isolation makes it impossible to resolve pronouns or track cumulative meaning, leading to outputs that may be linguistically correct but logically disconnected from the preceding dialogue.

Architectural Fusion of History

How do we break the one-sentence-at-a-time barrier? The most direct approach is modifying the neural network architecture to ingest a history of preceding context alongside the current audio frame.

Chiori Hori et al. introduced a "Context-expanded Transformer" that uses a sliding-window approach to incorporate previous acoustic and linguistic context. Instead of treating the current utterance as a standalone vector, the model's attention mechanism spans across historical frames, yielding significant error reductions in long-form monologues and dialogues.

Isolated Processing:
[Utterance t-1] -> Reset State -> [Utterance t] -> Reset State -> [Utterance t+1]

Contextual Fusion:
[Utterance t-1] ---\
                    --> [Attention Mechanism] --> [Accurate Transcript t]
[Utterance t]   ---/

For multi-party conversations, tracking context is more complex because it involves turn-taking and role dynamics. Siddharth Dalmia et al. address this by using a speaker-specific cross-attention mechanism (a matchLSTM) that explicitly tracks the history of both speakers to better predict the current utterance.

To enrich this history further, Siddharth Dalmia et al. later developed a contextual gating mechanism to fuse acoustic data with external pre-trained text embeddings like BERT. By gating these text embeddings, the model dynamically decides how much linguistic history is needed to resolve ambiguities in the current audio signal. Similarly, Bei Li et al. combine a Conformer architecture with a Cross-modal Conditional Variational Autoencoder (CVAE) to extract "role preference" and "topic coherence" from previous turns, allowing the system to naturally disambiguate similar-sounding words based on the conversational theme.

Session-Level Language Modeling and Rescoring

Instead of altering the acoustic encoder, another powerful strategy expands the scope of the Language Model (LM) that guides decoding. Traditional language models reset their internal states at sentence boundaries, but session-level language modeling treats the entire conversation as a single continuous sequence.

Wayne Xiong et al. developed a session-level LM using an LSTM that tracks the entire history of a conversation, including speaker changes and overlapping speech. This allows the system to maintain a continuous semantic thread across turns. Srikanth Raj Chetupalli refined this concept by introducing a gated attention mechanism over past sentences, allowing the model to selectively weigh which historical turns are most relevant to the current word prediction rather than treating all past tokens uniformly.

Where should this long-context language model be applied? It can happen either during the initial decoding search or as a post-processing rescoring step:

  1. Deep Integration (Beam Search): Robert Flynn explores integrating long-context Transformer Language Models directly into the beam search decoding process. By allowing the system to "remember" previous sentences during decoding—rather than just using them to re-score a list of candidates—the search algorithm avoids discarding the correct hypothesis early on.
  2. Second-Pass Regeneration: For systems where first-pass decoding must remain lightweight, second-pass rescoring can be completely rethought. Anthony Kum Hoe Tung et al. replace traditional rescoring with "second-pass regeneration." They feed the $n$-best hypotheses from an isolated ASR model along with a broad context passage into a generative Large Language Model (LLM), allowing the LLM to synthesize a highly accurate, contextually aware transcript.

Constraining the Search Space with Dialogue State

Conversations are not random sequences of sentences; they possess structural intent. Knowing what a speaker is trying to do provides massive predictive constraints for an ASR system.

Feng-Ju Chang et al. utilize the "Dialog Act"—the underlying intent of a conversation turn—to guide a contextual adapter. If the system knows that its own prior prompt strongly constrains the likely user response (such as asking for a contact name), the adapter filters the user's catalog and biases the attention mechanism toward personalized rare words.

This structural awareness is also vital for error recovery. When an ASR system mishears a command, users naturally repeat or reformulate their request. Hoang Long Nguyen et al. developed a pointer network that explicitly links sequential utterances, merging an incorrectly transcribed initial command with the user's follow-up correction. By treating the two turns as a single, connected interaction rather than isolated events, the system leverages natural user feedback to resolve its own errors.

This aligns with the broader paradigm shift advocated by Amos Azaria et al., who model conversation as a structured prediction task with latent discourse states, and Jatin Ganhotra et al., who encode full dialog history, speaker roles, and previous labels to drastically improve downstream intent recognition.

Overcoming the Efficiency Bottleneck

If cross-utterance context is so beneficial, why isn't every system using it? The primary obstacle is computational overhead. Processing long sequences of high-dimensional acoustic features across an entire conversation quickly runs into severe memory and latency bottlenecks.

To make long-context speech recognition practical, researchers have explored alternative, compressed representations of history:

  • Discrete Speech Tokens: Mingyu Cui et al. demonstrate that instead of passing continuous acoustic features from previous turns, one can use discrete speech tokens extracted from self-supervised learning (SSL) models like WavLM. This drastically compresses the historical context, achieving massive training speedups while retaining the necessary linguistic and acoustic cues.
  • Target-Language Text Context: In end-to-end speech translation, a similar efficiency trick involves modality switching. Amir Hussein et al. show that instead of feeding raw historical audio into the model, feeding previously translated target-language text back into the system bypasses memory bottlenecks while ensuring pronoun and stylistic consistency across a dialogue. This mirrors the approach taken by the authors of Towards Making the Most of Dialogue Characteristics for Neural Chat Translation, who train translation models to simultaneously predict the next response and identify the speaker, ensuring personality-consistent chat translations.

By treating conversation as a continuous, structured dialogue rather than a sequence of fragmented audio clips, modern speech systems are successfully overcoming acoustic degradation and semantic ambiguity.

Key Foundational Work

Ontology-based Contextual Coherence Scoring (2003) — contributed 1 concept elements to this direction.
A Speech Understanding System Based Upon A Co-Routine Parser (1975) — contributed 1 concept elements to this direction.
History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 60 papers
  2. Version 1 · 2026-10-04 · 111 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_isolated_utterance_processing_b5742358,
  title = {isolated utterance processing},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/isolated-utterance-processing/rdr_b574235833b44f75832ee6084b07d192},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_isolated_utterance_processing_b5742358_v2,
  title = {isolated utterance processing},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/isolated-utterance-processing/rdr_b574235833b44f75832ee6084b07d192/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake