Cross Utterance Context for Conversational Speech Recognition

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2111 papers119 conceptsfrozen 2026-10-04

Traditional Automatic Speech Recognition (ASR) systems operate under a "sentence-level independence" assumption. They treat every audio segment as an isolated island, resetting their internal state after every period or pause. While this simplifies the engineering, it ignores how humans actually listen. We use the flow of a conversation—the topic, the speaker’s accent, and previous turns—to resolve ambiguities.

Recent research has moved toward contextualized ASR, a paradigm that breaks the sentence barrier to leverage long-range linguistic and acoustic dependencies.

The Problem of Isolated Utterances

When a model processes speech in isolation, it lacks the "global" information needed to resolve phonetic similarities. For example, in a clinical setting, Milena Davudova and Ziyuan Cai found that foundation models like Whisper struggle with fragmented, single-word utterances because they rely heavily on surrounding linguistic context to "guess" the next word. Without that context, the models often hallucinate or fail entirely.

This isolation also affects endpointing—deciding when a user has finished talking. Traditional systems use simple silence thresholds, often cutting people off mid-thought. Ariya Rastrow and Björn Hoffmeister demonstrated that by incorporating the linguistic context of the entire search beam, a system can distinguish between a mid-sentence hesitation and a completed query.

Expanding the Transformer Horizon

To give models a "longer memory," researchers have developed architectures that explicitly look at previous utterances. Chiori Hori and Takaaki Hori introduced the Context-expanded Transformer, which uses a sliding-window approach to incorporate acoustic and linguistic cues from multiple consecutive utterances. This allows the model to adapt to a speaker's characteristics and the ongoing topic, reducing error rates by up to 15%.

Similarly, Zhao You and Dong Yu proposed a Persistent Memory model. Instead of clearing the cache after each sentence, the model stores historical information in a dedicated memory bank, allowing the Self-Attention Network (SAN) to "recall" patterns found across the entire dataset or conversation.

Fusing Dialogue History

In two-party conversations, the meaning of what Speaker B says is often locked in what Speaker A just said. Siddharth Dalmia and Florian Metze addressed this by using cross-attention mechanisms (like matchLSTM) to "listen" to the history of both speakers. By fusing the current acoustic signal with external text embeddings from previous turns, the model can leverage topical consistency to disambiguate similar-sounding words.

To make this efficient for streaming applications, Jiawen Kang developed attention pooling. This technique compresses the history of previous utterances into a compact, low-dimensional representation. This "summary" of the past provides the model with necessary context without the computational burden of processing every previous word in real-time.

Second Pass Regeneration with LLMs

Even when the initial transcription (the "first pass") is flawed, the broader context can be used to fix it. A growing trend involves using Large Language Models (LLMs) for second-pass rescoring.

Anthony Kum Hoe Tung and Yixuan Tang proposed a "regeneration" framework where an LLM like Mistral-7B receives the top $n$ hypotheses from the ASR and a passage of context. The LLM then synthesizes a final transcript that is semantically coherent with the conversation. While Julie Carson-Berndsen notes that LLMs can sometimes "over-correct" grammar, they significantly improve the logical consistency of the output, making the resulting text much more useful for downstream tasks like chatbots.

Collaborative Context and Graph Models

Context isn't just about what was said before; it's also about what sounds similar across a recording. Srinath Tankasala and Andreas Stolcke introduced cross-utterance rescoring using graph-based label propagation. By treating audio segments as nodes in a graph, the system can share high-confidence transcripts from one clip to resolve ambiguities in another similar-sounding clip.

This structural awareness is also vital for speaker attribution. Jixuan Wang and Frank Rudzicz showed that modeling segments as a graph allows the system to use the relationships between all segments in a session to identify speakers, rather than classifying them one-by-one in a vacuum.

Papers in this version

Showing 60 of 111 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 60 papers
  2. Version 1 · 2026-10-04 · 111 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_isolated_utterance_processing_b5742358,
  title = {isolated utterance processing},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/isolated-utterance-processing/rdr_b574235833b44f75832ee6084b07d192},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_cross_utterance_context_for_conversation_b5742358_v1,
  title = {Cross Utterance Context for Conversational Speech Recognition},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/cross-utterance-context-for-conversational-speech-recognition/rdr_b574235833b44f75832ee6084b07d192/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}