In automatic speech recognition (ASR), developers traditionally faced a strict architectural dichotomy. On one hand, applications like live captioning require streaming models—systems that operate with minimal latency by processing audio on the fly, restricted to looking only at past and immediate acoustic context. On the other hand, applications like meeting transcription favor offline (non-streaming) models, which process the entire audio utterance at once, leveraging future context to maximize accuracy.
As noted by Li et al. and Sharma et al., maintaining two distinct architectures for these use cases doubles engineering overhead, training costs, and deployment complexity. The research direction of unified, dual-mode ASR solves this dilemma by consolidating both capabilities into a single neural network with shared parameters. Crucially, this unified approach does not just save computational resources; the interplay between the two modes makes the real-time streaming mode significantly more accurate than if it were trained alone.
The Core Mechanism: Dynamic Chunking
To understand why a single model can handle both real-time streaming and offline processing, we must look at how the network controls its visibility into the future. In a standard Transformer or Conformer encoder, self-attention allows every audio frame to look at every other audio frame across the entire utterance. For an offline model, this is ideal. For a streaming model, it is impossible, as future frames have not yet been captured.
Unified architectures resolve this by using dynamic chunk-based training. Introduced by Yao et al. in the U2 framework, this technique varies the attention mask during training rather than keeping it fixed.
Offline Mode (Full Context):
[ Frame 1 ] <---> [ Frame 2 ] <---> [ Frame 3 ] <---> [ Frame 4 ]
Streaming Mode (Dynamic Chunks):
[ Chunk 1: F1, F2 ] ----> [ Chunk 2: F3, F4 ]
(No lookahead) (Can only look back at Chunk 1)
During a single training run, the model is exposed to a variety of chunk configurations for different batches:
- Full Context: The attention mask is completely open, allowing the model to act as an offline system.
- Limited Chunks: The audio is divided into small, independent time windows (e.g., 160ms). Frames within a chunk can attend to each other and to past chunks, but they are strictly barred from looking at future chunks.
By randomly sampling the chunk size during training, the shared encoder learns a flexible representation. It discovers how to extract local acoustic features when context is constrained, while retaining the capacity to exploit global context when the attention mask is fully opened. Similarly, Sharma et al. utilize dynamic "right-context" windows during training to smoothly control the amount of lookahead information the model is allowed to use.
Two-Pass Decoding: Immediate Feedback with Delayed Refinement
Operating a unified model in production requires a decoding strategy that satisfies both low latency and high accuracy. This is achieved through a two-pass decoding paradigm, which splits the output generation into two distinct phases fed by the shared encoder (Pang et al.; Yao et al.).
- The First Pass (Streaming): As audio streams in, the encoder operates in its chunk-limited mode. A fast, monotonic decoder—such as a Connectionist Temporal Classification (CTC) decoder or a Recurrent Neural Network Transducer (RNN-T)—processes these embeddings to emit text hypotheses immediately. This provides the user with instantaneous, real-time feedback.
- The Second Pass (Offline Rescoring): Once the speaker finishes talking (or a significant pause occurs), the attention mask is opened completely. An attention-based decoder takes the full, non-causal encoder embeddings and rescores the candidate sentences generated by the first pass.
This architecture ensures that the system provides immediate text that can dynamically correct itself a split second later as global context becomes available (Zhang et al.; Subramanian et al.). To ensure that the low-latency and full-context paths do not compromise each other's performance during joint optimization, Shakeel et al. demonstrated the necessity of branching the shared encoder into explicitly separate decoders for each mode.
Why Joint Training Enhances Streaming Accuracy
A key insight from this line of research is that a unified model often outperforms a standalone streaming model on real-time tasks. This counterintuitive result stems from how the shared weights are regularized during training.
When a streaming model is trained in isolation, it can easily get trapped in local acoustic ambiguities because it cannot see the upcoming words. However, in a dual-mode architecture, the shared encoder weights are simultaneously optimized to satisfy both the streaming and full-context objectives. The full-context path acts as a guide, forcing the shared encoder to learn deeper semantic and structural representations of speech.
To maximize this benefit, researchers use in-place knowledge distillation (KD). As proposed by Yu et al., when a batch is processed in full-context mode, it generates a set of highly accurate target distributions. When the exact same batch is processed in streaming mode, the model is penalized not only for misidentifying the words, but also if its streaming internal representations deviate from its own full-context representations.
[ Audio Input Batch ]
|
+---> (Streaming Mask) ----> Streaming Features <---+
| | (In-Place KD Loss)
+---> (Full-Context Mask) -> Full-Context Features -+
Shakeel et al. expanded on this by using similarity-preserving knowledge distillation, which ensures that the geometric relationships between different speech tokens learned in the offline path are preserved within the streaming path. This cross-mode knowledge transfer effectively allows the streaming mode to "anticipate" structural patterns in speech even when its physical lookahead window is blinded.
Alternative Paradigms and Future Horizons
The philosophy of unifying streaming and offline processing has expanded beyond standard Transformer and Conformer architectures into alternative sequence-modeling frameworks:
- State-Space Models: To avoid the quadratic computational complexity of self-attention over long audio sequences, Moriya et al. adapted the Mamba selective state-space model for chunk-wise processing (LC-BiMamba), achieving competitive dual-mode performance with linear computational scaling.
- Self-Supervised Learning: Rather than applying unification only during final task fine-tuning, Kutsakov et al. integrated dynamic chunking directly into the self-supervised pre-training phase of foundational speech models (GigaAM), ensuring that the underlying acoustic representations are natively optimized for both streaming and offline downstream tasks.
- Delayed Streams Modeling: Moving away from explicit attention masking entirely, Zeghidour et al. introduced a framework that treats audio and text as parallel, time-aligned token streams within a single decoder-only Transformer. By simply shifting the relative time delay between the input and output streams, the model naturally transitions from a low-latency streaming system to a high-accuracy system with lookahead context.
Key Foundational Work
Advances in Joint CTC-Attention Based End-to-End Speech Recognition with a Deep CNN Encoder and RNN-LM (2017) — contributed 1 concept elements to this direction.
Advancing Acoustic-to-Word CTC Model (2018) — contributed 1 concept elements to this direction.
Improving Attention Based Sequence-to-Sequence Models for End-to-End English Conversational Speech Recognition (2018) — contributed 1 concept elements to this direction.