Dual Latency Cascaded Encoders for Streaming Speech Recognition

20 concepts16 papers1 proposal

In automatic speech recognition (ASR), systems face a fundamental tension between latency and accuracy.

To provide real-time feedback—such as live captions during a conversation—a model must operate causally. It can only look at the audio frames it has received up to the current millisecond. However, language is inherently contextual; the meaning (and correct spelling) of a word often depends on words spoken seconds later. Non-streaming models exploit this by processing the entire utterance globally using bidirectional or non-causal attention, achieving significantly lower error rates at the cost of making the user wait until the sentence finishes.

To bridge this gap, modern ASR architectures use dual-latency cascaded encoders. Instead of choosing between streaming and non-streaming, these architectures run a fast, causal first pass to emit immediate hypotheses, followed by a non-causal second pass that refines the output using future context.


The Two-Pass Paradigm

The foundation of this approach relies on a two-pass decoding strategy. The system generates an immediate, low-latency prediction for the user and subsequently corrects it once more context becomes available.

Early implementations of this concept focused on splitting the decoding graph at the output of a single encoder. As explored by Gowda et al. and reviewed by Kim et al., a single uni-directional encoder can simultaneously feed two distinct decoders: a streaming decoder (like an RNN-Transducer) for real-time feedback, and a batch decoder (like an attention-based rescorer) that processes the full sequence globally.

                  ┌───► Streaming Decoder ───► Immediate Output
                  │
Audio ───► Causal Encoder
                  │
                  └───► Non-Streaming Decoder ───► Refined Output (Global Context)

In a prominent early formulation of this paradigm, Pang et al. combined an RNN-Transducer with a Listen, Attend and Spell (LAS) decoder to rescore the streaming hypotheses. Similarly, the open-source WeNet U2 framework proposed by Yao et al. unified streaming and non-streaming modes into a single model using a shared encoder trained with dynamic, chunk-based attention masks. This allowed the same network weights to serve both immediate chunk-by-chunk decoding and a final Transformer-based rescoring step.


Cascading the Encoders

While sharing a single causal encoder prevents operational overhead, it limits the second pass. Because the underlying encoder features are strictly causal, the second-pass decoder can only use future context during the attention step, not during the acoustic feature extraction itself.

To overcome this limitation, researchers shifted from a shared encoder to a cascaded encoder architecture. In this setup, the acoustic processing itself is split into a fast-slow hierarchy:

  1. The Causal (Fast) Encoder: Processes incoming audio with zero or minimal look-ahead to generate low-latency acoustic features for the first-pass streaming decoder.
  2. The Non-Causal (Slow) Encoder: Takes the intermediate representations from the causal encoder and processes them further, incorporating a temporal look-ahead window or full bidirectional context.
Audio ───► Causal Encoder ───┬───► Fast Streaming Decoder ───► Live Text
                             │
                             ▼
                       Non-Causal Encoder (Look-ahead) ───► Slow Decoder ───► Final Text

This architecture, detailed by Mahadeokar et al., allows the secondary encoder to effectively "wrap" around the primary encoder. By building directly on top of the causal features, the non-causal encoder only needs to learn how to refine those features with future context, rather than re-processing the raw audio from scratch.

This dual-head or dual-pass approach has been successfully integrated across various model backbones. For instance, Li et al. implemented a dual-head architecture within a Contextual Layer Trajectory LSTM, using a causal branch for instant predictions and a non-causal branch to capture future frames. In Transformer and Conformer spaces, Prabhavalkar et al. and Li et al. demonstrated that cascading a causal encoder with a look-ahead non-causal encoder consistently bridges the accuracy gap between streaming and offline models without delaying the initial token emission.


Why It Works: Feature Reuse and Deliberation

The cascaded architecture works efficiently because it treats the second pass as an incremental refinement step rather than an independent calculation. This provides several distinct advantages:

1. Asynchronous Look-Ahead

Instead of waiting for an entire sentence to finish, the non-causal encoder can operate with a fixed, multi-frame look-ahead window (e.g., a few hundred milliseconds or seconds of future context). As shown by Pang et al. and Li et al., this allows the model to correct acoustic ambiguities—such as homophones or rare words—shortly after they are spoken, updating the user interface asynchronously.

2. Streaming Deliberation

Because the first pass generates text hypotheses in real time, the second pass can leverage both acoustic and textual context. Mahadeokar et al. introduced a streaming deliberation module that takes the partial text hypotheses from the fast encoder, converts them into text embeddings, and combines them with the acoustic embeddings of the slow encoder. This allows the second pass to act as an implicit language model, correcting grammatical errors based on what the first pass thought it heard.

Causal Pass ───► Partial Text Hypotheses ───┐
                                            ▼
Non-Causal Pass ───► Acoustic Embeddings ───┴─► Shared Joiner ───► Refined Output

3. Component-Specific Optimization

Decoupling the streaming and non-streaming passes allows engineers to optimize each component independently based on its operational constraints. For example, Ding et al. demonstrated that quantization techniques (such as compressing weights to 4-bit precision) can be applied natively across a dual-encoder architecture, allowing the highly sensitive causal encoder and the more robust non-causal encoder to be calibrated separately for on-device deployment.


Expanding the Scope

The utility of cascaded encoders extends beyond standard single-speaker dictation. The architecture has become a vital tool for handling complex, multi-faceted speech tasks where online and offline requirements coexist.

In multi-talker scenarios, such as meeting transcriptions, Subramanian et al. showed that a cascaded architecture can run a causal first pass to provide immediate live captions for attendees, while a non-causal second pass incorporates speaker separation and unconstrained future context to generate a highly accurate offline meeting summary.

Furthermore, this paradigm is being used to adapt large, inherently non-streaming foundation models for real-time applications. Zhou et al. adapted OpenAI's Whisper model into a streaming-friendly system by prepending a fast Connectionist Temporal Classification (CTC) streaming pass to generate real-time hypotheses, while reserving Whisper's heavy attention-based decoder to rescore and finalize the text blocks asynchronously. Similarly, Kumar et al. utilized a monotonic chunk-based attention mechanism for low-latency decoding alongside a full-attention decoder for second-pass rescoring, proving the architecture's efficacy even when adapting models to low-resource languages.

By separating the immediate need for acoustic timing from the global need for linguistic context, cascaded encoders successfully eliminate the traditional trade-off between speed and accuracy in speech recognition.

Key Foundational Work

Two-Pass End-to-End Speech Recognition (2019) — contributed 2 concept elements to this direction.
A review of on-device fully neural end-to-end automatic speech recognition algorithms (2020) — contributed 2 concept elements to this direction.
High-Accuracy and Low-Latency Speech Recognition with Two-Head Contextual Layer Trajectory LSTM Model (2020) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_cascaded_encoder_architecture_74f7fada,
  title = {cascaded encoder architecture},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/cascaded-encoder-architecture/rdr_74f7fada49994d38b0aa4e872905e471},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_cascaded_encoder_architecture_74f7fada_v1,
  title = {cascaded encoder architecture},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/cascaded-encoder-architecture/rdr_74f7fada49994d38b0aa4e872905e471/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake