Two Pass Decoding via Lattice Rescoring

15 concepts12 papers1 proposal

When generating text from speech or translating between languages, systems must find the most likely sequence of words out of an astronomically large number of possibilities. If a vocabulary has 50,000 words, a sentence of just ten words yields $50,000^{10}$ potential combinations.

To find the optimal path through this search space, we want to use our most powerful, high-capacity models—such as deep Transformer language models or continuous-space neural networks. However, evaluating these complex models at every single branch of an unconstrained search tree is computationally impossible for real-time applications.

Two-pass decoding solves this dilemma by decoupling the problem into two distinct phases: a high-recall, low-latency search phase, followed by a high-precision evaluation phase.

Decoupling Search from Evaluation

The core intuition behind two-pass decoding is that a lightweight model can easily distinguish between completely implausible word sequences and reasonable candidates, even if it cannot perfectly identify the single best choice.

[Input Signal] 
      │
      ▼
┌───────────────┐
│    Pass 1     │  <── Fast, heavily pruned model
└───────────────┘
      │
      ▼  [Word Lattice / Compact Search Space]
┌───────────────┐
│    Pass 2     │  <── High-capacity model (Transformer, Neural LM)
└───────────────┘
      │
      ▼
[Final Output]

Instead of forcing a heavy model to explore the entire combinatorial tree, a fast, highly optimized model runs an initial pass to prune away 99.9% of the impossible paths. The output of this first pass is typically a word lattice—a directed acyclic graph that compactly represents a small, dense neighborhood of the most plausible hypotheses.

Once the infinite search space is compressed into a narrow lattice, the computational bottleneck disappears. A second, much more sophisticated model can then be deployed to evaluate and re-score only the paths remaining within that lattice.

Pass 1: Efficiently Constraining the Search Space

The primary goal of the first pass is speed and high recall. It must process the input rapidly while ensuring that the correct hypothesis is not accidentally pruned away. To achieve this, researchers use a variety of compression and simplification techniques.

One common approach is to use highly compact, lower-order n-gram language models and structured lexicons. For example, Kawahara et al. designed a system that utilizes a tree-structured lexicon and a bigram language model in the first pass to quickly generate a candidate space, which is only later refined by a cross-word triphone and trigram model. Similarly, Fohr et al. demonstrated that using a compact lexicon and a simple language model to generate an initial word lattice allows large-vocabulary speech recognition systems to maintain real-time performance.

Another powerful mechanism for accelerating the first pass is entropy pruning. Instead of simply dropping rare words, entropy pruning systematically removes transitions from a language model that contribute the least amount of information. Iglesias et al. showed that using an entropy-pruned language model to generate a candidate lattice via exact search reduces processing time by approximately 40% while preserving the structural paths necessary for high-quality translation.

In modern on-device applications, such as mobile speech recognition, this aggressive pruning is vital. Riley et al. and Salimbajevs et al. both demonstrated that running a heavily pruned, low-latency model on-device provides an immediate, responsive hypothesis that can be seamlessly refined during a second pass.

Pass 2: High-Capacity Rescoring

Once the first pass restricts the search space to a lattice, the second pass evaluates the remaining paths using models that would be far too slow to use during an exhaustive search. Because the lattice contains only a fraction of the vocabulary at any given state, the system can afford the high computational cost of continuous representations and deep architectures.

This strategy enabled the introduction of neural networks to sequence transduction tasks long before they were fast enough for direct decoding. Schwenk et al. pioneered this by projecting words into a continuous vector space and using a neural network to rescore translation lattices generated by traditional statistical machine translation systems. By avoiding the need to evaluate the neural network over the entire vocabulary tree at every step, they achieved significant accuracy gains without a prohibitive latency penalty.

As neural architectures evolved, the same two-pass paradigm allowed the integration of state-of-the-art models into traditional pipelines. Lüscher et al. utilized a two-step decoding process where an initial pass generates a search space, allowing an extremely deep, high-capacity Transformer language model to rescore the hypotheses.

Remarkably, the second pass is highly effective at recovering information lost during the initial pruning. Work by Zheng et al. and Riley et al. proved that a second-pass rescoring with a massive, uncompressed language model recovers almost all of the accuracy lost during an aggressive, low-latency first pass. Stallard et al. further refined this by using a detailed backward pass that applies richer acoustic and language models to the initial beam search hypotheses.

Beyond Scoring: Information Extraction and Adaptation

The benefits of two-pass decoding extend beyond balancing model size and search speed. The first pass provides a rough structural blueprint of the input sequence, which can be used to extract contextual metadata before the second pass begins.

For instance, Nouza et al. showed that an initial rapid pass can successfully identify speech segment boundaries, the speaker's language, and the audio channel type. This extracted metadata is then used to condition and optimize the acoustic and language models for the second, more precise transcription pass.

Additionally, this architecture facilitates speaker adaptation. As demonstrated by Lööf et al., an initial transcription pass using a fast Vocal Tract Length Normalization (VTLN) variant provides enough text context to calculate speaker-specific transformations, allowing the second pass to run with highly customized, discriminatively trained models.

By splitting the decoding problem into a fast search pass and a deep scoring pass, two-pass architectures achieve a Pareto-optimal trade-off: they deliver the accuracy of our largest, most complex models at a computational speed close to our fastest, most lightweight baselines.

Key Foundational Work

Recent progress of open-source LVCSR engine julius and Japanese model repository (2004) — contributed 1 concept elements to this direction.
The automatic news transcription system: ANTS, some real time experiments (2004) — contributed 1 concept elements to this direction.
Continuous Space Language Models for Statistical Machine Translation (2006) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_two_pass_decoding_56333be3,
  title = {two-pass decoding},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/two-pass-decoding/rdr_56333be3ea824bc3b1b9b84755aca13b},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_two_pass_decoding_56333be3_v1,
  title = {two-pass decoding},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/two-pass-decoding/rdr_56333be3ea824bc3b1b9b84755aca13b/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake