Where the Bidirectionality Bonus Comes From: A Controlled Test with Linear Recurrent Rescorers

Speech recognizers usually produce a short list of candidate transcripts and then ask a separately trained language model to pick the best one. That second step is called rescoring. A long-standing puzzle is that the language model with the best perplexity is often not the best rescorer. Perplexity measures how well a model predicts every next word on average. A rescorer is better judged by how often it ranks the right transcript first. A sharper version of this puzzle appeared recently. When Transformer rescorers are trained directly to reduce word errors, the bidirectional model (BERT, which reads the whole sentence) beats the left-to-right model (GPT-2) by a wide margin. When the same two models are scored by likelihood, they perform about the same. If that "bidirectionality bonus" is real, it says something basic: future context matters for telling transcripts apart, and it does little for averaged next-word prediction. The original comparison, however, also changed pretraining objective, data and model family at the same time. So it cannot say whether the bonus comes from reading in both directions at all.

We propose to settle this with a recurrent model whose reading direction is the only thing that changes. A linear recurrent network is a recurrent model whose state update is linear, so it can be trained in parallel and run in constant memory. It can be built in a causal form and a bidirectional form with matched parameters, pretraining tokens and tuning budget. We cross reading direction with the training objective: likelihood, minimum word error rate, and large-margin ranking. We then measure the gap between perplexity and word error rate inside each cell of that design. The result is a clean answer to where the bonus comes from. As a side product, it yields a rescorer that uses both directions in a single pass with no quadratic attention cost.

Core Claim

Our hypothesis is that the bidirectionality bonus is an interaction effect. Under likelihood scoring, a bidirectional linear recurrent model and a causal one of equal size should rank speech hypotheses about equally well. Under discriminative training, the bidirectional model should pull clearly ahead. The extra gain should be concentrated on errors that the words after them disambiguate. In speech these are mostly errors early in the utterance, where a left-to-right model has seen little context. We further predict that discriminative training loosens the link between validation perplexity and word error rate more for the bidirectional model than for the causal one. If so, selecting checkpoints by perplexity would systematically discard the best bidirectional rescorers. Each part of the claim can fail separately. A large direction gap under likelihood, or no gap under discriminative training, would each falsify the interaction story.

Positioning Against Prior Work

Two kinds of work motivate the proposal by documenting the perplexity-performance gap. Classic studies such as Neural network language models for conversational speech recognition and Discriminative language modeling using simulated ASR errors showed that lowering cross-entropy does not reliably lower recognition errors. Later work traded perplexity for task accuracy on purpose, all with causal recurrent models. Large Margin Neural Language Model used a hinge loss that forces the reference to outscore each competitor by a margin; it got worse perplexity but cut Wall Street Journal error from 7.69% to 5.48%. Improving Language Modeling with an Adversarial Critic used policy-gradient training against a critic network; it slightly raised perplexity while lowering Mandarin character error. Correction Focused Language Model Training up-weighted tokens predicted to be error-prone. None of these touched bidirectional context, and we import their objectives rather than their architectures.

The direct motivation is Discriminative Speech Recognition Rescoring with Pre-trained Language Models. It pooled BERT and GPT-2 representations into a single score trained with minimum word error rate. It reported that the bidirectional model gains much more from that training, reaching 4.31% on LibriSpeech test-clean from a 5.67% Whisper-tiny first pass. Its BERT/GPT-2 contrast confounds direction with pretraining recipe, which is exactly the confound our design removes. ASR Rescoring and Confidence Estimation with ELECTRA is a second discriminative bidirectional rescorer: it tags each word as likely wrong. It reached 3.98% on test-clean, but it is also Transformer-only and struggles with deleted words.

The method comes from the linear bidirectional recurrent literature. LION unifies bidirectional linear attention with bidirectional recurrent networks. It adds a correction term so the current token is not counted twice when forward and backward states merge. BLUR fuses forward and backward linear recurrent states through a nonlinear gate. DeltaFlow found that causal gated delta networks collapse to repetitive text in diffusion generation and bidirectional scans fix it. All three were trained only with likelihood, masked-token, or regression losses and never used for ranking hypotheses. DeltaFlow also complicates our story. There, bidirectionality improves perplexity, whereas the gap literature says task gains come at the cost of perplexity. Our design measures both quantities in every cell, so it can show whether these are two faces of one effect or genuinely different regimes. Finally, Bidirectional Recurrent Neural Networks as Generative Models supplies the key trick for the likelihood arm. It predicts each token from the forward state just before it and the backward state just after it, so the model cannot copy its own input.

Research Program

We build one backbone, a LION-style linear recurrent network of fixed depth and width, in two forms. The causal form runs a single left-to-right scan. The bidirectional form runs a forward and a backward scan with each scan's state width halved, so total parameters match. Both forms are pretrained on the same text corpus for the same number of tokens. The causal form uses next-token prediction. The bidirectional form uses the Berglund-style objective. This gives the bidirectional model a single-pass pseudo-likelihood, which avoids the $L$ masked passes that make BERT likelihood rescoring slow (6,115 ms per utterance versus 33 ms for pooled scoring in the Shivakumar et al. paper). For a hypothesis $x = (x_1, \dots, x_T)$ the bidirectional likelihood score is

$$ s_{\text{lik}}(x) = \sum_{t=1}^{T} \log p\big(x_t \mid h_f(t-1),\, h_b(t+1)\big), $$

where $h_f(t-1)$ is the forward state after reading $x_1,\dots,x_{t-1}$ and $h_b(t+1)$ is the backward state after reading $x_T,\dots,x_{t+1}$. The causal score is the ordinary sum of next-token log-probabilities.

From each pretrained model we fine-tune three rescorers on first-pass candidate lists. The likelihood rescorer keeps the pretraining loss on reference transcripts. The second rescorer pools the final forward and backward states into a scalar score $s_\theta(x)$. It is trained with minimum word error rate plus a small cross-entropy regularizer, following Shivakumar et al. The loss is

$$ \mathcal{L}_{\text{MWER}} = \sum_{i} P_\theta(x_i \mid \text{list})\,\big(E(x_i) - \bar{E}\big), $$

where $P_\theta$ is a softmax over the scores of the $N$ candidates in a list, $E(x_i)$ is the word error count of candidate $i$, and $\bar{E}$ is the list's mean error. The third rescorer uses the large-margin hinge objective of Huang et al. on the same pooled score. These six cells (two directions by three objectives) share hyperparameter search budgets.

For every checkpoint in every cell we record held-out perplexity, using pseudo-perplexity for the bidirectional model, and development word error rate. This lets us compute, per cell, the rank correlation between the two and the error rate lost by choosing checkpoints by perplexity. We then run three analyses aimed at mechanism. The first stratifies the word error gain of bidirectional over causal by the normalized position of the word where the chosen and the correct hypotheses differ. The second truncates the backward scan at test time to $k$ tokens of right context and traces error rate as $k$ grows. The third breaks errors into substitutions, insertions and deletions, and plots error against hypothesis length, which tests whether pooled bidirectional scores remove the length bias that Huang et al. had to correct with margins.

Baselines

The naive reference point is the first-pass recognizer with no rescoring. For LibriSpeech we regenerate Whisper-tiny candidate lists, for which Shivakumar et al. report 5.67% WER on test-clean. Every rescorer is evaluated on exactly these lists with the same interpolation-weight search on dev-clean. Beating it is necessary but uninteresting.

The strongest existing method is RescoreBERT trained with minimum word error rate plus cross-entropy, as in Discriminative Speech Recognition Rescoring with Pre-trained Language Models. Its reported figure is 4.31% WER on LibriSpeech test-clean from the Whisper-tiny first pass. It is the strongest competitor because it is the system that exhibited the bonus, and it already has low latency. We will both quote the published figure and rerun it on our regenerated lists, matched as closely as possible in parameter count and fine-tuning budget. Our pretraining corpus will inevitably differ from BERT's, so the rerun is a reference point, not the controlled comparison. A linear bidirectional rescorer within 0.1 absolute WER of this rerun, at equal parameters, would count as matching it. Doing better would be a bonus, not a requirement. ELECTRA rescoring at 3.98% test-clean (ASR Rescoring and Confidence Estimation with ELECTRA) used a different first pass. We therefore cite it as context only and do not treat it as a directly comparable number.

The controlled baseline that matters most for the scientific claim is the causal linear recurrent rescorer trained with the same three objectives. The bonus is defined relative to this baseline. It is matched in parameters, pretraining tokens and trial budget, and a second variant with doubled depth makes it compute-matched to the two-scan model. The bidirectional discriminative model beats it if its test WER is lower across three seeds by more than the seed spread. The claim further requires that this gap be significantly larger than the corresponding gap between the two likelihood rescorers.

Two ablations isolate the key ingredient, which is per-position fusion of past and future context inside one model. One ablation trains separate forward and backward causal rescorers and sums their scores. The other drops LION's double-counting correction from the bidirectional model. For the large-margin objective we also include the causal recurrent setup of Large Margin Neural Language Model on Wall Street Journal. There it reported 7.69% falling to 5.48% WER, and we ask whether bidirectionality adds to a gain already that large.

Why not just…?

Why not just use RescoreBERT, which is already fast? For the practical goal of a good short-utterance rescorer, RescoreBERT is a real threat, and we do not claim to beat its 33 ms latency on short utterances. Our main deliverable is the causal-versus-bidirectional answer, which RescoreBERT cannot give because its causal counterpart differs in every other respect. A secondary claim is that a linear rescorer's cost grows linearly with hypothesis length. We will test it by timing both systems on long-form, concatenated-utterance candidate lists. If the timing advantage does not appear at the lengths we can test, we will report that.

Why not just add a right-to-left language model to the left-to-right one? This is the cheapest way to give a rescorer future context, and it might capture most of the bonus. That is why the summed forward-plus-backward causal pair is an explicit ablation, trained with the same objectives and parameter total. If the pair matches the jointly fused model under minimum word error training, the conclusion is that future context matters but fusing it per position does not. That would still answer the main question, and it would also be the simpler system to recommend.

Why not just train a bigger causal model to lower perplexity? The gap literature argues against it. Huang et al. and Zhang et al. both got better recognition with worse perplexity. Perplexed by Perplexity found perplexity misleading for data selection, and Investigating the relationship between language model perplexity and IR precision-recall measures reported diminishing returns from lower perplexity. We test this directly with the doubled-depth causal arm. If that arm closes the gap, the bonus was a capacity effect rather than a direction effect.

Why not just reweight hard tokens in a causal model? Correction Focused Language Model Training gains 5.5% relative this way. The method still depends on an external error predictor and is sensitive to its weighting parameter, and it cannot see words after the error. We treat it as complementary rather than as a rival and leave combining it with our arms to later work, because it does not bear on the direction question.

Decisive Checks

The primary experiment runs on LibriSpeech test-clean and test-other with Whisper-tiny candidate lists. A replication on Wall Street Journal connects to the large-margin results. Each of the six cells and the ablations gets three seeds.

The first decisive check is the direction-by-objective interaction. The bidirectional-minus-causal WER gap should be near zero for likelihood rescorers and clearly larger for both discriminative rescorers. A gap under likelihood would mean bidirectionality helps regardless of objective. No gap under discriminative training would mean the Transformer bonus came from BERT's pretraining rather than its reading direction.

The second check is mechanism. The bidirectional gain should concentrate on early-position differences and grow with backward context length $k$ before leveling off. If the gain is uniform across positions, our explanation for it is wrong even if the interaction holds.

The third check concerns the perplexity-performance gap. In each cell, the rank correlation between checkpoint perplexity and WER should be weakest in the bidirectional discriminative cell. Perplexity-based checkpoint selection there should cost measurable WER relative to selecting checkpoints on development WER.

The fourth check is the forward-plus-backward ensemble. A single fused bidirectional model must beat the summed pair at equal parameters before we claim that joint per-position fusion is the ingredient that matters.