When working with sequence models, evaluating the probability or "goodness" of a sequence is a fundamental operation. We use it to rerank speech recognition hypotheses, score translation outputs, evaluate the fitness of protein mutations, and guide reinforcement learning agents.
However, a deep architectural tension exists between context and computational efficiency. Autoregressive models (like GPT) can compute joint probabilities in a single forward pass but are constrained to unidirectional context. Bidirectional models (like BERT) capture rich, all-to-all contextual dependencies but are notoriously expensive to score. To compute the exact pseudo-log-likelihood (PLL) of a sequence of length $N$ in a standard Masked Language Model (MLM), one must mask each token individually and run the model $N$ times. This $O(N)$ computational bottleneck makes bidirectional scoring impractical for real-time applications or high-throughput screening.
To resolve this, a prominent research direction focuses on single-pass sequence scoring. By restructuring attention mechanisms, leveraging energy-based formulations, or utilizing distillation, these techniques compress the computational complexity of sequence evaluation from $O(N)$ to $O(1)$ forward passes.
Architectural Masking for Parallel Evaluation
The most direct way to achieve single-pass scoring is to alter how information flows through the network's attention layers, ensuring that the model can compute multiple conditional probabilities simultaneously without information leakage.
An early foundation for this approach is found in the work of Mathieu Germain et al. on MADE: Masked Autoencoder for Distribution Estimation. By applying specific binary masks to the weight matrices of a standard feed-forward autoencoder, the authors enforced an autoregressive constraint. This allowed the network to compute the joint probability of an entire input vector in a single forward pass, bypassing the need for iterative sampling.
Modern Transformer architectures adapt this concept using specialized attention masks rather than static weight masks. For instance, Kaitao Song et al. introduced Transcormer: Transformer for Sentence Scoring with Sliding Language Modeling. Instead of masking tokens one-by-one across separate inference steps, Transcormer uses a sliding attention mask that allows every token to see its surrounding bidirectional context while preventing it from attending to itself. This enables the model to estimate the probability of every token in a sequence simultaneously in one forward pass.
Standard MLM Scoring (Requires N passes):
Pass 1: [MASK] token2 token3 token4 -> Predict token1
Pass 2: token1 [MASK] token3 token4 -> Predict token2
...
Single-Pass Masked Attention (Requires 1 pass):
Input: token1 token2 token3 token4
| | | |
v v v v
[ Specialized Attention Masking Layer ]
| | | |
v v v v
Output: P(t1|C) P(t2|C) P(t3|C) P(t4|C) (Simultaneous)
This strategy is highly versatile and extends to non-textual domains. In the context of generative diffusion models, Yair Schiff et al. developed the d2 framework in d2: Improved Techniques for Training Reasoning Diffusion Language Models. They designed an attention masking strategy (d2-AnyOrder) that calculates the exact trajectory likelihood of a sampling sequence in a single pass, unlocking efficient reinforcement learning for diffusion language models.
Similarly, for probabilistic autoregressive models, Conor Hassan et al. proposed Efficient Autoregressive Inference for Transformer Probabilistic Models. By decoupling static context encoding from sequential target prediction and implementing causal masking within a dedicated generation buffer, their method evaluates the joint log-density of an entire sequence in a single forward pass, drastically reducing memory and latency.
Energy-Based and Discriminative Formulations
Another way to bypass the $O(N)$ bottleneck is to change the learning objective itself, moving away from predicting tokens out of a massive vocabulary via a softmax layer.
In Pre-Training Transformers as Energy-Based Cloze Models, Kevin B. Clark et al. introduced Electric. Instead of calculating normalized probabilities, Electric treats the cloze task as an Energy-Based Model (EBM). Because the model outputs unnormalized energy scores for all positions at once, the pseudo-log-likelihood of an entire sequence can be aggregated in a single forward pass. Hong Liu 0024 et al. expanded on this in Exploring Energy-based Language Models with Different Architectures and Training Methods for Speech Recognition, demonstrating that EBMs trained with Dynamic Noise Contrastive Estimation can process entire sequences through bidirectional backbones without the computational overhead of iterative token masking.
Discriminative models offer a parallel advantage. Rather than generating text, a discriminator evaluates whether tokens in a sequence are correct or replaced. Hayato Futami et al. leveraged this in ASR Rescoring and Confidence Estimation with ELECTRA. Because the discriminator evaluates all tokens simultaneously, it can leverage full bidirectional context for automatic speech recognition (ASR) hypothesis rescoring in a single pass, running nearly 30 times faster than standard BERT-based rescoring. A similar principle underpins the unified architecture proposed by Erik Nijkamp et al. in SCRIPT: Self-Critic PreTraining of Transformers, which allows a single Transformer to estimate sequence-level pseudo-log-likelihoods in one pass by exploiting its internal discriminative capabilities.
Distillation and Representation Pooling
If an application requires the exact contextual representations of a heavy, multi-pass bidirectional model, researchers can use distillation to compress that iterative scoring behavior into a single-pass architecture.
In Masked Language Model Scoring, Julian Salazar et al. demonstrated that while calculating PLLs via iterative masking is highly accurate for rescoring speech and translation outputs, its latency is prohibitive. To solve this, the authors showed that the results of multiple masking passes can be distilled into a single inference step by training a linear estimator on top of a global context representation.
This concept is highly effective for deployment in streaming, real-time systems:
- Sentence-Level Tokens: In RescoreBERT: Discriminative Speech Recognition Rescoring with BERT, Liyan Xu et al. distilled bidirectional transformer scores into a single classification token (
[CLS]). This allowed the model to predict a sentence-level score in a single pass, bypassing iterative masking entirely while retaining bidirectional context. - Vector Pooling: Similarly, Prashanth Gurunath Shivakumar et al. showed in Discriminative Speech Recognition Rescoring with Pre-trained Language Models that pooling architectures can extract a single representative vector from an encoder and map it directly to a scalar score via a linear layer, slashing rescoring latency by over 99%.
High-Throughput Estimation in Biology and Reinforcement Learning
The need for single-pass likelihood estimation becomes even more acute when evaluating combinatorial spaces, such as predicting the fitness landscapes of proteins or calculating rewards in reinforcement learning.
In protein engineering, a common task is to predict how every possible single-point mutation affects a protein's stability or fitness. Evaluating these variants one-by-one using a Protein Language Model (pLM) is incredibly slow. To bypass this, Zeyuan Wang et al. developed DePLM: Denoising Protein Language Models for Property Optimization. Instead of mutating the sequence and re-running the model for each variant, DePLM calculates the marginal probabilities at the unmutated wild-type sequence. Because the model outputs a probability distribution over all 20 amino acids for every position in a single forward pass, the fitness of all potential single-site substitutions can be read off simultaneously.
Traditional Mutational Scanning (M positions * 20 amino acids = 20*M passes):
Pass 1: Run mutant (Position 1 -> Alanine)
Pass 2: Run mutant (Position 1 -> Cysteine)
...
Single-Pass Marginal Estimation (1 pass):
Input: [ Wild-Type Protein Sequence ]
|
v
[ Protein Language Model ]
|
v
Output: Matrix of size [Position × 20 Amino Acids]
(Contains scores for all possible single-point mutations at once)
Charles W. J. Pugh applied a similar single-pass marginal likelihood approach in From Likelihood to Fitness: Improving Variant Effect Prediction in Protein and Genome Language Models to efficiently estimate variant impacts while filtering out evolutionary noise. To make this process fully differentiable and high-throughput, Yuanxi Yu et al. introduced Venus-MAXWELL: Efficient Learning of Protein-Mutation Stability Landscapes using Protein Language Models. Their framework reformulates zero-shot likelihood comparisons into parallelized operations using Hadamard products and matrix subtractions, enabling the model to map out entire mutation landscapes simultaneously with up to a 1000x speedup.
Furthermore, Cade W Gordon et al. demonstrated in Protein Language Model Fitness is a Matter of Preference that by mathematically compensating for the noise distribution used during an MLM's training phase, one can approximate the true sequence pseudo-log-likelihood in a single forward pass. This provides a real-time confidence metric for model deployment without the linear computational cost of iterative masking.
Finally, this parallelization logic is crucial for scaling up sequence-based reinforcement learning. In ESRL: Efficient Sampling-Based Reinforcement Learning for Sequence Generation, Chenglong Wang et al. decoupled sequence generation from gradient calculation. By utilizing parallelized sequence likelihood estimation to calculate the log-probabilities of generated sequences in a single pass rather than token-by-token, they bypassed memory-intensive step-by-step gradient tracking, reducing memory consumption by up to 47%.
Summary of Core Mechanisms
| Mechanism | Representative Work | How it Achieves $O(1)$ Scoring |
|---|---|---|
| Specialized Attention Masking | Song et al. (Transcormer), Schiff et al. (d2) | Restructures the attention matrix so tokens can be evaluated in parallel without self-influence. |
| Energy-Based / Discriminative Objectives | Clark et al. (Electric), Futami et al. (P-ELECTRA) | Replaces the token-by-token vocabulary softmax with unnormalized energy or correctness scores computed in parallel. |
| Sequence-Level Distillation | Salazar et al., Xu et al. (RescoreBERT) | Compresses multi-pass token probabilities into a single global representation or classification token. |
| Wild-Type Marginal Mapping | Wang et al. (DePLM), Yu et al. (Venus-MAXWELL) | Uses the full vocabulary distribution at unmutated positions to read off the likelihood of all single-point mutations at once. |
Key Foundational Work
Pre-Training Transformers as Energy-Based Cloze Models (2020) — contributed 2 concept elements to this direction.
Masked Language Model Scoring (2020) — contributed 2 concept elements to this direction.
MADE: Masked Autoencoder for Distribution Estimation (2015) — contributed 1 concept elements to this direction.