The Mismatch Between Token Training and Sequence Evaluation

193 concepts152 papers

Autoregressive language models are almost universally trained using Maximum Likelihood Estimation (MLE) via cross-entropy loss. At every training step, the model looks at a ground-truth prefix and attempts to predict the single next token.

However, when we deploy these models for real-world tasks—such as machine translation, abstractive summarization, or dialogue generation—we do not evaluate them on their next-token classification accuracy. Instead, we judge them on the quality of the entire generated sequence using holistic, often non-differentiable metrics like BLEU, ROUGE, CIDEr, or human preference scores.

This creates a fundamental objective-metric misalignment. As observed by Michael Auli et al. in Decoder Integration and Expected BLEU Training for Recurrent Neural Network Language Models and Lijun Wu et al. in A Study of Reinforcement Learning for Neural Machine Translation, maximizing the local probability of individual tokens does not guarantee the optimization of global sequence quality.

Why Local Maximization Fails Globally

To understand why this misalignment degrades performance, we must look at the mathematical and structural differences between token-level training and sequence-level inference.

1. The Granularity Mismatch

Cross-entropy loss treats every token prediction as an independent event conditioned on the past. If a model is generating a summary, a sequence-level metric or a human judge evaluates the text based on overall coherence, factual accuracy, and lack of redundancy.

As Shentao Yang et al. explain in Preference-grounded Token-level Guidance for Language Model Fine-tuning, there is a severe credit assignment problem: MLE cannot easily map a holistic, sequence-level judgment back to the specific tokens that caused the generation to succeed or fail. A model can achieve high token-level predictive accuracy on training data while producing a complete sequence that lacks global utility or conversational flow.

2. The Search Disconnect

During training, a model benefits from teacher forcing—it always receives the ground-truth history as context. During inference, however, the model must generate text autoregressively, feeding its own potentially flawed predictions back into itself as context. To navigate this generation space, systems rely on decoding algorithms like greedy search or beam search.

Jiatao Gu et al. note in Neural Machine Translation with Gumbel-Greedy Decoding that standard training objectives do not tune model representations to work effectively with these test-time search strategies.

This disconnect leads to a striking paradox: finding the mathematically most probable sequence according to an MLE-trained model often yields degenerate, repetitive, or prematurely terminated outputs. As shown by Rémi Leblond et al. in Machine Translation Decoding beyond Beam Search, the highest-probability paths under pure likelihood estimation frequently correspond to low-quality text. In fact, empirical work by Jianfei He et al. in Empirical Analysis of Beam Search Curse and Search Errors with Model Errors in Neural Machine Translation reveals that ground-truth human references are regularly assigned lower probability by the model than its own flawed generations.

Bridging the Gap with Sequence-Level Objectives

To resolve this misalignment, researchers have developed frameworks that shift the optimization landscape from local token probabilities to global sequence metrics.

Standard MLE Training:
[Token 1] -> Predict [Token 2] -> Predict [Token 3] (Optimizes local cross-entropy)

Sequence-Level Training:
[Token 1 -> Token 2 -> Token 3] -> Evaluate Full Sequence -> Update Model (Optimizes global metric)

Direct Metric Optimization

Instead of hoping that token accuracy correlates with downstream metrics, some methods bake the evaluation metrics directly into the loss function. Maosong Sun et al. introduced this via Minimum Risk Training for Neural Machine Translation, an approach that minimizes the expected loss (or "risk") across a sampled pool of complete candidate translations. By optimizing the expectation of a metric like BLEU over the model's generation distribution, the network learns to prioritize global semantic preservation over exact token matching.

Similarly, Sam Wiseman et al. developed Sequence-to-Sequence Learning as Beam-Search Optimization, which integrates the beam-search decoding algorithm directly into the training loop, forcing the model to optimize for sequences that remain highly scored throughout the entire search process.

Reinforcement Learning and Self-Criticism

Because human evaluation and discrete metrics are non-differentiable, reinforcement learning (RL) is widely used to bridge the objective gap. In image captioning, Steven J Rennie et al. proposed Self-Critical Sequence Training for Image Captioning. Instead of relying on an external critic network, the model uses its own inference-time greedy prediction as a baseline to normalize rewards. If a sampled sequence scores higher than its greedy baseline on a metric like CIDEr, its token probabilities are reinforced; if it scores lower, they are penalized. This directly aligns the training signal with the final evaluation criteria.

Preference Alignment

When explicit automated metrics fail to capture human nuances, researchers scale sequence-level optimization to human preference data. Nisan Stiennon et al. demonstrated in Learning to summarize with human feedback that Reinforcement Learning from Human Feedback (RLHF) successfully bypasses the next-token prediction bottleneck by training models to maximize holistic human preference scores.

To avoid the training instability of online RL, Dhananjay Ram et al. introduced Sequence-level Large Language Model Training with Contrastive Preference Optimization. This technique uses contrastive objectives on full sequences to ensure that the model directly optimizes for sequence-level properties required for instruction-following, rather than focusing on local transition probabilities.

The Imitation Gap in Distillation

The objective-metric mismatch also compromises how knowledge is transferred between models. Standard knowledge distillation typically forces a smaller student model to match a larger teacher model's token-level probability distribution step-by-step.

However, Chen Jia demonstrates in Adversarial Moment-Matching Distillation of Large Language Models that this per-token matching creates an "imitation gap." Matching local token distributions does not guarantee that the student model will maintain long-term sequence coherence when generating text independently over extended horizons. By framing distillation as an adversarial moment-matching problem at the sequence level, the student learns to mimic the global generation dynamics and long-term behavior of the teacher, bypassing the limitations of localized token imitation.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 166 papers
  2. Version 1 · 2026-10-04 · 178 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_objective_metric_misalignment_objective__af727cd4,
  title = {objective-metric misalignment (objective-metric mismatch)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/objective-metric-misalignment-objective-metric-mismatch/rdr_af727cd4cb0d40fa9ef8ad04b050b5fa},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 2, frozen text):

@misc{lacuna_objective_metric_misalignment_objective__af727cd4_v2,
  title = {objective-metric misalignment (objective-metric mismatch)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/objective-metric-misalignment-objective-metric-mismatch/rdr_af727cd4cb0d40fa9ef8ad04b050b5fa/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-21},
}

See something wrong? Report a mistake