Global Normalization to Overcome Label Bias

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 256 papers68 conceptsfrozen 2026-10-04

In sequence modeling, we often face a choice between making decisions step-by-step or evaluating the entire sequence at once. Most modern autoregressive models, from traditional Maximum Entropy Markov Models (MEMMs) to standard Transformers, opt for the former: they are locally normalized. At every step, the model must distribute a fixed "probability budget" (summing to one) among all possible next tokens.

While computationally efficient, this local constraint introduces a fundamental structural flaw known as the label bias problem.

The Probability Trap

The intuition behind label bias is simple: if a model is forced to reach a decision at every step, it can become "trapped" by the topology of the state space. Mark Johnson in Joint and Conditional Estimation of Tagging and Parsing Models explains that when a state has only one or very few outgoing transitions, the requirement that those transitions sum to one forces the model to follow that path regardless of the actual input evidence.

Imagine a part-of-speech tagger in a state that only allows a "Verb" to follow. Even if the next word in the sentence is clearly a "Noun," a locally normalized model must assign a probability of 1.0 to the "Verb" transition because it has no other way to spend its probability budget at that specific step. As Christopher D Manning and Dan Klein 0001 demonstrate in Conditional Structure versus Conditional Estimation in NLP Models, this creates a systemic bias toward states with low branching factors, effectively "explaining away" the observations.

Observation Bias and Contextual Neglect

A related phenomenon is observation bias. In locally normalized models, a very strong local observation (like a word that is almost always a specific tag) can effectively override the influence of the preceding state. Manning and Klein argue that this prevents the model from using sequence context to resolve surrounding ambiguities. Because the model is focused on satisfying the local normalization constraint for the current token, it "forgets" to weigh how that choice fits into the global sequence.

This lack of global awareness is particularly damaging in tasks like Neural Machine Translation (NMT). David Chiang and Kenton Murray suggest in Correcting Length Bias in Neural Machine Translation that the tendency of NMT models to produce overly short sentences is a specific manifestation of label bias. The end-of-sentence token creates a "zero-entropy suffix" that the model cannot easily correct once it begins to terminate the sequence.

Global Normalization as the Solution

To fix these issues, researchers move from local normalization to global normalization. Instead of requiring each step to sum to one, a globally normalized model (like a Conditional Random Field) assigns a "score" to the entire sequence and normalizes only at the very end across all possible paths.

Charles Sutton and Andrew McCallum provide a comprehensive foundation for this in An Introduction to Conditional Random Fields. By abandoning local normalization, the model gains the ability to "look ahead." A path that looks promising early on but leads to a dead end can be penalized globally, something a local model cannot do because it has already "spent" its probability on the early steps.

This transition is not just theoretical; it leads to significant empirical gains:

  • Transition-Based Systems: Daniel Andor and colleagues show in Globally Normalized Transition-Based Neural Networks that simple feed-forward networks can outperform complex LSTMs if they are globally normalized, as it allows them to overcome the "label bias" that traps incremental parsers.
  • Sequence Labeling: Kai Hu and Zhijian Ou propose Neural CRF transducers to combine the long-range dependencies of RNNs with the global normalization of CRFs, preventing the model from making uncorrectable early mistakes.
  • Energy-Based Models: Recent work by Anton Bakhtin and Marc'Aurelio Ranzato in Residual Energy-Based Models for Text uses un-normalized energy functions to "correct" the outputs of standard language models, providing a global critic that evaluates sequence coherence.

Training for the Global View

The challenge of global normalization is computational: summing over all possible sequences is often intractable. Researchers have developed several strategies to bridge this gap:

  1. Beam-Search Optimization: Sam Wiseman and Alexander M Rush in Sequence-to-Sequence Learning as Beam-Search Optimization integrate the search algorithm into the training loop, forcing the model to learn how to handle the errors it makes during decoding.
  2. Differentiable Decoding: Ronan Collobert and Awni Hannun developed a fully differentiable beam search decoder, allowing gradients to flow through the global inference process.
  3. Hybrid Normalization: Models like HYPRO, introduced by Siqiao Xue and Hongyuan Mei in HYPRO: A Hybridly Normalized Probabilistic Model for Long-Horizon Prediction of Event Sequences, use a local model to propose sequences and a global energy function to refine them, balancing efficiency with global consistency.

By shifting the normalization from the local step to the global sequence, these models escape the "greedy" traps of label bias, enabling more coherent and contextually aware predictions across NLP and speech tasks.

Papers in this version

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 41 papers
  2. Version 1 · 2026-10-04 · 56 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_label_bias_bb016278,
  title = {label bias},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/label-bias/rdr_bb016278005746979b4d0a6b77742058},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_global_normalization_to_overcome_label_b_bb016278_v1,
  title = {Global Normalization to Overcome Label Bias},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/global-normalization-to-overcome-label-bias/rdr_bb016278005746979b4d0a6b77742058/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}