When building models for sequential tasks—such as speech recognition, text generation, or part-of-speech tagging—our standard tool is local normalization. At each step, the model computes a score for every possible next action and passes these scores through a softmax function to produce a valid probability distribution.
While computationally convenient, this step-by-step approach suffers from a fundamental flaw known as the label bias problem. By forcing the model to distribute a fixed probability budget at every single transition, local normalization inadvertently blinds the model to the global context of the sequence.
Global sequence normalization offers an elegant alternative. Instead of making independent, localized decisions, globally normalized models evaluate the entire sequence holistically, allowing them to recover from early mistakes and capture complex structural dependencies.
The Mechanics of Label Bias
To understand why local normalization fails, consider a model navigating a state space where it must choose a sequence of labels. At any given state, a locally normalized model must ensure that the probabilities of all outgoing transitions sum to exactly $1.0$.
This requirement introduces a severe structural distortion: states with low out-degree (fewer transitions) are inherently favored over states with high out-degree.
If a suboptimal path leads to a state with only one valid outgoing transition, a locally normalized model is forced to assign a transition probability of $1.0$ to that step, regardless of how poor the path is globally. As Sutton et al. observe, this means transitions from states with low out-degree are unfairly favored, preventing evidence from later parts of the sequence from effectively propagating backward to correct early mistakes.
Goyal frames this issue through the lens of a search beam. In a locally normalized system, the model is forced to spend its probability budget at every step. If a model enters a suboptimal branch of the search space, it cannot easily "zero out" the scores of that branch's successors. As a result, poor candidates continue to occupy valuable space in the decoding beam.
Similarly, Schmidt et al. note that local normalization forces a preference for states simply because they have limited transition options, rather than because they form a coherent global trajectory.
The Global Alternative: Path-Level Potentials
Global sequence normalization abandons the requirement of step-wise probability distributions. Instead, the model assigns unnormalized scalar scores—often called potentials or energies—to individual transitions or states. To find the probability of a complete sequence, the model sums these unnormalized scores along the entire path and divides the total by a global sequence-level normalization term, known as the partition function ($Z$).
$$P(Y|X) = \frac{\exp(\sum_{t} \text{score}(y_t, y_{t-1}, X))}{Z(X)}$$
Where $Z(X)$ is the sum of the exponentiated scores of all possible valid sequences for the input $X$.
Locally Normalized (Softmax at each step):
State A ───► [Softmax] ───► State B ───► [Softmax] ───► State C (Must sum to 1 at each step)
Globally Normalized (Energy accumulated across the path):
State A ───► Score: 2.4 ───► State B ───► Score: -0.8 ───► State C ───► Total Path Score: 1.6
│
Normalized globally over
all possible full paths (Z)
By computing a single partition function over the entire sequence, the model avoids committing to high-probability local paths that turn out to be suboptimal later on. As Andor et al. explain, this mechanism ensures that early decisions are penalized if they lead to globally improbable outcomes.
This formulation fundamentally alters the model's expressive capacity. Andor et al. demonstrate that globally normalized models possess a higher degree of expressivity than locally normalized ones, because no finite amount of local lookahead features can fully resolve the structural limitations imposed by local decision-making.
Why It Works: Error Recovery and Parameter Interdependency
The primary intuitive advantage of global normalization is its capacity for error recovery. In a directed, locally normalized model, if an early token or action is chosen incorrectly, that error propagates forward, frequently causing the model to derail (a phenomenon closely linked to exposure bias).
In contrast, a globally normalized model evaluates the complete trajectory. As Collins demonstrated with the Structured Perceptron, scoring the entire output sequence allows a system to recover from early sequence errors that would otherwise propagate catastrophically.
If a later stage in the sequence exhibits high confidence for a specific path, that downstream confidence can retroactively validate or invalidate earlier decisions. For instance, Raiman et al. leverage this property in extractive question answering, showing that global normalization across a beam search allows a reader to revise early sentence-selection errors if a later word-selection stage yields a highly confident match.
Furthermore, modern deep learning architectures rely heavily on shared dense parameters. When applied to neural networks, Zhou et al. point out that a global softmax function over entire action sequences is inherently more effective than summing local probabilities. Because the parameters are tightly coupled across time steps, a global objective ensures that the gradient updates reflect the global context and the complex interdependencies of those shared parameters.
Applications Across Domains
The transition from local to global normalization has driven performance breakthroughs across a wide variety of structured prediction tasks:
- Morphological Analysis and Sequence Labeling: In tasks like Japanese morphological analysis—where word boundaries are not explicitly marked—Kudo et al. showed that global normalization prevents models from systematically favoring shorter sequences or states with fewer transitions. More recently, the authors of Neural-Hidden-CRF combined global normalization with pretrained language models to build sequence labelers that are robust to noisy, weakly-supervised data.
- Transition-Based Parsing: Transition-based parsers make sequential decisions (such as Shift or Reduce) to construct a syntactic tree. Traditional greedy parsers suffer heavily from label bias. By applying global structured training objectives, both Zhou et al. and Weiss et al. demonstrated that reweighting local transition probabilities based on full-path scores leads to significantly more robust parsing structures.
- Speech Recognition (ASR): Real-time streaming speech recognition models are highly susceptible to label bias because they must make decisions on the fly. Variani et al. showed that introducing sequence-level global normalization dramatically reduces the performance gap between real-time streaming models and heavy, offline non-streaming models. Additionally, Chorowski et al. found that switching to global normalization relaxes strict local constraints, producing balanced probability distributions that integrate more naturally with external language models during decoding.
- Joint Entity and Relation Extraction: When predicting how entities in a sentence relate to one another, local classifiers often make logically inconsistent choices. Adel et al. solved this by treating the entity-relation-entity triplet as a single sequence, using global normalization to capture mutual dependencies and eliminate contradictory predictions.
The Nuances of Global Modeling
If global normalization is theoretically superior, why isn't it used everywhere? The primary challenge is computational. Calculating the partition function $Z$ requires summing over an exponentially large space of possible sequences. While exact dynamic programming (like the Forward algorithm in CRFs) works for simple linear chains, it becomes intractable for complex, long-range dependencies.
To circumvent this, researchers often combine local and global perspectives. For example, Li et al. introduce Residual Energy-Based Models, which use a standard autoregressive model to generate a set of candidate sequences, and then apply a global sequence-level energy score to re-rank those candidates. Similarly, Liu et al. utilize modern Transformer backbones to score entire sequences globally, bypassing local step-wise decompositions entirely during evaluation.
Interestingly, global normalization is not always a guaranteed win for every aspect of text generation. Gareev et al. discovered that the local "distortion" introduced by step-wise decoding methods (like top-$p$ or nucleus sampling) actually serves a practical purpose: it prevents models from falling into repetitive, overly brief, or incoherent loops, suggesting that the mathematical purity of global normalization requires careful calibration when applied to open-ended generation.
Key Foundational Work
Applying Conditional Random Fields to Japanese Morphological Analysis (2004) — contributed 1 concept elements to this direction.
Discriminative Training Methods for Hidden Markov Models: Theory and Experiments with Perceptron Algorithms (2002) — contributed 1 concept elements to this direction.