Latent State Expansion for Sequence Labeling

28 concepts21 papers1 proposal

In traditional sequence modeling, tasks like part-of-speech tagging, named entity recognition, or token classification map an input sequence directly to a sequence of discrete, observable labels. Standard architectures—such as linear-chain Conditional Random Fields (CRFs) or Hidden Markov Models (HMMs)—typically rely on a strict one-to-one mapping between their internal hidden states and these target labels.

However, treating target labels as monolithic, atomic entities introduces a severe representational bottleneck. As noted by Yu et al. (2008), flat label spaces fail to capture the predictable internal sub-structures and temporal phases that exist within a single complex tag (such as the distinct stages of a multi-word entity or a physical action). Furthermore, forcing a model to rely purely on first-order transitions between visible labels severely limits its ability to capture long-range context or non-linear feature interactions without causing an exponential explosion of the state space.

To overcome these limitations, a powerful paradigm has emerged: latent state expansion. By decoupling the internal state space from the visible label space—either by splitting labels into hidden sub-states or by factorizing the hidden layer into parallel, interacting variables—models can capture rich, high-dimensional sequence dynamics while remaining computationally tractable.

Latent State Refinement and Sub-structure Modeling

The simplest way to expand a state space is to assign multiple latent hidden states to each observable label. Instead of forcing the model to transition directly between coarse categories, the model transitions through a finer-grained, unobserved topology before mapping back to the target labels.

Standard Sequence Model:
[Input] ---> [ Monolithic Label ] ---> [ Monolithic Label ]

Latent State Expansion:
             /-- Latent State A1 --\        /-- Latent State B1 --\
[Input] ---> --- Latent State A2 --- --->  --- Latent State B2 --- ---> [Output Labels]
             \-- Latent State A3 --/        \-- Latent State B3 --/

As demonstrated by Thai et al. (2017) and Sun et al. (2008), this approach allows sequence labelers to capture fine-grained dependencies and sub-structures within sequences without altering the underlying label schema. For instance, in shallow parsing or named entity recognition, an entity label can be broken down internally into hidden sub-phases that track the beginning, middle, or end of a phrase.

Rush et al. (2013) formalize this mechanism through Refinement HMMs (R-HMMs), showing that associating each observable label with multiple hidden sub-states allows the model to capture complex dependencies that far exceed standard first-order Markov assumptions. This intuition remains highly relevant even in modern deep learning architectures. For example, the authors of ExLM (2025) apply a similar concept to masked language modeling by expanding a single masked token position into multiple latent representations. This expansion provides a broader semantic space capable of representing the inherent multimodality and ambiguity of missing context.

Factored State Representations

While splitting labels into sub-states works well for capturing internal sequential phases, some tasks require tracking multiple independent features or parallel streams of information simultaneously (e.g., tracking both part-of-speech tags and noun-phrase chunks). If we attempt to capture these combinations by creating a single joint state for every possible pair, the state space suffers a catastrophic combinatorial explosion.

The solution is factorization: decomposing the hidden state at each time step into multiple, distributed variables that run in parallel but interact locally.

  • Dynamic and Factorial CRFs: Sutton et al. (2004) introduced Dynamic Conditional Random Fields (DCRFs), which decompose the hidden state into multiple interacting variables. This allows the model to represent parallel sequences of information—such as different levels of linguistic annotation—within a single unified framework. Similarly, Lian et al. (2009) utilized Factorial CRFs to model concurrent activities, capturing dependencies both within a single timeline and across different layers at the same time step.
  • Factorial HMMs: On the generative side, Duh (2005) demonstrated that Factorial HMMs can jointly model distinct but related linguistic features as parallel processes that interact over time, drastically reducing error propagation compared to serial pipelines. Miller et al. (2011) leveraged this architecture to track independent discourse features and entity operations across a sequence without triggering a state-space explosion.
  • Multi-Layer Hidden Dynamics: Modern extensions, such as the Factored Latent-Dynamic CRFs developed by Neogi et al. (2022), use multiple hidden layers to model different facets of a single label sequence (such as distinct physical dimensions of a movement). This allows the system to capture fine-grained internal phases of an activity without requiring a single, unmanageably high-dimensional state space. This compositionality is also echoed by Yu et al. (2015), who composed word-level state models into a factorial structure to capture complex temporal dynamics in video descriptions.

Why It Works: Feature Expansion and Non-Linearity

Latent state expansion is not merely a trick to add more parameters; it fundamentally alters the geometric and algebraic capacity of the model.

Resolving Linear Inseparability

In a standard linear sequence model, the decision boundaries are constrained by the visible label space. If two different contexts lead to the same visible label but require different future transitions, a standard model experiences severe ambiguity.

As argued by Sun et al. (2009), latent variables function as feature expanders. By mapping the input data into a higher-dimensional hidden space, they allow the model to resolve contextual ambiguities that are linearly inseparable in the original label space. This high-dimensional mapping acts as a representational bridge, allowing the model to encode long-range context and complex structural dependencies (Sun et al., 2009) without the exponential computational cost of high-order conditional random fields.

Learning Distributed, Non-Linear Representations

By inserting a layer of hidden units between raw input features and output labels, models can learn distributed representations of the underlying data.

van der Maaten et al. (2011) and Kim et al. (2015) demonstrated this by incorporating layers of binary stochastic hidden units into CRFs (HUCRFs). This architecture enables the model to capture complex, non-linear feature interactions that standard linear sequence models cannot represent.

Discovering Unobserved Dynamics Automatically

Finally, this framework eliminates the need for manual, painstaking feature engineering. By allowing the hidden states to adapt dynamically during training, models like Latent-Dynamic CRFs (Cutugno et al., 2012) can automatically discover unobserved sequential dynamics directly from data. The model learns exactly how many sub-phases an action or token possesses and how they transition, optimizing the internal state representation purely to maximize the probability of the global sequence label.

Key Foundational Work

Speech Recognition with Dynamic Bayesian Networks (1998) — contributed 1 concept elements to this direction.
Bayesian Information Extraction Network (2003) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_internal_label_dynamics_cf192380,
  title = {internal label dynamics},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/internal-label-dynamics/rdr_cf192380da6c40d49188097158128f84},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_internal_label_dynamics_cf192380_v1,
  title = {internal label dynamics},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/internal-label-dynamics/rdr_cf192380da6c40d49188097158128f84/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake