Traditional Natural Language Processing treats text as a sequence of individual tokens. To identify a person's name or a chemical compound, we typically assign a label to each word—a process known as sequence labeling. However, this word-by-word approach struggles with "nested" structures, where one entity exists inside another (e.g., identifying "IL-2R alpha" as a protein within the larger "IL-2R alpha gene").
To solve this, researchers have shifted toward span-level modeling. Instead of labeling tokens, these models evaluate entire subsequences (spans) of text as single units. This shift allows models to capture the internal semantics of a phrase and naturally handle overlapping or nested entities.
The Exhaustive Search Problem
The most direct way to implement span-level extraction is to look at every possible subsequence in a sentence. In Deep Exhaustive Model for Nested Named Entity Recognition, Mohammad Golam Sohrab and Makoto Miwa demonstrated that by enumerating all spans and classifying them directly, models could outperform traditional sequence labelers on complex biomedical data.
However, this exhaustive approach introduces a significant computational bottleneck. For a sentence of length $n$, there are $O(n^2)$ possible spans. If a model also tries to identify relations between these spans—such as "who works for whom"—the search space explodes to $O(n^4)$. As noted by Wei Xu and colleagues in their work on SpanRel, this combinatorial explosion makes naive span models difficult to scale to long documents.
Pruning the Search Space
To make span-level extraction practical, the research has moved toward "Locate and Label" architectures. Rather than scoring every possible span, these models first identify high-probability "entity-rich" regions and then refine them.
Yongliang Shen and Xinyin Ma proposed a two-stage framework in Locate and Label: A Two-stage Identifier for Nested Named Entity Recognition that borrows from object detection in computer vision. By first proposing candidate spans and then classifying them, they significantly reduce the number of redundant calculations. Similarly, Lefei Wu and Qinghua Zheng introduced boundary regression, treating entity boundaries as continuous values that can be adjusted, much like bounding boxes in an image.
Beyond Entities: Relations and Graphs
The power of span representations extends beyond simple entity recognition. By treating spans as nodes in a graph, models can perform multiple extraction tasks simultaneously.
The DYGIE framework, developed by Yi Luan, David Wadden, and others in A general framework for information extraction using dynamic span graphs, uses this graph-based approach. It builds representations for spans and then dynamically updates them based on the relations and coreference links identified between them. This allows the model to use the fact that two spans are related to better understand what those spans actually are.
Efficiency through Language Modeling
Recent work has also explored reformulating span extraction as a language modeling problem to avoid explicit enumeration. Linyang Li and Tao Gui introduced EntLM in Template-free Prompt Tuning for Few-shot NER, which predicts "pivot words" at entity positions. This approach achieved a staggering 1930x speedup over template-based span search by moving the complexity from the search space into the model's internal vocabulary predictions.
Why It Works
The intuition behind the success of span-level models is twofold:
- Boundary Integrity: By looking at a span as a whole, the model can ensure that the start and end of an entity make sense together, reducing the "fragmented" predictions common in word-by-word tagging.
- Contextual Pooling: Span-level models often use max-pooling or attention over the tokens within a span. This creates a single vector that summarizes the entire phrase, making it easier for the model to compare that phrase against a dictionary or a relation template.
While the computational cost remains a challenge, the transition from token-level to span-level representations has become the standard for high-precision, structured information extraction in complex domains.