Token Level Sequence Labeling for Text Span Extraction

242 concepts223 papers5 proposals

When humans read text, we do not just process individual words in isolation; we group them into meaningful chunks. Whether we are identifying a named entity like "New York Stock Exchange," extracting a citation like "Smith et al., 2020," or isolating an argumentative claim in an essay, we are identifying text spans—contiguous sequences of tokens that vary in length and meaning.

If we try to extract these spans by classifying each word independently into a semantic category (such as Location or Citation), we run into a fundamental ambiguity known as the boundary problem. Consider a text that reads "Paris London." A naive token classifier might label both words as Location, but it fails to communicate whether "Paris London" is a single multi-word entity or two separate, adjacent entities.

Token-level sequence labeling schemes solve this problem by transforming structural span extraction into a standardized token classification task.

The Intuition Behind BIO Tagging

To resolve boundary ambiguities without complex structural parsing, the natural language processing community relies heavily on the Beginning-Inside-Outside (BIO) tagging scheme (sometimes called IOB).

Instead of predicting raw semantic classes, BIO introduces structural prefixes that dictate the position of a token relative to a span:

  • B- (Beginning): Marks the very first token of a target span.
  • I- (Inside): Marks any subsequent token that continues the span.
  • O (Outside): Marks tokens that do not belong to any target span.

By explicitly separating the start of an entity from its continuation, the model gains the capacity to delineate boundaries. As highlighted by Pilar et al. in their development of Cebuano entity baselines (CebuaNER) and Srirangam et al. in code-mixed social media analysis (Corpus Creation and Analysis for Named Entity Recognition in Telugu-English Code-Mixed Social Media Data), this representational strategy is what allows sequence models to distinguish between adjacent entities of the same type and accurately group multi-word phrases.

A Taxonomy of Tagging Schemes

Depending on the nature of the text and the target spans, researchers alter this basic grammar. Choosing a scheme changes the state space the model must learn, balancing prediction granularity against computational simplicity.

Text:       The   New   York   Stock   Exchange   opened
BIO:        O     B-Org I-Org  I-Org   I-Org      O
IO:         O     I-Org I-Org  I-Org   I-Org      O
BIEO:       O     B-Org I-Org  I-Org   E-Org      O

Inside-Outside (IO)

The simplest variant drops the B- tag entirely. Every token is either inside (I) or outside (O) a span. Liu et al. leveraged this scheme in Stanford MLab at SemEval-2021 Task 8: 48 Hours Is All You Need to rapidly extract quantities and units. While IO reduces the number of classes the model must predict—making it highly efficient for rapid development—it sacrifices the ability to separate adjacent entities of the same type.

Begin-Middle-End (BME) & BIEO

For tasks where the conclusion of a span is as structurally distinct as its beginning, adding an explicit End (E) tag helps. Zhang et al. utilized a BME scheme for hedge scope finding (Hedge Detection and Scope Finding by Sequence Labeling with Procedural Feature Selection), treating boundary prediction as a discrete classification of individual words. Similarly, Cejuela et al. found that a BIEO scheme was more effective for mapping the precise boundaries of natural language mutation mentions (nala: text mining natural language mutation mentions), where the termination of a biological term often contains critical structural cues.

Scaling from Entities to Macro-Structures

The beauty of token-level sequence labeling is its scale-agnostic nature. The same mechanism used to find a single-word gene mutation can be used to partition entire documents or map abstract discourse.

Document and Section Segmentation

Carvallo De Ferari et al. treated entire clinical document sections as "entities" using BIO tagging to partition raw text into contiguous, semantically labeled chunks (Automatic Section Classification in Spanish Clinical Narratives Using Chunked Named Entity Recognition).

Similarly, Hirohata et al. applied BI tagging to sentences to identify transitions in rhetorical roles within scientific abstracts (Identifying Sections in Scientific Abstracts using Conditional Random Fields), while Landes et al. used an IO encoding to classify text units into functional medical categories (A New Public Corpus for Clinical Section Identification: MedSecId). Even sentence boundary detection in ungrammatical text can be framed this way; Xu et al. used BIO tagging to isolate sentences from metadata noise in clinical notes (Automatic sentence segmentation of clinical record narratives in real-world data).

Argument and Discourse Mining

In computational linguistics, identifying the boundaries of a claim or premise is crucial. Stab et al. demonstrated that token-level sequence labeling could parse argumentative unit boundaries in persuasive essays (Parsing Argumentation Structures in Persuasive Essays), a strategy also mirrored by Guo et al. to track subjective statements and evidence spans in peer reviews (Automatic Analysis of Substantiation in Scientific Peer Reviews).

Why Token Tagging Outperforms Span Enumeration

Modern NLP offers alternative paradigms, such as span-based models, which enumerate all possible text segments and classify them. Why does token-level sequence tagging remain so dominant?

First, span enumeration suffers from a combinatorial explosion: a sentence of length $N$ has $O(N^2)$ possible spans. Token tagging, by contrast, scales linearly ($O(N)$) because it only requires one label per token.

Second, token tagging handles fragmentation gracefully. As noted by the authors of a study on domain-specific language models, token-level sequence tagging is significantly more robust than span-based prediction when long, complex strings are fragmented into numerous sub-word tokens during tokenization.

Finally, token tagging allows models to leverage sequential dependencies. By pairing a neural encoder (like BERT) with a Conditional Random Field (CRF) layer, the system does not just predict labels in isolation; it learns the legal transitions of the grammar (e.g., an I- tag can never follow an O tag). This architectural synergy is a cornerstone of top-performing systems, from clinical de-identification by Jiang et al. (A Deep Learning-Based System for the MEDDOCAN Task) to span-level hate speech detection by Khan et al. (Overview of the HASOC Subtrack at FIRE 2023: Identification of Tokens Contributing to Explicit Hate in English by Span Detection).

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 217 papers
  2. Version 1 · 2026-10-04 · 262 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_bio_tagging_scheme_429de3ee,
  title = {BIO tagging scheme},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/bio-tagging-scheme/rdr_429de3ee209c4c7c84f8c7311030036d},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_bio_tagging_scheme_429de3ee_v2,
  title = {BIO tagging scheme},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/bio-tagging-scheme/rdr_429de3ee209c4c7c84f8c7311030036d/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake