Sequence Labeling via Transformer Token Classification

29 concepts27 papers1 proposal

Traditional natural language processing architectures often treated sequence labeling—such as named entity recognition (NER), slot filling, and part-of-speech tagging—using sequential models like LSTMs or linear-chain Conditional Random Fields (CRFs). While effective, these models process text directionally or require complex decoding mechanics to capture dependencies between adjacent labels.

The widespread adoption of pre-trained Transformer encoders (such as BERT, RoBERTa, ELECTRA, and DeBERTa) introduced a more powerful paradigm: token classification. Instead of relying on sequential label propagation, a Transformer encoder generates dense, bidirectional contextual embeddings for every token in a sequence simultaneously. Because each token's vector representation is already deeply saturated with information from both its left and right context, a simple linear layer placed on top of these hidden states can make highly accurate, independent classification decisions.

By framing diverse structural, semantic, and syntactic problems as token-level classification tasks, researchers can build highly efficient, parallelizable models that excel across a wide array of domains.


Extracting Entities and Semantic Roles

The most direct application of this paradigm is mapping individual tokens to discrete entity boundaries or semantic categories. In this setup, an encoder processes a text sequence, and a token-classification head (such as the standard BertForTokenClassification architecture) maps the final hidden state of each token to a target label space. This is frequently paired with the BIO (Beginning, Inside, Outside) tagging format to delineate multi-token spans.

This approach has proven highly adaptable to specialized domains where entities are sparse or linguistically complex:

  • Domain-Specific NER: Almuslim et al. adapted token classification for legal text understanding, aligning tokenized subwords with expert legal annotations. In scientific literature, Rastogi et al. fine-tuned DeBERTa-v3-large with a token classification head to detect sparse software mentions. In clinical settings, Chuang et al. utilized RoBERTa to extract periodontitis diagnoses from unstructured notes, while Gupta applied RoBERTa-based token classification to multilingual pharmacovigilance tweets in Team Yseop at #SMM4H 2024: Multilingual Pharmacovigilance Named Entity Recognition and Relation Extraction.
  • Functional and Narrative Roles: Because Transformer layers capture abstract relational context, token classification can identify the structural or narrative roles that words play within a broader discourse. Cattan et al. appended token-level classifiers to pre-trained models to extract semantic concepts for spoken language understanding. Holur et al. used a token-classification head to determine the shifting narrative roles (such as "insiders" vs. "outsiders") of entities within social media text. Similarly, Wijngaard et al. fine-tuned a transformer to parse unstructured audio captions into structured semantic components ("who", "what", and "how").
  • Structural and Stylistic Tagging: Beyond semantic roles, token classification can isolate stylistic elements within long-form text. For instance, Stymne demonstrated in Direct Speech Identification in Swedish Literature and an Exploration of Training Data Type, Typographical Markers, and Evaluation Granularity that a BERT encoder fine-tuned via the Machamp sequence labeling toolkit can accurately tag direct speech spans in literature, even when explicit typographical markers like quotation marks are missing. For complex structural parsing, Rovera showed in EventNet-ITA: Italian Frame Parsing for Events that a fine-tuned Italian BERT model scales efficiently to large multi-label sequence spaces for event frame parsing without incurring massive computational overhead.

Mapping Structural Boundaries and Punctuation

A major conceptual hurdle in sequence labeling is applying token classification to elements that are not explicit words, such as punctuation marks, capitalization, or sentence boundaries. Researchers have resolved this by framing text structure as a property of word boundaries, using three primary methodologies:

1. Direct Token-to-Punctuation Mapping

The most straightforward framing treats punctuation as a trailing attribute of a word token. Each word in a raw, unpunctuated sequence is assigned a label corresponding to the punctuation mark that should follow it (e.g., NONE, COMMA, PERIOD).

Poláček et al. added a classification head to an ELECTRA-Small model to map output embeddings directly to discrete punctuation categories, creating a low-latency system for real-time streaming ASR captioning. Similarly, Znotins et al. leveraged BERT variants like RoBERTa and ELECTRA to jointly restore punctuation and capitalization in conversational Latvian transcripts. At an even finer granularity, Doostmohammadi et al. applied character-level token classification with BERT to predict word segmentation and zero-width non-joiner (ZWNJ) usage in Persian text.

2. Placeholder Insertion and Slot Filling

Instead of classifying the word token itself, some architectures explicitly modify the input sequence to represent potential boundary sites. Silva et al. inserted specific placeholder tokens into a text window to represent potential punctuation sites. The pre-trained language model generates a representation for this placeholder, which a classification head then maps to the correct punctuation mark based on the surrounding bidirectional context.

Alternatively, Rao et al. framed punctuation restoration as a masked language modeling (MLM) slot-filling task. Instead of predicting missing words, the model is trained to predict punctuation marks at designated word boundaries, leveraging the native pre-training objectives of language models.

3. Candidate Evaluation

For macro-level segmentation, such as identifying where a sentence ends, models can evaluate existing punctuation marks to determine if they serve a structural role. Wicks et al. framed sentence segmentation as a binary classification problem where a Transformer encoder processes the token contexts surrounding potential punctuation marks to evaluate whether they act as true sentence terminators.


Overcoming Class Imbalance and Label Dependencies

While token classification benefits from the representational power of Transformers, it faces two inherent challenges: severe class imbalance and the lack of explicit label-to-label constraints.

Mitigating Class Imbalance with Focal Loss

In tasks like punctuation prediction or sparse entity extraction, the dataset is overwhelmingly dominated by the negative or "null" class (i.e., tokens that do not precede a punctuation mark or belong to an entity). Standard Cross-Entropy loss struggles in this environment because the gradients are dominated by easy-to-classify negative examples, causing the model to underperform on rare target classes.

To solve this, Yi et al. integrated Focal Loss into a BERT-based sequence labeling model. Focal Loss introduces a modulating factor $(1 - p_t)^\gamma$ to the standard cross-entropy loss, where $p_t$ is the model's estimated probability for the correct class. When a token is easily classified (such as the thousands of standard words requiring no punctuation), $p_t$ is high, the modulating factor approaches zero, and its contribution to the gradient is down-weighted. This forces the model to focus its capacity on ambiguous boundaries and rare punctuation marks.

Eliminating Cascading Errors via Span-Based Classification

Traditional token classification models make independent decisions for each token or rely on sequential decoders like CRFs to enforce valid label transitions (e.g., preventing an I-PER tag from following an O tag). However, sequential decoders can suffer from cascading errors, where an early misclassification corrupts the predictions of subsequent tokens.

To bypass this limitation, Yu et al. proposed a span-based model called S-NER. Instead of assigning a label to every individual token, the architecture uses a pre-trained BERT encoder to generate contextualized token embeddings, which are then aggregated into fixed-length span representations. A simple multi-layer Feed-Forward Network (FFN) decoder then directly classifies the entity type of the entire span at once. By decoupling the architecture from token-by-token label dependencies, the model eliminates cascading misclassifications while maintaining the computational advantages of the underlying Transformer encoder.

Key Foundational Work

Joint Persian Word Segmentation Correction and Zero-Width Non-Joiner Recognition Using BERT (2020) — contributed 1 concept elements to this direction.
Extraction of Common-Sense Relations from Procedural Task Instructions using BERT (2021) — contributed 1 concept elements to this direction.
Information Extraction from Legal Documents: A Study in the Context of Common Law Court Judgements (2020) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_token_classification_for_punctuation_c3ae4f56,
  title = {token classification for punctuation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/token-classification-for-punctuation/rdr_c3ae4f56b5ed434cb375546df9db5a9a},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_token_classification_for_punctuation_c3ae4f56_v1,
  title = {token classification for punctuation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/token-classification-for-punctuation/rdr_c3ae4f56b5ed434cb375546df9db5a9a/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake