Character Level Sequence Labeling for Text Segmentation

42 concepts43 papers1 proposal

How do we find boundaries in continuous text? Consider a language like Chinese, where words are written sequentially without spaces, or an English word like unbelievable, which contains distinct morphological sub-components. A naive approach to finding these boundaries might involve a dictionary lookup or a sliding window that matches known words. However, this strategy fails catastrophically when encountering out-of-vocabulary (OOV) words, slang, or complex morphological variations.

The breakthrough idea that underpins modern text segmentation is to flip the problem on its head: instead of searching for words or segments globally, we classify individual characters locally. By assigning a position-indicator tag to every character in a sequence, text segmentation is elegantly reformulated as a standard sequence labeling task.

The Mechanics of Position-Indicator Tags

To transform a segmentation problem into a sequence labeling problem, we map structural boundaries to a discrete set of character tags. The most common formulation uses position-indicator tags to denote where a character sits relative to a segment boundary.

For instance, in Chinese Word Segmentation (CWS), early work by Nianwen Xue et al. introduced a four-tag scheme to indicate a character's position within a word: LL (left boundary), RR (right boundary), MM (middle of a word), and LR (a single-character word). Over time, this evolved into variations of the standard B (Begin), M (Middle), E (End), and S (Single) tag set, as explored by Irwin King et al. and Wen‐Jun Gao et al..

Alternatively, the problem can be simplified into a binary sequence labeling task. As demonstrated by Huihsin Tseng et al., characters can be labeled simply as word-initial or word-internal. Regardless of the exact tag set, this mapping converts a complex structural search into a sequence of local classification decisions.

Raw Text:      动 物 园 (Zoo)
BMES Scheme:   B M E
Binary Scheme: Initial Internal Internal

Why It Works: Contextual Windows and OOV Generalization

Why is this character-centric view so powerful? It allows us to deploy discriminative sequence classifiers—such as Maximum Entropy models, Conditional Random Fields (CRFs), and Hidden Markov Support Vector Machines (SVM-HMMs)—which excel at evaluating local context.

Instead of relying on a static lexicon, these models look at a sliding window of surrounding characters, extracting n-gram features and character transitions (Hwee Tou Ng et al.; Yuan Dong et al.). This local context provides strong clues about boundaries. For example, certain characters are highly likely to start a word but rarely end one.

Crucially, this enables the model to handle Out-of-Vocabulary (OOV) words. When a model encounters an unseen combination of characters, it does not fail; it uses the learned character-level transitions to infer the most probable boundaries. To further boost performance, Chia-Hui Chang showed that even simple dictionary-matching heuristics can be fed into CRFs as features, combining the raw coverage of lexicons with the generalization power of sequence labeling.

Generality Across Scales: From Morphemes to Paragraphs

The beauty of character-level sequence labeling is its scale-invariance. The exact same mechanism applies whether you are looking inside a single word or across an entire document:

  • Sub-word Morphology: In languages with complex word structures like Arabic, clitic segmentation can be framed as character-level classification to isolate prefixes, stems, and suffixes without manual morphological rules (Mona T. Diab et al.; Muhammad Abdul-Mageed et al.). Similarly, Jason S. Chang et al. applied CRFs to segment English words into deep Latin and Greek roots, while Helmut Schmid et al. used sequence tagging to identify syllable boundaries.
  • Internal Word Structure: Lun-Wei Ku et al. demonstrated that sequence labeling captures positional character context better than standard classification, allowing models to identify the semantic roles characters play within a word for sentiment analysis.
  • Macro-level Segmentation: The concept extends naturally to larger structural units. Chao-Lin Liu et al. used character-level CRFs to segment unpunctuated classical Chinese text into sentences, while Caroline Sporleder et al. framed paragraph identification as a binary boundary classification problem at sentence transitions.

Alternative Formulations: Gaps and Transitions

While tagging individual characters remains a dominant paradigm, researchers have explored alternative ways to structure the label space to optimize performance:

Inter-Character Gap Classification

Instead of labeling the characters themselves, Fandong Meng et al. proposed classifying the gaps between characters as either a boundary or a continuation. This binary formulation simplifies the label space and proves highly effective in semi-supervised or dictionary-bootstrapped settings, as it directly targets the decision point without needing to maintain complex position-indicator states.

Transition-Based Neural Models

With the advent of deep learning, the field expanded to incorporate transition-based architectures. Instead of assigning static tags to a pre-determined sequence, models developed by Meishan Zhang et al. and Jie Yang 0039 et al. process text incrementally from left to right. At each step, the model takes a binary action: either append the character to the current word or start a new word boundary. This allows the system to build up dynamic, word-level representations using LSTMs while maintaining the computational efficiency of sequential decision-making.

Key Foundational Work

Building a Large-Scale Annotated Chinese Corpus (2002) — contributed 1 concept elements to this direction.
An Empirical Study of Active Learning with Support Vector Machines for Japanese Word Segmentation (2002) — contributed 1 concept elements to this direction.
Chinese Sentence Tokenization Using Viterbi Decoder (1998) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_character_level_sequence_labeling_a275bbb5,
  title = {character-level sequence labeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/character-level-sequence-labeling/rdr_a275bbb54cba4646bf592932bbf06897},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_character_level_sequence_labeling_a275bbb5_v1,
  title = {character-level sequence labeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/character-level-sequence-labeling/rdr_a275bbb54cba4646bf592932bbf06897/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake