In most information extraction tasks, the data is overwhelmingly boring. Whether you are identifying named entities in a tweet, extracting gene splice sites from DNA, or pinpointing propaganda in a news article, the vast majority of your input consists of "background" tokens. In the standard BIO (Beginning, Inside, Outside) tagging scheme, this results in a massive "O" class that can account for 90% or more of the dataset.
This extreme class imbalance creates a "needle in a haystack" problem. Standard machine learning models, optimized via cross-entropy loss, often take the path of least resistance: they achieve high nominal accuracy by simply predicting the majority "background" class for everything. This bias leads to poor recall for the rare, critical tokens that actually matter.
The Failure of Token-Level Classification
Traditional sequence labeling treats every word as an independent classification sub-problem. As Sopan Khosla and colleagues noted in their work on propaganda detection, this granularity makes it difficult to delineate boundaries because the model is overwhelmed by non-propagandistic tokens. When a model sees a thousand "O" tags for every one "B-Entity" tag, the gradient signal from the entity is often drowned out.
Researchers like Gustavo Aguilar have shown that this is particularly acute in "noisy" environments like social media. In these settings, the diversity of entities is high, but their frequency is low, causing models to over-predict the majority class and fail to generalize to rare mentions.
Reframing Extraction as Search
One of the most effective ways to build intuition for solving this is to stop treating extraction as a labeling task and start treating it as a search or "pointing" task.
Instead of asking "What is this token?", models like SegBot, developed by Jing Li and Shafiq Joty, use Pointer Networks to identify segment boundaries. By "pointing" to the start and end of a span, the model bypasses the need to classify every internal token, effectively ignoring the "haystack" to focus on the "needle."
Similarly, Jin Huang introduced PMRC, which reframes Named Entity Recognition (NER) as a Machine Reading Comprehension (MRC) task. By asking the model a question (e.g., "Which spans refer to a Person?"), the problem shifts from a biased classification to a similarity-based retrieval, which is inherently more robust to the absence of many positive examples.
Harmonizing the Gradient
If we must stick to token-level labeling, we have to change how the model "feels" its mistakes. Standard cross-entropy treats a missed entity and a missed background token with equal weight, but since background tokens are everywhere, they dominate the learning process.
Huaishao Luo and the team behind GRACE addressed this using gradient harmonization. This technique, borrowed from computer vision, down-weights the loss contribution from "easy" examples (the ubiquitous background tokens) and focuses the model's attention on "hard" examples (the sparse target tokens).
Another approach is to change the objective function entirely. Ngoc Dang Nguyen and Lan Du proposed AUC Maximization for low-resource NER. By optimizing for the Area Under the Curve rather than simple accuracy, they force the model to rank entity tokens higher than non-entity tokens, which is a much more effective training signal when data is scarce and imbalanced.
Structural and Data-Level Interventions
Beyond loss functions, researchers have looked at the structure of the pipeline and the data itself:
- Sentence-Level Resampling: Xiaochen Wang and Yue Wang found that simply oversampling sentences that contain rare entities can balance the training signal without losing the linguistic context that Transformers require.
- Cascaded Tagging: Yu Wang proposed using Reinforcement Learning to "fix" the output of a standard tagger. The RL agent focuses specifically on minority tags, learning to refine the initial, biased predictions of the base model.
- Multi-Granular Features: In tasks like gene splicing identification, Qi-Jie Li demonstrated that looking at the "horizon" (the entire sequence) rather than individual points helps the model recognize biological patterns that are too sparse to be seen at the token level.
By moving away from the "one-label-per-word" paradigm and adopting strategies that explicitly account for the rarity of information, these researchers have made sequence labeling models more sensitive to the meaningful signals hidden within the noise of natural language.