The standard paradigm for extractive question answering (QA) is built on a clean but fragile assumption: the answer to a question exists as a single, contiguous block of text. In standard benchmarks like SQuAD, a model simply needs to identify two integers—the start index and the end index of the target span.
However, real-world information is rarely so neatly packaged. Consider asking a model to list the side effects of a medication from a medical paper, extract scattered data points from a complex financial table, or pull out key conclusions from a messy, multi-speaker meeting transcript. In these scenarios, the true answer is distributed across multiple, non-contiguous spans. Forcing a traditional pointer network into these domains presents a bad trade-off: the model must either extract a massive, noisy window containing irrelevant text or pick just one piece of the answer and discard the rest.
To solve this, a major research direction reframes extractive QA from a boundary-prediction problem into a token-level sequence tagging task. By making an independent classification decision for every single token, models can naturally extract an arbitrary number of disconnected answer spans in a single forward pass.
From Pointers to Token-Level Decisions
The core shift from single-span pointer networks to multi-span extraction requires changing how the model's output layer is interpreted. Instead of predicting two global probability distributions (one for the start and one for the end), a sequence tagging model assigns a label to every individual token in the document using standard sequence labeling schemes like BIO (Beginning, Inside, Outside), IOB2, BMEO, or BIOES.
Context: Headaches , nausea , and fatigue .
BIO Tags: B-ANS O B-ANS O O B-ANS O
In this formulation, a token labeled B-ANS marks the start of a distinct answer segment, I-ANS indicates its continuation, and O indicates that the token is entirely outside the answer.
This framing has deep roots but has gained significant traction with modern transformer architectures. Early foundations were laid by Yutaka Sasaki in Question Answering as Question-Biased Term Extraction: A New Approach toward Multilingual QA, which treated QA as a sequence-labeling task using the IOB2 scheme to bypass rigid, pre-defined question types. More recently, Elad Segal et al. demonstrated in A Simple and Effective Model for Answering Multi-span Questions that this simple architectural shift allows standard language models to naturally extract multiple non-contiguous text segments without requiring complex symbolic programs or specialized architectures.
Why Sequence Tagging Works
The elegance of sequence tagging lies in how it simplifies the optimization and decoding process for multi-span extraction.
- Full Differentiability: Because the task is reduced to token-level classification, the entire model can be trained using standard cross-entropy loss applied to each token. The model learns the contextual boundaries of answers organically through the loss gradient of individual tokens.
- Elimination of Fragile Heuristics: Traditional span-extraction models attempting to handle multiple answers typically predict the top-$K$ start and end indices. They then rely on complex post-processing heuristics or manual probability thresholds to filter out overlapping spans or determine exactly how many answers to return. As noted by Wonjin Yoon et al. in Sequence Tagging for Biomedical Extractive Question Answering, sequence tagging eliminates these fragile thresholds entirely. The token-level labels implicitly define both the boundaries and the total count of the spans in a single pass.
- Coherence via Global Constraints: While token decisions are made locally, models can incorporate global constraints to ensure the extracted spans make sense together. For example, Haonan Li et al. introduced MultiSpanQA: A Dataset for Multi-Span Question Answering, demonstrating that combining token-level sequence tagging with a global prediction head (such as predicting the total number of spans) significantly improves extraction accuracy and structural coherence.
Taming Messy and Structured Data
The sequence tagging paradigm has proven uniquely capable in specialized domains where text is inherently fragmented, conditional, or non-linear.
Biomedical Literature and Systematic Reviews
Biomedical questions frequently demand list-type answers (e.g., listing genes, symptoms, or patient cohorts). In KU-DMIS at BioASQ 9: Data-centric and model-centric approaches for biomedical question answering, Wonjin Yoon et al. showed that treating list-based QA as a token-level sequence tagging task allows models to identify an arbitrary number of answers without needing heuristic confidence thresholds. This approach was expanded by Hyunjae Kim et al. in Exploring Approaches to Answer Biomedical Questions: From Pre-processing to GPT-4, confirming that BIO-based sequence tagging provides a far more robust framework for extracting lists of distinct entities than standard span prediction.
Beyond simple lists, this mechanism is vital for automating systematic medical reviews. David Begert et al. demonstrated in Towards automating systematic reviews on immunization using an advanced natural language processing–based extraction system how multi-class sequence extraction can isolate specific, non-contiguous spans corresponding to structured PICO (Population, Intervention, Control, Outcomes) frameworks within unstructured scientific papers.
Conversational Transcripts
Dialogue is inherently messy; human speakers interrupt one another, change topics, and scatter answers across multiple turns. To extract a coherent answer, a model must stitch together disparate parts of a conversation. Archiki Prasad et al. tackled this in MeetingQA: Extractive Question-Answering on Meeting Transcripts, using IO tagging to capture non-contiguous answer spans across multi-speaker turns. Similarly, Changlin Yang et al. applied token-level tagging in Improving Knowledge Production Efficiency With Question Answering on Conversation to reconstruct clean FAQ databases from fragmented customer service chat logs.
Visually Rich and Conditional Documents
When documents contain visual structures like tables, charts, or complex layouts, relevant information becomes spatially fragmented. In DuReadervis: A Chinese Dataset for Open-domain Document Visual Question Answering, Le Qi et al. found that sequence labeling using BIOES tags accommodates complex document structures far better than pointer networks, especially when an answer is distributed across multiple isolated cells within a table. Furthermore, real-world answers are often conditional or hierarchical. To capture this complexity, Yiming Ju et al. introduced CMQA: A Dataset of Conditional Question Answering with Multiple-Span Answers, utilizing BIO sequence tagging to map out answers that depend on specific, fragmented sets of conditions within the text.
Expanding to General Information Extraction
The success of framing QA as sequence tagging has led researchers to reverse the paradigm: framing complex Information Extraction (IE) tasks as multi-span QA.
For example, Xiaoya Li et al. converted entity-relation extraction into a multi-turn QA task in Entity-Relation Extraction as Multi-Turn Question Answering. By using BMEO tagging, their model could ask a question like "Which entities are related?" and extract all distinct target spans in a single pass. For cross-lingual event extraction, Steven Fincke et al. developed a method in Language Model Priming for Cross-Lingual Event Extraction that prepends event triggers directly to the input text, using a BIO tagging layer on top of the primed sequence to pull out multiple event arguments simultaneously.
Alternative and Complementary Mechanisms
While token-level classification is the dominant approach for multi-span extraction, alternative architectures exist. In Ask-and-Verify: Span Candidate Generation and Verification for Attribute Value Extraction, Yan Liang et al. proposed a two-stage pipeline that uses multi-label classifiers to predict the top-$K$ most probable start and end indices independently, generating candidate spans that are subsequently filtered by a verification network.
Additionally, when working with discriminative models rather than standard encoders, evaluating multi-token spans requires different aggregation strategies. As shown by Mengzhou Xia et al. in Prompting ELECTRA: Few-Shot Learning with Discriminative Pre-Trained Models, multi-token sequences can be effectively evaluated by aggregating discriminator outputs or averaging hidden states across the entire candidate span, allowing for flexible handling of variable-length answers in few-shot settings.
Ultimately, whether through direct token classification or structured candidate generation, moving away from single-span constraints allows models to interact with text the way humans do: pulling scattered threads of information from a document and weaving them into a single, coherent answer.
Key Foundational Work
Question Answering as Question-Biased Term Extraction: A New Approach toward Multilingual QA (2005) — contributed 1 concept elements to this direction.
Two-Stage Synthesis Networks for Transfer Learning in Machine Comprehension (2017) — contributed 1 concept elements to this direction.