Abstract
Universal Information Extraction (UIE) models traditionally rely on rigid, exact text boundaries to identify entities, relations, and sentiments. However, human language is inherently ambiguous, and rigid boundary constraints often lead to sub-optimal performance due to annotation inconsistency. This paper introduces FSUIE (Fuzzy Span Universal Information Extraction), a framework that incorporates a fuzzy span mechanism to handle boundary ambiguity. By utilizing Fuzzy Span Loss (FSL) and Fuzzy Span Attention (FSA), the model achieves faster convergence and superior performance, particularly in low-resource scenarios, setting new benchmarks across several standard datasets.
Introduction
Information Extraction (IE) is the process of transforming unstructured text into structured data, encompassing tasks such as Named Entity Recognition (NER), Relation Extraction (RE), and Aspect Sentiment Triplet Extraction (ASTE). While modern UIE models have made significant strides, they typically treat text spans as binary: a token is either part of a span or it is not.
This "all-or-nothing" approach creates two primary challenges. First, annotation ambiguity: human annotators frequently disagree on exact boundaries (e.g., whether to include an adjective like "yellow" in the entity "yellow sports car"). Rigid models penalize reasonable predictions that do not perfectly match a specific label. Second, computational inefficiency: standard Transformer-based models use global attention, requiring the model to scan the entire sentence to identify localized spans, which ignores the prior knowledge that spans are usually short and semantically concentrated.
The FSUIE Framework
To address these limitations, the FSUIE framework introduces two innovative components designed to soften the boundaries of information extraction.
1. Fuzzy Span Loss (FSL)
Instead of the traditional "one-hot" target where only the exact boundary is considered correct, FSL employs a Gaussian distribution. This approach provides the model with "partial credit" for predicting boundaries in the immediate vicinity of the ground truth. By acknowledging that a word adjacent to the target is more relevant than a word at the opposite end of the sentence, FSL makes the model more robust to the quirks of individual annotators and improves generalization on unseen data.
2. Fuzzy Span Attention (FSA)
While standard attention mechanisms look at the entire input sequence simultaneously, FSA introduces a learnable mask function. This creates a "spotlight" effect, applying full attention to a local range and gradually attenuating (fading out) attention weights at the boundaries. This mechanism forces the model to prioritize local semantic information relevant to span construction, effectively filtering out global noise that can distract the model during the extraction process.
FSUIE is implemented on top of BERT-base and BERT-large architectures, with a single FSA layer added at the top to refine span decisions without significantly increasing the initial encoding time.
Experimental Results
The researchers evaluated FSUIE across three major IE tasks using the ACE2004, ACE2005, ADE (medical), and ASTE-Data-V2 datasets.
State-of-the-Art Performance
FSUIE achieved new state-of-the-art (SOTA) results on the ADE dataset for both NER and RE, even when utilizing only a BERT-base backbone. On the ACE04 dataset, the FSUIE-base model improved the Relation Extraction F1 score by 5.1 points over the baseline UIE model. Notably, FSUIE-base often outperformed significantly larger generative models, such as T5-v1.1-large, suggesting that architectural constraints on span identification are more effective than simply increasing parameter counts.
Low-Resource Robustness
The fuzzy mechanism proved particularly effective in data-scarce environments. In tests using only 1% of the training data, FSUIE’s Entity F1 score reached 70.09, compared to the baseline's 63.47. This indicates that the fuzzy span approach helps the model learn the underlying structure of language more efficiently when examples are limited.
Convergence and Synergy
Ablation studies confirmed that FSL and FSA are highly synergistic. While FSL addresses boundary uncertainty, FSA handles the localization of the search. Together, they lead to significantly faster convergence; by narrowing the search space early in the training process, the model spends less time processing irrelevant global information and more time refining span predictions.
Conclusion and Future Work
FSUIE demonstrates that incorporating "fuzziness" into information extraction models leads to more robust and efficient performance. By moving away from rigid boundaries, the framework better aligns with the natural ambiguity of human language.
Future research directions include exploring adaptive probability density functions beyond the Gaussian distribution to better match specific linguistic ambiguities. Additionally, the authors suggest refining the strategy for "single-sided" ambiguity, where only one end of a span is typically debated by annotators. Despite these areas for refinement, FSUIE represents a significant step forward in making universal information extraction more flexible and high-performing.
Summary updated