FSUIE: A Novel Fuzzy Span Mechanism for Universal Information Extraction

Tianshuo Peng, Zuchao Li, Lefei Zhang, Bo Du, Hai Zhao

TL;DR

FSUIE introduces a "fuzzy" approach to span-based Information Extraction (IE) to overcome the rigid constraints of traditional models. By replacing exact boundary targets with Gaussian distributions and implementing an adaptive local attention mechanism, the framework achieves state-of-the-art performance, faster convergence, and superior robustness in low-resource settings.

What Problem Does This Paper Address?

Information Extraction (IE)—the task of pulling structured data like entities and relationships from text—has recently moved toward "Universal Information Extraction" (UIE) frameworks. However, most UIE models suffer from two major issues:

  1. Annotation Ambiguity: Human annotators often disagree on exact boundaries. For example, in the phrase "a yellow sports car," one person might label "sports car" as the vehicle, while another labels "yellow sports car." Standard models use "teacher forcing" with rigid, one-hot targets, which penalizes the model for reasonable but slightly "off" predictions, creating a performance bottleneck.
  2. Global-Local Mismatch: Transformers are designed to capture global context across an entire sequence. However, IE tasks are inherently local; determining a span boundary usually requires focusing on a specific neighborhood of words. Standard global attention can introduce noise and ignore the limited-length nature of most IE spans.

Methods

The authors propose the Fuzzy Span Universal Information Extraction (FSUIE) framework, which introduces two novel components:

  • Fuzzy Span Loss (FSL): Instead of using a Dirac delta distribution (where only the exact ground-truth position has a probability of 1), FSL uses a Gaussian distribution centered on the ground truth. This quantitatively represents "correctness" as a gradient, acknowledging that positions closer to the target are more likely to be correct than those further away.
  • Fuzzy Span Attention (FSA): This is a specialized attention layer placed on top of the Transformer encoder. It uses a learnable mask function, $g_m$, to adaptively adjust the attention span length. Unlike standard sparse attention, it features an "attenuated" range where attention scores gradually decay rather than being abruptly truncated.
Figure 2: Illustration of exact boundary and fuzzy boundary.

Datasets

The framework was evaluated across three core IE tasks using four major datasets:

  • Named Entity Recognition (NER) & Relationship Extraction (RE): ACE2004, ACE2005, and ADE (a biomedical dataset).
  • Aspect Sentiment Triplet Extraction (ASTE): ASTE-Data-V2 (including 14lap, 14res, 15res, and 16res subsets).

Key Insights and Ideas

  • Embracing Uncertainty: The shift from "exact" to "fuzzy" boundaries reflects the reality of natural language. By using a Gaussian distribution for loss, the model learns a smoother decision boundary that is less sensitive to noisy human labels.
  • Local Focus for Local Tasks: The FSA mechanism confirms that for IE, "less is more." By forcing the model to focus on semantic information within a limited range of tokens, the authors reduce global noise and improve the model's ability to identify span-aware representations.
  • Synergistic Effects: Ablation studies showed that while FSL alone improves performance, combining it with FSA leads to the most substantial gains. FSA guides the model to filter critical information, while FSL allows it to extract more nuanced information from that filtered context.
Figure 5: Illustration of attention scores distribution in FSA layer. The extracting target are "walter rodgers" and "who".

Key Results

FSUIE demonstrated significant improvements over strong UIE baselines:

  • State-of-the-Art: FSUIE-large achieved new SOTA results on the ADE dataset for NER (93.08 F1) and several RE benchmarks.
  • Efficiency: Models with FSA converged significantly faster. As shown in the performance curves, FSUIE reaches high F1 scores in far fewer training steps than the baseline.
  • Low-Resource Robustness: In scenarios with only 1% of training data, FSUIE-base outperformed the standard UIE-base by a wide margin (e.g., 70.09 vs 63.47 F1 in NER), proving its ability to generalize from limited supervision.
Figure 4: NER performance of different models on ACE04 test set.

Limitations and Future Work

The authors acknowledge that the proposed framework is computationally and spatially more expensive due to the added complexity of the attention mechanism. Additionally, the study only explored a limited set of probability density functions for the fuzzy loss.

Future research could involve developing more targeted strategies for adjusting the information distribution and exploring more efficient implementations of the fuzzy attention mechanism to reduce the training overhead.

Summary updated

Abstract

Universal Information Extraction (UIE) has been introduced as a unified framework for various Information Extraction (IE) tasks and has achieved widespread success. Despite this, UIE models have limitations. For example, they rely heavily on span boundaries in the data during training, which does not reflect the reality of span annotation challenges. Slight adjustments to positions can also meet requirements. Additionally, UIE models lack attention to the limited span length feature in IE. To address these deficiencies, we propose the Fuzzy Span Universal Information Extraction (FSUIE) framework. Specifically, our contribution consists of two concepts: fuzzy span loss and fuzzy span attention. Our experimental results on a series of main IE tasks show significant improvement compared to the baseline, especially in terms of fast convergence and strong performance with small amounts of data and training epochs. These results demonstrate the effectiveness and generalization of FSUIE in different tasks, settings, and scenarios.
BibTeX
@inproceedings{peng2023fsuie,
  author = {Tianshuo Peng and Zuchao Li and Lefei Zhang and Bo Du and Hai Zhao},
  title = {{FSUIE: A Novel Fuzzy Span Mechanism for Universal Information Extraction}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.acl-long.902},
}
Figure