Abstract
Information Extraction (IE) is a critical task for structured data retrieval from unstructured text. While recent trends favor Universal IE (UIE) using generative language models, these models often neglect the intrinsic syntactic structure of language. We present LasUIE (Latent Adaptive Structure-aware UIE), a unified generative framework that induces and adapts latent dependency and constituency trees without external parsers. By leveraging task-specific syntactic structures through a three-stage training pipeline, LasUIE achieves state-of-the-art performance across 12 benchmarks and demonstrates significant robustness in low-resource scenarios.
Introduction
Information Extraction (IE)—encompassing Named Entity Recognition (NER), Relation Extraction (RE), and Event Extraction (EE)—is fundamental to knowledge graph construction. Traditional approaches relied on task-specific models, but the field has shifted toward Universal IE (UIE), treating all tasks as text-to-text generation. However, standard generative models often overlook syntactic grammar, which is vital for identifying entity boundaries (constituency syntax) and linking distant words (dependency syntax).
Existing methods that incorporate syntax typically rely on static, external parsers. These are often noisy and do not adapt to the specific needs of different IE tasks. LasUIE addresses these limitations by learning to induce its own task-specific, latent syntactic structures, allowing the model to "imagine" the most relevant grammatical framework for a given extraction goal.
Methods
LasUIE utilizes a three-stage training pipeline to integrate latent structure awareness into a generative backbone (e.g., T5).
- Pre-training: The model starts with a standard pre-trained language model backbone.
- Unsupervised Structure-aware Post-training: A Heterogeneous Structure Inductor (HSI) learns to predict latent dependency and constituency trees from raw text. This process is unsupervised; the model optimizes these structures to improve its internal text representation.
- Task-oriented Structure Fine-tuning: During task-specific training, the model employs Policy Gradients to treat syntactic structures as a "policy." It adjusts these structures based on the "reward" of extraction accuracy, effectively pruning irrelevant grammatical branches to focus on task-critical information.
A Structural Broadcaster (SB) module then compacts these latent trees into "forests" and injects them into the generation process via cross-attention, guiding the decoder to produce accurate structured outputs.
Results and Discussion
LasUIE was evaluated across 12 benchmarks covering NER, RE, Sentiment Analysis (AOP/ASTE), Role Labeling (ORL/SRL), and Event Extraction.
Key findings include:
- State-of-the-Art Performance: LasUIE (Large) achieved an average F1 score of 77.4, outperforming previous UIE models and specialized systems.
- Structural Synergy: Ablation studies confirmed that constituency features primarily aid boundary detection (NER), while dependency features are crucial for relational tasks (RE).
- Low-Resource Robustness: In 1-shot settings, LasUIE significantly outperformed baselines. For instance, on CoNLL03 (NER), it achieved a 39.4 F1 score, nearly doubling the performance of non-structure-aware models.
- Adaptive Pruning: Fine-tuning leads to sparser, more efficient internal trees, demonstrating the model's ability to isolate the structural skeleton relevant to the extraction task.
Conclusion
LasUIE demonstrates that latent, adaptive syntactic structures can significantly enhance universal information extraction. By moving away from static external parsers and toward task-specific structural induction, the model achieves superior performance and data efficiency. Future work may explore extending this latent structure approach to other complex NLP tasks such as summarization or translation.
Summary updated