TL;DR
The authors introduce USP, a unified "semantic contract" that bridges the gap between Information Extraction (IE) and Intent-Slot parsing. By normalizing 61 datasets into a shared representation and introducing five diagnostic "views," they demonstrate that standard evaluation metrics often mask significant structural failures in large language models (LLMs).
What Problem Does This Paper Address?
Structured semantic parsing—tasks like identifying entities (NER), relations (RE), events (EE), or user intents and slots (SLU)—is traditionally evaluated in silos. More importantly, evaluation is often shallow, focusing on whether a model picks the right category label while ignoring whether it correctly binds arguments to specific instances.
For example, a model might correctly identify a "Transaction" event but link the "Buyer" to the wrong "Amount." Current benchmarks often fail to penalize these structural misalignments, leading to an overestimation of model utility in real-world applications where data integrity and instance-level associations are critical.
Methods
The core of the paper is the USP Semantic Contract. This is a shared interface that represents categories, fields, values, and instance associations across all four task families.
The USP Contract
Instead of task-specific formats, every output follows a Category -> List of Argument Groups structure. This preserves the "conceptual identity" of an instance (e.g., a specific event or a specific tool call) without requiring complex, non-standardized identifiers.
Diagnostic Evaluation
Rather than relying on a single aggregate score, the authors propose five "views" that ask increasingly difficult questions about the model's output:
- Category Agreement: Did you pick the right label?
- Field Recognition: Did you identify the right slots?
- Field-Value Pairing: Did you get the right data in those slots?
- Instance-Bound Assignment: Did you group the right slots together for a specific instance?
- Complete-Instance Recovery: Did you get the entire structure perfectly right?
The authors fine-tuned a 29B parameter Mixture-of-Experts (MoE) model, Xing4.0-29B-Intent, using a unified autoregressive objective to handle these diverse tasks through a single schema-conditioned prompt.
Datasets
The researchers normalized 61 public datasets into the USP format, creating a massive, unified corpus for training and evaluation. This includes:
- 29 NER datasets (e.g., OntoNotes, MSRA)
- 16 RE datasets (e.g., DocRED, FewRel)
- 9 EE datasets (e.g., DuEE, CASIE)
- 7 Intent-Slot sources (e.g., ATIS, SNIPS, MultiWOZ)
They also introduced USP-Core-6K, a metadata-stratified subset designed for cost-efficient model screening without sacrificing representativeness.
Key Insights and Ideas
- The "Widening Gap": The most striking finding is that as the evaluation criteria become stricter (moving from Category to Complete-Instance matching), the performance gap between the best model and its competitors grows dramatically.
- Category Agreement is Deceptive: High scores in category matching often hide "structural deficits." A model might look competent at labeling but fail miserably at actually structuring the data it extracts.
- Unification without Dilution: The paper proves that heterogeneous tasks like NER and complex Event Extraction can share a single representation and learning objective without losing the nuances required for each specific task.
Key Results
- State-of-the-Art Performance: The fine-tuned Xing4.0-29B-Intent model achieved a task-balanced Field-Value Micro-F1 of 57.70, ranking first across all four task families and all five diagnostic views.
- Structural Superiority: While its lead over the next best system was only 2.93 points for simple category matching, that margin exploded to 20.73 points for complete-instance recovery.
- Robustness across Datasets: As shown in the per-dataset margin analysis, the model maintained a positive margin over the best alternatives across the vast majority of the 61 datasets.
Limitations and Future Work
- Conversion Fidelity: The authors acknowledge that while their converted records satisfy the USP contract, a formal "round-trip audit" (converting USP back to native formats) is needed to ensure no source semantics were lost during normalization.
- Schema Sensitivity: The current study uses fixed schemas. Future work is needed to explore how sensitive models are to changes in schema wording or the inclusion of detailed descriptions.
- Inference Budgets: The comparison includes proprietary APIs and open-weight models with different parameter counts; therefore, the results reflect performance under specific execution conditions rather than a strictly cost-matched benchmark.
Summary updated