Abstract
Open Information Extraction (OpenIE) models often struggle with "syntactic drift"—the variation in sentence structure for the same semantic content. We introduce ROBUST, a benchmark designed to evaluate knowledge invariance through a clique-based structure. Our evaluation reveals significant performance degradation across state-of-the-art models and LLMs, highlighting a critical need for structurally robust extraction methods.
Introduction
Open Information Extraction (OpenIE) is the task of pulling structured knowledge—usually in the form of $(a_1, p, a_2, ...)$ tuples—out of unstructured text. While significant strides have been made in extraction accuracy, most benchmarks evaluate models on a per-sentence basis. This ignores a critical real-world challenge: knowledge invariance. In practice, the same fact can be expressed in numerous ways. A model might successfully extract a fact from a simple active-voice sentence but fail if that same fact is presented in a complex passive construction or a relative clause.
The Challenge of Knowledge Invariance
The primary reason models fail is a sensitivity to sentence structure, or "syntactic drift." When the syntactic distribution shifts away from the training data, performance plummets. As shown in Figure 1, a model like OpenIE6 might perform well on a standard test sentence but fail on its paraphrases, even though the underlying knowledge remains identical.
The ROBUST Benchmark
To address this, we developed the ROBUST benchmark using a novel "knowledge-invariant clique" structure. ROBUST is built upon the foundation of the CaRB dataset and consists of 4,971 sentences organized into 1,272 cliques.
Methodology
- Clique Generation: Starting with seed sentences, we used a syntactically controllable paraphraser to generate multiple versions of each sentence, filtered for high syntactic diversity and semantic consistency.
- Human-in-the-loop Annotation: Human experts corrected the generated sentences and annotated ground-truth knowledge tuples for every sentence in the clique.
- Robustness Metric: Instead of averaging scores, we propose a metric that evaluates the worst performance within a clique. A model is only considered truly robust if it extracts the same knowledge accurately across all variations.
Experimental Results
We tested six representative OpenIE models and ChatGPT (GPT-3.5). The results reveal a universal "frustrating degradation" in performance:
- Universal Degradation: Every model showed a dramatic performance drop on ROBUST compared to CaRB, with an average F1 score decrease of 18 percentage points.
- LLM Fragility: Even ChatGPT (GPT-3.5) shows a significant robustness gap, with its score dropping from 0.516 on CaRB to 0.275 on ROBUST in a 10-shot setting.
- SpanOIE's Stability: SpanOIE demonstrated the least degradation (13.35 F1 drop), likely because its strategy of enumerating geometric spans is more resistant to syntactic shifts than labeling-based architectures.
Conclusion
Traditional metrics hide fragility by averaging results. By measuring clique-wise worst-case performance, ROBUST exposes how easily models are tripped up by simple rephrasing. Our study highlights a need for new model architectures that explicitly prioritize knowledge invariance over surface-level pattern matching.
Summary updated