Abstract
Tanya Goyal, Greg Durrett. Findings of the Association for Computational Linguistics: EMNLP 2020.
2020.
Introduction
Large language models (LLMs) are increasingly proficient at generating fluent text, yet they remain prone to "hallucinations"—the generation of content that is factually inconsistent with source material. Traditional evaluation metrics like ROUGE or BERTScore primarily measure lexical or semantic overlap, often failing to detect critical factual errors such as swapped subjects and objects. While Natural Language Inference (NLI) models attempt to address this through sentence-level entailment, they operate as "black boxes," offering no insight into which specific parts of a sentence are problematic.
The DAE Framework
To provide a more granular and interpretable approach to fact-checking, researchers from UT Austin developed Dependency Arc Entailment (DAE). This framework shifts the focus from holistic sentence evaluation to atomic semantic units: dependency arcs. By decomposing a sentence into its grammatical relationships (e.g., the link between a verb and its subject), DAE can pinpoint exactly where a model deviates from the truth.
The DAE pipeline consists of three main stages:
- Decomposition: A generated sentence is parsed into individual dependency arcs (e.g.,
nsubj(takeover → military)). - Arc-level Prediction: A pre-trained encoder (such as BERT or ELECTRA) processes the source text and the generated sentence. It then predicts an entailment score for every individual arc.
- Aggregation: A global factuality score for the sentence is derived by averaging the scores of all its constituent arcs.
Automatic Training Strategy
A significant hurdle in developing fine-grained fact-checkers is the scarcity of human-labeled data at the arc level. The authors overcame this using an automatic data creation strategy:
- Positive Examples: Arcs from high-quality human paraphrases are assumed to be factually entailed.
- Negative Examples: The researchers used a paraphrase model to generate multiple outputs. Arcs that appeared in low-quality (bottom-of-the-beam) generations—and were absent from both the source and the gold paraphrase—were treated as likely factual errors.
This approach enabled training on over 520,000 arc-level annotations derived from the ParaNMT-50M dataset.
Key Results and Insights
DAE demonstrates that dependency arcs serve as effective atomic semantic units for factuality. The model achieved 83.6% accuracy on the CNN/DM summary reranking task, nearly matching the human baseline of 83.9%. It significantly outperformed standard sentence-level NLI models and more complex QA-based metrics like QAGS.
One of the primary advantages of DAE is its interpretability. By evaluating each arc, the model provides a "heat map" of the sentence, allowing developers to see exactly which relationship (e.g., a misattributed adjective or a wrong subject) caused the factual failure. Furthermore, DAE proved more robust than standard NLI models, which are often distracted by simple lexical overlap rather than structural truth.
Limitations and Future Work
While highly effective, DAE has notable limitations:
- Omission Errors: It checks the truth of what is present but does not detect missing information from the source.
- Linguistic Nuance: The model can struggle with negation ("not") or quantification ("every") if these are not captured within the filtered arc set.
- Parser Dependency: The system's performance is inherently limited by the accuracy of the underlying dependency parser.
Future research may explore using more complex structured representations, such as Abstract Meaning Representation (AMR), to capture even deeper semantic relationships.
Summary updated