Introduction
Large Language Models (LLMs) often resolve linguistic ambiguity through statistical intuition rather than explicit logical reasoning. This paper introduces a new benchmark designed to evaluate whether models can navigate complex "definite descriptions"—phrases like "the head of state"—when they are subject to temporal shifts. By focusing on the distinction between de dicto and de re interpretations, the researchers highlight significant gaps in current AI reasoning capabilities.
The Core Ambiguity: De Dicto vs. De Re
The researchers focus on a specific linguistic puzzle involving temporal operators. Consider the sentence: "In 2010, the pope was a native speaker of German." This sentence is ambiguous depending on how the description "the pope" is interpreted:
- De dicto (of what is said): This refers to the individual who held the role at the specified time (Benedict XVI in 2010), making the statement true.
- De re (of the thing): This refers to the current holder of the role (Pope Francis) as they were in 2010. Since Francis is a native Spanish speaker, the statement would be false under this interpretation.
To solve these puzzles, a model must look at provided facts (premises) and determine which interpretation holds true without contradicting known regularities, such as the fact that a person's native language does not change over time.
Methodology and Dataset
The researchers developed a framework to generate these puzzles semi-automatically, resulting in the RADD-Wikidata-5-EN benchmark. The dataset consists of 500 instances derived from Wikidata across five distinct property pairs, such as military unit sizes and ship registry continents.
Each puzzle provides the LLM with a "closed-world" prompt containing:
- Premises: Factual relations from Wikidata.
- Regularities: Logical rules (e.g., "Military units do not change size class").
- The Question: An ambiguous sentence and two possible interpretations.
The study compared two prompting strategies: Direct answering and Chain-of-Thought (CoT), which requires the model to output its reasoning steps before providing a final answer.
Key Findings
The evaluation of models ranging from Koala-13B to GPT-4 revealed several critical insights into LLM reasoning capabilities.
The Chain-of-Thought Advantage
For almost all models, forced step-by-step reasoning significantly improved performance. GPT-4 emerged as the clear leader, achieving 83.2% accuracy when using CoT. Interestingly, the open-source OpenAssistant-33B outperformed GPT-3.5 in average accuracy (67.7% vs 64.8%), suggesting that model alignment and scale both play vital roles in logical disambiguation.
Inherent Biases and Reasoning Failures
A major discovery was the "De Dicto" bias: models have a strong inherent preference for literal role-based interpretations and often fail to even consider the de re (entity-based) possibility. Furthermore, error analysis showed that even advanced models like GPT-4 sometimes "hallucinate" logical constraints to justify incorrect answers, or provide post-hoc justifications for an answer they had already intuitively selected rather than reasoning their way to it.
Conclusion
While larger models like GPT-4 show promise in resolving temporal ambiguities, the significant performance gap between de dicto and de re interpretations suggests that LLMs still struggle with the underlying logic of how entities and roles evolve over time. Small models (under 30B parameters) performed near random levels, indicating that this type of logical disambiguation may be an emergent property of larger-scale models. Future work will expand this benchmark to include more languages and diverse types of linguistic ambiguity beyond temporal shifts.
Summary updated