Abstract
Large Language Models (LLMs) have demonstrated impressive capabilities in memorizing vast amounts of factual information. However, their ability to perform complex logical reasoning over these facts remains under-explored. This paper introduces CLR-Fact, a comprehensive evaluation framework designed to test LLMs on queries involving set intersections, unions, and negations derived from knowledge graphs. Our findings highlight a "logical asymmetry": LLMs are proficient at set unions but struggle significantly with intersections—the fundamental "filtering" step of complex logic. Furthermore, performance drops sharply in specialized domains like biomedicine compared to general knowledge.
Introduction
While LLMs can easily answer simple factual questions like "What is the capital of France?", real-world intelligence often requires synthesizing multiple facts to solve complex problems. For instance, a medical researcher might need to identify proteins associated with multiple specific diseases (an intersection of sets) or find drugs that interact with one gene but not another (negation). Existing benchmarks primarily measure fact retrieval (memorization), leaving a gap in our understanding of how LLMs perform complex logical reasoning over those facts. CLR-Fact addresses this by systematically measuring these capabilities across general and biomedical domains.
The CLR-Fact Framework
The researchers developed a three-step pipeline to evaluate complex reasoning:
- Logical Query Sampling: A recursive algorithm samples complex logical formulas (e.g., projection, intersection, union, negation) from existing Knowledge Graphs (KGs).
- Question Generation: Abstract formulas are translated into natural language using manually crafted "Relation Templates" to ensure the logical structure remains intact for LLM consumption.
- Systematic Evaluation: Eight state-of-the-art models, including GPT-4o, Llama-2, and Mixtral, were tested using prompting strategies such as In-Context Learning (ICL), Chain-of-Thought (CoT), and structural demonstration selection.
Datasets
The benchmark consists of 5,200 complex reasoning questions spanning 26 logical patterns, utilizing two primary sources:
- FB15k-237: A subset of Freebase covering general world knowledge (celebrities, movies, sports).
- PrimeKG: A large-scale biomedical knowledge graph containing data on drugs, diseases, and genes.
Key Insights
The Intersection Bottleneck
A primary finding is the "asymmetry" in set operations. LLMs are relatively capable of performing unions (combining sets) but fail significantly at intersections (finding common elements). Intersections require "subtractive" logic—actively excluding candidates that don't meet all criteria—which appears inherently more difficult for current transformer architectures.
Domain Sensitivity and Negation
Models perform significantly worse on biomedical data (PrimeKG) than on general knowledge (FB15k-237). This suggests that while LLMs "know" medical facts, their grasp of the relationships between those facts is shallower than their grasp of general history. Additionally, performance tanks when questions involve negation, as models struggle to process set complement operations.
Key Results
The evaluation revealed a clear "performance tax" for complexity. As reasoning depth (the number of "hops") increased, accuracy dropped consistently across all models.
- Top Performer: GPT-4o achieved the highest scores, though it still struggled with the biomedical domain (32.25% on FB15k-237 vs. 18.02% on PrimeKG).
- CoT Boost: Chain-of-Thought prompting provided a massive performance lift, particularly for the most complex queries, improving GPT-3.5-turbo's performance on multi-type variety questions by nearly 15%.
Conclusion
CLR-Fact exposes critical weaknesses in the logical reasoning capabilities of current LLMs, particularly regarding "subtractive" logic like intersections and negations. While Chain-of-Thought and structural alignment in prompting help, there is a significant gap between fact retrieval and complex reasoning. Future work should focus on improving LLM performance on these specific logical gates and exploring better integration of Retrieval-Augmented Generation (RAG) for multi-step reasoning.
Summary updated