CLR-Fact: Evaluating the Complex Logical Reasoning Capability of Large Language Models over Factual Knowledge

Tianshi Zheng, Jiaxin Bai, Yicheng Wang, Tianqing Fang, Yue Guo, Yauwai Yim, Yangqiu SongarXiv 2024 · DOI · arXiv

Abstract

Large Language Models (LLMs) have demonstrated impressive capabilities in memorizing vast amounts of factual information. However, their ability to perform complex logical reasoning over these facts remains under-explored. This paper introduces CLR-Fact, a comprehensive evaluation framework designed to test LLMs on queries involving set intersections, unions, and negations derived from knowledge graphs. Our findings highlight a "logical asymmetry": LLMs are proficient at set unions but struggle significantly with intersections—the fundamental "filtering" step of complex logic. Furthermore, performance drops sharply in specialized domains like biomedicine compared to general knowledge.

Introduction

While LLMs can easily answer simple factual questions like "What is the capital of France?", real-world intelligence often requires synthesizing multiple facts to solve complex problems. For instance, a medical researcher might need to identify proteins associated with multiple specific diseases (an intersection of sets) or find drugs that interact with one gene but not another (negation). Existing benchmarks primarily measure fact retrieval (memorization), leaving a gap in our understanding of how LLMs perform complex logical reasoning over those facts. CLR-Fact addresses this by systematically measuring these capabilities across general and biomedical domains.

The CLR-Fact Framework

The researchers developed a three-step pipeline to evaluate complex reasoning:

  1. Logical Query Sampling: A recursive algorithm samples complex logical formulas (e.g., projection, intersection, union, negation) from existing Knowledge Graphs (KGs).
  2. Question Generation: Abstract formulas are translated into natural language using manually crafted "Relation Templates" to ensure the logical structure remains intact for LLM consumption.
  3. Systematic Evaluation: Eight state-of-the-art models, including GPT-4o, Llama-2, and Mixtral, were tested using prompting strategies such as In-Context Learning (ICL), Chain-of-Thought (CoT), and structural demonstration selection.
Overview of the CLR-Fact evaluation framework.

Datasets

The benchmark consists of 5,200 complex reasoning questions spanning 26 logical patterns, utilizing two primary sources:

  • FB15k-237: A subset of Freebase covering general world knowledge (celebrities, movies, sports).
  • PrimeKG: A large-scale biomedical knowledge graph containing data on drugs, diseases, and genes.

Key Insights

The Intersection Bottleneck

A primary finding is the "asymmetry" in set operations. LLMs are relatively capable of performing unions (combining sets) but fail significantly at intersections (finding common elements). Intersections require "subtractive" logic—actively excluding candidates that don't meet all criteria—which appears inherently more difficult for current transformer architectures.

Domain Sensitivity and Negation

Models perform significantly worse on biomedical data (PrimeKG) than on general knowledge (FB15k-237). This suggests that while LLMs "know" medical facts, their grasp of the relationships between those facts is shallower than their grasp of general history. Additionally, performance tanks when questions involve negation, as models struggle to process set complement operations.

Key Results

The evaluation revealed a clear "performance tax" for complexity. As reasoning depth (the number of "hops") increased, accuracy dropped consistently across all models.

Performance comparison of various LLMs on FB15k-237 and PrimeKG datasets.
  • Top Performer: GPT-4o achieved the highest scores, though it still struggled with the biomedical domain (32.25% on FB15k-237 vs. 18.02% on PrimeKG).
  • CoT Boost: Chain-of-Thought prompting provided a massive performance lift, particularly for the most complex queries, improving GPT-3.5-turbo's performance on multi-type variety questions by nearly 15%.

Conclusion

CLR-Fact exposes critical weaknesses in the logical reasoning capabilities of current LLMs, particularly regarding "subtractive" logic like intersections and negations. While Chain-of-Thought and structural alignment in prompting help, there is a significant gap between fact retrieval and complex reasoning. Future work should focus on improving LLM performance on these specific logical gates and exploring better integration of Retrieval-Augmented Generation (RAG) for multi-step reasoning.

Summary updated

Abstract

While large language models (LLMs) have demonstrated impressive capabilities across various natural language processing tasks by acquiring rich factual knowledge from their broad training data, their ability to synthesize and logically reason with this knowledge in complex ways remains underexplored. In this work, we present a systematic evaluation of state-of-the-art LLMs' complex logical reasoning abilities through a novel benchmark of automatically generated complex reasoning questions over general domain and biomedical knowledge graphs. Our extensive experiments, employing diverse in-context learning techniques, reveal that LLMs excel at reasoning over general world knowledge but face significant challenges with specialized domain-specific knowledge. We find that prompting with explicit Chain-of-Thought demonstrations can substantially improve LLM performance on complex logical reasoning tasks with diverse logical operations. Interestingly, our controlled evaluations uncover an asymmetry where LLMs display proficiency at set union operations, but struggle considerably with set intersections - a key building block of logical reasoning. To foster further work, we will publicly release our evaluation benchmark and code.
BibTeX
@misc{zheng2024clr,
  author = {Tianshi Zheng and Jiaxin Bai and Yicheng Wang and Tianqing Fang and Yue Guo and Yauwai Yim and Yangqiu Song},
  title = {{CLR-Fact: Evaluating the Complex Logical Reasoning Capability of Large Language Models over Factual Knowledge}},
  year = {2024},
  eprint = {2407.20564},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2407.20564},
}
Figure