TL;DR
This paper introduces Logic-Thinker, a novel neuro-symbolic framework that bridges the gap between symbolic logic solvers and Large Language Models (LLMs) to generate more rigorous and concise reasoning chains (ThinkerCoT). Logic-Thinker achieves state-of-the-art performance on logical reasoning tasks and significantly reduces token output compared to existing "long chain-of-thought" methods, while also enhancing LLMs' general reasoning capabilities.
What Problem Does This Paper Address?
Large Language Models (LLMs) have shown impressive capabilities in various natural language tasks, but they often struggle with complex logical reasoning. While "long chain-of-thought" (LongCoT) prompting has improved reasoning by generating step-by-step explanations, it suffers from issues like redundancy, overthinking, and logical incoherence. These problems stem from LLMs' inherent difficulty in performing rigorous, deterministic logical deduction. This paper aims to equip LLMs with robust and concise logical reasoning abilities by integrating them with symbolic solvers.
Methods
Logic-Thinker is a neuro-symbolic framework composed of three core modules:
- Formulator: This module translates natural language logic reasoning problems into symbolic expressions. It includes specialized formulators for First-Order Logic (FOL) and Constraint Satisfaction Problems (CSP). For FOL, it uses a "semantic-enhanced FOL" (SeFOL) approach that incorporates semantic properties to mitigate information loss during formalization.
- LogicSolver: This module executes the symbolic expressions generated by the Formulator. It employs specialized solvers for FOL (using the Thinker engine) and CSP (using the Choco-solver library). The solver's execution process, including all reasoning paths, is recorded.
- CoT Generator: This module transforms the solver's execution process into concise and logically rigorous chains of thought, termed "ThinkerCoT." It extracts key reasoning paths and uses a style alignment procedure to ensure diverse language styles. For FOL, it converts the solver's tree structure into a directed acyclic graph (DAG) and traverses it. For CSP, it converts implications into a graph and uses a breadth-first search.
Datasets
The paper uses five logical reasoning datasets for evaluation:
- FOL: ProntoQA, ProofWriter, FOLIO
- CSP: LogicalDeduction, AR-LSAT
These datasets are standard benchmarks for evaluating logical reasoning capabilities.
Key Insights and Ideas
- Bridging Neuro-Symbolic and LLMs: The core idea is to leverage the deterministic and exhaustive reasoning of symbolic solvers and translate their internal processes into a format LLMs can understand and generate (ThinkerCoT). This overcomes the inherent limitations of LLMs in pure logical deduction.
- ThinkerCoT's Conciseness: ThinkerCoT is significantly more concise than traditional LongCoT, reducing token output by 73%-91% while maintaining or improving accuracy. This addresses the redundancy and inefficiency of current LLM reasoning methods.
- Rigor Enhances General Reasoning: Fine-tuning LLMs with ThinkerCoT data not only improves their logical reasoning but also enhances their broader reasoning capabilities in mathematics and science, demonstrating the value of rigorous logical thinking.
- Semantic Enhancement for FOL: The SeFOL approach, by incorporating semantic information like
subClassOf, significantly improves FOL formalization and reasoning accuracy, addressing a key weakness in previous neuro-symbolic methods.
Key Results
- State-of-the-Art Performance: Logic-Thinker achieves state-of-the-art (SOTA) performance on logical reasoning tasks, outperforming other neuro-symbolic systems and LLMs with CoT prompting.
- Accuracy Improvement: On average, Logic-Thinker achieved a significant accuracy improvement of 24.47% over direct GPT-4 reasoning and 15.68% over GPT-4 with CoT prompting.
- Token Reduction: ThinkerCoT achieved an overall accuracy improvement of 3.6% compared to LongCoT while reducing token output by 73%-91%.
- Enhanced General Reasoning: LLMs fine-tuned with ThinkerCoT showed substantial improvements on benchmarks for mathematics and science reasoning, with accuracy gains up to 37.73% on Multi-LogiEval.
Limitations and Future Work
- Formalization Information Loss: The formalization process still suffers from some degree of information loss, particularly affecting accuracy on datasets like FOLIO and AR-LSAT. Future work will focus on leveraging more semantic features to improve formalization quality.
- Unified Formalization: The current framework has separate formalization processes for FOL and CSP. The long-term goal is to develop a unified symbolic representation and a single solver for both types of problems, enabling a unified solution for logical reasoning tasks.
Summary updated