Abstract
Large Language Models (LLMs) often struggle with "unfaithful reasoning," where generated explanations do not logically lead to the final conclusion. We present Logic-LM, a neuro-symbolic framework that mitigates this by utilizing LLMs as translators to convert natural language into formal symbolic representations. By offloading the actual inference to deterministic symbolic solvers, Logic-LM ensures mathematical soundness and transparency. Our evaluation across five benchmarks shows that Logic-LM significantly outperforms standard prompting and Chain-of-Thought (CoT) methods, particularly as problem complexity and reasoning depth increase.
Introduction
While state-of-the-art LLMs like GPT-4 exhibit remarkable linguistic capabilities, they remain probabilistic "black boxes" that lack the 100% precision required for rigorous logical reasoning. In contrast, traditional symbolic solvers are perfectly faithful but require rigid, structured input that is difficult for humans to provide directly. Logic-LM bridges this gap by combining the natural language flexibility of LLMs with the logical rigor of symbolic solvers.
The Logic-LM Framework
Logic-LM operates through a three-stage pipeline designed to separate problem representation from resolution:
- Problem Formulator: The LLM translates natural language text into a symbolic representation, such as Logic Programming (Prolog), First-Order Logic (FOL), or Constraint Satisfaction Problems (CSP).
- Symbolic Reasoner: The structured formulation is executed by an external deterministic solver (e.g., Z3, Pyke, or Prover9). These solvers perform inference based strictly on the provided premises.
- Result Interpreter: The solver's output is translated back into a human-readable answer by the LLM.
Additionally, a Self-Refiner module allows the system to handle syntax errors. If a solver fails due to a malformed symbolic string, the error message is fed back to the LLM for automated debugging and reformulation.
Experimental Evaluation
We evaluated Logic-LM on five diverse logical reasoning datasets: ProofWriter and PrOntoQA (deductive reasoning), FOLIO (complex first-order logic), LogicalDeduction (constraint-based ordering), and AR-LSAT (analytical reasoning from law school admission tests).
Key Results
Logic-LM achieved an average performance boost of 39.2% over standard prompting and 18.4% over Chain-of-Thought (CoT) across all datasets. Notably, the framework showed significant synergy with advanced models; using GPT-4 as a backbone, Logic-LM improved accuracy on the challenging AR-LSAT dataset from approximately 35% to over 43%.
Robustness to Reasoning Depth
A critical weakness of standard LLMs is the rapid decline in accuracy as the number of reasoning steps increases. Logic-LM demonstrates remarkable stability in these scenarios. Once a problem is correctly translated into symbolic form, the external solver handles multi-step inference with the same precision as single-step problems.
Discussion and Conclusion
The success of Logic-LM highlights that LLMs are currently more effective as "translators" of logic than as "reasoners." By shifting the computational burden to symbolic solvers, we achieve more faithful and explainable results.
However, the system remains limited by the "translation bottleneck"—if the LLM misinterprets the initial natural language, the solver will perfectly solve the incorrect problem. Future work will explore more expressive logic frameworks and hybrid approaches like Statistical Relational Learning to better handle the nuances and uncertainties of real-world human language.
Summary updated