TL;DR
Existing logical reasoning systems for large language models (LLMs) excel at formal proofs over simple, unambiguous toy worlds, but break down when confronted with high semantic abstraction and conflicting viewpoints. The authors introduce LogicAgent, a framework that translates natural language into formal logic using structuralist semantics (Greimas’ Semiotic Square) to reason from multiple perspectives ($S_1$, contradictory $\neg S_1$, contrary $S_2$, and $\neg S_2$). Alongside it, they introduce RepublicQA, a philosophically grounded benchmark derived from Plato’s Republic that couples college-level linguistic complexity with multi-hop logical deduction.
What Problem Does This Paper Address?
While LLMs have made strides in multi-step deductive reasoning, standard benchmarks (like ProofWriter or ProntoQA) evaluate models in sterile, template-driven microworlds. These tasks test logical complexity (chains of deduction) while deliberately suppressing semantic complexity (abstract concepts, ambiguous contexts, and non-binary oppositions).
In real-world human discourse—such as legal analysis, ethics, and philosophy—reasoning requires grappling with nuanced definitions and distinct forms of opposition:
- Contradiction (e.g., true vs. false, where exactly one must hold under the law of excluded middle).
- Contrariety (e.g., just vs. unjust, where both statements cannot be simultaneously true, but both can be false).
Standard symbolic solvers and chain-of-thought methods frequently fail when premises require interpreting abstract semantics alongside deep reasoning chains.
Methods
LogicAgent bridges structuralist semiotics and classical First-Order Logic (FOL) through an automated three-stage agentic pipeline:
[ Natural Language Input ]
│
▼
1. Semantic Structuring Stage
• Constructs Greimas' Semiotic Square: S₁ (primary), ¬S₁ (contradictory),
S₂ (contrary), and ¬S₂ (sub-contrary).
• Validates formulas with an Existential Import Check (EIC) to prevent
vacuous truth, followed by a Context-Free Grammar (CFG) parser.
│
▼
2. Logical Reasoning Stage
• Translator: Maps premises to FOL predicates via typed conventions.
• Planner: Outlines a goal-directed reasoning trajectory without issuing a verdict.
• Solver: Deduces step-by-step intermediate conclusions for both S₁ and ¬S₁.
│
▼
3. Reflective Verification Stage
• Direct Resolution: Adopted if S₁ and ¬S₁ yield logically complementary verdicts.
• Quick Reflection: Triggered if one branch returns "Uncertain."
• Deep Reflection: Triggered when both branches conflict (e.g., both True or both False).
Resolves paradoxes using structural semiotic implications (S₁ ⇒ ¬S₂ and S₂ ⇒ ¬S₁).
Datasets
- RepublicQA (Introduced): 600 double-annotated instances grounded in Plato’s Republic, Aristotle’s Metaphysics, and the Nicomachean Ethics. It presents a balanced 3-way distribution (True: 32.5%, False: 30.0%, Uncertain: 37.5%) with a college-level reading grade (Flesch–Kincaid Grade Level of 11.94).
- Established Benchmarks: Evaluated for generalization across ProntoQA (5-hop), ProofWriter (depth-5, open-world assumption), FOLIO (expert-curated FOL/NL pairs), and ProverQA (hard split, 6–9 steps).
Key Insights and Ideas
- Contrary reasoning is fundamentally harder than contradiction: Current LLM-based solvers often confuse contradictory negations with contrary stances. Across all benchmarks, model accuracy on contrary relations consistently lagged behind overall performance.
- The "over-planning" threshold: On ProverQA, removing the planning module actually improved accuracy from 68.60% to 75.00%. While the dataset’s gold chains required 6–9 steps, the planner often generated over 10 steps, demonstrating that over-extended reasoning trajectories can introduce failure modes.
- Modality synergy: Ablation experiments revealed that purely symbolic logic (FOL-only) struggles on RepublicQA (dropping to 55.50%), because abstract philosophical discourse relies on contextual pragmatics. Conversely, removing FOL drops ProofWriter from 71.95% to 33.33%. Peak performance requires combining natural language's contextual grounding with FOL's inferential rigor.
Key Results
- RepublicQA Benchmark: LogicAgent outperformed all baseline models (including Aristotle, SymbCoT, and Logic-LM), achieving 82.50% on Qwen2.5-32B (+6.50 over Aristotle) and 87.00% on GPT-4o (+4.50 over Aristotle).
- Cross-Benchmark Generalization: Across standard logic benchmarks using Qwen2.5-32B, LogicAgent delivered an average gain of +7.05%:
- ProntoQA: 97.80% (+2.60 over best baseline)
- ProofWriter: 71.95% (+7.28 over best baseline)
- FOLIO: 79.90% (+7.97 over best baseline)
- ProverQA: 68.60% (+6.20 over best baseline)
- Ablation Studies: Removing the semiotic square caused the steepest performance decline (average score dropped from 75.74% to 67.58%), confirming the importance of multi-perspective semantic expansion.
Limitations and Future Work
- Planning Fragility: LogicAgent inherits the base LLM’s underlying planning constraints. When the model over-estimates reasoning depth or misplans deduction, semantic verification cannot always rescue the execution.
- Computational Cost: Multi-path deduction ($S_1$, $\neg S_1$, and conditionally $S_2$) and FOL formalization introduce noticeable overhead. The full system requires ~18.4k tokens per sample, and planning increases execution time by roughly 78%.
- Verification Inconsistency: At low rates, the reflective verification module experienced hallucinations, occasionally outputting a final verdict that contradicted its own generated intermediate verification steps.
Summary updated