Large Language Models (LLMs) often struggle with formal logic, not because they lack the capacity for complex operations, but because they prioritize statistical correlations over causal or structural reasoning. When faced with a high-complexity logical problem, models frequently fail to generalize or lose track of the task's rules.
A growing body of research suggests that the solution lies in complexity-ordered curriculum learning: a strategy that sequences tasks or demonstrations from simple to complex. By mastering basic logical "primitives" before attempting multi-step synthesis, models can stabilize their reasoning paths and navigate vast search spaces that would otherwise be computationally or logically intractable.
The Compositional Gap and the Need for Primitives
The fundamental challenge in logical reasoning is the "compositional gap." As Fu demonstrates, models may learn individual logical building blocks (primitives) but fail to combine them when presented with complex, unseen tasks. This failure is often rooted in how neural networks naturally generalize. Abbe et al. uncover a mathematical "Occam’s Razor" in gradient descent, showing that models have an inherent bias toward low-degree (simpler) solutions.
To overcome this, researchers use curricula to build "logical supports." By starting with simple Boolean inputs or basic rules, a model establishes a foundation that makes higher-order reasoning manageable. For example, Cheng argues that a graduated curriculum—moving from simple rules to complex proofs—improves generalization by helping the model distinguish causal links from statistical noise.
Staged Training for Robust Logic
In training scenarios, curriculum learning acts as a stabilizer. When teaching models to interact with Knowledge Graphs or formal logic, the search space for a correct answer grows exponentially with the number of logical steps.
- Logical Operations: Xia et al. show that starting with simple queries and gradually introducing multi-hop structures and negations prevents the model from being overwhelmed.
- Semantic Parsing: In robotics, Wang et al. utilize a curriculum of short, simple logical formulas to help models learn to map natural language to executable code.
- Formal Proofs: For lifelong learning in mathematics, Kumarappan et al. sequence tasks by proof length, allowing the model to learn the "grammar" of a domain before tackling advanced theorems.
This staged approach ensures that the model does not get lost in the "combinatorial explosion" of possible reasoning paths, as noted by Kumarappan et al..
In-Context Curricula: Prompting as Pedagogy
The principles of curriculum learning are equally effective when applied to prompt engineering, often referred to as In-Context Curriculum Learning (ICCL). Instead of providing random examples, researchers order few-shot demonstrations by their structural or logical complexity.
Wu 0001 et al. introduce the "Chain-of-Quizzes" (CoQ) framework, which treats examples as a "spiral" chain of increasing difficulty. This mimics human pedagogy, allowing the model to use simpler solved instances as context for more challenging steps. Similarly, Wang finds that sorting demonstrations from simple to complex improves performance on knowledge base question generation by establishing basic task patterns early in the prompt.
This sequencing is not just about ease; it is about task-set maintenance. Lewis et al. show that in-context curricula help models succeed at high-complexity cognitive tasks (like the 10-back task) by reinforcing the rules through progressive examples, proving that the failure of LLMs on these tasks is often a failure of instruction-following rather than a memory bottleneck.
Self-Generated Learning Paths
Recent work has even enabled models to construct their own curricula. Luo et al. propose a strategy where the model generates and solves simpler versions of a target problem first. By creating this internal mini-curriculum, the model builds a high-quality set of reasoning steps that guide it toward the final solution.
Whether through training or prompting, the evidence across these papers—from Ghosh et al. to Ma 0001 et al.—converges on a single intuition: to solve the most difficult logical problems, a model must first be guided through the simple ones.
Key Foundational Work
QuExEnt: Improved Zero-Shot Classification from Explanations Through Quantifier Modeling and Curriculum Learning (2022) — contributed 1 concept elements to this direction.
Learning a natural-language to LTL executable semantic parser for grounded robotics (2020) — contributed 1 concept elements to this direction.