Introduction
While Large Language Models (LLMs) have demonstrated significant reasoning potential through prompting techniques like Chain-of-Thought (CoT) and Tree-of-Thought (ToT), these approaches often encounter performance plateaus due to their inherent rigidity and heavy reliance on task-specific prompt engineering. Current paradigms typically follow predefined, static reasoning paths that fail to adapt to the evolving complexity of a problem as it unfolds.
The "Learn to Think" (L2T) framework addresses these bottlenecks by modeling the reasoning process as a dynamic graph. By integrating Graph Neural Networks (GNNs) and reinforcement learning, L2T enables LLMs to bootstrap their own logic. This allows for flexible navigation of problem spaces—including backtracking and strategy shifts—without requiring meticulously hand-crafted prompts for every new task.
The L2T Framework
The L2T architecture moves beyond static text generation by treating reasoning as a structured search problem guided by a learned policy.
1. Reasoning Process Graph (RPG)
At the heart of L2T is the Reasoning Process Graph $G = \{V, E\}$, where each node represents an intermediate "thought" or reasoning step. Unlike linear chains, this graph structure allows the model to explore multiple branches and revisit previous states. Nodes are dynamically classified into categories such as Continue, Stop, Backtrack, or Final Result.
2. GNN-based Mode Selection
To navigate this graph efficiently, L2T employs a lightweight Graph Neural Network (GNN). This module analyzes the current topology of the RPG to predict the optimal "reasoning mode." Instead of generating text directly, the GNN adjusts critical hyperparameters—such as temperature, top-p, and branching factors—allowing the LLM to adapt its creative or precise "thinking style" based on the current state of the problem.
3. Reinforcement Learning Optimization
The GNN is trained using Proximal Policy Optimization (PPO). The system is rewarded for reaching correct solutions and receives intermediate "shaping" rewards for logical progress. This training allows the model to internalize effective search strategies across diverse domains, effectively learning how to think rather than just what to output.
Experimental Evaluation
The authors benchmarked L2T against state-of-the-art methods (CoT, ToT, GoT, and AoT) using GPT-4o as the backbone. The evaluation spanned four distinct task types:
- Sudoku (3x3 to 5x5): Logic and constraint satisfaction.
- Game of 24: Mathematical search and arithmetic.
- TruthQuest: Logical verification and suppositional reasoning.
- Creative Writing: Open-ended conceptual generation.
A key component of the evaluation was the "w/o TSP" (without Task-Specific Prompts) setting, which tested the framework's ability to generalize without manual guidance.
Results and Key Insights
L2T demonstrated superior performance and efficiency across all benchmarks:
- Accuracy: In 4x4 Sudoku, L2T achieved 98.46% accuracy, significantly outperforming ToT's 72.31%. In the Game of 24, it reached 80.42% accuracy, surpassing the next best method (AoT) at 74.23%.
- Token Efficiency: L2T is significantly more economical. For the Game of 24, it used an average of 4.68k tokens per case, compared to 11.60k for ToT and 7.56k for GoT.
- Robustness to Prompting: While traditional methods saw performance drops of up to 50% when task-specific prompts were removed, L2T remained stable. Its logic is embedded in the GNN's policy rather than the text prompt.
- Adaptive Strategies: Analysis showed the GNN naturally learned task-appropriate behaviors, such as increasing diversity (higher temperature/top-p) for creative writing while maintaining precision for mathematical tasks.
Conclusion
The L2T framework represents a shift from static "prompting" to dynamic "learning" in LLM reasoning. By leveraging graph representation learning, models can navigate complex logic spaces with greater flexibility and efficiency. While the GNN requires an initial training phase, the resulting system is more robust, token-efficient, and capable of solving complex problems without the need for rigid, hand-crafted instructions.
Summary updated