Introduction
Post-training large language models (LLMs) using reinforcement learning (RL), such as Reinforcement Learning from Human Feedback (RLHF), typically relies on a single scalar reward based on the final outcome: "Did the model produce the correct answer?" While effective for many tasks, this "outcome-only" approach often overlooks the internal logical structure required for complex reasoning. For tasks like logic puzzles, the sequence of deductions is as critical as the final solution.
Previous research indicates that models trained on "solver-ordered" data—where steps follow a logical progression—significantly outperform those trained on randomized steps. However, high-quality ordered datasets are not always available for supervised fine-tuning (SFT). This paper investigates whether logical structure can be "injected" during the RL phase using a simple scalar reward, even when the initial SFT data is unordered.
Methods
The researchers employed a two-stage training pipeline focused on the Zebra puzzle dataset (Einstein’s puzzles), which requires assigning attributes to entities based on clues.
Datasets
The study focused on the Zebra puzzle dataset. To ensure consistency in measuring trajectory order, the authors filtered the dataset to include only puzzles requiring exactly 9 actions to solve.
1. Standard Fine-Tuning (SFT)
A GPT-2 style Transformer was first trained on Zebra puzzles where the target sequences were randomly shuffled. This ensured the model had no prior exposure to a logical step-by-step progression before the RL phase.
2. GRPO Post-Training
The model was then refined using Group Relative Policy Optimization (GRPO). A dual-reward system was introduced:
- Solved Reward ($R_{solve}$): A binary reward (1 for success, 0 for failure).
- Order Reward ($R_{order}$): A scalar reward that increases as the model’s output sequence aligns with a "canonical" logical solver's order.
To balance these heterogeneous rewards, the authors developed Bootstrapped Reward Scaling. By calculating the mean of each reward on a validation set at the start of RL, they normalized the components. This ensures that a specific mixture ratio (e.g., 99:1) accurately reflects the intended influence of each reward, preventing one from dominating the other due to scale differences.
Key Results
The experimental results demonstrate that incorporating even a minor ordering signal significantly boosts performance over optimizing for correctness alone.
- SFT Baseline (Random Order): 0.279 accuracy.
- RL (Correctness Only - 1:0 ratio): 0.326 accuracy.
- RL (Mixed Reward - 99:1 ratio): 0.363 accuracy.
The study found that the optimal performance peak occurred at a 99% solve : 1% order weighting. This suggests that a "light shaping" signal is sufficient to guide the RL process toward more robust reasoning trajectories. Notably, every mixture including an ordering component outperformed the "solve-only" baseline.
Discussion and Limitations
The ability to learn logical progression from scalar rewards—without ever seeing correctly ordered text sequences during training—is a significant finding. However, several limitations remain:
- Scalability: The experiments were conducted on a relatively small model (GPT-2) and a specific puzzle type. Future work should test these "ordering hints" on larger architectures and more diverse reasoning tasks.
- Dynamic Calibration: The current bootstrapping method uses fixed scaling factors. Implementing dynamic scales that update as the policy evolves could further improve stability.
- Multi-modal Trajectories: The current approach assumes a single "canonical" order. Adapting the reward system to recognize multiple valid logical paths is a necessary step for generalizing to more complex real-world problems.
Summary updated