Introduction
Large Language Models (LLMs) have demonstrated remarkable capabilities in complex reasoning tasks. However, a significant bottleneck persists: as these models scale, they often struggle to adhere to strict user-specified output formats, such as JSON, XML, or YAML. This "instruction-following degradation" creates a counter-intuitive trade-off where a model might solve a difficult mathematical proof correctly but fail to wrap the final answer in the required machine-readable tags.
DICE (guiDing SLMs to thInk with Chain-of-thought corrEction) addresses this by decoupling the reasoning and formatting processes. By employing a small language model (SLM) as a dedicated corrector, DICE significantly improves both format adherence and content accuracy without requiring expensive fine-tuning of the primary LLM.
The Format-Reasoning Trade-off
Empirical evidence suggests that formatting accuracy does not scale linearly with model size. As shown in Figure 1, format accuracy for the Qwen2.5 series peaks at mid-scale (approximately 14B parameters) and declines sharply as models reach 32B and 72B parameters. This "Scale Paradox" highlights that while larger models possess superior underlying reasoning skills, their ability to maintain strict structural constraints during long-form reasoning is compromised.
The DICE Framework Architecture
The DICE framework introduces a collaborative pipeline where the LLM serves as the "Thinker" and a trained SLM serves as the "Formatter/Corrector." Instead of directly parsing the LLM's output, the SLM is trained to perform "Chain-of-Thought Correction." It analyzes the LLM's reasoning path, identifies potential errors, and then generates a reformatted, corrected output.
Dual-Tuning Strategy
The SLM is optimized using a two-stage training process:
- Supervised Fine-Tuning (SFT): The model learns the basic "analyze-then-answer" pattern using a structured dataset. If the SLM initially fails to generate a correct rationale, it is provided with ground-truth "hints" to facilitate the learning of valid reasoning paths.
- Group Relative Policy Optimization (GRPO): This reinforcement learning stage uses a reward function to penalize format and content errors. By calculating relative rewards within groups of outputs, DICE avoids the need for a separate critic model and prevents the SLM from prioritizing rationale length over accuracy.
Experimental Evaluation
DICE was evaluated across five diverse reasoning benchmarks, including GSM8K, MATH, CommonsenseQA (CSQA), MedQA-zh, and StrategyQA. The framework demonstrated exceptional generalizability across XML, JSON, and YAML formats.
Key Results
- Format Accuracy: On the MATH dataset, where standard In-Context Learning (ICL) often yields less than 5% format accuracy, DICE achieved a near-perfect 99.6%.
- Content Improvement: Beyond structural fixes, the SLM's ability to "double-check" reasoning led to an average content correctness boost of 29.4%.
- Efficiency and Generalizability: Utilizing a 1.5B parameter SLM, DICE introduces only a ~20% inference latency overhead. Furthermore, an SLM trained on one model's outputs (e.g., Qwen) can effectively correct outputs from different architectures like Llama-3 or GPT-4, making it a highly portable solution.
Conclusion
DICE provides a lightweight, effective solution to the instruction-following challenges inherent in large-scale reasoning models. By leveraging a dual-tuned SLM to guide the correction process, the framework eliminates the "alignment tax" typically associated with fine-tuning while setting new state-of-the-art benchmarks for structured reasoning tasks.
Summary updated