DICE: Structured Reasoning in LLMs through SLM-Guided Chain-of-Thought Correction

Yiqi Li, Yusheng Liao, Zhe Chen, Yanfeng Wang, Yu Wang

TL;DR

This paper introduces DICE, a novel framework that uses small language models (SLMs) to guide large language models (LLMs) in generating structured outputs for complex reasoning tasks. DICE effectively addresses the trade-off between LLMs' reasoning capabilities and their adherence to specific output formats, achieving state-of-the-art performance.

What Problem Does This Paper Address?

Large Language Models (LLMs) excel at complex reasoning but often struggle with strictly adhering to specific output formats (e.g., JSON, XML). This instruction-following ability degrades as LLMs scale up, leading to a counter-intuitive trade-off where larger, more capable models are sometimes worse at following instructions. Fine-tuning LLMs directly is computationally expensive and often infeasible for API-only models. Existing methods for adapting LLMs to structured tasks often fail to balance reasoning quality with format adherence or suffer from high mis-correction rates. This paper tackles the challenge of making LLMs reliably produce structured outputs for reasoning tasks without sacrificing their core reasoning abilities or requiring extensive fine-tuning.

Methods

DICE employs a two-stage approach:

  1. Structured Chain-of-Thought (CoT) Adaptation Dataset Construction:
    • LLMs first generate natural language responses to avoid format-induced errors.
  2. Then, trained SLMs analyze these responses. A two-stage process generates rationales and refines predicted answers, guided by the LLM's output and the ground truth. This process aims to create high-quality CoT data that helps SLMs learn to correct and structure outputs.
  3. Dual-Tuning Strategy for SLMs:
    • Supervised Fine-Tuning (SFT): SLMs are initially fine-tuned using the constructed CoT dataset to learn format specifications and the analyze-then-answer pattern.
  4. Group Relative Policy Optimization (GRPO): A more granular fine-tuning method is applied to further optimize the SLM. This stage focuses on rewarding correct format and final answers, while downplaying the rationale's direct contribution to the reward, to prevent the SLM from over-prioritizing rationale generation.

During inference, the trained SLM systematically analyzes and refines the LLM's natural language output into the desired structured format.

Datasets

The paper evaluates DICE on five reasoning benchmarks:

  • GSM8K: Math word problems.
  • MATH: More challenging math problems.
  • CommonsenseQA: Commonsense reasoning.
  • MedQA-zh: Domain-specific medical reasoning (simplified Chinese subset).
  • StrategyQA: Strategic reasoning.

For structured output generation, these datasets are adapted to XML, JSON, and YAML formats. The paper also mentions the construction of a new structured CoT adaptation dataset using a two-stage rationale generation process.

Key Insights and Ideas

  • Scale-Instruction Following Trade-off: The paper highlights a surprising negative correlation between LLM scale and instruction-following ability in reasoning tasks. Larger models, while better at reasoning, can become worse at adhering to specific output formats.
  • SLM as a "Corrector": DICE leverages SLMs not just to generate outputs but to actively correct and structure LLM outputs, acting as a sophisticated post-processing step.
  • Analyze-then-Answer Paradigm: The core innovation lies in guiding SLMs to first analyze LLM outputs and then generate the structured answer, rather than simply trying to generate the structured output directly.
  • "Alignment Tax" Avoidance: By not fine-tuning the LLM itself, DICE circumvents the "alignment tax" (performance degradation on other tasks) and preserves the LLM's general knowledge.
  • Plug-and-Play Adaptability: DICE is designed to be a lightweight framework that can adapt LLMs to structured tasks without modifying the LLM's parameters, making it broadly applicable.

Key Results

  • DICE achieves significant improvements, with average gains of 35.4% in format accuracy and 29.4% in content accuracy compared to LLMs with 2-shot prompts.
  • It consistently outperforms other baselines across nearly all evaluated datasets, achieving state-of-the-art (SOTA) performance.
  • DICE demonstrates near-perfect format accuracy (approaching 100%) across evaluated datasets, while simultaneously improving content accuracy.
  • The framework shows strong generalizability, adapting various large models to XML-constrained reasoning tasks with average F-Acc exceeding 99.5%.
  • DICE achieves superior performance in both F-Acc and C-Acc compared to collaboration-based baselines, with average content accuracy gains of 18.4%, 16.1%, and 3.6% over Aligner, BBox-Adapter, and CoBB, respectively.
  • While introducing a modest latency overhead (approx. 20% compared to ICL), the performance gains (content accuracy improvement over 70% on MATH) are substantial.

Limitations and Future Work

  • Computational Overhead: The approach introduces additional computational overhead as it involves both an LLM and an SLM. For simple questions, invoking the LLM might be unnecessary.
  • Future Work: The authors suggest incorporating a mechanism to assess question complexity beforehand, selectively engaging the LLM only when necessary to optimize resource usage.

Summary updated

Abstract

When performing reasoning tasks with user-specific requirements, such as strict output formats, large language models (LLMs) often prioritize reasoning over adherence to detailed instructions. Fine-tuning LLMs on supervised datasets to address this is impractical due to high computational costs and limited parameter access. To tackle this, we propose DICE, a lightweight framework that guides small language models (SLMs) to refine LLMs' outputs through chain-of-thought (CoT) correction. DICE decouples the process by first prompting LLMs to generate natural language responses, then using trained SLMs to analyze and refine these outputs to meet structured output specifications. This framework preserves LLMs' broad knowledge and reasoning capabilities while ensuring the outputs conform to user demands. Specifically, DICE first constructs structured CoT adaptation datasets via a two-stage method and subsequently applies a dual-tuning strategy to fine-tune SLMs for generating structured outputs in an analyze-then-answer pattern. Experiments demonstrate that DICE improves the average format accuracy and content correctness of LLM outputs by 35.4\% and 29.4\%, respectively, achieving state-of-the-art (SOTA) performance over other competitive baselines.
BibTeX
@inproceedings{li2025dice,
  author = {Yiqi Li and Yusheng Liao and Zhe Chen and Yanfeng Wang and Yu Wang},
  title = {{DICE: Structured Reasoning in LLMs through SLM-Guided Chain-of-Thought Correction}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure