Fewer is More: Boosting LLM Reasoning with Reinforced Context Pruning

Xijie Huang, Li Lyna Zhang, Kwang‐Ting Cheng, Mao Yang, Yang, Mao

Abstract: Large Language Models (LLMs) have shown impressive capabilities, yet they still struggle with math reasoning. In this work, we propose CoT-Influx, a novel approach that pushes the boundary of few-shot Chain-of-Thoughts (CoT) learning to improve LLM mathematical reasoning. Motivated by the observation that adding more concise CoT examples in the prompt can improve LLM reasoning performance, CoT-Influx employs a coarse-to-fine pruner to maximize the input of effective and concise CoT examples. The pruner first selects as many crucial CoT examples as possible and then prunes unimportant tokens to fit the context window. A math reasoning dataset with diverse difficulty levels and reasoning steps is used to train the pruner, along with a math-specialized reinforcement learning approach. As a result, by enabling more CoT examples with double the context window size in tokens, CoT-Influx significantly outperforms various prompting baselines across various LLMs (LLaMA2-7B, 13B, 70B) and 5 math datasets, achieving up to 4.55% absolute improvements. Remarkably, without any fine-tuning, LLaMA2-70B with CoT-Influx surpasses GPT-3.5 and a wide range of larger LLMs (PaLM, Minerva 540B, etc.) on the GSM8K. CoT-Influx serves as a plug-and-play module for LLMs and is compatible with most existing reasoning prompting techniques, such as self-consistency and self-verification.

Introduction

While Large Language Models (LLMs) have made massive strides in natural language tasks, multi-step mathematical reasoning remains a frontier challenge. Researchers have found that providing "Chain-of-Thought" (CoT) examples—step-by-step solutions—significantly helps models navigate complex logic. However, a significant bottleneck exists: the finite context window. Standard prompts can only accommodate a few detailed examples before running out of space, and simply adding more examples often leads to truncated responses or decreased accuracy as the model's output space is "squeezed."

Existing solutions like context window extension require expensive fine-tuning, while standard prompt compression techniques (e.g., LLMLingua) often accidentally remove tokens vital for math, such as numerical values or formatting cues. To address this, we introduce CoT-Influx, a plug-and-play module designed to maximize information density by selecting the most effective examples and stripping away redundant natural language filler.

Methodology: Coarse-to-Fine Reinforced Pruning

CoT-Influx utilizes a two-stage "coarse-to-fine" pruner to optimize the prompt for a specific target LLM. The architecture is designed to be model-aware, recognizing that a 7B model may require different support than a 70B model.

  1. Shot Pruner (Coarse): This stage selects the most helpful "shots" or examples from a large candidate pool for the specific target model.
  2. Token Pruner (Fine): Within the selected examples, this stage identifies and removes redundant natural language tokens while strictly preserving crucial mathematical and structural tokens.
The CoT-Influx framework overview, showing the pipeline from example generation to reinforced pruning.

The REINFORCE Training Loop

Because pruning is a discrete operation, standard gradient descent cannot be used. Instead, we employ REINFORCE, a reinforcement learning algorithm. The pruner is trained using a multi-objective reward function that balances three key factors: LLM next-token prediction loss, reasoning accuracy ($R_{Acc}$), and length constraints to prevent exceeding the context window. To stabilize training, we utilize a "difficulty-aware" filter, initially training the pruner on easier problems to ensure the RL agent receives consistent positive signals.

Experimental Setup and MRD³ Dataset

To train the pruner effectively, we developed the Math Reasoning Dataset with Diverse Difficulty (MRD³). We curated 9.7K question-answer pairs from benchmarks like GSM8K and AQuA, then used GPT-4 to "evolve" these problems—rewriting them to vary in complexity and depth. This diversity allows the pruner to learn which types of examples best assist models of different sizes.

Key Results and Insights

CoT-Influx demonstrates that "fewer is more" when it comes to token count but "more is more" when it comes to the number of examples. By pruning redundant words (e.g., shortening "Let's think step by step. Step 1: Find the total..." to "1: Find total..."), the system can fit nearly double the number of few-shot examples into the same token limit.

Performance of CoT-Influx on LLaMA2-7B and 13B, showing significant accuracy gains over standard 8-shot and TopK baselines.

Our evaluations across five major math benchmarks (GSM8K, AddSub, MultiArith, Svamp, and Singleeq) yielded several key findings:

  • Outperforming Proprietary Models: LLaMA2-70B with CoT-Influx achieved 59.6% accuracy on GSM8K, surpassing GPT-3.5 (57.1%) and massive models like PaLM and Minerva (540B) without any fine-tuning of the target LLM.
  • Model-Specific Preferences: Smaller models (LLaMA2-7B) perform better when provided with simpler CoT examples, whereas larger models (70B) benefit more from complex, high-difficulty examples.
  • Efficiency: The pruner achieved a 4.28x compression ratio on examples with negligible overhead, adding only ~1.4GB of VRAM usage and a 1.7% increase in latency.

Conclusion

CoT-Influx provides a highly portable, effective method for boosting LLM reasoning. By treating prompt optimization as a reinforced pruning task, it enables models to leverage more examples without the computational cost of context window extension. Future work will explore applying this reinforced pruning approach to other domains, such as code generation and long-form document analysis.

Summary updated

Abstract

Large Language Models (LLMs) have shown impressive capabilities, yet they still struggle with math reasoning. In this work, we propose CoT-Influx, a novel approach that pushes the boundary of few-shot Chain-of-Thoughts (CoT) learning to improve LLM mathematical reasoning. Motivated by the observation that adding more concise CoT examples in the prompt can improve LLM reasoning performance, CoT-Influx employs a coarse-to-fine pruner to maximize the input of effective and concise CoT examples. The pruner first selects as many crucial CoT examples as possible and then prunes unimportant tokens to fit the context window. A math reasoning dataset with diverse difficulty levels and reasoning steps is used to train the pruner, along with a math-specialized reinforcement learning approach. As a result, by enabling more CoT examples with double the context window size in tokens, CoT-Influx significantly outperforms various prompting baselines across various LLMs (LLaMA2-7B, 13B, 70B) and 5 math datasets, achieving up to 4.55% absolute improvements. Remarkably, without any fine-tuning, LLaMA2-70B with CoT-Influx surpasses GPT-3.5 and a wide range of larger LLMs (PaLM, Minerva 540B, etc.) on the GSM8K. CoT-Influx serves as a plug-and-play module for LLMs and is compatible with most existing reasoning prompting techniques, such as self-consistency and self-verification.
BibTeX
@misc{huang2023fewer,
  author = {Xijie Huang and Li Lyna Zhang and Kwang‐Ting Cheng and Mao Yang and Yang, Mao},
  title = {{Fewer is More: Boosting LLM Reasoning with Reinforced Context Pruning}},
  year = {2023},
  eprint = {2312.08901},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2312.08901},
}
Figure