Boosting LLM Reasoning: Push the Limits of Few-shot Learning with Reinforced In-Context Pruning

Xijie Huang, Li Lyna Zhang, Kwang-Ting Cheng, Mao Yang

Abstract

Large Language Models (LLMs) demonstrate enhanced reasoning capabilities when provided with multiple Chain-of-Thought (CoT) examples. However, the limited context window of these models often restricts the number of examples that can be included, leading to a performance plateau or decline as token limits are reached. This paper introduces CoT-Influx, a novel "coarse-to-fine" pruning framework designed to maximize information density within the prompt. By utilizing reinforcement learning to strip away redundant tokens while preserving critical logical and numerical data, CoT-Influx enables models to process up to six times more CoT examples. Experimental results show that a LLaMA2-70B model using CoT-Influx outperforms significantly larger models like PaLM (540B) on mathematical reasoning benchmarks without any fine-tuning.

Introduction

Multi-step mathematical reasoning remains a significant challenge for Large Language Models. While "few-shot" prompting—typically using around 8 examples—is a standard technique to guide reasoning, researchers frequently encounter a "few-shot boundary." As more examples are added, the context window fills up, forcing a trade-off between the number of examples and the length of the model's generated response.

Existing approaches to this problem, such as context extension or general-purpose prompt compression, often suffer from high computational costs or the loss of vital logical tokens. CoT-Influx addresses these issues by asking: Can we improve reasoning by fitting more, higher-quality, and more concise examples into the existing context window?

Methodology

CoT-Influx is a plug-and-play module that treats the target LLM as a black box. It employs a two-stage, coarse-to-fine pruning strategy to optimize the prompt:

  1. Shot Pruner (Coarse): Selects the most relevant examples ("shots") from a candidate pool for a given question.
  2. Token Pruner (Fine): Removes unimportant natural language tokens (e.g., "the", "then") from the selected examples while strictly preserving critical numerical and logical tokens.
Overview of the CoT-Influx framework, featuring a coarse-to-fine pruning strategy trained via reinforcement learning.

The system utilizes a lightweight BERT-Large model for text embeddings and a two-layer Multi-Layer Perceptron (MLP) for pruning decisions. Because pruning is a discrete process, the framework is trained using Reinforcement Learning (REINFORCE). The reward function balances LLM next-token prediction loss, reasoning accuracy ($R_{Acc}$), and length constraints. To ensure training stability, a Difficulty-aware Data Filter is used to curriculum-train the agent, starting with simpler problems.

Datasets

To address the lack of diversity in existing CoT datasets, the authors developed the Math Reasoning Dataset with Diverse Difficulty (MRD³). This dataset contains 9.7K pairs derived from GSM8K, MAWPS, and AQuA, which were "evolved" using GPT-4 to create a wide range of difficulty levels (1 to 10). Evaluation was conducted across five standard benchmarks: GSM8K, AddSub, MultiArith, Svamp, and Singleeq.

Results and Key Insights

The implementation of CoT-Influx yielded several significant findings:

  • Information Density: By pruning redundant words, the framework could fit up to 48 examples into a space typically reserved for 8, significantly boosting in-context learning.
  • Model-Specific Preferences: Smaller models like LLaMA2-7B were found to perform better with simpler examples (difficulty 3-4), while larger models like LLaMA2-70B thrived on high-difficulty examples.
  • Token Fragility: CoT-Influx successfully learned to protect "crucial tokens" (digits, signs) while discarding "function words," a critical distinction for mathematical tasks where binary logic is paramount.
Performance comparison of CoT-Influx against standard few-shot and TopK baselines on GSM8K across varying shot counts.

Quantitatively, LLaMA2-70B with CoT-Influx achieved 59.6% on GSM8K, surpassing GPT-3.5 (57.1%) and the much larger PaLM 540B (56.5%). The pruner achieved an average 4.28x compression ratio, effectively expanding the context window with negligible overhead (~1.4GB VRAM, 1.7% latency increase).

Conclusion

CoT-Influx demonstrates that reinforced in-context pruning is a highly effective, efficient, and model-agnostic way to push the limits of LLM reasoning. While performance remains dependent on the quality of initial CoT examples, the method offers a significant leap in prompt optimization. Future work will explore more sophisticated data evolution strategies and the application of reinforced pruning to other complex domains like legal analysis and code generation.

Summary updated

Abstract

Large Language Models (LLMs) have shown impressive capabilities, yet they still struggle with math reasoning. In this work, we propose CoT-Influx, a novel approach that pushes the boundary of few-shot Chain-of-Thoughts (CoT) learning to improve LLM mathematical reasoning. Motivated by the observation that adding more concise CoT examples in the prompt can improve LLM reasoning performance, CoT-Influx employs a coarse-to-fine pruner to maximize the input of effective and concise CoT examples. The pruner first selects as many crucial CoT examples as possible and then prunes unimportant tokens to fit the context window. A math reasoning dataset with diverse difficulty levels and reasoning steps is used to train the pruner, along with a math-specialized reinforcement learning approach. As a result, by enabling more CoT examples with double the context window size in tokens, CoT-Influx significantly outperforms various prompting baselines across various LLMs (LLaMA2-7B, 13B, 70B) and 5 math datasets, achieving up to 4.55% absolute improvements. Remarkably, without any fine-tuning, LLaMA2-70B with CoT-Influx surpasses GPT-3.5 and a wide range of larger LLMs (PaLM, Minerva 540B, etc.) on the GSM8K. CoT-Influx serves as a plug-and-play module for LLMs and is compatible with most existing reasoning prompting techniques, such as self-consistency and self-verification.
BibTeX
@misc{huang2023boosting,
  author = {Xijie Huang and Li Lyna Zhang and Kwang-Ting Cheng and Mao Yang},
  title = {{Boosting LLM Reasoning: Push the Limits of Few-shot Learning with Reinforced In-Context Pruning}},
  year = {2023},
  eprint = {2312.08901},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2312.08901},
}
Figure