Abstract
Large Language Models (LLMs) demonstrate enhanced reasoning capabilities when provided with multiple Chain-of-Thought (CoT) examples. However, the limited context window of these models often restricts the number of examples that can be included, leading to a performance plateau or decline as token limits are reached. This paper introduces CoT-Influx, a novel "coarse-to-fine" pruning framework designed to maximize information density within the prompt. By utilizing reinforcement learning to strip away redundant tokens while preserving critical logical and numerical data, CoT-Influx enables models to process up to six times more CoT examples. Experimental results show that a LLaMA2-70B model using CoT-Influx outperforms significantly larger models like PaLM (540B) on mathematical reasoning benchmarks without any fine-tuning.
Introduction
Multi-step mathematical reasoning remains a significant challenge for Large Language Models. While "few-shot" prompting—typically using around 8 examples—is a standard technique to guide reasoning, researchers frequently encounter a "few-shot boundary." As more examples are added, the context window fills up, forcing a trade-off between the number of examples and the length of the model's generated response.
Existing approaches to this problem, such as context extension or general-purpose prompt compression, often suffer from high computational costs or the loss of vital logical tokens. CoT-Influx addresses these issues by asking: Can we improve reasoning by fitting more, higher-quality, and more concise examples into the existing context window?
Methodology
CoT-Influx is a plug-and-play module that treats the target LLM as a black box. It employs a two-stage, coarse-to-fine pruning strategy to optimize the prompt:
- Shot Pruner (Coarse): Selects the most relevant examples ("shots") from a candidate pool for a given question.
- Token Pruner (Fine): Removes unimportant natural language tokens (e.g., "the", "then") from the selected examples while strictly preserving critical numerical and logical tokens.
The system utilizes a lightweight BERT-Large model for text embeddings and a two-layer Multi-Layer Perceptron (MLP) for pruning decisions. Because pruning is a discrete process, the framework is trained using Reinforcement Learning (REINFORCE). The reward function balances LLM next-token prediction loss, reasoning accuracy ($R_{Acc}$), and length constraints. To ensure training stability, a Difficulty-aware Data Filter is used to curriculum-train the agent, starting with simpler problems.
Datasets
To address the lack of diversity in existing CoT datasets, the authors developed the Math Reasoning Dataset with Diverse Difficulty (MRD³). This dataset contains 9.7K pairs derived from GSM8K, MAWPS, and AQuA, which were "evolved" using GPT-4 to create a wide range of difficulty levels (1 to 10). Evaluation was conducted across five standard benchmarks: GSM8K, AddSub, MultiArith, Svamp, and Singleeq.
Results and Key Insights
The implementation of CoT-Influx yielded several significant findings:
- Information Density: By pruning redundant words, the framework could fit up to 48 examples into a space typically reserved for 8, significantly boosting in-context learning.
- Model-Specific Preferences: Smaller models like LLaMA2-7B were found to perform better with simpler examples (difficulty 3-4), while larger models like LLaMA2-70B thrived on high-difficulty examples.
- Token Fragility: CoT-Influx successfully learned to protect "crucial tokens" (digits, signs) while discarding "function words," a critical distinction for mathematical tasks where binary logic is paramount.
Quantitatively, LLaMA2-70B with CoT-Influx achieved 59.6% on GSM8K, surpassing GPT-3.5 (57.1%) and the much larger PaLM 540B (56.5%). The pruner achieved an average 4.28x compression ratio, effectively expanding the context window with negligible overhead (~1.4GB VRAM, 1.7% latency increase).
Conclusion
CoT-Influx demonstrates that reinforced in-context pruning is a highly effective, efficient, and model-agnostic way to push the limits of LLM reasoning. While performance remains dependent on the quality of initial CoT examples, the method offers a significant leap in prompt optimization. Future work will explore more sophisticated data evolution strategies and the application of reinforced pruning to other complex domains like legal analysis and code generation.
Summary updated