Abstract: Parameter-Efficient finetuning (PEFT) enhances model performance on downstream tasks by updating a minimal subset of parameters. Representation finetuning (ReFT) methods further improve efficiency by freezing model weights and optimizing internal representations with fewer parameters than PEFT, outperforming PEFT on several tasks. However, ReFT exhibits a significant performance decline on mathematical reasoning tasks. To address this problem, the paper demonstrates that ReFT's poor performance on mathematical tasks primarily stems from its struggle to generate effective reasoning prefixes during the early inference phase. Moreover, ReFT disturbs the numerical encoding and the error accumulates during the CoT stage. Based on these observations, this paper proposes Bias-REstrained Prefix Representation FineTuning (BREP ReFT), which enhances ReFT's mathematical reasoning capability by truncating training data to optimize the generation of initial reasoning prefixes, intervening on the early inference stage to prevent error accumulation, and constraining the intervention vectors' magnitude to avoid disturbing numerical encoding. Extensive experiments across diverse model architectures demonstrate BREP's superior effectiveness, efficiency, and robust generalization capability, outperforming both standard ReFT and weight-based PEFT methods on the task of mathematical reasoning.
Introduction
Parameter-Efficient Finetuning (PEFT) methods like LoRA have become the standard for adapting Large Language Models (LLMs). However, a newer paradigm called Representation Finetuning (ReFT) offers even greater efficiency. Instead of modifying weights, ReFT "intervenes" on the model's internal hidden states during the forward pass using a simple linear transformation: $Scaling \cdot (Layer\ output) + bias$.
Despite its efficiency, ReFT has historically struggled with mathematical reasoning, often trailing LoRA by double digits. This research identifies two primary causes for this failure: misleading initial reasoning prefixes and the disturbance of internal numerical encodings.
The "Math Problem" in Representation Finetuning
Standard ReFT often fails to generate a strong starting "thought" (the first few tokens), which leads the entire Chain-of-Thought (CoT) process astray. Furthermore, LLMs represent numbers in specific linear subspaces. ReFT’s intervention vectors can be too "loud," effectively corrupting the model's ability to maintain numerical logic as reasoning steps accumulate.
As shown in Figure 1, standard ReFT (left) can lead to "disturbing numerical encoding" where intermediate calculations like $200 / 20$ result in incorrect values (e.g., 10 instead of 15 in a subsequent step due to corrupted state), ultimately causing an error. BREP ReFT (right) addresses this by using optimal reasoning prefixes and constraining the intervention magnitude.
Methodology: BREP ReFT
To overcome these limitations, the authors introduce BREP ReFT, which incorporates two novel strategies:
1. Prefix Training & Early-Stage Intervention
Research indicates that the first few tokens of a reasoning path are the most critical for setting the correct trajectory. BREP truncates training data to focus exclusively on these "prefixes." During inference, the model only applies interventions to the first $k$ tokens. This provides the necessary "nudge" to start the problem correctly while preventing the accumulation of intervention-induced errors during long-form generation.
2. Bias Constraint Training via PID Control
To solve the "numerical disturbance" issue, the authors treat the magnitude of the intervention vector as a control problem. They implement a Proportional-Integral-Derivative (PID) controller to dynamically adjust the training loss, ensuring the intervention's $L_2$ norm stays within a "Goldilocks zone."
Empirical results show that there is a quantifiable threshold beyond which interventions start causing calculation errors. As illustrated in Figure 6, peak performance is achieved when the $L_2$ norm of the bias is constrained to approximately 0.608. This magnitude is strong enough to steer the model's reasoning without scrambling its internal arithmetic logic.
Experimental Results
BREP ReFT was evaluated across diverse benchmarks including GSM8K, SVAMP, and MATH500 using architectures like Llama 3 and Qwen 2.5.
- Accuracy Gains: On GSM8K, BREP achieved 82.8% with Llama3-8B, outperforming LoRA (81.1%) and standard LoReFT (78.8%).
- Complex Reasoning: On the difficult MATH500 benchmark, BREP (42.8%) showed a significant lead over LoRA (39.3%).
- Generalization: Unlike LoRA, which sometimes saw performance drops on non-math tasks (e.g., a 5.1% drop on GPQA), BREP maintained or improved performance on out-of-domain benchmarks like BoolQ and PIQA.
Conclusion
BREP ReFT demonstrates that representation finetuning can be highly effective for complex reasoning tasks if the interventions are carefully restrained. By focusing on early-stage prefixes and maintaining "neural faithfulness" through bias constraints, BREP provides a more efficient and robust alternative to traditional weight-based finetuning for mathematical domains.
Summary updated