TL;DR
Instruction-tuned LLMs under 14B parameters notoriously lag behind classical encoder-only models like BERT on core natural language understanding (NLU) benchmarks. This paper demonstrates that framing NLU adaptation as a reinforcement learning (RL) problem using Proximal Policy Optimization (PPO) over lightweight LoRA adapters bridges this gap—outperforming supervised fine-tuning, beating BERT-large on GLUE and SuperGLUE, and even surpassing GPT-4o on out-of-domain transfer tasks.
What Problem Does This Paper Address?
While modern decoder-only large language models excel at text generation and few-shot reasoning, smaller models (<14B parameters, such as LLaMA-2-7B-chat) struggle on structured NLU benchmarks like GLUE and SuperGLUE. In zero-shot and few-shot setups, these models frequently underperform standard encoder models like BERT-base, often exhibiting strong label biases and fragile output formatting.
Standard supervised fine-tuning (SFT) using cross-entropy next-token prediction helps, but it frequently overfits to specific token distributions, causing the model to lose general instruction-following abilities and degrade overall language modeling quality. The authors investigate whether reward-driven reinforcement learning can better align pre-trained representations with task-level correctness without suffering from SFT’s drawbacks.
Methods
The authors reframe classification and regression NLU tasks into an RL environment:
- Task Formulation & Action Space: Tasks are structured with system prompts that instruct the model to wrap outputs within structured tags (e.g.,
<Judgement> Yes </Judgement>). Autoregressive token generation constitutes the sequence of actions, and final answers are parsed using regular expressions. - RL Optimization: The policy is fine-tuned using Proximal Policy Optimization (PPO) on top of Low-Rank Adaptation (LoRA) layers ($r=16$), keeping the base backbone frozen. A value-head critic model estimates the Generalized Advantage Estimation (GAE).
- Reward Structure:
- Classification: Binary reward ($+1$ for exact match, $-1$ for incorrect format/wrong prediction).
- Regression (STS-B): Continuous bounded reward $R = 2.5 - |\hat{y}_i - y_i| \in [-2.5, 2.5]$.
- Comparisons with GRPO & Reward Modeling: The authors compare PPO against Group Relative Policy Optimization (GRPO) and explore a learned reward model (BERT-base trained on GPT-4o pairwise rankings of answer rationales).
Datasets
- Primary Benchmarks: GLUE (MNLI, QQP, QNLI, SST-2, CoLA, STS-B, MRPC, RTE, WNLI) and SuperGLUE (BoolQ, CB, COPA, MultiRC, ReCoRD, RTE, WiC, WSC).
- Out-of-Domain Zero-Shot Generalization:
- Sentiment: Financial PhraseBank, Labelled Financial News, Mental Health, and Emotion datasets.
- NLI: Babi-nli and SIGA-nli.
- Instruction Following & Transfer: Amazon Review dataset (1–5 star rating classification).
- Language Modeling & QA: WikiText-2 (perplexity) and SQuAD (extractive question answering).
Key Insights and Ideas
- Preserving the Policy Distribution: SFT pushes the policy directly toward specific training tokens, which narrows the model’s learned distribution and severely harms out-of-distribution generalization. In contrast, PPO’s probability ratio clipping and KL-penalty retain proximity to the base reference model, preserving both intrinsic language modeling ability and broader instruction following.
- Mitigating Label Bias: Few-shot prompting often causes smaller LLMs to latch onto class distribution biases. Reinforcing exact end-task outcomes via RL removes these biases without requiring manual prompt engineering.
- PPO vs. GRPO Trade-Off: While GRPO eliminates the critic network and saves memory, its requirement to sample multiple trajectories per prompt increases training wall-clock time (+25% per-step runtime over SFT). PPO incurs only a ~4% runtime overhead per step.
- The Reward Model Trade-off: Training PPO against a learned reward model (scoring generated explanations) yielded significantly higher explanation quality according to GPT-4o (4.10 vs. 3.48), but degraded raw classification accuracy (dropping from 96.4% to 89.7%), pointing to susceptibility to reward hacking or reward model noise.
Key Results
- GLUE & SuperGLUE Dominance: On GLUE, PPO single-task fine-tuning on LLaMA-2-7B achieved an average score of 84.8, substantially outperforming SFT (78.5), few-shot prompting (58.7), zero-shot prompting (46.1), and beating BERT-large (82.1). On SuperGLUE, PPO reached 78.3, exceeding BERT-large++ (71.5).
- Cross-Architecture Consistency: Similar large performance jumps were verified on Qwen2.5-7B-Instruct and MPT-7B-chat.
- Outperforming GPT-4o Out-of-Domain: LLaMA-2 models fine-tuned on a single task with PPO (SST-2 or MNLI) generalized to unseen datasets, beating GPT-4o by 4.3%–4.6% on average, including gains of +7.3% on the Mental Health dataset and +10.8% on SIGA-nli.
- Retaining General Capabilities: On WikiText-2, PPO’s perplexity (6.966) remained virtually identical to the base model (6.939), while SFT worsened to 7.384. On the Amazon Review format-shift test, PPO achieved 39.35% accuracy, whereas SFT collapsed to <1%.
Limitations and Future Work
- Model Scale: The empirical study was constrained to models $\le$ 13B/14B parameters due to compute limitations.
- Rule-Based Ground Truth Reliance: The most effective setup relied on binary exact-match rewards against labeled ground truth, rather than an unsupervised or fully generalizable reward model.
- Reward Hacking in Complex Rewards: Early experiments integrating learned reward models for joint explanation and classification led to degraded classification accuracy, underscoring the challenge of designing robust multi-objective reward models for NLU.
Summary updated