Improving the Language Understanding Capabilities of Large Language Models Using Reinforcement Learning

Bokai Hu, Sai Ashish Somayajula, Xin Pan, Zihan Huang, Pengtao XieSubmission to ICLR 2025 · Withdrawn

TL;DR

Instruction-tuned LLMs under 14B parameters notoriously lag behind classical encoder-only models like BERT on core natural language understanding (NLU) benchmarks. This paper demonstrates that framing NLU adaptation as a reinforcement learning (RL) problem using Proximal Policy Optimization (PPO) over lightweight LoRA adapters bridges this gap—outperforming supervised fine-tuning, beating BERT-large on GLUE and SuperGLUE, and even surpassing GPT-4o on out-of-domain transfer tasks.

What Problem Does This Paper Address?

While modern decoder-only large language models excel at text generation and few-shot reasoning, smaller models (<14B parameters, such as LLaMA-2-7B-chat) struggle on structured NLU benchmarks like GLUE and SuperGLUE. In zero-shot and few-shot setups, these models frequently underperform standard encoder models like BERT-base, often exhibiting strong label biases and fragile output formatting.

Standard supervised fine-tuning (SFT) using cross-entropy next-token prediction helps, but it frequently overfits to specific token distributions, causing the model to lose general instruction-following abilities and degrade overall language modeling quality. The authors investigate whether reward-driven reinforcement learning can better align pre-trained representations with task-level correctness without suffering from SFT’s drawbacks.

Methods

The authors reframe classification and regression NLU tasks into an RL environment:

  • Task Formulation & Action Space: Tasks are structured with system prompts that instruct the model to wrap outputs within structured tags (e.g., <Judgement> Yes </Judgement>). Autoregressive token generation constitutes the sequence of actions, and final answers are parsed using regular expressions.
  • RL Optimization: The policy is fine-tuned using Proximal Policy Optimization (PPO) on top of Low-Rank Adaptation (LoRA) layers ($r=16$), keeping the base backbone frozen. A value-head critic model estimates the Generalized Advantage Estimation (GAE).
  • Reward Structure:
  • Classification: Binary reward ($+1$ for exact match, $-1$ for incorrect format/wrong prediction).
  • Regression (STS-B): Continuous bounded reward $R = 2.5 - |\hat{y}_i - y_i| \in [-2.5, 2.5]$.
  • Comparisons with GRPO & Reward Modeling: The authors compare PPO against Group Relative Policy Optimization (GRPO) and explore a learned reward model (BERT-base trained on GPT-4o pairwise rankings of answer rationales).

Datasets

  • Primary Benchmarks: GLUE (MNLI, QQP, QNLI, SST-2, CoLA, STS-B, MRPC, RTE, WNLI) and SuperGLUE (BoolQ, CB, COPA, MultiRC, ReCoRD, RTE, WiC, WSC).
  • Out-of-Domain Zero-Shot Generalization:
  • Sentiment: Financial PhraseBank, Labelled Financial News, Mental Health, and Emotion datasets.
  • NLI: Babi-nli and SIGA-nli.
  • Instruction Following & Transfer: Amazon Review dataset (1–5 star rating classification).
  • Language Modeling & QA: WikiText-2 (perplexity) and SQuAD (extractive question answering).

Key Insights and Ideas

  • Preserving the Policy Distribution: SFT pushes the policy directly toward specific training tokens, which narrows the model’s learned distribution and severely harms out-of-distribution generalization. In contrast, PPO’s probability ratio clipping and KL-penalty retain proximity to the base reference model, preserving both intrinsic language modeling ability and broader instruction following.
  • Mitigating Label Bias: Few-shot prompting often causes smaller LLMs to latch onto class distribution biases. Reinforcing exact end-task outcomes via RL removes these biases without requiring manual prompt engineering.
  • PPO vs. GRPO Trade-Off: While GRPO eliminates the critic network and saves memory, its requirement to sample multiple trajectories per prompt increases training wall-clock time (+25% per-step runtime over SFT). PPO incurs only a ~4% runtime overhead per step.
  • The Reward Model Trade-off: Training PPO against a learned reward model (scoring generated explanations) yielded significantly higher explanation quality according to GPT-4o (4.10 vs. 3.48), but degraded raw classification accuracy (dropping from 96.4% to 89.7%), pointing to susceptibility to reward hacking or reward model noise.

Key Results

  • GLUE & SuperGLUE Dominance: On GLUE, PPO single-task fine-tuning on LLaMA-2-7B achieved an average score of 84.8, substantially outperforming SFT (78.5), few-shot prompting (58.7), zero-shot prompting (46.1), and beating BERT-large (82.1). On SuperGLUE, PPO reached 78.3, exceeding BERT-large++ (71.5).
  • Cross-Architecture Consistency: Similar large performance jumps were verified on Qwen2.5-7B-Instruct and MPT-7B-chat.
  • Outperforming GPT-4o Out-of-Domain: LLaMA-2 models fine-tuned on a single task with PPO (SST-2 or MNLI) generalized to unseen datasets, beating GPT-4o by 4.3%–4.6% on average, including gains of +7.3% on the Mental Health dataset and +10.8% on SIGA-nli.
  • Retaining General Capabilities: On WikiText-2, PPO’s perplexity (6.966) remained virtually identical to the base model (6.939), while SFT worsened to 7.384. On the Amazon Review format-shift test, PPO achieved 39.35% accuracy, whereas SFT collapsed to <1%.

Limitations and Future Work

  • Model Scale: The empirical study was constrained to models $\le$ 13B/14B parameters due to compute limitations.
  • Rule-Based Ground Truth Reliance: The most effective setup relied on binary exact-match rewards against labeled ground truth, rather than an unsupervised or fully generalizable reward model.
  • Reward Hacking in Complex Rewards: Early experiments integrating learned reward models for joint explanation and classification led to degraded classification accuracy, underscoring the challenge of designing robust multi-objective reward models for NLU.

Summary updated

Abstract

Large language models (LLMs), primarily built on decoder-only transformer architectures, excel in natural language generation tasks and have shown promise in adapting to diverse downstream tasks using zero-shot and few-shot prompting techniques. However, these prompting methods often fall short on natural language understanding (NLU) tasks, where smaller encoder-only models like BERT-base consistently outperform LLMs on benchmarks such as GLUE and SuperGLUE. In this paper, we explore two approaches—supervised fine-tuning and proximal policy optimization (PPO)—to enhance the NLU capabilities of LLMs. To reduce the computational cost of full-model fine-tuning, we integrate low-rank adaptation (LoRA) layers, restricting updates to these layers during both supervised fine-tuning and PPO stages. In the supervised fine-tuning approach, task-specific prompts are concatenated with input queries and ground-truth labels from the NLU training corpus, optimizing the model using the next-token prediction objective. Despite this, LLMs still underperform compared to encoder-only models like BERT-base on several NLU tasks. To address this gap, we employ PPO, a reinforcement learning technique that treats each token generation as an action and evaluates the sequence of generated tokens using a reward function based on their alignment with ground-truth answers. PPO then updates the model to maximize these rewards, effectively aligning its outputs with the correct labels. Our experiments with the LLAMA2-7B model demonstrate that PPO-based fine-tuning significantly improves performance, delivering an average gain of 6.3 points over supervised fine-tuning on the GLUE benchmark. PPO surpasses zero-shot prompting by 38.7 points and few-shot prompting by 26.1 points on GLUE, while also outperforming these baselines by 28.8 and 28.5 points on SuperGLUE. Additionally, PPO exceeds the performance of BERT-large, a strong baseline, with an average improvement of 2.7 points on GLUE and 9.3 points on SuperGLUE. These improvements are consistent across models such as Qwen2.5-7B and MPT-7B, highlighting PPO’s robustness and effectiveness in enhancing the NLU capabilities of LLMs.
BibTeX
@misc{hu2024improving,
  author = {Bokai Hu and Sai Ashish Somayajula and Xin Pan and Zihan Huang and Pengtao Xie},
  title = {{Improving the Language Understanding Capabilities of Large Language Models Using Reinforcement Learning}},
  year = {2024},
  eprint = {2410.11020},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.11020},
}
Figure