Q-Probe: A Lightweight Approach to Reward Maximization for Language Models

Kenneth Li, Samy Jelassi, Hugh Zhang, Sham M. Kakade, Martin Wattenberg, David Brandfonbrener

Abstract

Q-Probe is a lightweight method for adapting large language models (LLMs) to specific tasks by training a tiny linear layer on the model's internal embeddings. By reweighting multiple candidate completions through a novel "direct policy learning" objective, Q-Probe significantly improves performance in coding and math tasks—often outperforming standard fine-tuning in data-scarce scenarios—while keeping the base model entirely frozen.

Introduction

Adapting a pre-trained LLM for a specific goal, such as writing bug-free code or following human preferences, usually requires either computationally expensive fine-tuning or simple prompting that often hits a performance ceiling. Q-Probe addresses this by leveraging the observation that LLMs are frequently better at discriminating between good and bad answers than they are at generating the perfect answer on the first try. The method seeks a middle ground: a system that is as easy to train as a simple classifier but as effective as reinforcement learning.

Methods

The core of Q-Probe is a simple linear function (a "probe") trained on the model’s residual stream—the internal vector representations (embeddings) the model produces for a given input.

Inference and Training

At test time, the system draws $k$ candidate completions from the frozen LLM. The Q-Probe assigns a score to each completion based on its embedding. These scores are converted into a probability distribution via softmax, and one completion is sampled. This acts as a "smart" version of rejection sampling.

The Q-Probe architecture: candidate completions are generated by a frozen LLM, scored by a linear probe on internal features, and reweighted via softmax for final selection.

For training, the authors introduce a Direct Policy Learning ($L_{PG}$) objective. This importance-weighted policy gradient loss forces the probe to contrast different completions for the same prompt. By training the probe to recognize why one sample is better than another in context, it aligns much better with the actual selection process used during inference than standard reward prediction ($L_Q$).

Experimental Setup

The researchers evaluated Q-Probe across several demanding domains:

  • Code Generation: Training on MBPP (Mostly Basic Python Problems) and testing on HumanEval.
  • Mathematical Reasoning: Using GSM-8K grade-school math word problems.
  • Human Preferences: Using Anthropic HH, OpenAssistant, and Stanford Human Preferences (SHP) datasets to test alignment.

Key Results and Insights

The Contrastive Advantage

The most significant finding is that training the probe to rank samples ($L_{PG}$) is far more effective than training it to predict absolute reward values ($L_Q$). By focusing on the relative quality of completions for a specific prompt, the probe ignores "easy" vs. "hard" prompt noise and focuses on the features that lead to success.

Extreme Data Efficiency

Q-Probe is remarkably sample-efficient. On the MBPP coding benchmark, the method achieved a 0.4 test reward with only 50 training prompts. This makes it an ideal candidate for niche tasks where human-labeled data is scarce.

Comparison of different loss objectives across varying numbers of training prompts. $L_{PG}$ consistently outperforms $L_Q$ and $L_{CE}$, demonstrating superior data efficiency.

Performance Benchmarks

  • Coding: On MBPP, a Q-Probe on top of Code-LLaMA-7B achieved 46% accuracy, compared to the base model's 29%. When combined with 5-shot prompting, it reached 52%, outperforming LoRA fine-tuning.
  • Human Preferences: Q-Probe outperformed strong offline RL baselines like PPO and DPO by roughly 6% in win rate as judged by GPT-4.
  • Speed: Training a Q-Probe takes seconds to minutes on a single GPU because it only involves a linear layer.

Limitations and Future Work

The primary drawback of Q-Probe is inference-time compute. Because the method requires generating $k$ samples (often $k=48$) to pick the best one, it is slower and more expensive to run at scale than a single-completion model.

Future directions include distillation, where the Q-Probe is used to identify high-quality data for fine-tuning the base model, potentially baking the probe's wisdom into the model weights to eliminate the need for multiple samples at runtime.

Summary updated

BibTeX
@inproceedings{li2024q,
  author = {Kenneth Li and Samy Jelassi and Hugh Zhang and Sham M. Kakade and Martin Wattenberg and David Brandfonbrener},
  title = {{Q-Probe: A Lightweight Approach to Reward Maximization for Language Models}},
  booktitle = {International Conference on Machine Learning},
  year = {2024},
  url = {https://openreview.net/forum?id=gxOQEMRbRa},
}
Figure