ARM: Alignment with Residual Energy-Based Model

Bo Pang, Caiming Xiong, Yingbo ZhouNAACL 2024 · DOI

Abstract

Aligning Large Language Models (LLMs) with human preferences is a critical final step in developing helpful AI assistants. While Reinforcement Learning from Human Feedback (RLHF) via Proximal Policy Optimization (PPO) is the industry standard, it is computationally intensive and complex to implement. We present ARM (Alignment with Residual Energy-Based Model), a framework that treats the target aligned policy as an Energy-Based Model (EBM). By sampling from this "tilted" distribution and training via Direct Preference Optimization (DPO), ARM achieves the performance of on-policy reinforcement learning with the implementation simplicity of supervised fine-tuning.

Introduction

The current paradigm for LLM alignment typically involves two main paths: online reinforcement learning (e.g., PPO) and offline preference optimization (e.g., DPO). PPO is an "on-policy" method that allows the model to explore its own generations, but it requires maintaining four models in memory simultaneously and optimizes a mathematically challenging reverse Kullback–Leibler (KL) divergence. Conversely, DPO is an "off-policy" method that is simpler to implement but relies on static datasets, missing the benefits of active exploration.

ARM addresses this gap by providing a middle ground. It enables on-policy exploration without the massive resource overhead of traditional RL, leveraging a residual energy-based formulation to bridge the gap between supervised learning and reinforcement learning.

Methodology

The ARM framework operates through a three-stage process designed to approximate an ideal aligned policy $\pi^*$ without the instability of online RL.

The ARM framework: SFT and RM are combined via Exponential Tilting to form a Residual EBM, which is then sampled via SNIS to create a training dataset for DPO.
  1. Target Policy Definition: ARM defines the target distribution as a Residual Energy-Based Model. This model "tilts" the base Supervised Fine-Tuned (SFT) model toward higher rewards using a learned reward model (RM).
  2. Importance Sampling: Direct sampling from an EBM is non-trivial. ARM employs Self-Normalizing Importance Sampling (SNIS) to generate $n$ candidate responses from the SFT model and resample them based on their reward scores. This probabilistic approach acts as a refined version of "Best-of-n" sampling.
  3. Optimization via Forward KL: Unlike PPO's reverse KL objective, ARM minimizes the forward KL divergence. This encourages the model to cover the full distribution of high-reward responses, reducing the risk of mode collapse. The final policy is trained using a modified version of DPO on the resampled high-quality data.

Experimental Setup

The researchers evaluated ARM across three diverse benchmarks:

  • AlpacaFarm: Simulating general instruction-following capabilities.
  • Reddit TL;DR: Testing text summarization performance.
  • Anthropic Helpful-Harmless (HH): Assessing dialogue quality and safety at scale.

The study also investigated ARM's performance in low-resource environments (using as few as 2,000 preference samples) and non-pairwise feedback scenarios (e.g., binary "thumbs up/down"), where traditional DPO is not directly applicable.

Results and Discussion

ARM consistently outperformed established baselines, including PPO and DPO, across all tested benchmarks.

Main results comparing ARM (Ours) against SFT, PPO, DPO, and other baselines across three major datasets.

On the Anthropic HH dataset, ARM achieved a 77.7% win rate, a significant improvement over DPO (67.3%) and PPO (63.6%). Similar gains were observed in summarization tasks, where ARM reached a 71.0% win rate on TL;DR compared to DPO's 65.8%.

Key Insights

  • Robustness through Forward KL: By using forward KL, ARM maintains better response diversity compared to methods that suffer from the mode-seeking behavior of reverse KL.
  • On-Policy Efficiency: ARM effectively transforms DPO into an on-policy method. By learning from the best versions of its own outputs, the model benefits from exploration without the complexity of a full RL loop.
  • Data Versatility: ARM's reliance on reward scoring rather than explicit pairs allows it to utilize binary feedback, making it more flexible for real-world data collection where pairwise comparisons may be unavailable.

Conclusion and Future Work

ARM represents a significant step toward more accessible and efficient LLM alignment. While it avoids the memory overhead of PPO, the sampling phase (Best-of-n) does introduce a computational cost during data preparation. Future work may explore reducing this overhead or refining the surrogate reward functions to better handle complex multi-way rankings. Despite these challenges, ARM's ability to deliver state-of-the-art performance with a simpler implementation makes it a compelling alternative for the final stages of LLM training.

Summary updated

BibTeX
@inproceedings{pang2024arm,
  author = {Bo Pang and Caiming Xiong and Yingbo Zhou},
  title = {{ARM: Alignment with Residual Energy-Based Model}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.naacl-long.455},
}
Figure