Introduction
Aligning Large Language Models (LLMs) with human values is a critical challenge in AI development. Traditional methods like Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO) primarily rely on the Bradley-Terry model. This model evaluates the reward difference between a "chosen" and a "rejected" response. However, this approach suffers from sparse feedback—assuming rewards only occur at the end of a sequence—and inefficient data utilization, as it ignores the intrinsic quality of individual demonstrations. These limitations often lead to "reward hacking," where models exploit reward function quirks (e.g., verbosity) rather than achieving true helpfulness.
To address these issues, this paper introduces Approximated Variational Alignment (AVA). AVA treats LLM alignment as a Bayesian Inverse Reinforcement Learning (BIRL) problem, modeling absolute rewards for individual demonstrations and accounting for intermediate, step-by-step rewards.
The AVA Framework
The core of AVA is the AVRIL (Approximated Variational Reward Imitation Learning) framework. Unlike standard architectures, AVA utilizes a TQR (Transformer-Q-Reward) architecture.
As shown in the architecture diagram, a standard Transformer backbone is augmented with two specialized heads: a Q-value head to predict the value of the next token and a Reward head to predict immediate rewards for state-action pairs. A key innovation is the TD-Error Constraint, which ensures consistency between the reward and the policy. This constraint forces the reward received after a token to match the change in the model's internal value of the sentence, acting as a powerful regularizer against reward hacking.
AVA is implemented through two primary pipelines:
- AVA-d (Demonstration): Learns directly from high-quality single demonstrations.
- AVA-p (Preference): A contrastive version designed for traditional preference pairs that still models individual rewards.
Experimental Setup
The researchers evaluated AVA across several benchmarks and model scales. Preference datasets included Anthropic Harmless, Anthropic Helpful, and OpenAI-Summary. For demonstration-based learning, Alpaca-GPT-4 and Math-GPT-4o were used. The framework's scalability was tested using the Google FLAN-T5 family (Small, Base, Large), alongside evaluations on GPT-2, BART-base, and Llama-2-7b-chat.
Results and Discussion
AVA demonstrates significant improvements in reward modeling accuracy and policy alignment. On the Anthropic Helpful task using a BART backbone, AVA-p achieved 74.84% accuracy, substantially outperforming the Bradley-Terry baseline's 67.56%.
A critical finding is the scalability of the approach. As illustrated in the performance trends, the gap between AVA and traditional baselines widens as the model size increases. This suggests that AVA is particularly well-suited for larger, next-generation LLMs.
Key insights from the results include:
- Dense Training Signals: By modeling rewards at every token rather than just at the end of a sequence, AVA provides a much richer signal for the model to learn from.
- Learning from Demonstrations: Unlike DPO, which requires negative examples, AVA-d can learn effectively from single, high-quality demonstrations.
- Robustness: Qualitative analysis indicates that the TD-error constraint produces more robust reward functions that are less susceptible to over-optimization.
Conclusion
Approximated Variational Alignment (AVA) offers a promising alternative to standard pairwise alignment methods. By leveraging Bayesian Inverse Reinforcement Learning and intermediate rewards, it improves data efficiency and mitigates reward hacking. While direct optimization showed marginal gains in some tasks, the overall framework provides a more consistent and scalable path for LLM alignment. Future work will focus on refining direct optimization objectives and handling noisy or conflicting human feedback.
Summary updated