Temper and Tilt Lead to SLOP: Reward Hacking Mitigation with Inference-Time Alignment

Ye Wang, Jing Liu, Toshiaki Koike-Akino

This paper, presented at the AIWILD workshop at ICML 2026, introduces a novel framework for aligning Large Language Models (LLMs) at inference time. By generalizing existing sampling techniques into a "Sharpened Logarithmic Opinion Pool" (SLOP), the authors provide a lightweight way to ensemble multiple models and mitigate the common problem of "reward hacking."

TL;DR

The authors introduce SLOP, a method that ensembles multiple generative and reward models at inference time by adjusting their "temperature" and "tilt." By calibrating these ensemble weights against a small "gold" dataset, the system can detect and even invert flawed reward models, significantly improving robustness against reward hacking without expensive retraining.

What Problem Does This Paper Address?

Aligning LLMs with human values typically requires Reinforcement Learning from Human Feedback (RLHF), which is computationally expensive and rigid. A popular, cheaper alternative is inference-time alignment (e.g., Best-of-N sampling), where a "proxy" reward model selects the best output from several candidates generated by a reference LLM.

However, these proxy reward models are often imperfect. When a model is optimized to maximize a flawed proxy, it often finds "shortcuts" that yield high scores but poor actual performance—a phenomenon known as reward hacking. This paper seeks to make inference-time alignment more robust by effectively combining multiple signals and "tuning" the influence of the reference model versus the reward model.

Methods

The authors propose two primary technical extensions to standard alignment:

  1. Tempered and Tilted Alignment: Standard methods "tilt" the output distribution toward high rewards. The authors add a "tempering" parameter ($\alpha$) to the reference model itself. This allows the system to adjust how much it trusts the original model's "common sense" versus the reward model's signal.
  2. Sharpened Logarithmic Opinion Pool (SLOP): They generalize this into an ensemble of $m$ "experts" (which can be LLMs or reward models). Unlike traditional opinion pools, SLOP allows for:
  3. Sharpening: Weights can be greater than 1 to amplify confident experts.
  4. Inversion: Weights can be negative. If a reward model is consistently wrong (anti-aligned), SLOP can learn to do the opposite of what that reward model suggests.
  5. Calibration Algorithm: The authors provide a gradient-ascent algorithm to optimize these weights ($\omega$) using a tiny "gold" dataset (e.g., 200 samples). This allows the system to adapt to shifting objectives or failing reward models without fine-tuning the underlying LLMs.

Datasets

The researchers evaluated SLOP on two main benchmarks:

  • ScienceQA (SQA): A multiple-choice visual question-answering dataset. They used this to test robustness by pairing a Vision-Language Model (VLM) with a synthetic "noisy" reward model.
  • GSM8K: A benchmark of grade-school math word problems. Here, they tested the ensembling of four different small LLMs (Gemma-3-1B, Qwen2-1.5B, Qwen3-0.6B, and Phi-3.5-mini).

Key Insights and Ideas

  • The Power of Negative Weights: Perhaps the most "aha" moment is the utility of negative weights. In the ScienceQA experiments, when the proxy reward model's accuracy dropped below 40% (worse than random guessing), the calibrated SLOP learned a negative weight for it. By "inverting" the bad advice, SLOP actually performed better than if it had ignored the reward model entirely.
  • Reference Model as an Anchor: By treating the reference model's log-likelihood as just another "expert" score in the pool, the system can automatically decide when to stick to the base model's training and when to deviate based on reward signals.
  • Lightweight Continual Adaptation: Because the calibration only involves tuning a few scalar weights, it serves as a "lightweight continual-adaptation mechanism." If a reward target shifts, you don't need to re-run RLHF; you just re-calibrate the SLOP weights.

Key Results

  • Mitigating Hacking: In the ScienceQA task using LLaVA-1.5-7B, SLOP consistently outperformed baselines. While a pure "proxy-only" approach saw performance collapse as the reward model became less accurate, SLOP gracefully degraded back to the base model's performance or used negative weights to stay above it.
  • Superior Ensembling: On the GSM8K math task, SLOP sampling outperformed standard Best-of-N (BoN) and simple aggregated rewards. For example, when using Qwen2-1.5B as the generator, SLOP achieved higher accuracy than any individual model's scoring could provide.
  • Hard Selection: The authors found that "hard selection" (picking the single candidate with the highest SLOP score) often yielded even better results than sampling from the SLOP distribution.

Limitations and Future Work

  • Scale and Diversity: The experiments focused on relatively small models (under 7B parameters). The authors acknowledge that future work needs to broaden these experiments to larger models and more diverse tasks.
  • Ensemble Correlation: The paper notes that the impact of correlation or diversity within the ensemble (e.g., using models from the same family vs. different families) remains an open question for further investigation.
  • Proof of Concept: The authors describe the work as a "proof-of-concept," suggesting that while the theoretical framework is robust, its application in massive-scale production environments still needs to be tested.

Summary updated

Abstract

Inference-time alignment techniques offer a lightweight alternative or complement to costly reinforcement learning, while enabling continual adaptation as alignment objectives and reward targets evolve. Existing theoretical analyses justify these methods as approximations to sampling from distributions optimally tilted toward a given reward model. We extend these techniques by introducing reference-model temperature adjustment, which leads to further generalization of inference-time alignment to ensembles of generative and reward models combined as a sharpened logarithmic opinion pool (SLOP). To address reward hacking, we propose an algorithm for calibrating SLOP weight parameters and experimentally demonstrate that it improves robustness while preserving alignment performance.
BibTeX
@inproceedings{wang2026temper,
  author = {Ye Wang and Jing Liu and Toshiaki Koike-Akino},
  title = {{Temper and Tilt Lead to SLOP: Reward Hacking Mitigation with Inference-Time Alignment}},
  booktitle = {AIWILD @ ICML},
  year = {2026},
  url = {https://openreview.net/forum?id=o2FXghirw3},
}
Figure