Offline Safe Policy Optimization From Heterogeneous Feedback

Ze Gong, Pradeep Varakantham, Akshat KumararXiv 2025 · DOI · arXiv

Aligning autonomous agents with human values usually requires a delicate dance of reward engineering. When safety is added to the mix, the dance becomes a tightrope walk. A new research paper from SIAT and Singapore Management University, titled "Offline Safe Policy Optimization From Heterogeneous Feedback," proposes a way to skip complex reward/cost modeling and train safe, effective agents directly from simple human feedback.

The Challenge of Long-Horizon Safety

In standard Reinforcement Learning from Human Feedback (RLHF), the process typically involves two steps: learning a reward model from human preferences and then using that model to train a policy. For safety-critical applications (Safe RLHF), a third step is added: learning a cost model to keep the agent within safe bounds.

However, this approach hits a significant wall in long-horizon continuous control tasks, such as robotics or autonomous driving. Errors in the learned reward and cost models compound over time. If a reward model is slightly inaccurate, the agent might exploit a "loophole" that appears optimal to the model but is disastrous in reality. Furthermore, asking humans to provide pairwise comparisons for safety (e.g., "Is crash A worse than crash B?") is often impractical. Binary labels ("This was safe" vs. "This was unsafe") are much easier to collect but difficult to integrate into standard constrained RL frameworks.

The PreSa Framework

To address these issues, the researchers introduced PreSa (Preference and Safety Alignment), an offline reinforcement learning framework that learns safe policies directly from human pairwise preferences and binary safety labels.

Comparison of standard Offline PbRL and the proposed Offline Safe POHF framework.

Key Innovations:

  1. Direct Policy Learning: Instead of the traditional "learn model, then optimize" pipeline, PreSa optimizes the policy directly using a Lagrangian paradigm. It treats preference data as the objective to maximize and safety labels as constraints to satisfy.
  2. Heterogeneous Feedback: PreSa utilizes two distinct data types: pairwise preferences for task performance and binary safety labels for constraint adherence. This bypasses the need for an explicit cost function.
  3. Adapting HALOs: The authors adapted Human-Aware Losses (HALOs)—originally designed for fine-tuning Large Language Models—to sequential continuous control. They define a "trajectory segment utility" that compares the log-probability of an action under the current policy against a reference policy.
  4. Feasible Policy Set: By treating safety as a classification problem (predicting if a segment is safe), PreSa implicitly defines a "feasible set" of policies likely to be safe without requiring continuous cost values.

Experimental Evaluation

The authors evaluated PreSa across 29 different tasks using the DSRL (Datasets for Safe RL) benchmark, including SafetyGym (robots navigating hazards), BulletGym (drones and ants in critical environments), and MetaDrive (autonomous driving). They also collected real human evaluations for driving tasks in the Highway environment.

Results: Balancing Helpfulness and Harmlessness

PreSa demonstrated a superior ability to balance "helpfulness" (reward) and "harmlessness" (safety) compared to state-of-the-art baselines.

Ratio of safe tasks achieved by PreSa compared to various baselines in SafetyGym and BulletGym.
  • Outperforming Ground Truth: Surprisingly, PreSa often outperformed standard Offline Safe RL methods that had access to perfect ground-truth reward and cost functions. This suggests that the direct optimization approach is more robust than traditional constrained RL.
  • Safety Adherence: In SafetyGym and BulletGym, PreSa consistently produced "safe" agents (normalized cost < 1.0) while maintaining higher rewards than baselines like Safe-RLHF.
  • Real-World Performance: In tests with real human feedback, PreSa achieved a 1.0 task completion score and 0.85 safety score, significantly beating the Safe-RLHF baseline (0.87 completion / 0.63 safety).
  • Noise Robustness: The algorithm remained effective even when 10-30% of the human labels were intentionally flipped, demonstrating resilience to noisy or inconsistent human annotators.

Limitations and Future Directions

While PreSa offers a robust alternative to model-based RLHF, it still relies on the Bradley-Terry assumption, which may not fully capture the complexity of human psychology. Additionally, solving the Lagrangian optimization problem can be computationally intensive and sensitive to hyperparameters. Future work will explore simplifying the optimization process and testing the framework on higher-dimensional robotic tasks. By avoiding explicit reward/cost models, PreSa provides a promising path toward more reliable and human-aligned autonomous systems.

Summary updated

Abstract

Offline Preference-based Reinforcement Learning (PbRL) learns rewards and policies aligned with human preferences without the need for extensive reward engineering and direct interaction with human annotators. However, ensuring safety remains a critical challenge across many domains and tasks. Previous works on safe RL from human feedback (RLHF) first learn reward and cost models from offline data, then use constrained RL to optimize a safe policy. While such an approach works in the contextual bandits settings (LLMs), in long horizon continuous control tasks, errors in rewards and costs accumulate, leading to impairment in performance when used with constrained RL methods. To address these challenges, (a) instead of indirectly learning policies (from rewards and costs), we introduce a framework that learns a policy directly based on pairwise preferences regarding the agent's behavior in terms of rewards, as well as binary labels indicating the safety of trajectory segments; (b) we propose \textsc{PreSa} (Preference and Safety Alignment), a method that combines preference learning module with safety alignment in a constrained optimization problem. This optimization problem is solved within a Lagrangian paradigm that directly learns reward-maximizing safe policy \textit{without explicitly learning reward and cost models}, avoiding the need for constrained RL; (c) we evaluate our approach on continuous control tasks with both synthetic and real human feedback. Empirically, our method successfully learns safe policies with high rewards, outperforming state-of-the-art baselines, and offline safe RL approaches with ground-truth reward and cost.
BibTeX
@misc{gong2025offline,
  author = {Ze Gong and Pradeep Varakantham and Akshat Kumar},
  title = {{Offline Safe Policy Optimization From Heterogeneous Feedback}},
  year = {2025},
  eprint = {2512.20173},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.20173},
}
Figure