BECON: Belief-Conditioned Constrained Multi-Objective Reinforcement Learning under Drifting Preferences and Budgets

Bui Trong Duc, Huynh Thi Thanh BinharXiv 2026 · arXiv

TL;DR

BECON is a new framework for constrained Multi-Objective Reinforcement Learning (MORL) that handles shifting user preferences and safety budgets. By treating these signals as noisy, unreliable observations and inferring the true "latent" intent from interaction history, BECON achieves significantly better performance and safety in dynamic environments where goals and constraints are constantly in flux.

What Problem Does This Paper Address?

In the ideal world of reinforcement learning, an agent is given a clear objective and a fixed safety budget. In the real world, things are messier. Imagine an autonomous vehicle: the passenger’s preference for speed versus comfort might drift during a trip, and safety tolerances (the "budget") might shift depending on whether the car is on a highway or a narrow residential street.

Current MORL algorithms typically assume "full task observability"—meaning they treat the preference and budget signals they receive as ground truth. However, these signals are often "unreliable proxies": they can be noisy, delayed, or incomplete. This creates a signal-reality gap. If an agent blindly follows a stale or noisy preference signal, it might optimize for the wrong goal or miss emerging safety risks. The authors tackle this by reframing constrained MORL as a partially observed task-specification problem.

Methods

The authors propose BECON (Belief-Conditioned framework), which moves away from the "blind trust" model. Instead of treating the external task signal as a command, BECON treats it as a noisy observation.

Key components of the architecture include:

  • History-Conditioned Belief: A recurrent encoder (specifically a GRU) processes the recent history of states, actions, rewards, and costs to form a "belief" ($z_t$) about the true underlying task.
  • Uncertainty-Gated Fusion: This is the "secret sauce." The model estimates its own epistemic uncertainty. If the history is short or ambiguous, the agent defers to the external signal. If the history is informative and the external signal seems "off," the agent uses its inferred belief to correct the signal.
  • Context-Adaptive Constraint Enforcement: Standard constrained RL uses a global "Lagrange multiplier" to penalize safety violations. BECON uses a context-dependent dual variable that adapts the safety penalty based on the inferred feasibility of the current situation.
Overview of the BECON architecture showing the history encoder, uncertainty-gated fusion, and the actor-critic components.

Datasets

The researchers evaluated BECON on a diverse suite of benchmarks designed to test complex trade-offs:

  • Continuous Control (MuJoCo): Standard robotic tasks like Ant, HalfCheetah, Hopper, and Humanoid, modified for multiple objectives (e.g., speed vs. energy efficiency).
  • Building Energy Control: A high-dimensional task involving nine objectives to balance comfort and energy use in a multi-zone building.
  • Discrete Tasks: Minecart and MO-LunarLander.

Crucially, they tested these in both static settings and drift settings, where preferences and budgets were refreshed every 50 steps with added noise and observation delays.

Key Insights and Ideas

  • Arbitration is Essential: The most important "aha moment" is that agents shouldn't just be conditioned on history; they should use history to arbitrate between conflicting signals. This allows the agent to ignore a "stale" command if its recent experience suggests the user's intent has changed.
  • The Bias-Variance Trade-off in History: The paper provides a theoretical analysis showing that while longer histories reduce statistical noise, they increase "staleness" bias when tasks are drifting. The uncertainty-gated fusion helps the agent find the "sweet spot" automatically.
  • Unified Latent Context: By using a single latent belief to inform both preference optimization and constraint enforcement, the agent develops a more holistic understanding of what is both desirable and feasible in the current context.

Key Results

BECON consistently outperformed state-of-the-art baselines like COLA and C-MORL across the board.

  • Superior Feasible Fronts: In static evaluations, BECON improved the Feasible Hypervolume (FHV)—a measure of how well the agent covers the space of optimal, safe solutions—by an average of 98.4% over the second-best method.
  • Robustness to Drift: In the challenging drift scenarios, BECON achieved the highest scalarized returns and utility while maintaining a zero violation rate on most benchmarks.
  • Efficiency: As shown in the performance table below, BECON (rightmost column) consistently achieves higher utility (FUT) and hypervolume (FHV) while keeping mean costs (MC) low compared to competitors.
Table 1: Reference-threshold feasible-front evaluation of BECON against MORL baselines

Limitations and Future Work

While BECON is a significant step forward, the authors note a few limitations:

  • Stage-wise Constraints: The current model focuses on "per-step" safety limits. It doesn't yet handle "cumulative" budgets, like a robot needing to finish a task within a total battery limit.
  • Task Identifiability: The framework assumes that the task can be inferred from history. In environments where rewards and costs are extremely sparse or uninformative, the belief module might struggle.
  • Future Directions: The authors suggest extending the framework to handle joint task-environment drift (where the physics of the world changes alongside the goals) and exploring more complex long-horizon resource management.

Summary updated

Abstract

Constrained preference-conditioned MORL typically assumes full task observability, treating preferences and safety budgets as stationary task signals. In real-world deployment, however, these signals are often unreliable proxies: intent may drift, safety tolerances may shift with context, and exposed descriptors may be noisy, delayed, or incomplete. We therefore formulate constrained preference-conditioned MORL as a partially observed task-specification problem, where the agent must navigate a fundamental conflict between unreliable external signals and latent interaction dynamics. We propose \textbf{BECON}, a belief-conditioned framework for constrained MORL with drifting preferences and budgets, which infers latent task context from recent histories to estimate preferences and safety budgets. These estimates are fused with the exposed task signal via uncertainty-dependent gates, enabling the policy to defer to observations under high epistemic uncertainty while correcting them when history is informative. The fused context jointly conditions preference optimization and context-adaptive constraint enforcement. Experiments on diverse multi-objective control tasks demonstrate that BECON consistently improves over state-of-the-art MORL baselines, achieving stronger robustness and constraint satisfaction under task drift.
BibTeX
@misc{duc2026becon,
  author = {Bui Trong Duc and Huynh Thi Thanh Binh},
  title = {{BECON: Belief-Conditioned Constrained Multi-Objective Reinforcement Learning under Drifting Preferences and Budgets}},
  year = {2026},
  eprint = {2611.12053},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.12053},
}
Figure