An Empirical Investigation of Value-Based Multi-objective Reinforcement Learning for Stochastic Environments

Kewen Ding, Peter Vamplew, Cameron Foale, Richard Dazeley

Introduction

In standard Reinforcement Learning (RL), an agent seeks to maximize a single scalar reward. However, real-world scenarios often involve competing objectives, such as balancing speed against safety or cost against quality. Multi-objective RL (MORL) addresses these trade-offs. This paper investigates value-based MORL under the Scalarised Expected Reward (SER) criterion, where the objective is to maximize the utility of the average outcome over many episodes.

The researchers identify two critical barriers to convergence in stochastic environments:

  1. Local Decision-Making: Standard Q-learning makes decisions based on the current local state. In stochastic settings, the optimal global policy for the average reward may require actions that appear sub-optimal when viewed in isolation from a single state.
  2. Noisy Q-value Estimates: Environmental randomness leads to noisy estimates. In MORL, particularly with non-linear preferences, minor estimation noise can cause an agent to oscillate between radically different reward vectors, leading to significant instability.

The Space Traders Benchmark

To analyze these failures, the study utilizes the Space Traders MOMDP, a benchmark environment specifically designed to highlight the shortcomings of value-based MORL.

The Space Traders MOMDP environment transitions and rewards.

As shown in Figure 1, the environment consists of two states (A and B) where an agent chooses between "Direct," "Indirect," and "Teleport" actions to deliver cargo. The complexity arises from the stochastic nature of these actions and the multi-dimensional reward vectors associated with success and failure.

Detailed reward and probability structure for the Space Traders environment.

Figure 2 details the specific trade-offs. For instance, the "Indirect" action in State A offers a guaranteed but low reward, while "Direct" offers a higher potential reward but carries a risk of failure. Under the SER criterion, the agent must account for the global average across all possible paths, a task that traditional local Q-learning struggles to perform.

Methodological Approaches

The researchers evaluated four distinct approaches within a tabular MOQ-learning framework:

  • Baseline MOQ-learning: An extension of Q-learning using vector rewards, conditioning action selection on rewards accumulated within the current episode.
  • Reward Engineering: Manually modifying reward signals (e.g., adding "hints" about failure probabilities) to provide better local information.
  • MOSS (Multi-objective Stochastic State Q-learning): A novel heuristic that tracks global statistics—such as average return over all episodes and state visitation probabilities—to help the agent weigh local actions against the "big picture."
  • Policy Options: A "meta-action" approach where the agent selects an entire deterministic policy at the start of an episode, bypassing local decision-making entirely.

Additionally, the team used Learning Rate Decay (reducing $\alpha$ to zero) as a diagnostic tool to isolate the impact of noise from algorithmic logic.

Key Insights and Results

The "Band-aid" Failure

While reward engineering can improve performance on a specific problem, the authors demonstrate its lack of generality. Adding a single intermediate state (the "3-state variant") caused engineered solutions to fail, proving they are not robust fixes for the underlying SER paradox.

The SER Paradox and Noise

To optimize for the average of all runs, an agent must effectively know what happens in runs where it doesn't visit the current state. This global requirement is fundamentally at odds with the local nature of traditional Q-learning. Furthermore, noise acts as a stability killer; it can lead an agent to believe a policy failing its primary safety threshold is optimal, causing behavior to oscillate wildly.

Performance Comparison

  • Baseline Failure: Standard MOQ-learning found the optimal policy in only 1 out of 20 trials. Even with a decaying learning rate to force stability, it stabilized on the wrong policy 100% of the time.
  • MOSS Success: The MOSS algorithm significantly improved performance, finding the optimal policy in 15 out of 20 trials on the original benchmark.
  • The Gold Standard: Only the Policy Options approach combined with Decayed Learning Rate achieved a 100% success rate across all benchmarks, confirming that both local decision-making and estimation noise must be addressed for perfect convergence.

Conclusion

The study concludes that while Policy Options provide a robust solution, they suffer from the "curse of dimensionality" in larger state spaces. Future work should explore Policy-based MORL (using policy gradients to optimize the policy as a whole) and Distributional RL. By learning the full distribution of rewards rather than just the expected average, agents may gain the context necessary to overcome the noisy estimate problem and make SER-optimal decisions.

Summary updated

BibTeX
@article{ding2025empirical,
  author = {Kewen Ding and Peter Vamplew and Cameron Foale and Richard Dazeley},
  title = {{An Empirical Investigation of Value-Based Multi-objective Reinforcement Learning for Stochastic Environments}},
  journal = {Knowl. Eng. Rev.},
  year = {2025},
  doi = {10.1017/s0269888925100052},
}
Figure