Reinforcement Learning on Multiple Correlated Signals

Tim Brys, Ann NowéAAAI 2014 · DOI

Abstract

Traditional Reinforcement Learning (RL) typically relies on a single scalar reward signal to guide agent behavior. However, many complex tasks naturally involve multiple objectives or can be augmented with heuristic signals. This paper introduces the framework of Correlated Multi-Objective Markov Decision Processes (CMOMDPs), where objectives are not in conflict but are instead strongly correlated. We propose a novel approach involving multi-objectivization through potential-based reward shaping and an adaptive objective selection mechanism. This method allows agents to dynamically prioritize the most reliable signals based on confidence estimates, eliminating the need for manual weight tuning and improving learning robustness across various benchmarks.

1. Introduction

In standard Reinforcement Learning, an agent's goal is to maximize a cumulative scalar reward. While this simplifies the optimization target, it often masks the underlying complexity of real-world tasks which are inherently multi-faceted. For instance, autonomous driving requires balancing speed, safety, and fuel efficiency. Traditionally, these are treated as Multi-Objective MDPs where trade-offs lead to a Pareto front of solutions.

This research identifies a distinct and highly practical sub-class of these problems: Correlated Multi-Objective Markov Decision Processes (CMOMDPs). In a CMOMDP, the multiple objectives are positively correlated; improving one generally leads to the improvement of others. The primary challenge in these environments is not navigating trade-offs, but rather leveraging these multiple information sources to accelerate convergence toward a single optimal policy.

2. The CMOMDP Framework

The authors argue that almost any single-objective MDP can be transformed into a CMOMDP through a process called multi-objectivization. By adding auxiliary "shaping" rewards—such as distance-to-goal heuristics—the agent is provided with more frequent and informative feedback.

To ensure that these additional signals do not distract the agent from the true goal, the framework utilizes potential-based reward shaping. This mathematical approach is guaranteed to preserve the total order of policies, ensuring that the optimal policy for the multi-objective problem remains identical to the optimal policy of the original single-objective task.

3. Adaptive Objective Selection

A central difficulty in multi-objective RL is "scalarization"—the process of summing different reward signals into a single value. This usually requires tedious manual tuning of weights (e.g., is a 1-unit increase in speed worth a 5-unit decrease in fuel consumption?).

To bypass this, the paper proposes Adaptive Objective Selection:

  1. Parallel Learning: The agent maintains separate Q-functions (value estimates) for every objective simultaneously.
  2. Confidence Estimation: At each decision point, the agent calculates its confidence in each objective’s estimate. Confidence is derived from the variance of reward distributions or specific properties of the function approximator (such as tile-coding).
  3. Dynamic Ranking: Instead of summing rewards, the agent selects the objective it is currently most confident in to rank and select its next action.

This mechanism makes the agent scale-invariant. Because the selection is based on confidence rather than raw reward magnitude, there is no need to normalize different reward signals to the same range.

4. Experimental Evaluation

The framework was evaluated across several RL benchmarks to test its versatility:

  • Traffic Light Control: A natural CMOMDP where metrics like average delay and total throughput are inherently correlated. The adaptive agent outperformed standard single-objective learners by effectively switching focus between these metrics as traffic patterns changed.
  • Pursuit Domain: A predator-prey simulation used to validate multi-objectivization. By adding shaping rewards, the agent learned the pursuit task faster and more reliably.
  • Mountain Car & Keepaway Soccer: Ongoing work in these domains further validates that the approach scales to classic control and multi-agent coordination problems.

5. Key Insights and Discussion

One of the most significant findings is the agent’s state-space awareness. The adaptive selection process allows the agent to learn that certain signals are more reliable in specific regions of the environment. For example, a distance-based heuristic might be highly reliable at the start of a task but become noisy or misleading near the goal; the agent can autonomously learn to ignore the heuristic and focus on the primary reward as it nears completion.

Furthermore, the decisions made by the adaptive selector provide interpretability. By observing which objective the agent "trusts" at any given time, researchers can gain intuitive insights into which heuristics are most effective in different phases of the learning process.

6. Conclusion and Future Work

This paper demonstrates that treating RL problems as CMOMDPs offers a powerful path toward more efficient and less brittle learning. By moving away from manual weight tuning and toward confidence-based objective selection, agents can better navigate complex reward landscapes.

Future research will focus on providing a more rigorous mathematical definition of the CMOMDP class and exploring ensemble techniques to combine multiple predictors rather than just selecting one. Additionally, the authors aim to scale the framework to high-complexity environments such as StarCraft to test the limits of adaptive objective selection in massive state spaces.

Summary updated

BibTeX
@inproceedings{brys2014reinforcement,
  author = {Tim Brys and Ann Nowé},
  title = {{Reinforcement Learning on Multiple Correlated Signals}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2014},
  doi = {10.1609/aaai.v28i1.8773},
}
Figure