Orchestrated Value Mapping for Reinforcement Learning

Mehdi Fatemi, Arash Tavakoli

Reinforcement Learning (RL) agents frequently encounter environments with "extreme" reward structures—where signals are either exceptionally sparse or vary drastically in scale. Standard algorithms like Q-Learning often struggle in these settings because the "action gap" (the mathematical difference in estimated value between the optimal and suboptimal actions) becomes inconsistent or vanishingly small.

In their ICLR 2022 paper, Orchestrated Value Mapping for Reinforcement Learning, Mehdi Fatemi and Arash Tavakoli introduce a powerful theoretical framework that allows agents to learn across multiple mathematical spaces simultaneously. This approach effectively handles diverse reward structures while maintaining strict mathematical guarantees that the agent will converge on the optimal strategy.

The Challenge of Extreme Rewards

Standard RL algorithms estimate action values ($Q$) in the standard space of real numbers. However, real-world environments often present complex reward structures that introduce two primary challenges:

  1. Non-uniform Action Gaps: If the value of the best action is 1000.1 and the second best is 1000.0, the agent may struggle to distinguish between the two, as the absolute difference is dwarfed by the overall magnitude.
  2. Scale Sensitivity: Standard deep RL networks (like DQN) are highly sensitive to reward magnitudes. If rewards range from 0.1 to 5,000, the network's gradients can become highly unstable during training.

Previous solutions, such as Log Q-Learning, attempted to map values to a logarithmic space to magnify small differences and "zoom in" on the action gap. Unfortunately, these methods introduced mathematical biases—specifically, Jensen’s gap—which prevented them from converging to the true optimal policy in stochastic environments.

The Orchestrated Value Mapping (OVM) Framework

The authors introduce Orchestrated Value Mapping (OVM), a framework built on two foundational pillars:

  1. Value Mapping ($f$): Instead of learning the standard $Q$-value, the agent learns a mapped value $\tilde{Q} = f(Q)$. The mapping function $f$ can be drawn from a broad class of functions, provided it is a smooth bijection.
  2. Linear Reward Decomposition: The total reward $r$ is split into $L$ distinct channels ($r = \sum \lambda_j r^{(j)}$). Each channel can be processed through its own unique mapping function, allowing the agent to view the same reward signal through multiple mathematical "lenses" simultaneously.

The Technical Breakthrough: Decoupled Averaging

The core technical contribution of OVM is its novel update rule. Traditionally, averaging stochastic updates in a mapped space introduces bias. OVM circumvents this by decoupling the averaging: it performs the stochastic averaging in the original, unmapped space before applying the mapping function. This decoupled approach ensures the algorithm remains strictly convergent, a property the authors prove mathematically.

LogLinDQN: The Best of Both Worlds

To demonstrate OVM's utility, the authors introduce a specific instantiation called LogLinDQN.

While logarithmic mapping preserves action gaps for small rewards, its derivative shrinks for large values, excessively compressing large rewards and rendering the agent "blind" to significant gains (a flaw that causes LogDQN to fail in dense-reward games like Skiing). LogLinDQN solves this by employing a piecewise mapping function: it operates logarithmically for small values to preserve action gaps, and linearly for large values to maintain sensitivity to big rewards.

Experimental Results

The authors evaluated their approach on 55 different games in the Atari 2600 suite via the Arcade Learning Environment (ALE), testing across a spectrum of sparse-reward challenges (e.g., Montezuma's Revenge) and dense-reward games.

  • Superior Robustness: LogLinDQN consistently outperformed both standard DQN and LogDQN. It successfully unified the strengths of its predecessors, handling sparse rewards as effectively as LogDQN and dense rewards as well as standard DQN.
  • Competitive with State-of-the-Art: Despite being a relatively simple modification to the standard DQN architecture, LogLinDQN achieved performance competitive with Rainbow—a highly complex algorithm integrating six distinct RL enhancements (such as distributional RL and prioritized replay).
  • Human-Normalized Scores: Across the 55-game suite, LogLinDQN demonstrated significant improvements in both mean and median human-normalized scores compared to prior baselines.

Limitations and Future Directions

While OVM opens up a highly flexible design space, it does come with limitations. Selecting the appropriate mapping functions ($f$) and decomposition weights ($\lambda$) currently requires domain knowledge and hyperparameter tuning. Furthermore, while the convergence proofs are rigorous for tabular settings (where every state-action pair can be mapped), theoretical guarantees for deep RL function approximation remain an open challenge.

Looking forward, the authors suggest exploring adaptive channel weights, allowing agents to dynamically learn which mapping "lens" is most effective for their current state in the environment. This could further automate the orchestration process, paving the way for even more robust reinforcement learning agents.

Summary updated

Abstract

We present a general convergent class of reinforcement learning algorithms that is founded on two distinct principles: (1) mapping value estimates to a different space using arbitrary functions from a broad class, and (2) linearly decomposing the reward signal into multiple channels. The first principle enables incorporating specific properties into the value estimator that can enhance learning. The second principle, on the other hand, allows for the value function to be represented as a composition of multiple utility functions. This can be leveraged for various purposes, e.g. dealing with highly varying reward scales, incorporating a priori knowledge about the sources of reward, and ensemble learning. Combining the two principles yields a general blueprint for instantiating convergent algorithms by orchestrating diverse mapping functions over multiple reward channels. This blueprint generalizes and subsumes algorithms such as Q-Learning, Log Q-Learning, and Q-Decomposition. In addition, our convergence proof for this general class relaxes certain required assumptions in some of these algorithms. Based on our theory, we discuss several interesting configurations as special cases. Finally, to illustrate the potential of the design space that our theory opens up, we instantiate a particular algorithm and evaluate its performance on the Atari suite.
BibTeX
@inproceedings{fatemi2022orchestrated,
  author = {Mehdi Fatemi and Arash Tavakoli},
  title = {{Orchestrated Value Mapping for Reinforcement Learning}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=c87d0TS4yX},
}
Figure