Off-Policy Reinforcement Learning with High Dimensional Reward

D. N. Lee, Michael R. KosorokarXiv 2024 · DOI · arXiv

TL;DR

This paper extends Distributional Reinforcement Learning (DRL) to handle high-dimensional and even infinite-dimensional rewards by proving that the distributional Bellman operator remains a "contraction" (it converges to a stable solution) in complex mathematical spaces called Banach spaces. The authors provide a practical algorithm that approximates these high-dimensional rewards in lower-dimensional Euclidean spaces, allowing RL agents to optimize for complex, multi-faceted objectives like balancing medical efficacy against specific side-effect thresholds.

What Problem Does This Paper Address?

Most Reinforcement Learning (RL) algorithms are "scalar-centric"—they try to maximize a single number representing the expected total reward. However, the real world is rarely that simple.

The authors identify two major gaps:

  1. Multi-dimensional Objectives: In fields like finance or medicine, we often care about multiple outcomes simultaneously (e.g., maximizing a stock's return while minimizing its volatility, or maximizing drug effectiveness while strictly limiting three different side effects).
  2. Infinite-Dimensional Rewards: Some rewards are inherently continuous structures, such as 3D brain imaging data or time-series signals, which cannot be easily squashed into a single scalar without losing vital information.

While "Distributional RL" (DRL) exists to model the full range of possible outcomes rather than just the average, it has traditionally been limited to simple Euclidean spaces. This paper provides the mathematical "rigging" needed to move DRL into the much broader world of high-dimensional and infinite-dimensional reward spaces.

Methods

The paper bridges the gap between abstract functional analysis and practical machine learning through three main contributions:

  • Banach Space Framework: The authors define rewards and returns within a separable Banach space (a broad category of mathematical spaces that includes standard vectors as well as infinite-dimensional functions).
  • The Contraction Proof: They rigorously prove that the distributional Bellman operator is a contraction mapping in these spaces. In plain English: they prove that if you keep updating your "guess" about the reward distribution using their formula, you are guaranteed to eventually arrive at the correct, unique answer.
  • Hypercube Approximation: Since we can't actually compute in infinite dimensions, they prove a "Hypercube Approximation Theory." This shows that any random variable in a Banach space can be projected onto a finite-dimensional "box" (hypercube) in Euclidean space with an arbitrarily small error.
  • Max-Sliced Wasserstein Distance: To compare high-dimensional distributions efficiently, they use the "max-sliced" version of the Wasserstein metric. Instead of calculating the distance between two complex 3D clouds of data points directly (which is computationally expensive), they project the clouds onto 1D lines and measure the distance there, picking the "worst-case" projection to ensure accuracy.

Datasets

The paper primarily uses synthetic simulations to validate its theoretical claims:

  • 2D Simulation Environment: A custom-built environment with a 2-dimensional state space and a 2-dimensional reward space.
  • Brain Imaging Motivation: While not used for training, the authors reference 3D brain activity data from the NeuroVault repository (specifically a study on first-episode psychosis patients) to illustrate the real-world need for infinite-dimensional reward modeling.

Key Insights and Ideas

  • Beyond the Mean: The "Aha!" moment here is that by modeling the entire distribution of a multi-dimensional reward, you can optimize for "Utility Functions" that a standard RL agent couldn't even understand—like maximizing the 25th percentile of a return to ensure a "safety floor."
  • The Power of Projection: The paper demonstrates that even if a reward is infinitely complex (like a continuous image), a finite-dimensional approximation is "good enough" for the Bellman operator to work effectively.
  • Mathematical Generality: By proving their theorems for Banach spaces, the authors have created a "future-proof" foundation. Their math applies to standard vectors, Hilbert spaces, and even spaces of continuous functions.

Key Results

  • Convergence in Scenario 1: When the environment's rules (transitions and rewards) were known, the algorithm's estimate of the 2D return distribution converged rapidly. The distance between the estimate and the truth fell below 0.5 within just 10 iterations.
  • Robustness in Scenario 2: Even when the environment was unknown—requiring the agent to learn transitions via Neural Networks and rewards via regression—the algorithm successfully estimated the return distributions as it gathered more data.
  • Policy Optimization: In a test where the agent had to maximize a complex utility (the median of one reward plus a probability threshold for another), the proposed algorithm identified an optimal policy that outperformed 95% of all other candidate policies.

Limitations and Future Work

  • The Curse of Dimensionality: The current algorithm uses "bins" (discrete supports) to represent distributions. As the number of reward dimensions grows, the number of bins required grows exponentially, which could become a bottleneck.
  • Scaling with Transformers: The authors acknowledge that "table-based" updates (storing values for every state) are inefficient. They suggest that future work should integrate Transformer-based architectures (like Decision Transformers) to handle the sequence modeling and high-dimensional distributions more elegantly.
  • Real-World Testing: The current results are limited to 2D simulations. The next major step is applying this framework to the high-dimensional medical imaging or financial datasets that motivated the theory.

Summary updated

Abstract

Conventional off-policy reinforcement learning (RL) focuses on maximizing the expected return of scalar rewards. Distributional RL (DRL), in contrast, studies the distribution of returns with the distributional Bellman operator in a Euclidean space, leading to highly flexible choices for utility. This paper establishes robust theoretical foundations for DRL. We prove the contraction property of the Bellman operator even when the reward space is an infinite-dimensional separable Banach space. Furthermore, we demonstrate that the behavior of high- or infinite-dimensional returns can be effectively approximated using a lower-dimensional Euclidean space. Leveraging these theoretical insights, we propose a novel DRL algorithm that tackles problems which have been previously intractable using conventional reinforcement learning approaches.
BibTeX
@misc{lee2024off,
  author = {D. N. LEE and Michael R. Kosorok},
  title = {{Off-Policy Reinforcement Learning with High Dimensional Reward}},
  year = {2024},
  eprint = {2408.07660},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2408.07660},
}
Figure