RD$^2$: Reward Decomposition with Representation Decomposition

Zichuan Lin, Derek Yang, Li Zhao, Tao Qin, Guangwen Yang, Tie-Yan LiuNeurIPS 2020 · OpenReview

This NeurIPS 2020 paper presents RD² (Reward Decomposition with Representation Disentanglement), a novel framework for breaking down a single scalar reward signal into multiple, more interpretable sub-rewards. By ensuring each sub-reward is tied to a unique and minimal set of state features, the authors improve the sample efficiency and interpretability of reinforcement learning (RL) agents.

The Credit Assignment Problem

In many RL tasks, the agent receives a single reward (e.g., a score) that is actually the sum of several different events—like picking up a coin, defeating an enemy, and reaching a checkpoint in a game. If an agent only sees the total sum, it faces a difficult credit assignment problem: which action caused which part of the reward?

Previous "Reward Decomposition" methods tried to solve this, but they were often policy-dependent. They required the agent to behave in specific ways or needed the ability to reset the environment to arbitrary states to "tease out" the reward structure. RD² aims to discover these sub-rewards in a policy-independent manner, meaning it can learn the reward structure from any transition data, regardless of how the agent is currently acting.

The Minimal Supporting Principle

The core idea behind RD² is the Minimal Supporting Principle. The authors argue that a "good" decomposition is one where each sub-reward is explained by the smallest possible set of features (compactness) and where different sub-rewards don't rely on the same features (uniqueness).

To implement this, the authors use a deep learning architecture that includes:

  1. Feature Extraction: A convolutional neural network (CNN) processes raw images into a feature map.
  2. Learnable Masks: For each potential sub-reward channel, the model learns a spatial mask that "picks out" relevant parts of the feature map.
  3. Sub-reward Predictors: Small networks that take the masked features and predict a sub-reward value.

The training objective consists of three primary loss functions:

  • Reconstruction ($L_{sum}$): Ensures the sum of predicted sub-rewards equals the actual observed total reward.
  • Compactness ($L_{mini}$): Uses an entropy-based surrogate loss to force the masks to be as sparse as possible.
  • Diversity ($L_{div}$): Encourages different channels to focus on different features, preventing multiple channels from learning the same thing.

Experimental Setup

The authors evaluated RD² on two main platforms:

  • Monster-Treasure (Toycase): A custom mini-gridworld environment where the agent gets positive rewards for treasure and negative rewards for hitting a monster. This provides a "ground truth" to verify if the algorithm correctly identifies the two distinct reward sources.
  • Atari Domain: Specifically chosen games with multi-reward structures, including UpNDown, Gopher, Asterix, and Centipede. These serve as complex, high-dimensional benchmarks where the true reward structure is latent (hidden).

Key Insights and Results

Feature-Reward Coupling: The most powerful insight is that reward decomposition shouldn't just be about splitting a number; it should be about disentangling the representations that cause those numbers. If one reward comes from a "monster" and another from a "treasure," the agent's internal representation should reflect that physical separation.

Accurate Decomposition: In the Monster-Treasure toy case, RD² perfectly separated the monster penalty from the treasure reward, even when both events happened simultaneously (resulting in a total reward of zero).

Combined results for the Monster-Treasure toy case showing reward error and mask convergence.

Improved Sample Efficiency: When combined with Q-learning, RD² significantly outperformed Rainbow (a state-of-the-art RL baseline) and DRDRL across several Atari games. The agent reached higher scores much earlier in the training process.

Meaningful Saliency: Because the model uses masks on feature maps, the researchers can generate saliency maps. In UpNDown, the model successfully assigned one channel to "hitting flags" and another to "jumping on cars," with the saliency maps clearly highlighting the respective objects for each channel.

Limitations and Future Work

  • Assumption of Factored Structure: RD² assumes that the reward can be decomposed into parts based on local features. If a reward is "global" or depends on a complex entanglement of all features at once, this method might struggle.
  • Fixed Number of Channels: The user must pre-specify the number of sub-reward channels ($K$). An ideal future version would automatically discover the optimal number of channels.
  • Multi-Agent Scaling: The authors suggest that exploring reward decomposition in multi-agent settings—where the state might have a natural graph structure—is a promising next step.

Summary updated

Abstract

Reward decomposition, which aims to decompose the full reward into multiple sub-rewards, has been proven beneficial for improving sample efficiency in reinforcement learning. Existing works on discovering reward decomposition are mostly policy dependent, which constrains diverse or disentangled behavior between different policies induced by different sub-rewards. In this work, we propose a set of novel reward decomposition principles by constraining uniqueness and compactness of different state features/representations relevant to different sub-rewards. Our principles encourage sub-rewards with minimal relevant features, while maintaining the uniqueness of each sub-reward. We derive a deep learning algorithm based on our principle, and term our method as RD$^2$, since we learn reward decomposition and representation decomposition jointly. RD$^2$ is evaluated on a toy case, where we have the true reward structure, and some Atari environments where reward structure exists but is unknown to the agent to demonstrate the effectiveness of RD$^2$ against existing reward decomposition methods.
BibTeX
@inproceedings{lin2020rd,
  author = {Zichuan Lin and Derek Yang and Li Zhao and Tao Qin and Guangwen Yang and Tie-Yan Liu},
  title = {{RD$^2$: Reward Decomposition with Representation Decomposition}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=B-uG8omJHCi},
}
Figure