This NeurIPS 2020 paper presents RD² (Reward Decomposition with Representation Disentanglement), a novel framework for breaking down a single scalar reward signal into multiple, more interpretable sub-rewards. By ensuring each sub-reward is tied to a unique and minimal set of state features, the authors improve the sample efficiency and interpretability of reinforcement learning (RL) agents.
The Credit Assignment Problem
In many RL tasks, the agent receives a single reward (e.g., a score) that is actually the sum of several different events—like picking up a coin, defeating an enemy, and reaching a checkpoint in a game. If an agent only sees the total sum, it faces a difficult credit assignment problem: which action caused which part of the reward?
Previous "Reward Decomposition" methods tried to solve this, but they were often policy-dependent. They required the agent to behave in specific ways or needed the ability to reset the environment to arbitrary states to "tease out" the reward structure. RD² aims to discover these sub-rewards in a policy-independent manner, meaning it can learn the reward structure from any transition data, regardless of how the agent is currently acting.
The Minimal Supporting Principle
The core idea behind RD² is the Minimal Supporting Principle. The authors argue that a "good" decomposition is one where each sub-reward is explained by the smallest possible set of features (compactness) and where different sub-rewards don't rely on the same features (uniqueness).
To implement this, the authors use a deep learning architecture that includes:
- Feature Extraction: A convolutional neural network (CNN) processes raw images into a feature map.
- Learnable Masks: For each potential sub-reward channel, the model learns a spatial mask that "picks out" relevant parts of the feature map.
- Sub-reward Predictors: Small networks that take the masked features and predict a sub-reward value.
The training objective consists of three primary loss functions:
- Reconstruction ($L_{sum}$): Ensures the sum of predicted sub-rewards equals the actual observed total reward.
- Compactness ($L_{mini}$): Uses an entropy-based surrogate loss to force the masks to be as sparse as possible.
- Diversity ($L_{div}$): Encourages different channels to focus on different features, preventing multiple channels from learning the same thing.
Experimental Setup
The authors evaluated RD² on two main platforms:
- Monster-Treasure (Toycase): A custom mini-gridworld environment where the agent gets positive rewards for treasure and negative rewards for hitting a monster. This provides a "ground truth" to verify if the algorithm correctly identifies the two distinct reward sources.
- Atari Domain: Specifically chosen games with multi-reward structures, including UpNDown, Gopher, Asterix, and Centipede. These serve as complex, high-dimensional benchmarks where the true reward structure is latent (hidden).
Key Insights and Results
Feature-Reward Coupling: The most powerful insight is that reward decomposition shouldn't just be about splitting a number; it should be about disentangling the representations that cause those numbers. If one reward comes from a "monster" and another from a "treasure," the agent's internal representation should reflect that physical separation.
Accurate Decomposition: In the Monster-Treasure toy case, RD² perfectly separated the monster penalty from the treasure reward, even when both events happened simultaneously (resulting in a total reward of zero).
Improved Sample Efficiency: When combined with Q-learning, RD² significantly outperformed Rainbow (a state-of-the-art RL baseline) and DRDRL across several Atari games. The agent reached higher scores much earlier in the training process.
Meaningful Saliency: Because the model uses masks on feature maps, the researchers can generate saliency maps. In UpNDown, the model successfully assigned one channel to "hitting flags" and another to "jumping on cars," with the saliency maps clearly highlighting the respective objects for each channel.
Limitations and Future Work
- Assumption of Factored Structure: RD² assumes that the reward can be decomposed into parts based on local features. If a reward is "global" or depends on a complex entanglement of all features at once, this method might struggle.
- Fixed Number of Channels: The user must pre-specify the number of sub-reward channels ($K$). An ideal future version would automatically discover the optimal number of channels.
- Multi-Agent Scaling: The authors suggest that exploring reward decomposition in multi-agent settings—where the state might have a natural graph structure—is a promising next step.
Summary updated