TL;DR
This paper introduces a novel method for learning reward functions for AI systems using human preferences, specifically by employing differentiable decision trees. This approach aims to make the learned reward functions interpretable, allowing us to understand why an AI makes certain decisions, which is crucial for safety-critical applications.
What Problem Does This Paper Address?
The core problem this paper tackles is the "black box" nature of many modern AI reward learning systems. While neural networks can learn complex reward functions from human feedback (like preferences), their internal workings are often opaque. This lack of interpretability makes it difficult to trust that the AI's learned objective truly aligns with human intentions, especially in high-stakes scenarios like autonomous driving or assistive robotics. The authors want to bridge this gap by creating reward functions that are both expressive enough to capture nuanced human preferences and interpretable enough for humans to understand and verify.
Methods
The authors propose using Differentiable Decision Trees (DDTs) for reward learning. Unlike traditional decision trees, DDTs are designed to be fully differentiable, allowing them to be trained end-to-end using gradient descent.
Here's a breakdown of their approach:
- Differentiable Decision Trees: These trees combine the interpretability of decision trees with the flexibility of neural networks.
- Internal Nodes: Each internal node uses a linear layer with learnable parameters (weights and bias) and a sigmoid activation to produce a "routing probability." This probability determines how likely an input is to go down the left or right branch. A parameter $\beta$ controls the "softness" of these decisions.
- Leaf Nodes: Each leaf node represents a potential reward output. They are modeled as a softmax distribution over a discrete set of reward values.
- Reward Prediction: The final reward for a given input is a weighted sum of the rewards at each leaf node, where the weights are determined by the path probabilities through the tree. This provides a "soft" reward estimate, crucial for differentiability.
- Training with Preferences: The DDT is trained using pairwise preference labels over trajectories. If trajectory $T_i$ is preferred over $T_j$, the model is trained to assign a higher reward to $T_i$ than $T_j$. This is achieved using a cross-entropy loss derived from the Bradley-Terry model.
Datasets
The paper evaluates their method on two domains:
- MNIST Gridworld: A custom environment where grid cells are associated with MNIST digits. Trajectories are sequences of these grid cells. Preferences are generated based on the sum of ground truth labels of the images within each trajectory. This domain is used to test learning from high-dimensional visual inputs.
- OpenAI Gym's Cartpole: A classic reinforcement learning environment. For this experiment, they generate suboptimal trajectories using a random policy. A synthetic preference labeler is created based on whether the cart position and pole angle are within desired ranges. This domain is used to test learning from low-dimensional state inputs.
Key Insights and Ideas
- Interpretability through Tree Structure: The core idea is that the hierarchical structure of a decision tree naturally lends itself to interpretation. Each split in the tree represents a decision criterion, making it easier to trace the reasoning behind a reward prediction.
- Differentiability is Key: Making the decision tree differentiable is crucial for enabling end-to-end training with gradient descent, which is a standard and powerful technique in machine learning. This avoids the multi-stage, indirect optimization required by non-differentiable tree methods.
- Bridging High and Low Dimensions: The method demonstrates the ability to learn interpretable reward functions from both low-dimensional (Cartpole) and high-dimensional visual (MNIST Gridworld) state spaces.
- Tree Structure as a Diagnostic Tool: The authors suggest that the learned tree structure itself can be used to assess how well the learned reward function aligns with human preferences. For instance, if a split in the tree is based on a feature that humans wouldn't intuitively use for preference, it might indicate misalignment.
Key Results
- MNIST Gridworld: The learned DDT of depth 1 successfully learned to branch based on features that visually corresponded to zeros and ones in the MNIST digits, indicating it learned interpretable routing features.
- Cartpole: The learned DDT of depth 2 showed that the model learned to approximate preferences related to the pole angle but paid less attention to the cart position. This suggests a potential misalignment between the learned reward and the true underlying preferences, which the authors could identify by inspecting the tree.
Limitations and Future Work
- Limitations:
- The paper acknowledges that while their approach is more interpretable than black-box neural networks, the interpretability is still limited by the depth and complexity of the learned tree.
- The Cartpole experiment revealed a potential misalignment, highlighting that the learned reward might not perfectly capture human intent, and the tree structure is used to diagnose this.
- Future Work:
- Applying the framework to more complex, real-world tasks.
- Leveraging more sophisticated techniques to analyze the decisions made within the reward DDT.
- Exploring tasks with more complex reward functions requiring more than two reward classes or deeper trees.
- Distilling existing black-box neural network reward functions into interpretable DDTs.
Summary updated