TL;DR
This paper introduces Preference-Guided Deep Q-Network (PGDQN), a novel exploration strategy for Deep Q-Learning (DQN) that learns an "action preference" alongside Q-values. This preference guides exploration to be more efficient and less biased than traditional methods, leading to faster learning and better performance across various environments.
What Problem Does This Paper Address?
Deep Reinforcement Learning (DRL), particularly Deep Q-Networks (DQN), has achieved remarkable success in complex tasks. However, a critical challenge lies in exploration: how can an agent efficiently discover optimal actions without getting stuck in local optima or wasting time on suboptimal exploration? Traditional methods like epsilon-greedy explore randomly, while more advanced techniques like NoisyNets or energy-based exploration can introduce biases or struggle with multi-modal Q-value landscapes. This paper aims to bridge this gap by developing a stochastic exploration method that is both efficient and unbiased, directly learning an action preference that aligns with the underlying Q-value landscape.
Methods
The core innovation is a dual-architecture for DQN.
- Q-branch: This is a standard DQN component that learns the state-action values (Q-values).
- Preference branch: This novel component learns an "action preference" distribution. This preference is not a direct probability of choosing an action but rather a learned signal indicating how favorable an action is.
These two branches share a common embedding network, allowing them to learn from the same state representations. The exploration strategy is a preference-guided epsilon-greedy policy. With probability $(1-\epsilon)$, the agent takes the greedy action (highest Q-value). With probability $\epsilon$, it samples an action proportional to its learned preference.
The preference branch is trained to minimize the KL divergence between the action preference and the corresponding Q-values. This is achieved by maximizing an entropy-regularized objective function, which theoretically guarantees a policy improvement property, similar to standard Q-learning, and prevents premature convergence due to inaccurate Q-values early in training. The authors also propose a method to automatically tune the temperature parameter controlling the entropy.
Datasets
The paper evaluates PGDQN on two sets of environments:
- Classic Control Tasks: CartPole, MountainCar, and Acrobot.
- Atari Games: Alien, Pong, Berzerk, CrazyClimber, FishingDerby, and Amidar.
These are standard benchmarks in DRL research, and the paper uses their default settings. No new benchmarks were introduced.
Key Insights and Ideas
- Learning Action Preference: The key insight is that instead of just exploring randomly or with fixed noise, learning an explicit "action preference" that mirrors the Q-value landscape can lead to more intelligent and efficient exploration.
- Unbiased Exploration: By designing the preference learning objective to minimize statistical distance to Q-values and proving the policy improvement theorem, the method avoids introducing bias into the core DQN learning process.
- Adaptive Exploration: The learned preference can adapt to the Q-value landscape, becoming unimodal when a single action is clearly superior or multimodal when multiple actions are promising, which is a significant advantage over methods like NoisyNets.
- Shared Embedding Benefits: Using a shared embedding network for both Q-value and preference learning demonstrably improves performance, suggesting that learning from multiple perspectives enhances representation learning.
Key Results
PGDQN consistently outperforms several strong DQN baselines (DQN, D2QN, V-D D3QN, NoisyNet-DQN) across nine environments.
- Performance Improvement: PGDQN shows significant performance gains, with improvements of up to 136% in the Amidar environment.
- Data Efficiency: PGDQN is substantially more data-efficient, consuming up to 85% fewer frames to reach baseline performance levels (e.g., in Berzerk).
- Faster Convergence: The training curves in Figure 2 clearly show PGDQN converging much faster than the baselines.
- Lower Variance: PGDQN also exhibits lower variance in its learning curves, indicating more stable training.
- Robustness: Ablation studies suggest PGDQN is robust to hyperparameter choices for the preference branch update frequency and learning rate.
Limitations and Future Work
- FishingDerby Anomaly: The paper notes a slight degradation in efficiency and performance for PGDQN in the FishingDerby environment, suggesting that while generally superior, it's not universally optimal.
- Generalization to Continuous Actions: The current method is designed for discrete action spaces.
- Future Work: The authors suggest extending PGDQN to human-in-the-loop (HIL) RL frameworks and applying it to complex real-world tasks like autonomous driving, potentially incorporating personalized driving styles. They also highlight the potential for combining human policy with action preference for decision-making problems.
Summary updated