Abstract
Reinforcement learning (RL) often struggles in environments characterized by sparse rewards or shifting goal distributions. We present DIRECT (Discriminative Reward Co-Training), a framework that leverages an agent's past successful trajectories to create a dense, internal surrogate reward signal. By training a GAN-style discriminator to distinguish between current behavior and high-performing historical data, DIRECT provides continuous guidance that enables agents to navigate sparse reward landscapes and adapt rapidly to environmental changes. Our evaluations on AI Safety Gridworlds demonstrate that DIRECT consistently outperforms standard RL baselines in both stability and adaptation speed.
Introduction
Reinforcement Learning typically relies on a well-defined reward function to guide an agent. However, real-world scenarios present two major hurdles: sparse rewards, where feedback is only received at the end of long action sequences, and distributional shifts, where the environment or goal changes post-training. Standard agents often wander aimlessly in the former and fail to adapt in the latter. DIRECT addresses these challenges by using an agent's own "greatest hits" to shape a smoother, more informative reward landscape.
Methodology
The DIRECT architecture integrates three core components to facilitate self-imitation through reward shaping:
- Self-Imitation Buffer ($B$): A memory bank that stores the top-performing trajectories (sequences of states and actions) achieved by the agent, ranked by their total return.
- Discriminator Network ($D$): Inspired by Generative Adversarial Networks (GANs), this network is trained concurrently with the agent to distinguish between trajectories from the current policy and the "beneficial" trajectories in the buffer.
- Discriminative Reward Shaping: The agent's reward is an interpolation of the environment reward ($r_{env}$) and the discriminator's output ($D$): $$r^*_t = \chi D(s, a, G) + (1 - \chi) r_{env}$$
By using the discriminator's verdict as a reward, the agent receives a dense signal at every step, steering it toward regions of the state space that previously led to success.
Experimental Setup
The authors evaluated DIRECT using the AI Safety Gridworlds benchmark. The environments consist of a starting point (blue), obstacles/lava (red), and a goal (green). To test robustness, three variations were used:
- Training Environment: The baseline configuration.
- Obstacle Shift: The layout of the lava changes, blocking previously optimal paths.
- Target Shift: The goal moves to a new location, requiring the agent to re-evaluate its learned policy.
Key Results and Insights
- Superiority in Sparse Settings: In sparse reward gridworlds, DIRECT was the only algorithm to consistently converge to optimal behavior. Baselines like PPO, A2C, and DQN often failed to find the goal entirely.
- The Discriminator as a Compass: The discriminator fills the void in sparse settings by rewarding the agent for moving in ways that resemble successful past runs, even if the goal hasn't been reached yet.
- Emergent Safety: The discriminator learned to assign near-zero rewards to actions leading into lava. Consequently, the agent learned safety as a byproduct of imitating successful (non-lethal) experiences rather than through explicit constraints.
- Reliability: DIRECT achieved a 100% success rate in reaching targets without hitting obstacles, whereas the best-performing baseline (PPO) reached the target only 50% of the time in certain shifted scenarios.
Limitations and Future Work
While effective, DIRECT introduces a "double optimization loop" that can make initial training slower than standard PPO in dense-reward environments. Additionally, if the discriminator becomes too accurate too quickly, the reward signal can saturate, leaving the policy without gradients to learn from. Future research will explore techniques to balance discriminator training and increase sample diversity within the buffer to prevent overfitting to single "lucky" paths.
Summary updated