DIRECT: Learning from Sparse and Shifting Rewards using Discriminative Reward Co-Training

Philipp Altmann, Thomy Phan, Fabian Ritz, Thomas Gabor, Claudia Linnhoff-PopienarXiv 2023 · DOI · arXiv

Abstract

Reinforcement learning (RL) often struggles in environments characterized by sparse rewards or shifting goal distributions. We present DIRECT (Discriminative Reward Co-Training), a framework that leverages an agent's past successful trajectories to create a dense, internal surrogate reward signal. By training a GAN-style discriminator to distinguish between current behavior and high-performing historical data, DIRECT provides continuous guidance that enables agents to navigate sparse reward landscapes and adapt rapidly to environmental changes. Our evaluations on AI Safety Gridworlds demonstrate that DIRECT consistently outperforms standard RL baselines in both stability and adaptation speed.

Introduction

Reinforcement Learning typically relies on a well-defined reward function to guide an agent. However, real-world scenarios present two major hurdles: sparse rewards, where feedback is only received at the end of long action sequences, and distributional shifts, where the environment or goal changes post-training. Standard agents often wander aimlessly in the former and fail to adapt in the latter. DIRECT addresses these challenges by using an agent's own "greatest hits" to shape a smoother, more informative reward landscape.

Methodology

The DIRECT architecture integrates three core components to facilitate self-imitation through reward shaping:

  1. Self-Imitation Buffer ($B$): A memory bank that stores the top-performing trajectories (sequences of states and actions) achieved by the agent, ranked by their total return.
  2. Discriminator Network ($D$): Inspired by Generative Adversarial Networks (GANs), this network is trained concurrently with the agent to distinguish between trajectories from the current policy and the "beneficial" trajectories in the buffer.
  3. Discriminative Reward Shaping: The agent's reward is an interpolation of the environment reward ($r_{env}$) and the discriminator's output ($D$): $$r^*_t = \chi D(s, a, G) + (1 - \chi) r_{env}$$
The DIRECT Agent architecture, illustrating the flow between the environment, the policy (π), the discriminator (D), and the self-imitation buffer (B).

By using the discriminator's verdict as a reward, the agent receives a dense signal at every step, steering it toward regions of the state space that previously led to success.

Experimental Setup

The authors evaluated DIRECT using the AI Safety Gridworlds benchmark. The environments consist of a starting point (blue), obstacles/lava (red), and a goal (green). To test robustness, three variations were used:

  • Training Environment: The baseline configuration.
  • Obstacle Shift: The layout of the lava changes, blocking previously optimal paths.
  • Target Shift: The goal moves to a new location, requiring the agent to re-evaluate its learned policy.
The Target Shift scenario, where the goal (green) is moved to a new location, testing the agent's ability to adapt to shifting rewards.

Key Results and Insights

  • Superiority in Sparse Settings: In sparse reward gridworlds, DIRECT was the only algorithm to consistently converge to optimal behavior. Baselines like PPO, A2C, and DQN often failed to find the goal entirely.
  • The Discriminator as a Compass: The discriminator fills the void in sparse settings by rewarding the agent for moving in ways that resemble successful past runs, even if the goal hasn't been reached yet.
  • Emergent Safety: The discriminator learned to assign near-zero rewards to actions leading into lava. Consequently, the agent learned safety as a byproduct of imitating successful (non-lethal) experiences rather than through explicit constraints.
  • Reliability: DIRECT achieved a 100% success rate in reaching targets without hitting obstacles, whereas the best-performing baseline (PPO) reached the target only 50% of the time in certain shifted scenarios.

Limitations and Future Work

While effective, DIRECT introduces a "double optimization loop" that can make initial training slower than standard PPO in dense-reward environments. Additionally, if the discriminator becomes too accurate too quickly, the reward signal can saturate, leaving the policy without gradients to learn from. Future research will explore techniques to balance discriminator training and increase sample diversity within the buffer to prevent overfitting to single "lucky" paths.

Summary updated

Abstract

We propose discriminative reward co-training (DIRECT) as an extension to deep reinforcement learning algorithms. Building upon the concept of self-imitation learning (SIL), we introduce an imitation buffer to store beneficial trajectories generated by the policy determined by their return. A discriminator network is trained concurrently to the policy to distinguish between trajectories generated by the current policy and beneficial trajectories generated by previous policies. The discriminator's verdict is used to construct a reward signal for optimizing the policy. By interpolating prior experience, DIRECT is able to act as a surrogate, steering policy optimization towards more valuable regions of the reward landscape thus learning an optimal policy. Our results show that DIRECT outperforms state-of-the-art algorithms in sparse- and shifting-reward environments being able to provide a surrogate reward to the policy and direct the optimization towards valuable areas.
BibTeX
@misc{altmann2023direct,
  author = {Philipp Altmann and Thomy Phan and Fabian Ritz and Thomas Gabor and Claudia Linnhoff-Popien},
  title = {{DIRECT: Learning from Sparse and Shifting Rewards using Discriminative Reward Co-Training}},
  year = {2023},
  eprint = {2301.07421},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2301.07421},
}
Figure