Never Forget: Balancing Exploration and Exploitation via Learning Optical Flow

Hsuan-Kung Yang, Po-Han Chiang, Kuan-Wei Ho, Min-Fong Hong, Chun-Yi LeearXiv 2019 · arXiv

Introduction

Reinforcement Learning (RL) agents frequently encounter the "sparse reward" problem, where meaningful feedback from the environment is rare. To encourage exploration, researchers often employ intrinsic curiosity modules that provide a "bonus" reward for visiting novel states. The standard Intrinsic Curiosity Module (ICM) generates this bonus based on errors in predicting the next frame. However, ICM suffers from "catastrophic forgetting": as the model learns new areas, it forgets old ones, leading the agent to redundantly re-explore familiar territory. This paper introduces the Flow-based Intrinsic Curiosity Module (FICM), which leverages optical flow estimation to create a more stable and persistent curiosity signal.

Methods: The Flow-based Intrinsic Curiosity Module (FICM)

The core innovation of FICM is shifting the curiosity signal from frame generation to motion correspondence. Instead of predicting a future state, the module estimates the movement of pixels (optical flow) between consecutive frames $s_t$ and $s_{t+1}$.

FICM Architecture and Agent Loop

As shown in the figure above, FICM predicts both forward ($F_{forward}$) and backward ($F_{backward}$) optical flow. A warping function $W$ uses these flows to reconstruct the frames. The intrinsic reward $r_t^i$ is defined as the reconstruction error—the difference between the warped image and the actual image. High error indicates a motion pattern the agent has not yet mastered, signaling a novel state. The authors propose two variants: FICM-S, which uses a simple encoder-decoder on stacked frames, and FICM-C, which employs a correlation layer to explicitly compare feature maps. Notably, FICM is action-independent, simplifying the architecture compared to the original ICM.

Experimental Setup

The proposed method was evaluated in the ViZDoom (DoomMyWayHome-v0) environment across two difficulty levels:

  1. Sparse Reward: A 9-room maze where the agent must find a specific item.
  2. Very Sparse Reward: The agent starts significantly further from the goal, making random discovery highly unlikely.

Agents processed raw $42 \times 42$ grayscale images. Performance was measured by the total reward achieved over 15 million timesteps.

Key Results and Discussion

The experimental results demonstrate that FICM effectively solves the catastrophic forgetting problem that plagues standard ICM.

Performance comparison in Sparse and Very Sparse settings

As illustrated in the plots, the standard ICM baseline experiences a sharp performance crash after approximately 10 million timesteps in the Sparse setting. In contrast, both FICM-S and FICM-C maintain peak performance throughout the entire training duration. In the "Very Sparse" setting, the gap is even more pronounced: while the pixel-based ICM baseline fails to learn entirely, both FICM variants successfully reach the goal and maintain stability.

Beyond stability, FICM is highly efficient. FICM-S achieves superior results to ICM while utilizing only ~56% of the parameters. Qualitative "footprint" analysis confirmed that FICM successfully memorizes terrain; unlike ICM, where visited paths eventually appear "new" (high reward) again, FICM correctly identifies them as familiar, preventing redundant exploration.

Conclusion

By prioritizing correspondence (how things move) over generation (what things look like), the Flow-based Intrinsic Curiosity Module provides a more robust signal for exploration. FICM eliminates the performance drops associated with catastrophic forgetting and enables successful learning in environments with extreme reward sparsity. Future work will explore the application of flow-based curiosity in non-visual tasks and more complex, high-dimensional dynamic environments.

Summary updated

Abstract

Exploration bonus derived from the novelty of the states in an environment has become a popular approach to motivate exploration for deep reinforcement learning agents in the past few years. Recent methods such as curiosity-driven exploration usually estimate the novelty of new observations by the prediction errors of their system dynamics models. Due to the capacity limitation of the models and difficulty of performing next-frame prediction, however, these methods typically fail to balance between exploration and exploitation in high-dimensional observation tasks, resulting in the agents forgetting the visited paths and exploring those states repeatedly. Such inefficient exploration behavior causes significant performance drops, especially in large environments with sparse reward signals. In this paper, we propose to introduce the concept of optical flow estimation from the field of computer vision to deal with the above issue. We propose to employ optical flow estimation errors to examine the novelty of new observations, such that agents are able to memorize and understand the visited states in a more comprehensive fashion. We compare our method against the previous approaches in a number of experimental experiments. Our results indicate that the proposed method appears to deliver superior and long-lasting performance than the previous methods. We further provide a set of comprehensive ablative analysis of the proposed method, and investigate the impact of optical flow estimation on the learning curves of the DRL agents.
BibTeX
@misc{yang2019never,
  author = {Hsuan-Kung Yang and Po-Han Chiang and Kuan-Wei Ho and Min-Fong Hong and Chun-Yi Lee},
  title = {{Never Forget: Balancing Exploration and Exploitation via Learning Optical Flow}},
  year = {2019},
  eprint = {1901.08486},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1901.08486},
}
Figure