Introduction
Reinforcement Learning (RL) agents frequently encounter the "sparse reward" problem, where meaningful feedback from the environment is rare. To encourage exploration, researchers often employ intrinsic curiosity modules that provide a "bonus" reward for visiting novel states. The standard Intrinsic Curiosity Module (ICM) generates this bonus based on errors in predicting the next frame. However, ICM suffers from "catastrophic forgetting": as the model learns new areas, it forgets old ones, leading the agent to redundantly re-explore familiar territory. This paper introduces the Flow-based Intrinsic Curiosity Module (FICM), which leverages optical flow estimation to create a more stable and persistent curiosity signal.
Methods: The Flow-based Intrinsic Curiosity Module (FICM)
The core innovation of FICM is shifting the curiosity signal from frame generation to motion correspondence. Instead of predicting a future state, the module estimates the movement of pixels (optical flow) between consecutive frames $s_t$ and $s_{t+1}$.
As shown in the figure above, FICM predicts both forward ($F_{forward}$) and backward ($F_{backward}$) optical flow. A warping function $W$ uses these flows to reconstruct the frames. The intrinsic reward $r_t^i$ is defined as the reconstruction error—the difference between the warped image and the actual image. High error indicates a motion pattern the agent has not yet mastered, signaling a novel state. The authors propose two variants: FICM-S, which uses a simple encoder-decoder on stacked frames, and FICM-C, which employs a correlation layer to explicitly compare feature maps. Notably, FICM is action-independent, simplifying the architecture compared to the original ICM.
Experimental Setup
The proposed method was evaluated in the ViZDoom (DoomMyWayHome-v0) environment across two difficulty levels:
- Sparse Reward: A 9-room maze where the agent must find a specific item.
- Very Sparse Reward: The agent starts significantly further from the goal, making random discovery highly unlikely.
Agents processed raw $42 \times 42$ grayscale images. Performance was measured by the total reward achieved over 15 million timesteps.
Key Results and Discussion
The experimental results demonstrate that FICM effectively solves the catastrophic forgetting problem that plagues standard ICM.
As illustrated in the plots, the standard ICM baseline experiences a sharp performance crash after approximately 10 million timesteps in the Sparse setting. In contrast, both FICM-S and FICM-C maintain peak performance throughout the entire training duration. In the "Very Sparse" setting, the gap is even more pronounced: while the pixel-based ICM baseline fails to learn entirely, both FICM variants successfully reach the goal and maintain stability.
Beyond stability, FICM is highly efficient. FICM-S achieves superior results to ICM while utilizing only ~56% of the parameters. Qualitative "footprint" analysis confirmed that FICM successfully memorizes terrain; unlike ICM, where visited paths eventually appear "new" (high reward) again, FICM correctly identifies them as familiar, preventing redundant exploration.
Conclusion
By prioritizing correspondence (how things move) over generation (what things look like), the Flow-based Intrinsic Curiosity Module provides a more robust signal for exploration. FICM eliminates the performance drops associated with catastrophic forgetting and enables successful learning in environments with extreme reward sparsity. Future work will explore the application of flow-based curiosity in non-visual tasks and more complex, high-dimensional dynamic environments.
Summary updated