Abstract: In this paper, we focus on a prediction-based novelty estimation strategy upon the deep reinforcement learning (DRL) framework, and present a flow-based intrinsic curiosity module (FICM) to exploit the prediction errors from optical flow estimation as exploration bonuses. We propose the concept of leveraging motion features captured between consecutive observations to evaluate the novelty of observations in an environment. FICM encourages a DRL agent to explore observations with unfamiliar motion features, and requires only two consecutive frames to obtain sufficient information when estimating the novelty. We evaluate our method and compare it with a number of existing methods on multiple benchmark environments, including Atari games, Super Mario Bros., and ViZDoom. We demonstrate that FICM is favorable to tasks or environments featuring moving objects, which allow FICM to utilize the motion features between consecutive observations. We further ablatively analyze the encoding efficiency of FICM, and discuss its applicable domains comprehensively.
Introduction
Deep Reinforcement Learning (DRL) agents often struggle in environments with sparse rewards, where feedback is infrequent. Intrinsic curiosity serves as a mechanism to provide exploration bonuses, encouraging agents to discover novel states. While existing methods like next-frame prediction or Random Network Distillation (RND) have shown promise, they often face challenges with high-dimensional state spaces or fail to capture temporal dynamics effectively. This paper introduces the Flow-based Intrinsic Curiosity Module (FICM), which leverages optical flow prediction errors to evaluate the novelty of observations. By focusing on motion features between consecutive frames, FICM provides a biologically inspired exploration strategy that prioritizes dynamic elements in the environment.
The Flow-based Intrinsic Curiosity Module (FICM)
The core idea of FICM is to use the prediction error of optical flow as an intrinsic reward. Unlike methods that predict entire future frames, FICM estimates the apparent motion between two consecutive observations, $o_t$ and $o_{t+1}$.
The architecture consists of a Flow Predictor ($G$) that estimates both forward ($F_{forward}$) and backward ($F_{backward}$) optical flow. A warping function then uses these predicted flows to reconstruct the frames (e.g., warping $o_{t+1}$ back to estimate $o_t$). The intrinsic reward $r^i_t$ is calculated as the sum of the reconstruction errors (flow loss). High prediction error indicates unfamiliar motion patterns, signaling novelty and providing a higher exploration bonus to the agent.
Experimental Evaluation
The authors evaluated FICM across several benchmarks:
- Atari 2600: Including hard-exploration games (e.g., Montezuma’s Revenge) and motion-heavy games (e.g., Enduro).
- Super Mario Bros.: A standard platformer for testing curiosity.
- ViZDoom: Specifically the DoomMyWayHome-v0 environment with sparse and very sparse rewards.
Results and Insights
FICM demonstrates a natural ability to direct an agent's attention toward moving objects, such as enemies or projectiles, which are often critical for task success. This is particularly evident in motion-centric environments.
Key findings include:
- Pure Curiosity Performance: In scenarios with only intrinsic rewards, FICM-driven agents outperformed baselines like RND in motion-heavy Atari games like Enduro and CrazyClimber.
- Efficiency in Sparse Environments: In ViZDoom's "very sparse" reward configuration, FICM achieved faster and more reliable convergence than Inverse Dynamics Features (IDF).
- Temporal Novelty: By requiring only two consecutive frames, FICM captures change directly, making it more effective than single-frame methods in dynamic settings.
Limitations and Conclusion
Despite its strengths, FICM faces challenges in environments with significant background motion (the "noisy TV" problem), where irrelevant movement might distract the agent. Additionally, it provides minimal rewards for novel but stationary objects, which explains its lower performance in games like Montezuma’s Revenge that require interacting with static keys or doors.
In conclusion, the Flow-based Intrinsic Curiosity Module offers a robust approach to exploration in DRL by prioritizing motion novelty. By leveraging optical flow prediction errors, FICM effectively guides agents toward dynamic, task-relevant features. Future work may involve combining FICM with reasoning-based curiosity or noise-filtering mechanisms to enhance its applicability across a broader range of environments.
Summary updated