Investigation of Factorized Optical Flows as Mid-Level Representations

Hsuan-Kung Yang, Tsu-Ching Hsiao, Ting-Hsuan Liao, Hsu-Shen Liu, Li-Yuan Tsao, Tzu-Wen Wang, Shan-Ya Yang, Yu-Wen Chen, Huang-Ru Liao, Chun-Yi Lee

Introduction

In the field of robotic learning, a central debate exists between end-to-end learning (mapping raw pixels directly to actions) and modular design (extracting mid-level representations to guide control). While modular designs offer greater flexibility and interpretability, they are often limited by the quality of the representations used, such as depth maps or semantic segmentation. Most current mid-level representations focus on spatial features from individual frames, largely ignoring the rich temporal information available between frames.

This paper investigates factorized optical flow as a powerful mid-level representation for Deep Reinforcement Learning (DRL) agents. By decoupling raw optical flow into "ego-flow" (self-motion) and "object-flow" (motion of others), the authors demonstrate that agents can achieve superior performance and generalization, particularly in dynamic environments with fast-moving obstacles.

The Problem: The Ambiguity of Raw Flow

Optical flow captures the pattern of apparent motion of objects, surfaces, and edges in a visual scene. However, raw optical flow ($F_{raw}$) is a "noisy" signal for a moving robot because it conflates two distinct sources of motion: the robot's own movement (ego-motion) and the independent movement of other objects.

For example, if a robot turns a corner while a pedestrian crosses the street, the raw flow vector for the pedestrian is a diagonal resultant of both movements. This makes it difficult for a DRL agent to isolate the pedestrian's specific trajectory. Furthermore, in most robotic scenarios, $F_{raw}$ is dominated by ego-motion. This "bully" signal often drowns out the subtle motion of distant or small obstacles, leading to failures in high-speed obstacle avoidance.

Methodology: Flow Factorization

The core contribution of this research is a mathematical procedure to decompose raw optical flow into two distinct components:

  1. Ego-Flow ($F_{ego}$): This component represents the flow caused solely by the camera's translation and rotation. It provides information about the robot's own movement and the static geometry of the environment.
  2. Object-Flow ($F_{obj}$): This represents the residual flow after ego-motion is subtracted, capturing the "pure" movement of dynamic objects like cars or pedestrians.

To compute $F_{ego}$, the authors utilize the robot's known odometry (rotation and translation matrices), camera intrinsics, and depth values to project where pixels would move if the world were entirely static. Subtracting this calculated $F_{ego}$ from the observed $F_{raw}$ isolates the $F_{obj}$.

The researchers implemented this framework using the Unity engine and ML-Agents, training agents with the Soft Actor-Critic (SAC) algorithm. They evaluated various input combinations, including Semantic Segmentation ($S$), Depth ($D$), Raw Flow ($F_{raw}$), and the factorized components ($F_{ego}, F_{obj}$).

Experimental Setup and Results

The agents were tested across four distinct urban navigation environments designed to challenge different aspects of robotic control:

  • Straight Road: Basic navigation with crossing pedestrians.
  • S-Turn: Testing lane-keeping without explicit semantic markers.
  • H-Shaped Pathways: Complex intersections to prevent path overfitting.
  • Three-Way Junction: Navigation around static obstacles (trees, parked cars) and dynamic agents.

Key Findings

  • Superior Generalization: Agents trained with factorized flow ($F_{ego} + F_{obj}$ as separate channels) exhibited remarkable speed generalization. While agents using other representations struggled when pedestrians moved faster than they did during training, the factorized flow agents maintained high success rates.
  • Geometry Inference: Interestingly, $F_{ego}$ alone proved to be a strong proxy for geometry. In static environments, agents could infer relative depth and the "causality" of their own actions simply by observing how their movement changed the visual field.
  • The Need for Complementarity: While flow is excellent for motion detection, it lacks the precise boundary definition provided by Semantic Segmentation. Agents using only flow maps had higher "Out-of-Bound" (OOB) errors, frequently clipping sidewalks. The most robust performance was achieved by ensembling representations (e.g., Flow + Segmentation + Depth).
  • Sensitivity to Temporal Resolution: The study found that flow-based agents are highly sensitive to frame rates. A drop from 12 FPS to 6 FPS caused success rates to plummet by up to 50%, as the displacement between frames became too erratic for the flow predictors to handle accurately.

Conclusion and Future Work

Factorizing optical flow into ego and object components provides a significant advantage for robotic navigation in dynamic scenes. By separating self-motion from the motion of others, DRL agents can better interpret complex environments and generalize to novel speeds.

However, the current method relies on accurate depth and odometry metadata. Future research should explore the robustness of this factorization when using noisy, estimated depth or odometry data, which is more representative of real-world sensor limitations. Additionally, developing more sophisticated neural architectures to better integrate these multi-modal, high-dimensional inputs remains a promising avenue for improving robotic autonomy.

Summary updated

BibTeX
@inproceedings{yang2022investigation,
  author = {Hsuan-Kung Yang and Tsu-Ching Hsiao and Ting-Hsuan Liao and Hsu-Shen Liu and Li-Yuan Tsao and Tzu-Wen Wang and Shan-Ya Yang and Yu-Wen Chen and Huang-Ru Liao and Chun-Yi Lee},
  title = {{Investigation of Factorized Optical Flows as Mid-Level Representations}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=nP1BmB1NzOR},
}
Figure