OmniFlow: Human Omnidirectional Optical Flow

Roman Seidel, Andre Apitzsch, Gangolf HirtzWorkshops @ CVPR 2021 · DOI

Abstract

OmniFlow is a large-scale synthetic dataset containing over 23,000 image pairs designed to facilitate the training of optical flow algorithms for omnidirectional (fisheye) cameras. By simulating realistic human activities within 3D indoor environments with wide-angle distortions, OmniFlow provides the necessary ground truth to address the unique challenges of wide-angle lenses in robotics and surveillance, which are poorly represented in standard perspective-based datasets.

Introduction

Optical flow estimation—the process of determining pixel-level motion between consecutive frames—is fundamental to computer vision. While deep learning models have achieved state-of-the-art performance, they require vast amounts of labeled data. In real-world settings, obtaining pixel-perfect motion labels is nearly impossible, necessitating the use of synthetic datasets.

Most existing datasets, such as FlyingChairs or MPI Sintel, focus on standard perspective views. However, many practical applications like 360° security monitoring or indoor robot navigation utilize fisheye lenses with a Field of View (FOV) exceeding 180°. These lenses introduce significant non-linear distortions that perspective-trained models fail to generalize to. Furthermore, there is a distinct lack of datasets focusing on human-centric motion (e.g., walking, sitting, falling) within these distorted views.

Methods

The researchers developed a robust data generation pipeline using the Blender rendering engine to create naturalistic 3D indoor environments.

The OmniFlow data generation pipeline using the Blender rendering engine.

The pipeline incorporates several key elements:

  • Omnidirectional Simulation: A virtual camera employs a fisheye-equidistant model with a 180° FOV. The camera is programmed to track the "hip bone" of human characters, ensuring the subject remains centered while capturing environmental distortion.
  • Domain Randomization: To improve real-world generalization, the system randomizes textures, lighting conditions, and the placement of household objects in every scene.
  • Human Animation: Realistic movements (sitting, walking, falling) are sourced from the CMU Motion Capture Database. Characters are generated via MakeHuman to ensure diversity in body types and clothing.
  • Ground Truth Extraction: Blender’s "vector pass" in the Cycles engine allows for the precise export of forward and backward optical flow for every pixel.

The OmniFlow Dataset

The primary contribution of this work is the OmniFlow dataset, which offers a significant advancement over previous specialized datasets in terms of resolution and volume.

Comparison of OmniFlow with other prominent optical flow datasets.

OmniFlow consists of 23,653 image pairs at a high resolution of 2048 × 2048 pixels. It spans 321 distinct scenes, each containing 75 frames of human activity. Unlike automotive-focused datasets like KITTI, OmniFlow is specifically tailored for indoor, human-centric, omnidirectional applications. For evaluation, the authors utilized the RAFT (Recurrent All Pairs Field Transforms) architecture, pre-training on standard datasets before fine-tuning on OmniFlow.

Results and Key Insights

The study yielded several important findings regarding model adaptation to distorted imagery:

  • Data Efficiency: Models do not require the entire dataset to adapt to the omnidirectional domain. Fine-tuning on just 5,000 image pairs was sufficient to significantly reduce the End-Point Error (EPE) on fisheye imagery.
  • Implicit Distortion Handling: The results suggest that rather than designing complex new architectures for fisheye lenses, existing state-of-the-art models can "learn" to handle these distortions if provided with appropriate synthetic training data.
  • Human-Centric Motion: By utilizing MoCap data, the dataset captures complex, non-rigid motion of human limbs, providing a more challenging and realistic training signal than the rigid motion found in many other datasets.
  • Test-Time Augmentation (TTA): Applying transformations like scaling and flipping during inference consistently improved flow estimation accuracy across all tested subsets.

Conclusion and Future Work

While OmniFlow provides a powerful tool for omnidirectional flow estimation, there are avenues for expansion. Currently, backgrounds remain static within scenes, and most sequences feature a single human subject. Future work could introduce dynamic environments and multi-human interactions. Ultimately, the high-quality optical flow provided by OmniFlow serves as a foundation for higher-level tasks, such as fine-grained human activity recognition in healthcare and smart home settings.

Summary updated

BibTeX
@inproceedings{seidel2021omniflow,
  author = {Roman Seidel and Andre Apitzsch and Gangolf Hirtz},
  title = {{OmniFlow: Human Omnidirectional Optical Flow}},
  booktitle = {Workshops @ CVPR},
  year = {2021},
  doi = {10.1109/cvprw53098.2021.00407},
}
Figure