Unsupervised Generation of Optical Flow Datasets from Videos in the Wild

Hoang-An Le, Tushar Nimbhorkar, Thomas Mensink, Anil S. Baslamisli, Sezer Karaoglu, Theo GeversarXiv 2018 · DOI · OpenReview

Abstract: There hardly exists any large-scale datasets with dense optical flow of non-rigid motion from real-world imagery as of today. The reason lies mainly in the required setup to derive ground truth optical flows: a series of images with known camera poses along its trajectory, and an accurate 3D model from a textured scene. Human annotation is not only too tedious for large databases, it can simply hardly contribute to accurate optical flow. To circumvent the need for manual annotation, we propose a framework to automatically generate optical flow from real-world videos. The method extracts and matches objects from video frames to compute initial constraints, and applies a deformation over the objects of interest to obtain dense optical flow fields. We propose several ways to augment the optical flow variations. Extensive experimental results show that training on our automatically generated optical flow outperforms methods that are trained on rigid synthetic data using FlowNet-S, LiteFlowNet, PWC-Net, and RAFT.

Introduction

Deep learning has revolutionized optical flow estimation, yet the performance of Convolutional Neural Networks (CNNs) remains heavily dependent on the availability of massive, high-quality datasets. A critical bottleneck in the field is the scarcity of large-scale datasets containing dense optical flow for non-rigid motion in real-world imagery. Obtaining ground-truth flow for real-world video is practically impossible via traditional sensors, and manual pixel-level annotation is unfeasible for humans.

Consequently, the field has relied on synthetic datasets like FlyingChairs or CGI movies like MPI-Sintel. However, these datasets often feature simplistic rigid motions or lack the texture complexity of the real world, leading to poor generalization when networks encounter the complex deformations of natural scenes. This paper introduces a framework to automatically generate realistic, non-rigid optical flow data from real-world videos, bridging the gap between synthetic training and real-world application.

Methodology: Automated Flow Generation

The proposed pipeline circumvents the need for manual annotation by extracting, matching, and deforming objects from existing video sequences to create ground-truth flow labels.

Overview of the proposed pipeline for generating optical flow datasets: (A) Image segmentation, (B) Image matching, (C) Image deformation, (D) Image warping, and (E) Background generation.

The pipeline operates in five primary stages:

  1. Segmentation: Objects of interest are extracted from video frames (e.g., frame $t$ and $t+\Delta$) using an off-the-shelf model like Mask R-CNN.
  2. Matching: Sparse point-to-point correspondences between the extracted objects are established using DeepMatching, a training-free algorithm robust to non-rigid deformations.
  3. Deformation: An "As-Rigid-As-Possible" (ARAP) algorithm deforms a grid over the first object, guided by the matched points. Interpolating this grid produces a dense optical flow field.
  4. Warping: To ensure a mathematically perfect correspondence between the image and the flow label, the authors do not use the original second frame for training. Instead, they warp the first object using the generated flow field. This guarantees that the flow is the exact ground truth for the transformation.
  5. Background Generation: The original and warped objects are composited onto random static backgrounds to create the final training pairs.

Key Insights and Data Augmentation

A significant finding of this work is that imperfections in the matching and deformation steps can be leveraged as features. Because the first image is warped using the generated flow field, any visual artifacts are perfectly aligned with the flow label. Training on these artifacts actually enhances the robustness of neural networks to real-world noise and occlusions.

Examples of texture randomization applied to objects (bears and dogs) to encourage class-agnostic motion learning.

Furthermore, the authors introduce texture randomization. By replacing an object's texture with random images or noise during dataset generation, the network is forced to learn the underlying physics of motion rather than relying on specific object appearances. This "class-agnostic" training approach significantly improves the model's ability to generalize to unseen object categories.

Experimental Results

The authors introduced the DMO (Deepmatching-Maskrcnn-OpticalFlow) dataset, containing 55,000 image pairs. They evaluated several state-of-the-art architectures—FlowNet-S, LiteFlowNet, PWC-Net, and RAFT—trained on DMO versus the standard FlyingChairs baseline.

  • Generalization: Models trained on DMO consistently outperformed those trained on FlyingChairs when tested on the MPI-Sintel benchmark. For example, PWC-Net's End-Point-Error (EPE) improved from 33.58 to 31.56 on the Sintel final pass.
  • Non-rigid Motion: On the HumanFlow dataset, which focuses on complex human body motion, PWC-Net trained on DMO achieved an EPE of 0.26, compared to 0.30 for the FlyingChairs baseline.
  • Real-world Performance: On the KITTI 2015 benchmark, PWC-Net trained on DMO reduced EPE from 5.36 to 3.18, demonstrating the superior transferability of models trained on real-world textures and non-rigid motion statistics.

Conclusion

This research demonstrates that high-quality training data for optical flow can be effectively bootstrapped from real-world videos without human intervention. By leveraging ARAP deformation and a strategic warping approach, the proposed framework generates dense, non-rigid ground truth that effectively bridges the domain gap. The resulting DMO dataset enables current architectures to achieve superior generalization across both synthetic and real-world benchmarks, providing a scalable solution for future optical flow research.

Summary updated

Abstract

There hardly exists any large-scale datasets with dense optical flow of non-rigid motion from real-world imagery as of today. The reason lies mainly in the required setup to derive ground truth optical flows: a series of images with known camera poses along its trajectory, and an accurate 3D model from a textured scene. Human annotation is not only too tedious for large databases, it can simply hardly contribute to accurate optical flow. To circumvent the need for manual annotation, we propose a framework to automatically generate optical flow from real-world videos. The method extracts and matches objects from video frames to compute initial constraints, and applies a deformation over the objects of interest to obtain dense optical flow fields. We propose several ways to augment the optical flow variations. Extensive experimental results show that training on our automatically generated optical flow outperforms methods that are trained on rigid synthetic data using FlowNet-S, LiteFlowNet, PWC-Net, and RAFT. Datasets and implementation of our optical flow generation framework are released at https://github.com/lhoangan/arap_flow
BibTeX
@misc{le2018unsupervised,
  author = {Hoang-An Le and Tushar Nimbhorkar and Thomas Mensink and Anil S. Baslamisli and Sezer Karaoglu and Theo Gevers},
  title = {{Unsupervised Generation of Optical Flow Datasets from Videos in the Wild}},
  year = {2018},
  eprint = {1812.01946},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1812.01946},
}
Figure