Learning Human Optical Flow

Anurag Ranjan, Javier Romero, Michael J. BlackarXiv 2018 · arXiv

TL;DR

Researchers have developed a specialized optical flow algorithm tailored specifically for human motion, moving away from the "one-size-fits-all" approach of generic flow methods. By creating a massive synthetic dataset of realistic human movements and training a modified, end-to-end version of SPyNet, they achieved a 30% improvement in accuracy while maintaining real-time performance suitable for mobile devices.

What Problem Does This Paper Address?

Optical flow—the pattern of apparent motion of objects in a visual scene—is a fundamental building block for computer vision tasks like action recognition and pose estimation. Traditionally, it has been treated as a generic, low-level problem. However, humans are not generic objects; we are complex, articulated entities that move quickly, self-occlude, and adopt a vast range of poses.

Generic optical flow methods often struggle with these human-specific nuances, resulting in blurred estimates that miss sharp details like moving hands or legs. The core challenge is the lack of high-quality training data: it is nearly impossible to capture "ground truth" dense optical flow for real humans in natural environments.

Methods

The authors tackle the data scarcity problem by building a sophisticated synthetic data pipeline.

  1. Data Generation: They used the SMPL body model to generate 100,000 unique human shapes and animated them using motion capture data. These models were rendered in Blender against realistic indoor backgrounds from the LSUN dataset.
  2. Ground Truth Flow: To get perfect labels, they utilized Blender’s "Vector pass," which calculates the exact motion of every pixel between frames.
  3. Realism Augmentation: To ensure the model generalizes to the real world, they introduced "imperfections" during rendering, including camera motion, motion blur, and Gaussian image blur.
  4. Architecture: They based their model on SPyNet (Spatial Pyramid Network), which uses a pyramid of small neural networks to estimate flow at different resolutions.
The pipeline for generating synthetic human motion data with ground truth flow.

What’s new? While the original SPyNet uses fixed bilinear operators for upsampling and downsampling, this paper introduces learnable convolutional layers ($u$ and $d$) and a differentiable warping operator. This makes the entire spatial pyramid architecture end-to-end trainable, allowing all levels of the pyramid to optimize jointly.

The modified Spatial Pyramid Network architecture featuring learnable upsampling and downsampling layers.

Datasets

The primary contribution is the Human Flow dataset, which contains 146,020 frame pairs of humans in motion.

  • Training: 135,153 frames.
  • Testing: 10,867 frames.
  • Resolution: 256 x 256 (optimized for deep learning efficiency).

The model was pre-trained on the Flying Chairs dataset before being fine-tuned on Human Flow. For evaluation, the authors compared their results against benchmarks like MPI Sintel, KITTI, and Flying Things.

Key Insights and Ideas

  • Specialization Wins: Training a network specifically on human motion statistics significantly outperforms generic models. The "HumanFlow" model captures sharp limb movements that generic methods like EpicFlow or FlowNetS often blur out.
  • Synthetic Realism is Essential: The authors found that adding camera noise and motion blur to synthetic renders was crucial for the model to generalize to real-world video sequences.
  • Efficiency Matters: By choosing a spatial pyramid structure, the authors kept the model extremely compact (only 7.8 MB). This proves that you don't need massive, multi-hundred-megabyte models to achieve state-of-the-art results in specialized domains.

Key Results

  • Accuracy: The proposed method achieved an Average End Point Error (AEPE) of 0.1164 on the Human Flow dataset, outperforming the original SPyNet (0.2066) and other top methods like Flow Fields (0.1709).
  • Speed: The model runs at 32 fps (31ms per frame pair) on an NVIDIA TitanX, making it viable for real-time applications.
  • Qualitative Superiority: As seen in visual comparisons, the model is significantly better at resolving the motion of legs, hands, and heads, even under partial occlusion.
Figure 1: (a) We simulate human motion in virtual world creating an extensive dataset with images (top row) and flow fields (bottom row); color coding from [2]. (b) We train an existing deep network for human motion estimation and show that it performs better when trained on our dataset and (c) generalizes to human motions in real world scenes.

Limitations and Future Work

While a major step forward, the authors note several areas for improvement:

  • Subtle Motions: The current model struggles with very fine movements, such as those in the face and fingers.
  • Crowds: The dataset primarily focuses on single individuals; future work is needed to handle multiple interacting people and complex 3D scene dynamics.
  • Appearance Diversity: The current textures are somewhat limited. Adding 3D clothing, accessories, and more diverse outdoor scenarios would further improve robustness.

The code and dataset have been made available, providing a new foundation for researchers working on human-centric computer vision.

Summary updated

Abstract

The optical flow of humans is well known to be useful for the analysis of human action. Given this, we devise an optical flow algorithm specifically for human motion and show that it is superior to generic flow methods. Designing a method by hand is impractical, so we develop a new training database of image sequences with ground truth optical flow. For this we use a 3D model of the human body and motion capture data to synthesize realistic flow fields. We then train a convolutional neural network to estimate human flow fields from pairs of images. Since many applications in human motion analysis depend on speed, and we anticipate mobile applications, we base our method on SpyNet with several modifications. We demonstrate that our trained network is more accurate than a wide range of top methods on held-out test data and that it generalizes well to real image sequences. When combined with a person detector/tracker, the approach provides a full solution to the problem of 2D human flow estimation. Both the code and the dataset are available for research.
BibTeX
@misc{ranjan2018learning,
  author = {Anurag Ranjan and Javier Romero and Michael J. Black},
  title = {{Learning Human Optical Flow}},
  year = {2018},
  eprint = {1806.05666},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1806.05666},
}
Figure