Beyond Brightness Constancy: Learning Noise Models for Optical Flow

Dan Rosenbaum, Yair WeissarXiv 2016 · OpenReview

Abstract

Optical flow is typically estimated by minimizing a "data cost" and an optional regularizer. While there has been much work on different regularizers many modern algorithms still use a data cost that is not very different from the ones used over 30 years ago: a robust version of brightness constancy or gradient constancy. In this paper we leverage the recent availability of ground-truth optical flow databases in order to learn a data cost. Specifically we take a generative approach in which the data cost models the distribution of noise after warping an image according to the flow and we measure the "goodness" of a data cost by how well it matches the true distribution of flow warp error. Consistent with current practice, we find that robust versions of gradient constancy are better models than simple brightness constancy but a learned GMM that models the density of patches of warp error gives a much better fit than any existing assumption of constancy. This significant advantage of the GMM is due to an explicit modeling of the spatial structure of warp errors, a feature which is missing from almost all existing data costs in optical flow. Finally, we show how a good density model of warp error patches can be used for optical flow estimation on whole images. We replace the data cost by the expected patch log-likelihood (EPLL), and show how this cost can be optimized iteratively using an additional step of denoising the warp error image. The results of our experiments are promising and show that patch models with higher likelihood lead to better optical flow estimation.

Introduction

At the core of almost every optical flow algorithm is an optimization problem that balances a regularizer (which assumes flow fields are smooth) and a data cost (which measures how well pixels in the first image match the warped pixels in the second image). For over 30 years, the field has relied on hand-crafted data costs, such as assuming the brightness or the gradient of a pixel remains constant between frames.

However, in real-world scenes with large motions and occlusions, these assumptions fail. When an object occludes a background, the difference between the original image and the warped second image—the "warp error"—is not just random white noise. It contains distinct spatial structures, such as oriented edges. Existing data costs treat errors pixel-by-pixel, ignoring this spatial structure. This work leverages modern datasets with ground-truth optical flow to learn a generative noise model that captures the true distribution of these errors.

Visualization of optical flow components: input images, ground-truth flow, resulting warp error $D_v$, and extracted patches used for learning the GMM noise model.

Methodology: Learning the Warp Error Distribution

The authors reframe the optical flow data cost as a generative probability density model of the "warp error." Traditional data costs like L2 brightness constancy are mathematically equivalent to assuming the warp error follows simple, pixel-independent Gaussian distributions. To capture spatial structure, the authors propose:

  1. Patch-Based GMM: They extract $8 \times 8$ patches of ground-truth warp error and train a Gaussian Mixture Model (GMM) with up to 100 components. This allows the model to learn the specific spatial patterns (like edges) that occur during occlusions.
  2. Expected Patch Log-Likelihood (EPLL): To apply this patch model to full images, the EPLL framework is used to sum the log-likelihood of all overlapping patches in the warp error image.
  3. Optimization via Half-Quadratic Splitting: The non-convex cost function is optimized using an alternating technique. An "r-step" performs image denoising on the warp error using the learned GMM, while a "v-step" solves a standard optical flow problem on the denoised target.

Experiments and Results

The model was trained and evaluated using the MPI Sintel Dataset. The authors used 708 image pairs for training the GMM and evaluated the final pipeline on a subset of 20 images.

Key Insights

  • Structured Noise: Warp errors are highly structured. Because of occlusions, they frequently contain oriented edges rather than just white noise.
  • GMMs as Occlusion Detectors: While high-weight GMM components capture flat patches, lower-weight components explicitly model edges at specific orientations, effectively learning the physics of occluding objects.
  • Likelihood Correlates with Performance: There is a direct correlation between how well a noise model fits held-out warp error patches (log-likelihood) and its performance in full-image optical flow estimation.

Quantitative Results

The learned GMM with 100 components (GMM100) significantly outperformed traditional models like Brightness or Gradient Constancy in log-likelihood. When integrated into the EPLL framework, GMM100 yielded the lowest Average End-Point-Error (EPE) among tested patch models, achieving results comparable to top-performing reference algorithms even with a simple spatial regularizer.

Conclusion

This research demonstrates that replacing hand-crafted data costs with learned, generative noise models significantly improves optical flow estimation. By explicitly modeling the spatial structure of warp errors, GMMs can better handle complex phenomena like occlusions. Future work will focus on more powerful optimization techniques and combining these learned data costs with advanced non-local flow regularizers.

Summary updated

Abstract

Optical flow is typically estimated by minimizing a "data cost" and an optional regularizer. While there has been much work on different regularizers many modern algorithms still use a data cost that is not very different from the ones used over 30 years ago: a robust version of brightness constancy or gradient constancy. In this paper we leverage the recent availability of ground-truth optical flow databases in order to learn a data cost. Specifically we take a generative approach in which the data cost models the distribution of noise after warping an image according to the flow and we measure the "goodness" of a data cost by how well it matches the true distribution of flow warp error. Consistent with current practice, we find that robust versions of gradient constancy are better models than simple brightness constancy but a learned GMM that models the density of patches of warp error gives a much better fit than any existing assumption of constancy. This significant advantage of the GMM is due to an explicit modeling of the spatial structure of warp errors, a feature which is missing from almost all existing data costs in optical flow. Finally, we show how a good density model of warp error patches can be used for optical flow estimation on whole images. We replace the data cost by the expected patch log-likelihood (EPLL), and show how this cost can be optimized iteratively using an additional step of denoising the warp error image. The results of our experiments are promising and show that patch models with higher likelihood lead to better optical flow estimation.
BibTeX
@misc{rosenbaum2016beyond,
  author = {Dan Rosenbaum and Yair Weiss},
  title = {{Beyond Brightness Constancy: Learning Noise Models for Optical Flow}},
  year = {2016},
  eprint = {1604.02815},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1604.02815},
}
Figure