Neuroexplicit Diffusion Models for Inpainting of Optical Flow Fields

Tom Fischer, Pascal Peter, Joachim Weickert, Eddy Ilg

Introduction

The field of computer vision currently navigates a tension between two dominant paradigms: deep learning and classical physics-based modeling. While deep learning offers state-of-the-art performance, it often functions as a "black box" requiring vast amounts of labeled data. Conversely, models based on Partial Differential Equations (PDEs) provide transparency and physical grounding but frequently struggle with the noise and complexity of real-world imagery.

In the paper "Neuroexplicit Diffusion Models for Inpainting of Optical Flow Fields", the authors propose a hybrid "neuroexplicit" architecture. This approach integrates the structural constraints of PDEs with the adaptive learning capabilities of neural networks to address the challenge of optical flow inpainting—the process of filling missing motion data in sparse flow fields.

Methodology

The core innovation is the Neuroexplicit approach, which avoids direct prediction of flow values. Instead, a neural network is tasked with estimating the parameters that govern a physical diffusion process.

Architecture Overview

The system consists of two primary components:

  1. Neural Component (Diffusion Tensor Module): A U-Net architecture processes a reference image to predict a diffusion tensor ($D$) and a discretization parameter ($\alpha$).
  2. Explicit Component: These predicted parameters are fed into an anisotropic diffusion PDE. This ensures that the resulting inpainting adheres to stable, mathematically defined rules of information propagation.
The Neuroexplicit architecture combines a neural U-Net for parameter estimation with an explicit image-driven inpainting process across multiple scales.

To enhance performance and speed, the model utilizes a coarse-to-fine resolution pyramid and a Fast Semi-Iterative (FSI) scheme, making the iterative diffusion process computationally efficient within a deep learning pipeline.

Experiments and Results

The model was evaluated against several benchmarks, including FlyingThings3D for training, and Sintel and KITTI 2015 for testing generalization and real-world robustness.

Performance and Generalization

The neuroexplicit model demonstrated significant improvements over existing baselines. On the Sintel dataset, it achieved an 11–27% reduction in Endpoint Error (EPE) compared to explicit PDE methods. More strikingly, it outperformed purely data-driven models like GANs and Probabilistic Diffusion by over 40%.

Qualitative comparison showing that the Neuroexplicit method produces sharper boundaries and more accurate flow fields compared to both classical PDE-based and modern neural baselines.

Key Insights

  • Data Efficiency: By leveraging physical constraints, the model requires significantly fewer training samples to reach peak performance compared to purely neural architectures.
  • Robustness to Sparsity: The model maintains high accuracy across varying mask densities (from 1% to 10%), a scenario where purely data-driven models often fail to generalize.
  • Real-World Application: On the KITTI dataset, the model handled non-uniform LiDAR-based sparsity effectively, matching the robustness of classical methods while minimizing outliers.

Conclusion

By bridging the gap between neural networks and explicit physical models, the authors have developed a highly efficient and robust framework for optical flow inpainting. This "neuroexplicit" paradigm suggests a promising direction for future research: embedding physical layers into end-to-end vision pipelines to combine the best of both worlds.

Summary updated

BibTeX
@inproceedings{fischer2024neuroexplicit,
  author = {Tom Fischer and Pascal Peter and Joachim Weickert and Eddy Ilg},
  title = {{Neuroexplicit Diffusion Models for Inpainting of Optical Flow Fields}},
  booktitle = {International Conference on Machine Learning},
  year = {2024},
  url = {https://openreview.net/forum?id=3FKEtlX4aM},
}
Figure