Learning to Estimate Single-View Volumetric Flow Motions without 3D Supervision

Erik Franz, Barbara Solenthaler, Nils Thuerey

TL;DR

This paper introduces Neural Global Transport (NGT), a novel deep learning approach that can estimate 3D fluid flow and density from a single camera view, without requiring any 3D ground truth for training. By cleverly combining image-based losses, adversarial training, and a learned prototype volume, NGT achieves stable, long-term motion estimation that rivals computationally expensive optimization-based methods.

What Problem Does This Paper Address?

Estimating the motion and density of volumetric fluids (like smoke or water) is crucial for many applications, from computer graphics to scientific simulation. Traditionally, this requires multiple camera views or complex, time-consuming optimization processes on explicit 3D data. The core challenge this paper tackles is how to perform this estimation from a single camera view, which is far more practical for data acquisition but introduces significant ambiguity, particularly regarding depth. Existing single-view methods often struggle with this depth ambiguity or still rely on 3D ground truth for training, limiting their applicability.

Methods

The authors propose an end-to-end deep learning framework called Neural Global Transport (NGT). The key innovations are:

  • Unsupervised 3D Flow Estimation: NGT trains a neural network to predict a dense 3D velocity field from a sequence of monocular images. Crucially, this is done without explicit 3D velocity targets. Instead, it leverages a differentiable rendering process and an image-based loss to ensure the predicted flow transports the estimated volume to match the observed images.
  • Addressing Depth Ambiguity: To overcome the inherent depth ambiguity of single-view inputs, NGT employs an adversarial discriminator network. This discriminator helps regularize the estimated volumetric densities, pushing them towards plausible appearances without requiring specific 3D ground truth.
  • Prototype Volume Guidance: A novel aspect is the use of a "prototype" density volume, generated by a separate network ($G_p$) from a single input image. This prototype acts as a soft constraint and an additional input to the velocity estimation network ($G_u$), helping to stabilize the learning process and guide the motion estimation, especially for long sequences.
  • Multi-Scale Architecture: The velocity estimator ($G_u$) uses a multi-scale architecture, similar to optical flow networks, to efficiently handle displacements at different levels of detail.
  • Divergence-Free Velocity: To ensure physical plausibility, the velocity field is constrained to be divergence-free by predicting it as the curl of a vector potential, a technique that avoids explicit divergence losses.

Datasets

  • Synthetic Dataset: A dataset of 30 randomized smoke plume simulations generated using Mantaflow at a resolution of 128x192x128.
  • Real-World Dataset: The ScalarFlow dataset, which contains real-world captures of scalar transport flows. The authors use the first 30 scenes, downsampling images to approximately match the projected grid resolution.

Key Insights and Ideas

  • Adversarial Training for Depth Regularization: Using an adversarial discriminator not only helps generate realistic densities but also implicitly tackles the depth ambiguity by forcing the network to produce plausible volumetric structures that are consistent across different views (even if only one is observed directly).
  • Prototype Volume as a Stabilizer: The learned prototype volume, while not a perfect reconstruction, provides a crucial "anchor" for the velocity estimation. This guidance is key to achieving stable long-term predictions, preventing the learned motion from diverging.
  • Single-View is Sufficient (with the right priors): The work demonstrates that with carefully designed losses (image-based, adversarial) and learned priors (prototype volume, divergence-free velocity), a single view can indeed be sufficient for estimating complex 3D volumetric motion.

Key Results

  • Quantitative Performance: NGT achieves competitive results on both synthetic and real-world datasets, often outperforming other learned methods (like RapidGen) and approaching the quality of computationally expensive optimization-based methods (like ScalarFlow and GlobTrans) in terms of metrics like LPIPS and FID. For instance, on the ScalarFlow dataset, NGT achieves an FID of 134, outperforming RapidGen (186) and ScalarFlow (153), while being significantly faster than optimization methods.
  • Qualitative Performance: NGT produces visually plausible and stable volumetric flow sequences from single camera views. It successfully captures the intricate 3D structures of smoke plumes, even in challenging real-world scenarios. The paper shows qualitative comparisons where NGT's results are visually comparable to or better than other methods, especially in terms of realism and temporal coherence.
  • Efficiency: NGT offers a significant speed advantage over optimization-based methods, achieving single-pass inference after training, whereas methods like GlobTrans require hours per scene.

Limitations and Future Work

  • Limited to Specific Materials: The current renderer only supports "white smoke" and doesn't handle anisotropic scattering, self-illumination, or other complex material properties.
  • No Obstructions: The transport model currently assumes advection without obstructions, limiting its applicability to scenarios with clear flows.
  • Future Work: The authors suggest extending the method to support obstacles and multi-phase flows. They also propose passing information about future time steps to further improve long-term stability. The current method could also be extended to support more complex rendering features.

Summary updated

Abstract

We address the challenging problem of jointly inferring the 3D flow and volumetric densities moving in a fluid from a monocular input video with a deep neural network. Despite the complexity of this task, we show that it is possible to train the corresponding networks without requiring any 3D ground truth for training. In the absence of ground truth data we can train our model with observations from real-world capture setups instead of relying on synthetic reconstructions. We make this unsupervised training approach possible by first generating an initial prototype volume which is then moved and transported over time without the need for volumetric supervision. Our approach relies purely on image-based losses, an adversarial discriminator network, and regularization. Our method can estimate long-term sequences in a stable manner, while achieving closely matching targets for inputs such as rising smoke plumes.
BibTeX
@inproceedings{franz2023learning,
  author = {Erik Franz and Barbara Solenthaler and Nils Thuerey},
  title = {{Learning to Estimate Single-View Volumetric Flow Motions without 3D Supervision}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=2vmGv5wPDBZ},
}
Figure