Learning to Estimate Single-View Volumetric Flow Motions without 3D Supervision

Aleksandra Franz, Barbara Solenthaler, Nils Thuerey

Overview

Researchers have developed a deep learning framework called Neural Global Transport (NGT) that reconstructs 3D fluid motion—such as rising smoke—from a single 2D video, without ever seeing 3D ground truth during training. By combining differentiable rendering, physics-based constraints, and adversarial training, the model achieves high-quality 3D reconstructions in minutes that previously required hours of optimization.

The Challenge of 3D Fluid Capture

Capturing the 3D motion of fluids like smoke, clouds, or fire is notoriously difficult. Traditionally, this requires complex multi-camera setups in calibrated labs to resolve "depth ambiguity"—the geometric reality that many different 3D shapes can look identical from a single 2D perspective.

While deep learning has revolutionized 2D motion estimation (such as optical flow), applying it to 3D fluids is hindered by a severe lack of data: we simply do not have "ground truth" 3D motion data for real-world fluid captures. Previous attempts either relied on synthetic data, which often fails to translate to reality, or slow, scene-specific optimization that had to be restarted from scratch for every new video. This paper tackles the challenge of training a neural network to perform "single-pass" inference from a monocular (single-view) video using only 2D observations.

Methods: Neural Global Transport (NGT)

The authors propose an end-to-end unsupervised framework consisting of two main components:

  1. Density Estimator ($G_\rho$): A custom 2D-to-3D UNet that generates a "prototype volume" from a single image. This provides a rough 3D anchor for the fluid's shape.
  2. Velocity Estimator ($G_u$): A multi-scale network that predicts the 3D velocity field. It uses a recursive approach, refining the motion estimate across different resolutions.
Neural Global Transport Architecture

To make this architecture work without 3D supervision, the authors introduce several key technical innovations:

  • Differentiable Rendering: The model uses a ray-marching renderer to project its 3D volumetric guesses back into 2D. It then compares these projections to the original video frames to calculate the loss.
  • Adversarial Training: To solve depth ambiguity, the framework employs a discriminator network. If the reconstructed 3D volume looks "fake" or "implausible" from a side view, the discriminator penalizes it. This forces the generator to create realistic 3D structures and prevents the "pancake effect"—a common failure mode where single-view networks flatten the 3D volume against the camera plane to easily satisfy the 2D loss.
  • Hard Physics Constraints: The velocity field is constrained to be divergence-free by defining it as the curl of a vector potential. This ensures the fluid respects mass conservation, meaning smoke doesn't spontaneously vanish or appear.
  • Global Transport: Instead of looking at just two frames, the model is trained to advect (move) the volume over time, ensuring long-term stability.

Datasets and Key Insights

The model was evaluated on two primary datasets: Synthetic Plumes (30 randomized simulations of buoyant smoke for quantitative comparison) and ScalarFlow (a large-scale dataset of real-world smoke captures). While the model only used one view for training on ScalarFlow, the other available views were used to validate the accuracy of the 3D reconstruction from "unseen" angles.

A crucial insight from the research is the power of prototype volumes ($\tilde{\rho}$). By generating a 3D guess for every frame, the network has a consistent target to move toward. This prevents the motion estimation from drifting or exploding over long sequences. Furthermore, unlike classic methods that optimize a single scene for hours, NGT learns the underlying patterns of fluid motion, allowing it to process entirely new videos in a single forward pass.

Key Results

Qualitative Comparison of Reconstructions
  • Speed: NGT performs inference for a 120-frame sequence in approximately 4 minutes. In contrast, the state-of-the-art optimization method (GlobTrans) takes 37.5 hours for the same task—a speedup of nearly 600x.
  • Accuracy: On the ScalarFlow dataset, NGT achieved a significantly lower FID (Fréchet Inception Distance) of 134 compared to the previous best-learned model (RapidGen) at 186, indicating much more realistic reconstructions that closely match the reference targets.
  • Stability: The inclusion of prototype volumes allowed the model to maintain stable fluid structures over 120+ frames, whereas models without this guidance often saw their volumes "evaporate" or distort after just a few steps.

Limitations and Future Work

While highly effective, the current rendering model is limited to "white smoke" (isotropic scattering) and cannot yet handle fire, multi-phase fluids like water splashes, or complex lighting effects like self-shadowing. Additionally, the transport model assumes an open domain without obstacles. Future work could involve teaching the network to navigate fluid flow around 3D objects and passing information about future frames back into the current estimate to further improve motion accuracy.

Summary updated

Abstract

We address the challenging problem of jointly inferring the 3D flow and volumetric densities moving in a fluid from a monocular input video with a deep neural network. Despite the complexity of this task, we show that it is possible to train the corresponding networks without requiring any 3D ground truth for training. In the absence of ground truth data we can train our model with observations from real-world capture setups instead of relying on synthetic reconstructions. We make this unsupervised training approach possible by first generating an initial prototype volume which is then moved and transported over time without the need for volumetric supervision. Our approach relies purely on image-based losses, an adversarial discriminator network, and regularization. Our method can estimate long-term sequences in a stable manner, while achieving closely matching targets for inputs such as rising smoke plumes.
BibTeX
@inproceedings{franz2023learning,
  author = {Aleksandra Franz and Barbara Solenthaler and Nils Thuerey},
  title = {{Learning to Estimate Single-View Volumetric Flow Motions without 3D Supervision}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=2vmGv5wPDBZ},
}
Figure