Image Based Reconstruction of Liquids from 2D Surface Detections

Florian Richter, Ryan K. Orosco, Michael C. Yip

Abstract

In this work, we present a solution to the challenging problem of reconstructing liquids from image data. The challenges in reconstructing liquids, which is not faced in previous reconstruction works on rigid and deforming surfaces, lies in the inability to use depth sensing and color features due the variable index of refraction, opacity, and environmental reflections. Therefore, we limit ourselves to only surface detections (i.e. binary mask) of liquids as observations and do not assume any prior knowledge on the liquids properties. A novel optimization problem is posed which reconstructs the liquid as particles by minimizing the error between a rendered surface from the particles and the surface detections while satisfying liquid constraints. Our solvers to this optimization problem are presented and no training data is required to apply them. We also propose a dynamic prediction to seed the reconstruction optimization from the previous time-step. We test our proposed methods in simulation and on two new liquid datasets which we open source so the broader research community can continue developing in this under explored area.

Introduction

While 3D reconstruction has matured significantly for rigid and deformable objects, liquids remain a significant challenge. Traditional depth sensors like LiDAR or Kinect often fail when encountering fluids because of unpredictable refraction and reflection. Furthermore, the visual appearance of a liquid is highly dependent on its environment, rendering color-based feature tracking unreliable. This research addresses these hurdles by proposing a method to reconstruct 3D liquid geometry using the most basic visual information available: 2D binary masks (surface detections).

Methodology: Physics-in-the-Loop Optimization

The core of the approach is a particle-based optimization framework that requires no pre-training. By modeling the liquid as a collection of 3D particles, the system can naturally handle complex topological changes like splashing or stream breakage.

The optimization process leverages differentiable rendering via the Pulsar toolkit. This allows the system to iteratively adjust particle positions so that their rendered 2D silhouette matches the observed image mask. To ensure the resulting 3D volume is physically plausible and doesn't simply flatten against the camera view, the authors enforce two critical physical constraints:

  1. Collision Constraints: Particles are restricted to the known boundaries of the environment using Signed Distance Fields.
  2. Density Constraints: Inspired by Smoothed Particle Hydrodynamics (SPH), these ensure the liquid maintains a consistent resting density, enforcing incompressibility.
Overview of the physical and image-based constraints used in the optimization framework.

Additionally, a dynamic prediction step seeds the optimization for each new frame based on the previous state, and adaptive branching allows the system to dynamically add or remove particles to better fit the observed volume.

Experimental Datasets

The method was validated across three distinct scenarios:

  • Simulated Fountain: A multi-step fountain used to establish ground truth metrics and test scalability with over 10,000 particles.
  • Endoscopic Liquid: Real-world data of red-dyed water in a silicon cavity, captured via a surgical endoscope to simulate medical environments.
  • Pouring Milk: A challenging real-world scenario involving chocolate milk being poured into a mug, testing the system's ability to handle falling streams and partial occlusions by the container.

Results and Discussion

The proposed method demonstrated high accuracy and robustness. In simulations, it achieved a 3D Intersection over Union (IoU) of approximately 70%, maintaining stability even during turbulent transitions between fountain steps.

Reconstruction results for the endoscopic liquid (top) and pouring milk (bottom) datasets.

A key finding was the effectiveness of the density constraint. While other fluid registration methods often become unstable during turbulent motion, this simpler constraint consistently maintained "liquid-like" behavior and incompressibility across all trials. The system also proved capable of handling significant occlusions, successfully reconstructing the liquid volume at the bottom of a mug even when obscured by the rim.

Conclusion

This work demonstrates that 2D silhouettes, when combined with fundamental physical priors, are sufficient for high-fidelity 3D liquid reconstruction. By bypassing the need for complex depth sensing or massive training datasets, this approach offers a practical path forward for robotic interaction with fluids in diverse fields, from automated service to robotic surgery. Future work will explore the integration of Graph Neural Networks (GNNs) to further enhance fluid dynamics modeling and temporal consistency.

Summary updated

Abstract

In this work, we present a solution to the challenging problem of reconstructing liquids from image data. The challenges in reconstructing liquids, which is not faced in previous reconstruction works on rigid and deforming surfaces, lies in the inability to use depth sensing and color features due the variable index of refraction, opacity, and environmental reflections. Therefore, we limit ourselves to only surface detections (i.e. binary mask) of liquids as observations and do not assume any prior knowledge on the liquids properties. A novel optimization problem is posed which reconstructs the liquid as particles by minimizing the error between a rendered surface from the particles and the surface detections while satisfying liquid constraints. Our solvers to this optimization problem are presented and no training data is required to apply them. We also propose a dynamic prediction to seed the reconstruction optimization from the previous time-step. We test our proposed methods in simulation and on two new liquid datasets which we open source <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> https://github.com/ucsdarclab/liquid_reconstruction so the broader research community can continue developing in this under explored area.
BibTeX
@inproceedings{richter2022image,
  author = {Florian Richter and Ryan K. Orosco and Michael C. Yip},
  title = {{Image Based Reconstruction of Liquids from 2D Surface Detections}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2022},
  url = {https://openreview.net/forum?id=yNdirrxwif},
}
Figure