Perceptual Evaluation of Liquid Simulation Methods

Kiwon Um, Xiangyu Hu, Nils Thuerey

Abstract

This paper proposes a novel framework to evaluate fluid simulation methods based on crowd-sourced user studies in order to robustly gather large numbers of opinions. The key idea for a robust and reliable evaluation is to use a reference video from a carefully selected real-world setup in the user study. By conducting a series of controlled user studies and comparing their evaluation results, we observe various factors that affect the perceptual evaluation. Our data show that the availability of a reference video makes the evaluation consistent. We introduce this approach for computing scores of simulation methods as visual accuracy metric. As an application of the proposed framework, a variety of popular simulation methods are evaluated.

Introduction

In computer graphics, the ultimate goal of a fluid simulation is often "believability." However, the field has traditionally relied on two types of metrics: mathematical (e.g., how well does this solve the Navier-Stokes equations?) and performance-based (e.g., how fast does it run?).

There is a massive gap between a low error-rate in a solver and a human viewer thinking, "Yes, that looks like real water." Human perception is notoriously fickle; a viewer might prefer a physically inaccurate simulation simply because the rendering has attractive reflections. This paper tackles the challenge of creating a reliable, unbiased metric for "visual accuracy" that can survive different rendering styles and subjective human bias.

Methodology

The authors propose a framework based on crowd-sourced pair-wise comparisons. The process begins by generating multiple simulations using different algorithms. These are then presented to participants in a controlled user study interface to derive a final visual accuracy score.

Overview of the perceptual evaluation framework, from simulation generation to crowd-sourced scoring.
  1. The Setup: Participants are shown three videos: a real-world "Reference" video and two different computer-generated simulations (A and B).
  2. The Task: Users must choose which simulation is "closer" to the reference.
  3. The Math: The authors use the Bradley-Terry model to convert thousands of these binary choices into a single "visual accuracy" score for each simulation method.

The "secret sauce" here is the Reference Video. The authors hypothesized that without a ground-truth reference, users would judge simulations based on "style" (rendering). With a reference, users focus on the "motion" (the simulation itself). To prove this, they ran studies across three different crowd-sourcing platforms (Amazon Mechanical Turk, CrowdFlower, and Microworkers) to ensure their results were platform-independent.

Experimental Setup

The study focused on two classic fluid dynamics benchmarks that have well-documented real-world experimental data:

  • Breaking-dam: A column of water collapsing and hitting an obstacle, creating violent splashes.
  • Sloshing-wave: A tank partially filled with water undergoing periodic tilting, creating complex wave patterns.
Physical experimental setups for the Breaking-dam (left) and Sloshing-wave (right) benchmarks.

The authors used these setups to evaluate seven major simulation categories, including Eulerian methods (like FLIP and APIC) and Lagrangian methods (like SPH and IISPH).

Discussion of Perceptual Factors

A significant finding is the "Anchor" Effect: providing a reference video makes perceptual evaluation stable. When users were asked to pick a "preferred" video without a reference, their opinions shifted wildly depending on whether the water was rendered as opaque or transparent. With a reference, the scores for the underlying simulation remained consistent regardless of the rendering "skin."

Furthermore, the study highlights that physics does not always equal perception. Some methods that are mathematically "better" at conserving energy or volume don't necessarily score higher in visual accuracy. The paper also provides empirical evidence for industry "rules of thumb," such as confirming that "particle skinning" (generating a mesh at 2x the simulation resolution) is the perceptual "sweet spot."

Comparative Analysis

  • Lagrangian vs. Eulerian: When comparing simulations with similar particle counts, Lagrangian methods (specifically IISPH and SPH) consistently received higher visual accuracy scores than Eulerian methods like FLIP.
  • The Efficiency Flip: However, when the authors constrained the computational budget (giving each method the same amount of time to calculate a frame), the results flipped. FLIP and APIC became the winners because they could handle much higher resolutions within the same time frame compared to the more expensive SPH variants.
  • Splash Modeling: The authors tested a neural-network-based splash model (MLFLIP). They found it significantly boosted visual accuracy scores, proving that sub-grid splash details are a primary cue for human realism.

Conclusion

While the framework is robust, boundaries remain regarding the scope of liquids (water-like only), participant demographics (expert vs. novice), and rendering styles. Future work could involve using these perceptual scores as a loss function for machine learning models to automatically tune simulations for maximum realism.

Summary updated

Abstract

This paper proposes a novel framework to evaluate fluid simulation methods based on crowd-sourced user studies in order to robustly gather large numbers of opinions. The key idea for a robust and reliable evaluation is to use a reference video from a carefully selected real-world setup in the user study. By conducting a series of controlled user studies and comparing their evaluation results, we observe various factors that affect the perceptual evaluation. Our data show that the availability of a reference video makes the evaluation consistent. We introduce this approach for computing scores of simulation methods as visual accuracy metric. As an application of the proposed framework, a variety of popular simulation methods are evaluated.
BibTeX
@article{um2017perceptual,
  author = {Kiwon Um and Xiangyu Hu and Nils Thuerey},
  title = {{Perceptual Evaluation of Liquid Simulation Methods}},
  journal = {ACM Trans. Graphics},
  year = {2017},
  url = {https://openreview.net/forum?id=p04DmFaObQm},
}
Figure