TL;DR
This paper introduces a two-dimensional framework to categorize physical hallucinations in video models into "low-density sampling risk" and "high-density drifting risk." By addressing both risks simultaneously through a novel sampler (HA-Sampler) and a likelihood-based evaluator (HA-Scorer), the authors significantly improve the physical plausibility of generated videos across multiple state-of-the-art backbones.
What Problem Does This Paper Address?
Modern video generative models produce visually stunning results, but they frequently "hallucinate" impossible physics—objects might teleport, disappear, or gain energy out of nowhere. While researchers have tried to fix this using internal model signals (refining the model's own "preferences") or external physics checkers (guiding the model toward real-world laws), there hasn't been a unified theory explaining why these hallucinations happen or how these two approaches interact.
The authors identify a fundamental gap: a model might fail because it samples from an "unfamiliar" low-probability region (low-density sampling risk), or it might fail because the regions it finds "familiar" are actually physically incorrect (high-density drifting risk). Standard methods often tackle one but ignore the other.
Methods
The paper proposes a dual-pronged approach based on their new risk framework:
- Two-Dimensional Risk Framework:
- Low-Density Sampling Risk: Captures cases where the model generates something it doesn't "believe" in (low likelihood), like a ball suddenly jumping positions.
- High-Density Drifting Risk: Captures cases where the model generates something it thinks is likely, but which violates physics, like a ball bouncing progressively higher.
- HA-Sampler (Hallucination-Aware Sampler): A training-free sampling method that integrates external physical guidance directly into the latent refinement process. It uses gradients from an external physics loss to steer the noise perturbation, ensuring the model stays in high-likelihood regions that are also physically valid.
- HA-Scorer (Hallucination-Aware Scorer): A method to evaluate generated videos by calculating their exact log-likelihood under the model. It leverages Continuous Normalizing Flow (CNF) mathematics, using a coarse ODE solver and a finite-difference Hutchinson estimator to make likelihood calculation computationally feasible for high-dimensional video data.
Datasets
The methods were evaluated on two primary benchmarks:
- Physics-IQ: A benchmark for image-to-video generation that compares model continuations against real-world physical recordings using metrics like Spatial IoU and MSE.
- VideoPhy: A text-to-video benchmark consisting of 344 prompts designed to test "Physical Commonsense" (PC) across different material interactions (Solid-Solid, Fluid-Fluid, etc.).
Key Insights and Ideas
- Complementary Strengths: The "aha moment" is that internal refinement and external guidance are not redundant. Refinement acts like a "denoising" step for the model's intent, while guidance acts as a "correction" toward reality. Using one without the other often leads to a trade-off where you either get high-likelihood nonsense or low-likelihood physics.
- Likelihood as a Proxy for Risk: The authors demonstrate that a model’s internal likelihood is a surprisingly strong indicator of physical consistency. If a model "knows" a video is unlikely, that video is probably a hallucination.
- Training-Free SOTA: Both HA-Sampler and HA-Scorer are "plug-and-play" and do not require retraining the massive underlying video models, making them highly practical for existing pipelines.
Key Results
- Benchmark Dominance: HA-Sampler achieved the highest overall scores on both Physics-IQ and VideoPhy. On the Wan2.2-TI2V-5B backbone, it exceeded the base model's Physics-IQ score by +2.10 points.
- Superior Selection: HA-Scorer proved more effective at picking the most physically plausible video from a pool of candidates than standard Vision-Language Models (VLMs) like Qwen-VL or Gemma.
- Qualitative Improvement: Qualitative results show that HA-Sampler successfully prevents common errors like objects changing color mid-motion or disappearing into thin air, which were prevalent in baseline "Predict-and-Perturb" (P&P) methods.
Limitations and Future Work
- Diversity vs. Plausibility: The authors note a potential trade-off: by pushing the model toward high-density, physically valid regions, the diversity of generated outputs might decrease. Preserving creative "long-tail" but valid generations remains an open challenge.
- Computational Overhead: While HA-Scorer is optimized, it still requires multiple model forward passes (NFE), which may be slow for real-time applications.
- Future Applications: The authors suggest these risk measures could be used to filter synthetic data for training robotics models, ensuring that the "world simulators" robots learn from are physically accurate.
Summary updated