Abstract
The IntPhys Challenge aims to evaluate how well algorithms capture 'common sense' about the physical world by measuring the ability to detect violations of intuitive physics in dynamic multi-object visual scenes. One approach to this problem is to define or learn a detailed model of the observations and dynamics and to then detect violations of that model. While viable, this approach poses challenges in acquiring an accurate enough model that can handle detailed non-linear object interactions, such as visual occlusion and collisions. In this work, we consider an alternative approach, the Surprise and Explain (SnE) framework, which aims for simplicity while remaining highly flexible. The key idea is to exploit the assumption that, for the vast majority of time, objects follow simple dynamic models, e.g. linear dynamics. Further, when the simple dynamics are occasionally violated ('surprises') due to non-linear interactions, e.g. collisions and occlusion, it is assumed that there is a small set of detectable explanations for the surprise. Violations of intuitive physics then correspond to surprises for which an explanation cannot be inferred. This paper develops an instantiation of the SnE framework and demonstrates its potential in the IntPhys Challenge by placing 2nd at the time of this paper's submission.
Introduction
Human infants naturally possess "intuitive physics"—the common-sense rules governing how objects move, collide, and persist even when out of sight. Replicating this capability in AI is a significant challenge. Traditional methods often attempt to build complex, high-fidelity physics engines to predict every frame of a video. However, modeling every non-linear interaction, such as complex collisions or subtle lighting changes, is computationally expensive and error-prone. The IntPhys Challenge addresses this by requiring models to distinguish between physically "possible" and "impossible" scenes, such as objects vanishing or changing shape mid-motion.
The Surprise and Explain (SnE) Framework
The authors propose the Surprise and Explain (SnE) framework, which shifts the focus from exhaustive prediction to explaining deviations from simple expectations. The framework is composed of three primary modules:
- Perception: This module identifies and tracks objects across frames using a Hungarian algorithm based on spatial proximity.
- Dynamics: A simple linear model predicts an object's next position and velocity, assuming constant velocity motion as the default state.
- Explanation of Surprise (XoS): When an observation deviates significantly from the linear prediction (a "surprise"), this module seeks a physical explanation. Motion-based surprises (e.g., an object stopping) are checked against potential occluders or collisions. Appearance-based surprises (e.g., shape changes) are handled by a Siamese Network that compares current and previous object states.
The final plausibility of a scene is determined by a Violation of Expectation (VOE) signal. If a surprise occurs and the XoS module cannot find a valid physical explanation, the VOE signal spikes, marking the scene as impossible.
Experimental Evaluation
The SnE framework was evaluated on the IntPhys Challenge 2019 dataset, which includes 11,250 training videos of possible scenarios and test sets structured as "k-uplets" (groups of possible and impossible videos). The dataset tests three pillars of intuitive physics: object permanence, shape constancy, and spatio-temporal continuity.
Results
The SnE framework proved highly competitive, placing 2nd overall on the IntPhys 2019 Leaderboard. Notably, in Block O2, which tests specific physical concepts, the model achieved the highest score among all competitors, with a relative error of 0.08 and an absolute error of 0.17.
The results demonstrate that a "simplicity" heuristic—assuming linear dynamics and only invoking complex reasoning during surprises—can outperform end-to-end deep learning models that attempt to learn physics from scratch.
Conclusion
The SnE framework highlights the effectiveness of using surprise as a filter for physical reasoning. By categorizing surprises into explainable (occlusions/collisions) and unexplainable (teleportation/vanishing) events, the model mimics human-like physical intuition. However, the system currently relies on hand-engineered explanation rules and is sensitive to the quality of initial object tracking. Future work will focus on learning these explanations and latent representations directly from visual data to improve robustness in more chaotic, real-world environments.
Summary updated