Abstract
Dynamics prediction, which is the problem of predicting future states of scene objects based on current and prior states, is drawing increasing attention as an instance of learning physics. To solve this problem, Region Proposal Convolutional Interaction Network (RPCIN), a vision-based model, was proposed and achieved state-of-the-art performance in long-term prediction. RPCIN only takes raw images and simple object descriptions, such as the bounding box and segmentation mask of each object, as input. However, despite its success, the model's capability can be compromised under conditions of environment misalignment. In this paper, we investigate two challenging conditions for environment misalignment: Cross-Domain and Cross-Context by proposing four datasets that are designed for these challenges: SimB-Border, SimB-Split, BlenB-Border, and BlenB-Split. The datasets cover two domains and two contexts. Using RPCIN as a probe, experiments conducted on the combinations of the proposed datasets reveal potential weaknesses of the vision-based long-term dynamics prediction model. Furthermore, we propose a promising direction to mitigate the Cross-Domain challenge and provide concrete evidence supporting such a direction, which provides dramatic alleviation of the challenge on the proposed datasets.
Introduction
Dynamics prediction—the ability of an AI to watch a video and predict where objects will be in the future—is a cornerstone of "learning physics" from pixels. State-of-the-art models like the Region Proposal Convolutional Interaction Network (RPCIN) perform exceptionally well when the test environment is identical to the training one. However, real-world deployment rarely offers such consistency. This research identifies a critical gap: Environment Misalignment, which occurs when the testing environment differs from the training one in ways that break the model's learned physical intuition.
The Environment Misalignment Challenge
Environment misalignment is categorized into two distinct types:
- Cross-Domain: The visual "style" changes (e.g., moving from a simple 2D simulation to a realistic 3D render) while the underlying physics remains the same.
- Cross-Context: The physical layout changes (e.g., adding a new wall or a divider to a billiard table) that alters how objects interact with their surroundings.
To benchmark these challenges, four new datasets based on a billiard ball simulation were introduced:
- SimB-Border/Split: Simple 2D simulations with either a standard border or a vertical "split bar" divider.
- BlenB-Border/Split: Realistic 3D versions of the same layouts rendered in Blender.
Methodology: Bridging the Gap with Semantic Masks
Standard vision-based models often overfit to specific visual statistics. To address the Cross-Domain challenge, this paper proposes mapping raw RGB images to a common intermediate representation: semantic segmentation masks. By training the dynamics model on these abstract masks rather than raw pixels, the model becomes invariant to superficial visual changes like lighting or textures, focusing purely on geometry and movement.
Three types of masks were evaluated: Ground-truth masks (GT-Mask), supervised segmentation masks (Sup-Mask), and self-supervised masks generated via k-means clustering (Self-Mask).
Experimental Results and Analysis
Experiments using RPCIN as a probe revealed that vision-based models are surprisingly fragile. Even changing the normalization layer (Batch Norm vs. Layer Norm) fails to bridge the gap when a model trained on 2D shapes is tested on 3D renders.
Key Findings
- The Fragility of RGB: As shown in the results, moving from SimB to BlenB using raw RGB causes long-term prediction error (P2) to spike (e.g., from ~9.5 to ~23.5). Normalization techniques like Instance Norm (IN) or Group Norm (GN) offer some improvement over Batch Norm (BN) but do not solve the fundamental misalignment.
- Masks as a Universal Language: Semantic masks act as a physics-centric abstraction. Using GT-Masks effectively neutralized the domain shift, keeping error rates stable across SimB and BlenB. Notably, Self-Masks achieved performance comparable to supervised versions, proving the viability of this approach for real-world applications where labels are scarce.
- The Cross-Context Barrier: Unlike domain shifts, context shifts remain a significant hurdle. Models trained on a "Border" context often "hallucinate" in a "Split" context, predicting that balls will pass through walls because they have not learned the environment as a set of physical constraints.
Conclusion
While visual domain shifts can be largely mitigated using intermediate semantic representations, the challenge of Cross-Context misalignment persists. Future work must focus on disentangling object dynamics from environmental constraints to ensure that AI "physicists" can truly generalize to new physical layouts. Creating matching synthetic and real-world datasets remains a critical next step for the field.
Summary updated