APT: Atomic Physical Transitions for Causal Video-Language Understanding

Shang Wu, Haoran Lu, Songling Liu, Chenwei Xu, Lie Lu, Pranav Maneriker, Fan Du, Zhaoran Wang, Han LiuarXiv 2026 · arXiv

TL;DR

Current vision-language models (VLMs) can easily recognize what happens in a video (e.g., "a ball bouncing") but struggle to understand how it happens step-by-step. This paper introduces Atomic Physical Transitions (APTs)—ordered chains of timestamped, mechanism-typed state changes—and shows that even frontier models have dismal zero-shot physical reasoning capabilities. To bridge this gap, the authors develop APT-Tune, a parameter-efficient training recipe that dramatically improves a VLM's physical grounding and transition tracking without sacrificing general video question-answering performance.

What Problem Does This Paper Address?

When humans watch a physical event, we don't just see a static label; we intuitively segment the action based on underlying physical principles like gravity, friction, and collision forces. However, most video AI benchmarks only evaluate models using clip-level labels or final outcomes. A label like "bounce" hides a complex, causal sequence of events: support loss, contact onset, elastic rebound, and final settling.

Event labels vs. APT chains

The authors identify two major challenges in current video-language understanding:

  1. The Representation Bottleneck: Existing VLMs completely miss these fine-grained, intermediate state transitions.
  2. Specialist Collapse (Event-Level Forgetting): When researchers try to fine-tune VLMs directly on structured physical transition chains (like JSON outputs), the models overfit to the specific text format. They become "format specialists" and lose their general ability to answer ordinary video questions or descriptions.

Methods

To make physical transitions learnable and reusable, the authors propose APT-Tune, a parameter-efficient supervised fine-tuning (SFT) recipe (utilizing only 11M LoRA parameters on a Qwen3-VL-2B backbone). APT-Tune introduces two core design principles:

  1. Image-Pad-Aware Masking: In standard multimodal SFT, the model's loss function inadvertently spends capacity trying to reconstruct hundreds of placeholder "image pad" tokens. APT-Tune shifts the loss calculation to focus exclusively on the structured text answer sequence, preventing wasted model capacity.
  2. Format-Conditional Co-Training: To prevent format specialization and forgetting, the training mixture balances structured APT JSON chains with standard multiple-choice questions (MCQ) and short natural language descriptions.

Additionally, the framework uses Mechanism-Conditioned Decoding. Instead of guessing a transition label directly from pixels, the model is prompted to first identify the active physical domain (e.g., Gravity & Projectile) and then decode the specific transition type (e.g., free_fall_onset), enforcing a physically grounded causal hierarchy.

APT-Tune Design Principles

Datasets

The authors introduced APT-Bench, a diagnostic dataset containing 27,303 timed transition instances across 1,246 trials.

To build it reliably and at scale, they created a mixed-source pipeline:

  • Human Annotations: Human labels on real and synthetic videos (including CLEVRER) anchor the exact visual boundaries and semantics of the transitions.
  • Simulator Ground Truth: Physics simulators (Physion++ and Isaac Sim) generate diverse, domain-randomized scenes where exact object poses, velocities, and contact forces are known. An LLM-assisted parser converts these raw simulator traces into the shared APT schema.

The dataset covers 14 distinct transition types across 4 core macroscopic mechanics domains: Contact & Collision, Gravity & Projectile, Friction, Surface & Medium, and Rotation & Stability.

Key Insights and Ideas

  • Clip-Level Competence $\neq$ Physical Understanding: Commercial frontier models (including GPT-5, Gemini 3.1, and Claude 3.5 Sonnet) achieved a maximum of only 15.2% zero-shot recall on tracking these basic physical transitions.
  • Missing, Not Jittering: The low performance of zero-shot models isn't because their timestamps are slightly off (timing jitter); they simply fail to propose or recognize the transitions altogether.
  • Scale Alone Won't Fix It: Scaling open-source models from 2B to 8B parameters yielded virtually zero improvement in zero-shot transition recall, proving that transition tracking is a representation bottleneck rather than a capacity issue.
  • Structure as a Supervision Signal: Structured chains shouldn't be viewed merely as an output format, but as a human-aligned supervision signal. Co-training on them actually improves the model's internal representations of physics, which transfers positively to non-APT tasks.

Key Results

  • Dramatic Performance Boost: Applying APT-Tune to Qwen3-VL-2B raised its transition recall from 10.0% to 38.1% and its F1 score from 14.8% to 38.8%.
  • Accurate Cardinality: The model achieved higher recall without overpredicting; it emitted an average of 21.5 predictions per clip against a ground-truth mean of 22.3.
  • Strong Out-of-Distribution Transfer: Without seeing any of its data during training, the APT-tuned model improved performance across all sub-tasks of PhysBench (a real-world physical property benchmark) by an average of 11.1 to 18.2 percentage points. It also boosted Something-Something v2 MCQ accuracy by 15 percentage points over the base model.

Limitations and Future Work

  • Physics Coverage: The current taxonomy is strictly scoped to visually localizable, macroscopic rigid-body mechanics. It does not cover fluid dynamics, deformable objects, material fracture, or thermal, chemical, optical, and electromagnetic changes.
  • Multi-Object Density: Most video clips in the benchmark feature relatively simple scenes with 2–3 interacting objects (maximum of 5). It does not yet systematically stress dense multi-object environments with severe mutual occlusions or massive interaction cascades.
  • Alternative Training Objectives: The authors focused on structured SFT as a baseline. They note that future work could explore timestamp-aware regression losses or reinforcement learning (RL) formulations to further refine temporal localization.

Summary updated

Abstract

Physical events are not understood by their names alone, but by the causal state changes that compose them. A clip-level label such as "bounce" can be correct while hiding the process that makes the event physically valid, from support loss and contact onset to rebound and settling. To make this hidden process explicit, we introduce Atomic Physical Transitions (APTs): minimal, temporally localized state changes that bind a visible cue to an active physical mechanism and before/after dynamical regimes. An APT chain represents a video as an ordered causal transition sequence rather than a single aggregate event label: event labels tell what happened; APT chains explain why it happened. To make APTs learnable by VLMs, we construct mixed-source APT data from human annotations and simulator ground truth, covering 14 transition types across contact, gravity, friction, and rotation/stability, with 27,303 timed instances over 1,246 trials. Using this data, we find that current VLMs miss transition-level physics, with zero-shot recall at most 14% and errors dominated by missed transitions. Direct fine-tuning on APT chains improves transition detection but causes event-level forgetting, indicating that the model learns a specialized answer format rather than a reusable physical representation. We therefore propose APT-Tune, a parameter-efficient recipe that teaches VLMs to use causal transitions without forgetting how to answer video questions. It combines image-pad-aware supervision, format-conditional co-training, and mechanism-conditioned domain-to-type decoding to make APT learning format-robust and physically grounded. With only 11 M LoRA parameters on Qwen3-VL-2B, APT-Tune substantially improves APT recall while also improving event-level video transfer. These results show that APTs are not a new answer format, but a human-aligned causal supervision signal for physical video understanding.
BibTeX
@misc{wu2026apt,
  author = {Shang Wu and Haoran Lu and Songling Liu and Chenwei Xu and Lie Lu and Pranav Maneriker and Fan Du and Zhaoran Wang and Han Liu},
  title = {{APT: Atomic Physical Transitions for Causal Video-Language Understanding}},
  year = {2026},
  eprint = {2606.18586},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2606.18586},
}
Figure