Agentic Physics-Grounded Video Generation via Memory-Augmented In-Context Reasoning

Xikang Guan, Shaocong Yang, Yanqi Bao, Can Wang, Zheng Gu, Zhengqing Chen, Kai Chen, Jing Huo, Yang GaoarXiv 2026 · arXiv

TL;DR

PhysRepair is a new agentic framework that improves the physical realism of AI-generated videos by adopting a "generate–diagnose–repair" workflow. Instead of forcing physical constraints during the initial synthesis, it identifies physical glitches after generation and uses causal reasoning combined with a memory of past successful repairs to surgically fix inconsistencies.

What Problem Does This Paper Address?

While modern video generators (like Sora or Gen-3) produce visually stunning results, they frequently hallucinate "impossible" physics: objects might pass through one another, move before being touched, or ignore gravity.

Existing solutions typically try to bake physics into the model before generation—either through massive amounts of simulation data or complex prompts. However, this often limits the model's creative diversity and fails to catch errors that only emerge during the generation process. The authors identify two core challenges in fixing these "post-hoc" errors:

  1. What to repair? A simple spatial mask isn't enough; you need to know how a physical failure propagates through time and across different objects.
  2. How to repair? Different physical failures (e.g., a splashing liquid vs. a bouncing ball) require different tools, simulation settings, and strategies that are hard to pre-define.

Methods

PhysRepair introduces an agentic loop that treats the initial video as a draft to be refined. The system consists of three main stages:

  • Evidence-Grounded Diagnosis: The system uses a suite of perception tools (tracking, segmentation, optical flow) to identify exactly where and when a physical law was broken.
  • Spatiotemporal Causal Constraints: To decide what to fix, the agent builds an object-centric causal graph. It identifies the first invalid state and traces its "causal descendants." For example, if a ball is saved by a goalkeeper but the net still ripples as if a goal was scored, the agent identifies both the ball's trajectory and the net's motion as needing repair.
  • Experience-Augmented In-Context Learning (ICL): To decide how to fix it, the agent uses two types of memory. Short-term memory tracks the current repair attempts, while Long-term memory stores successful strategies from previous, similar physical failures. The agent retrieves these "lessons" to select the right simulation parameters or generation conditions.

As shown in the comparison below, this differs from traditional methods by allowing for localized, adaptive corrections rather than global regenerations.

Figure 1: Comparison of physics-grounded video generation paradigms. (a) Plan and Prompt Refine iteratively updates generation plans or physics-aware prompts, while (b) Simulator Pro- gram Refine iteratively updates physical programs or simulator configurations; both impose phys- ical guidance before synthesis and may require repeated regeneration. (c) PhysRepair instead fol- lows a generate–diagnose–repair paradigm: it diagnoses physical failures from the generated video (what to repair?) and performs localized, causally constrained repair, while leveraging prior repair experience to adapt the intervention and preserve generated already-valid content (how to repair?).

The final repair is guided by a physics simulator (like PyBullet or Blender) to create a "counterfactual" physical state, which is then used to re-condition the video generator for a final, physically plausible result.

Figure 2: Overview of PhysRepair. Starting from an initially generated video, a Physics Repair Agent diagnoses the realized physical failure and performs localized counterfactual repair. Spa- tiotemporal causal constraints determine what may change and what must remain fixed, while experience-augmented in-context learning retrieves relevant repair experience to adapt how to re- pair. The resulting counterfactual physical state guides video regeneration, followed by physics- and-preservation verification and experience distillation for subsequent repair.

Datasets

The authors evaluated PhysRepair on Physics-IQ Verified, a benchmark containing 198 instances across 66 real-world physical experiments. The benchmark is diverse, covering:

  • Solid mechanics (collisions, friction)
  • Fluid dynamics (pouring, splashing)
  • Optics (reflections, shadows)
  • Thermodynamics and Magnetism

Key Insights and Ideas

  • Causality over Geometry: You cannot fix physical errors by looking at frames in isolation. By tracing causal effects through a spatiotemporal graph, PhysRepair ensures that "downstream" effects of a collision (like a box moving after being hit) are corrected alongside the collision itself.
  • The "Agentic" Advantage: By using an LLM-based agent to orchestrate tools and reason about failures, the system can adapt to a wide variety of physical domains without needing to retrain the underlying video model.
  • Memory as a Scaling Law: As the system repairs more videos, its long-term memory grows. The authors found that the number of iterations needed to fix a video decreases over time as the agent learns from its past "experience."

Key Results

PhysRepair consistently outperformed both general-purpose video models and specialized physics-aware methods.

  • Quantitative: On the Physics-IQ Verified benchmark, PhysRepair achieved a top score of 43.6, surpassing the strongest baseline (Cosmos3) and significantly improving upon its own backbone model, MiniMax H3 (which scored 39.8).
  • Spatiotemporal Accuracy: The most significant gains were in Spatiotemporal IoU (ST), which jumped from 22.8 to 34.7. This indicates that the model is much better at getting the timing of physical events right.
  • Qualitative: In ablation studies, the authors showed that without the "Causal Constraints," the model might fix a primary error (like removing a ghost ball) but leave behind secondary errors (like a net that still moves). The full PhysRepair system resolves these dependencies for a "clean" save.
Figure 4: Qualitative ablation of Spatiotemporal Causal Constraints. Without causal constraints, repair may correct the immediate target while leaving causally affected states unresolved or disrupt- ing valid interactions. Full PhysRepair jointly constrains editable and preserved states across the interaction, enabling complete and localized physical repair.

Limitations and Future Work

  • Agent Dependency: The system's success is tied to the reasoning capabilities of the underlying multimodal LLM. A "weaker" agent might make incorrect diagnostic or tool-use decisions.
  • Inference Cost: Because it is an iterative "generate-diagnose-repair" process, it is computationally more expensive than one-pass generation methods.
  • Safety and Provenance: As video repair becomes more realistic, the authors note the increasing difficulty in distinguishing synthetic content from real recordings, highlighting the need for robust digital provenance mechanisms.

Summary updated

Abstract

Physical plausibility is essential for realistic video generation. Existing methods impose physical guidance before synthesis via simulations or refined prompts, but this can limit dynamic diversity, miss errors that emerge only after generation, and require repeated regeneration when inconsistencies remain. We therefore study a generate-diagnose-repair paradigm that preserves the original generation capability and selectively repairs physical inconsistencies afterward, raising two key questions: what to repair and how to repair? To address them, we introduce PhysRepair, an agentic video generation framework that combines causal reasoning over the observed physical failure with prior repair experience to enable localized and adaptive correction. Specifically, Spatiotemporal Causal Constraints identify the first invalid physical state and its causally affected states, producing a repair specification that guides physically plausible local regeneration while preserving unaffected content. Experience-augmented In-context Learning combines the ongoing repair context with relevant experience from previous repairs, allowing the agent to adapt its repair decisions across diverse physical failures and accumulate completed trajectories for future cases. Experiments on Physics-IQ Verified across solid mechanics, fluid dynamics, optics, thermodynamics, and magnetism demonstrate that PhysRepair consistently improves the physical plausibility of generated videos over existing methods. Our project page is available at https://xxkkxxx.github.io/PhysRepair-Website/.
BibTeX
@misc{guan2026agentic,
  author = {Xikang Guan and Shaocong Yang and Yanqi Bao and Can Wang and Zheng Gu and Zhengqing Chen and Kai Chen and Jing Huo and Yang Gao},
  title = {{Agentic Physics-Grounded Video Generation via Memory-Augmented In-Context Reasoning}},
  year = {2026},
  eprint = {2610.23677},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.23677},
}
Figure