TL;DR
PhyPrompt is a two-stage reinforcement learning framework that fixes a major flaw in text-to-video (T2V) models: their tendency to ignore the laws of physics. By automatically refining simple user prompts into detailed, physics-aware descriptions, PhyPrompt enables existing models to generate realistic motion and interactions—outperforming even massive models like GPT-4o and DeepSeek-V3 while using a fraction of the parameters.
What Problem Does This Paper Address?
Current state-of-the-art text-to-video generators produce visually stunning clips, but they are often "physical hallucinations." Objects might teleport, liquids might fail to fill containers, and gravity is frequently treated as a suggestion rather than a law.
The authors argue that this isn't necessarily a limitation of the video models themselves, but a prompting bottleneck. T2V models are trained on highly detailed captions, but users typically provide brief, underspecified prompts (e.g., "pouring wine"). Without explicit instructions to "raise the liquid level steadily," the model defaults to the path of least resistance, often resulting in physically impossible scenes. Manual prompt engineering requires expert physics knowledge and doesn't scale, creating a need for an automated "physics-aware" rewriter.
Methods
PhyPrompt introduces a two-stage pipeline to transform a Large Language Model (LLM) into a specialized prompt refiner:
- Physics-Focused Chain-of-Thought (CoT) SFT: The researchers first fine-tune a base model (Qwen2.5) on a curated dataset. They use GPT-4o1 to generate reasoning chains that explain why certain physical laws (like momentum or surface tension) lead to specific visual outcomes. This teaches the model the "logic" of physics before it starts writing prompts.
- RL with a Dynamic Curriculum: The model is further refined using Group Relative Policy Optimization (GRPO). The breakthrough here is the Dynamic Multi-Objective Reward. Usually, optimizing for physics (Physical Commonsense, or PC) causes the model to forget the original user intent (Semantic Adherence, or SA). PhyPrompt uses a curriculum that initially prioritizes SA to build a "semantic scaffold," then progressively shifts focus to PC.
This staged approach allows the model to discover "compositional prompt structures" that satisfy both intent and physics—regions of the prompt space that standard single-objective training cannot reach.
Datasets
- PhyGenBench: Used as the foundation for the 160-example CoT dataset used during the Supervised Fine-Tuning (SFT) stage.
- VideoPhy2 / VideoPhy2-Auto: The primary benchmark for evaluation and the source of the reward signal for RL. It provides human-aligned automated scores for both Semantic Adherence and Physical Commonsense.
- Vimeo25M: Mentioned in the context of the T2V models (like Lavie) used for testing.
Key Insights and Ideas
- The "Latent Physicist" Hypothesis: The paper proves that T2V models already "know" physics; they just need to be told to use that knowledge. Providing explicit physical constraints in the prompt acts as a trigger for the model's latent capabilities.
- Synergy over Trade-offs: In many AI tasks, you trade accuracy for style. Here, the authors found that a dynamic curriculum creates a superadditive effect. By learning semantics first and physics second, the model actually performs better on both metrics than models trained on only one.
- Model-Agnostic Transfer: Perhaps the most impressive finding is that PhyPrompt is "zero-shot" transferable. A rewriter trained on CogVideoX-2B outputs works remarkably well when paired with entirely different architectures like Lavie or VideoCrafter2. This suggests that "physical priors" in language are universal across different video synthesis engines.
Key Results
- Superior Performance: PhyPrompt-7B achieved a 40.8% joint success rate (meeting both semantic and physical bars) on VideoPhy2, an 8.6 percentage point gain over the baseline.
- Efficiency: Despite having only 7B parameters, PhyPrompt outperformed GPT-4o (+3.8% joint) and DeepSeek-V3 (+2.2% joint), the latter of which is nearly 100x larger.
- Physics Gains: The framework improved physical commonsense scores by 11 percentage points (from 55.8% to 66.8%) while simultaneously improving semantic adherence.
- Zero-Shot Success: When tested on the VideoCrafter2 model, PhyPrompt provided a 16.8% improvement over original prompts without any additional fine-tuning for that specific generator.
Limitations and Future Work
- Evaluator Dependency: The RL process relies heavily on the
VideoPhy2-Autoevaluator. If the evaluator has blind spots regarding certain physical phenomena, the rewriter might learn to exploit those biases. - Small SFT Seed: The initial CoT dataset is relatively small (160 examples). Scaling this dataset could potentially lead to even more nuanced physical reasoning.
- Temporal Consistency: While the paper focuses on physical laws, broader temporal consistency (preventing flickering or style drift over long videos) remains a separate, ongoing challenge in the T2V field.
- Future Directions: The authors suggest this approach could be vital for domains requiring high-fidelity dynamics, such as robotics simulation and scientific visualization.
Summary updated