PhyPrompt: RL-based Prompt Refinement for Physically Plausible Text-to-Video Generation

Shang Wu, Chenwei Xu, Zhuofan Xia, Weijian Li, Lie Lu, Pranav Maneriker, Fan Du, Manling Li, Han Liu

TL;DR

PhyPrompt is a two-stage reinforcement learning framework that fixes a major flaw in text-to-video (T2V) models: their tendency to ignore the laws of physics. By automatically refining simple user prompts into detailed, physics-aware descriptions, PhyPrompt enables existing models to generate realistic motion and interactions—outperforming even massive models like GPT-4o and DeepSeek-V3 while using a fraction of the parameters.

What Problem Does This Paper Address?

Current state-of-the-art text-to-video generators produce visually stunning clips, but they are often "physical hallucinations." Objects might teleport, liquids might fail to fill containers, and gravity is frequently treated as a suggestion rather than a law.

The authors argue that this isn't necessarily a limitation of the video models themselves, but a prompting bottleneck. T2V models are trained on highly detailed captions, but users typically provide brief, underspecified prompts (e.g., "pouring wine"). Without explicit instructions to "raise the liquid level steadily," the model defaults to the path of least resistance, often resulting in physically impossible scenes. Manual prompt engineering requires expert physics knowledge and doesn't scale, creating a need for an automated "physics-aware" rewriter.

Methods

PhyPrompt introduces a two-stage pipeline to transform a Large Language Model (LLM) into a specialized prompt refiner:

  1. Physics-Focused Chain-of-Thought (CoT) SFT: The researchers first fine-tune a base model (Qwen2.5) on a curated dataset. They use GPT-4o1 to generate reasoning chains that explain why certain physical laws (like momentum or surface tension) lead to specific visual outcomes. This teaches the model the "logic" of physics before it starts writing prompts.
  2. RL with a Dynamic Curriculum: The model is further refined using Group Relative Policy Optimization (GRPO). The breakthrough here is the Dynamic Multi-Objective Reward. Usually, optimizing for physics (Physical Commonsense, or PC) causes the model to forget the original user intent (Semantic Adherence, or SA). PhyPrompt uses a curriculum that initially prioritizes SA to build a "semantic scaffold," then progressively shifts focus to PC.

This staged approach allows the model to discover "compositional prompt structures" that satisfy both intent and physics—regions of the prompt space that standard single-objective training cannot reach.

Datasets

  • PhyGenBench: Used as the foundation for the 160-example CoT dataset used during the Supervised Fine-Tuning (SFT) stage.
  • VideoPhy2 / VideoPhy2-Auto: The primary benchmark for evaluation and the source of the reward signal for RL. It provides human-aligned automated scores for both Semantic Adherence and Physical Commonsense.
  • Vimeo25M: Mentioned in the context of the T2V models (like Lavie) used for testing.

Key Insights and Ideas

  • The "Latent Physicist" Hypothesis: The paper proves that T2V models already "know" physics; they just need to be told to use that knowledge. Providing explicit physical constraints in the prompt acts as a trigger for the model's latent capabilities.
  • Synergy over Trade-offs: In many AI tasks, you trade accuracy for style. Here, the authors found that a dynamic curriculum creates a superadditive effect. By learning semantics first and physics second, the model actually performs better on both metrics than models trained on only one.
  • Model-Agnostic Transfer: Perhaps the most impressive finding is that PhyPrompt is "zero-shot" transferable. A rewriter trained on CogVideoX-2B outputs works remarkably well when paired with entirely different architectures like Lavie or VideoCrafter2. This suggests that "physical priors" in language are universal across different video synthesis engines.

Key Results

  • Superior Performance: PhyPrompt-7B achieved a 40.8% joint success rate (meeting both semantic and physical bars) on VideoPhy2, an 8.6 percentage point gain over the baseline.
  • Efficiency: Despite having only 7B parameters, PhyPrompt outperformed GPT-4o (+3.8% joint) and DeepSeek-V3 (+2.2% joint), the latter of which is nearly 100x larger.
  • Physics Gains: The framework improved physical commonsense scores by 11 percentage points (from 55.8% to 66.8%) while simultaneously improving semantic adherence.
  • Zero-Shot Success: When tested on the VideoCrafter2 model, PhyPrompt provided a 16.8% improvement over original prompts without any additional fine-tuning for that specific generator.

Limitations and Future Work

  • Evaluator Dependency: The RL process relies heavily on the VideoPhy2-Auto evaluator. If the evaluator has blind spots regarding certain physical phenomena, the rewriter might learn to exploit those biases.
  • Small SFT Seed: The initial CoT dataset is relatively small (160 examples). Scaling this dataset could potentially lead to even more nuanced physical reasoning.
  • Temporal Consistency: While the paper focuses on physical laws, broader temporal consistency (preventing flickering or style drift over long videos) remains a separate, ongoing challenge in the T2V field.
  • Future Directions: The authors suggest this approach could be vital for domains requiring high-fidelity dynamics, such as robotics simulation and scientific visualization.

Summary updated

Abstract

State-of-the-art text-to-video (T2V) generators frequently violate physical laws despite high visual quality. We show this stems from insufficient physical constraints in prompts rather than model limitations: manually adding physics details reliably produces physically plausible videos, but requires expertise and does not scale. We present PhyPrompt, a two-stage reinforcement learning framework that automatically refines prompts for physically realistic generation. First, we fine-tune a large language model on a physics-focused Chain-of-Thought dataset to integrate principles like object motion and force interactions while preserving user intent. Second, we apply Group Relative Policy Optimization with a dynamic reward curriculum that initially prioritizes semantic fidelity, then progressively shifts toward physical commonsense. This curriculum achieves synergistic optimization: PhyPrompt-7B reaches 40.8\% joint success on VideoPhy2 (8.6pp gain), improving physical commonsense by 11pp (55.8\% to 66.8\%) while simultaneously increasing semantic adherence by 4.4pp (43.4\% to 47.8\%). Remarkably, our curriculum exceeds single-objective training on both metrics, demonstrating compositional prompt discovery beyond conventional multi-objective trade-offs. PhyPrompt outperforms GPT-4o (+3.8\% joint) and DeepSeek-V3 (+2.2\%, 100$\times$ larger) using only 7B parameters. The approach transfers zero-shot across diverse T2V architectures (Lavie, VideoCrafter2, CogVideoX-5B) with up to 16.8\% improvement, establishing that domain-specialized reinforcement learning with compositional curricula surpasses general-purpose scaling for physics-aware generation.
BibTeX
@misc{wu2026phyprompt,
  author = {Shang Wu and Chenwei Xu and Zhuofan Xia and Weijian Li and Lie Lu and Pranav Maneriker and Fan Du and Manling Li and Han Liu},
  title = {{PhyPrompt: RL-based Prompt Refinement for Physically Plausible Text-to-Video Generation}},
  year = {2026},
  eprint = {2603.03505},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.03505},
}
Figure