CounterSketch: Physics-Grounded Counterfactual Reasoning for Sketch-to-Video Generation

Hien Trinh, Tan-Hiep To, Dinh-Khoi Vo, Tam V. Nguyen, Minh-Triet Tran, Trung-Nghia LearXiv 2026 · arXiv

TL;DR

CounterSketch is a new framework that generates physically plausible sketch animations from natural-language "What if...?" queries. By combining causal event-chain reasoning with a physics-based trajectory solver, it ensures that hypothetical interventions—like removing a table or tilting the ground—result in realistic, cascading physical consequences that guide video synthesis.

What Problem Does This Paper Address?

Traditional sketch-to-video methods are great at following direct instructions (e.g., "the ball rolls right"), but they struggle with counterfactual reasoning. If you show a model a sketch of a ball on a table and ask, "What if the table disappeared?", most models fail to infer the physical consequence: the ball should fall.

The core challenge is that physical relations like support, contact, and collision are dynamic. A single intervention (removing the table) triggers a chain of events that must be propagated across all interacting objects. Existing generative models often treat scene conditions as fixed or rely on "black-box" predictions from Vision-Language Models (VLMs), which frequently hallucinate physically impossible motion.

Methods

CounterSketch tackles this by separating physical reasoning from visual synthesis through a five-stage pipeline:

  1. Sketch Scene Grounding: A VLM parses the input sketch into a structured scene graph, identifying objects and their initial relations (e.g., "Ball A is supported by Table B").
  2. Counterfactual Event-Chain Reasoning: The framework translates the "What if?" prompt into a "vector intervention" that modifies the scene graph. It then infers a sequence of physical events (e.g., falling, hitting the ground, bouncing).
  3. Constraint Compilation & Physics Rollout: These events are converted into formal geometric and physical constraints. The system solves a constrained dynamics problem (using the KKT system) to compute continuous, physically accurate object trajectories.
  4. Verification & Closed-Loop Repair: The generated trajectories are checked against geometric and semantic constraints. If a ball passes through a floor (a violation), the system performs local replanning or parameter adjustment to fix it.
  5. Trajectory-Conditioned Synthesis: The verified trajectories are projected into image-space bounding boxes, which serve as explicit spatial guidance for a downstream video diffusion model (specifically, MoSketch).
Fig. 1: CounterSketch interprets a counterfactual intervention, reasons about its physical consequences, and generates a sketch video whose motion follows the resulting trajectory.

Datasets

The authors introduce CounterSketchBench, a new benchmark designed to evaluate counterfactual reasoning in sketch animation.

  • Composition: 80 "What if...?" questions based on 20 unique vector-sketch scenes.
  • Categories: It covers six intervention types: object removal, position changes, orientation (e.g., tilting the scene), size/geometry modifications, relation/constraint changes (e.g., detaching a string), and initial dynamics.

Key Insights and Ideas

  • Physics as a "Middleman": Instead of asking a VLM to predict pixels or even bounding boxes directly, CounterSketch uses the VLM only for high-level causal reasoning. The heavy lifting of motion planning is handed off to a classical physics engine, ensuring the output obeys gravity and collision laws.
  • Causal Propagation: The framework excels at "cascading" effects. As shown in the qualitative results, changing one element (like the orientation of the ground) correctly updates the trajectories of all objects interacting with that surface.
  • Explicit vs. Implicit Constraints: By using a scene graph and formal constraints, the model avoids the "floaty" or inconsistent motion common in pure diffusion-based video generators.
Qualitative results showing how different interventions (removal, repositioning, etc.) lead to distinct physical outcomes.

Key Results

The researchers conducted an ablation study comparing CounterSketch against a VLM-only baseline (using Gemini 1.5 Flash).

  • Text-to-Video (T2V) Alignment: CounterSketch achieved a score of 0.2011, significantly outperforming the VLM-only baseline (0.1682). This indicates the generated videos much more accurately reflect the "What if?" prompt.
  • Dynamic Degree: The physics-grounded approach produced much more intense and realistic motion (0.7750) compared to the VLM's more static predictions (0.5625).
  • Qualitative Success: The model successfully handled complex scenarios, such as removing the bottom box of a stack, which correctly caused the boxes above to tumble—a feat difficult for models without explicit physical reasoning.

Limitations and Future Work

While powerful, CounterSketch has a few hurdles to clear:

  • Grounding Sensitivity: The system's success depends heavily on the VLM's ability to correctly interpret the initial sketch. Ambiguous drawings can lead to incorrect scene graphs.
  • Latent Properties: It is difficult to infer physical properties like mass, friction, or elasticity from a simple 2D sketch. Currently, the model assumes default values.
  • Rigid Body Focus: The current framework is designed for rigid objects. Extending this to non-rigid motion (like a bouncing jelly or a flowing cloth) is a primary goal for future work.
  • Computational Cost: Complex scenes with many interacting objects increase the reasoning and optimization time.

Summary updated

Abstract

Counterfactual sketch animation explores how a static sketch evolves under a hypothetical intervention. We present CounterSketch, a physics-grounded framework that transforms natural-language ``What if...?'' queries into physically plausible sketch animations. The framework integrates VLM-guided scene grounding, causal event-chain reasoning, constraint-based trajectory generation, and verification. It models physical transitions to update the scene graph, computes continuous object trajectories, and validates them against physical, geometric, and semantic constraints. The verified trajectories are then projected into object-level motion bounding boxes to guide downstream sketch-video synthesis. Experiments on CounterSketchBench (80 questions) demonstrate that physics-grounded trajectories provide stronger motion guidance than direct VLM predictions.
BibTeX
@misc{trinh2026countersketch,
  author = {Hien Trinh and Tan-Hiep To and Dinh-Khoi Vo and Tam V. Nguyen and Minh-Triet Tran and Trung-Nghia Le},
  title = {{CounterSketch: Physics-Grounded Counterfactual Reasoning for Sketch-to-Video Generation}},
  year = {2026},
  eprint = {2611.04244},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.04244},
}
Figure