PhysDiff-VTON: Cross-Domain Physics Modeling and Trajectory Optimization for Virtual Try-On

Shibin Mei, Bingbing NiNeurIPS 2025 · OpenReview

Virtual Try-On (VTON) technology has experienced a massive leap forward thanks to diffusion models. However, generating photorealistic images where clothing drapes naturally and retains fine details remains a significant challenge. Most current models treat garment warping as a purely geometric task, leading to biomechanically implausible wrinkles or "floating" clothes. Furthermore, the iterative denoising process of diffusion models often acts like a low-pass filter, blurring out high-frequency details like fabric textures, logos, and intricate patterns—a phenomenon known as texture erosion.

To address these dual challenges of physical plausibility and detail preservation, researchers have introduced PhysDiff-VTON, a novel diffusion-based framework that integrates the laws of physics directly into the generative process.

Bridging Data-Driven Generation and Physical Realism

PhysDiff-VTON departs from traditional post-processing fixes by embedding physical constraints directly into the latent space of the diffusion model. The framework introduces three core innovations:

1. Pose-Guided Deformable Warping

Instead of relying on rigid transformations (like Thin-Plate Splines), this module predicts spatial offsets conditioned on human pose semantics. By utilizing cross-attention mechanisms, it injects "material adaptability" into the model. This essentially teaches the network that stiffer fabrics, such as denim, should deform and move differently than softer fabrics, like silk, when a joint bends.

2. Wavelet-Enhanced Spectral Fidelity

To combat the blurring of high-frequency details, the researchers integrated Haar wavelet transforms into the UNet’s skip connections. By decomposing features into high- and low-frequency subbands, the model acts as a "computational microscope." It explicitly protects fine details—such as sharp edges, micro-textures, and complex patterns—ensuring they survive the denoising process.

3. Potential-Regularized Path Optimization (PRPO)

Inspired by the "principle of least action" in physics, PRPO is a novel sampling strategy that reinterprets the denoising trajectory as a variational problem. Rather than treating diffusion sampling as independent noise-reduction steps, PRPO views it as a continuous path. By adding potentials for temporal smoothness, spatial total variation, and multiscale self-similarity, it steers the diffusion process toward energetically favorable states, preventing erratic noise artifacts and ensuring structural consistency.

State-of-the-Art Performance

PhysDiff-VTON was rigorously evaluated on two industry-standard benchmarks: VITON-HD (containing 11,647 high-resolution person-garment pairs) and DressCode (featuring diverse categories like upper-body, lower-body, and dresses).

The framework achieved state-of-the-art results across major metrics. On the VITON-HD dataset, it reached an LPIPS of 0.093 and an FID of 6.21, significantly outperforming previous leading models like IDM-VTON. It also demonstrated superior semantic alignment (CLIP-I scores), indicating that the generated garments closely match the target images. Remarkably, despite the added physics modeling, inference time remains highly competitive at roughly 11 seconds per batch.

Qualitatively, PhysDiff-VTON excels where previous models fail. Visual comparisons reveal much cleaner text on graphic shirts, realistic stacking of fabric at the elbows, and perfectly preserved patterns on complex clothing.

Qualitative comparison of PhysDiff-VTON against StableVTON and IDM-VTON

Future Directions

While PhysDiff-VTON establishes a new paradigm for synthesizing photorealistic try-on images, the authors note that more advanced evaluation metrics may be needed to fully capture the nuances of fabric dynamics. Future research directions include extending this physics-integrated approach to video virtual try-on, where temporal consistency is paramount, and exploring the generation of diverse garment materials with extreme physical properties, such as highly transparent or reflective fabrics.

Summary updated

Abstract

We present PhysDiff-VTON, a diffusion-based framework for image-based virtual try-on that systematically addresses the dual challenges of garment deformation modeling and high-frequency detail preservation. The core innovation lies in integrating physics-inspired mechanisms into the diffusion process: a pose-guided deformable warping module simulates fabric dynamics by predicting spatial offsets conditioned on human pose semantics, while wavelet-enhanced feature decomposition explicitly preserves texture fidelity through frequency-aware attention. Further enhancing generation quality, a novel sampling strategy optimizes the denoising trajectory via least action principles, enforcing temporal coherence, spatial smoothness, and multi-scale structural consistency. Comprehensive evaluations across multiple datasets demonstrate significant improvements in both geometric plausibility and perceptual quality compared to existing approaches. The framework establishes a new paradigm for synthesizing photorealistic try-on images that adhere to physical constraints while maintaining intricate garment details, advancing the practical applicability of diffusion models in fashion technology.
BibTeX
@inproceedings{mei2025physdiff,
  author = {Shibin Mei and Bingbing Ni},
  title = {{PhysDiff-VTON: Cross-Domain Physics Modeling and Trajectory Optimization for Virtual Try-On}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=fbTGCEfztk},
}
Figure