Abstract
In fashion e-commerce, capturing high-quality video for every fabric variation is logistically challenging. This paper presents a "one-click" solution for realistic video garment re-texturing using pattern-guided UV recovery. By employing a temporary printed grid pattern and a novel neural architecture featuring a Jacobian-based $L_{0.5}$ loss and blended-weight MLPs, the system recovers dense UV coordinates and shading from single-view smartphone video. The approach effectively preserves complex wrinkles and folds, significantly outperforming existing 3D reconstruction and human parsing baselines.
Introduction
The rapid pace of modern e-commerce demands efficient content creation. Traditionally, showcasing a single garment design in multiple fabrics required either expensive, multi-day photoshoots or labor-intensive manual video editing. While 3D reconstruction offers a potential alternative, current methods often struggle with the complex, non-rigid deformations of loose clothing and the temporal jitter inherent in frame-by-frame processing.
Researchers from Zhejiang University and Adobe Research have proposed a middle ground: a pattern-based approach that enables high-fidelity re-texturing from a single-view smartphone video. By using a temporary, color-coded grid pattern, their system can automatically swap textures while perfectly preserving the original garment's physical behavior, including every wrinkle, fold, and shadow.
The Challenge of Dense UV Estimation
The core technical hurdle in video re-texturing is dense UV estimation—determining the exact correspondence between every pixel on a moving garment and a 2D texture map. This is particularly difficult for loose clothing due to:
- Complex Geometry: Deep wrinkles and self-occlusions create discontinuities in the UV space.
- Temporal Instability: Independent per-frame estimation often results in "flickering" or "swimming" textures.
- Hardware Accessibility: Most high-end solutions require specialized multi-camera rigs, limiting their use "in the wild."
Methodology
The proposed pipeline leverages a garment printed with a specific color-coded grid to facilitate dense correspondence detection.
Enhanced Correspondence and Temporal Smoothing
To improve detection, the authors implemented a "graph-growing" strategy that increases detection recall by approximately 20%, even in motion-blurred areas or near garment edges. To ensure temporal smoothness, the system utilizes a Blended-Weight MLP. Instead of a standard neural network, this architecture uses weights interpolated over time via cubic splines, effectively eliminating texture jitter while reducing the model's memory footprint to 12% of what would be required for separate per-frame models.
Jacobian $L_{0.5}$ Loss
A key innovation is the use of a Jacobian-based $L_{0.5}$ loss function. While standard $L_2$ smoothing tends to blur sharp edges, the $L_{0.5}$ loss encourages sparsity in the gradient of the UV map. This allows the model to accurately represent sharp discontinuities—such as the "seams" created by overlapping fabric folds—without distorting the surrounding texture.
Shading Preservation
To maintain realism, a U-Net architecture extracts a shading layer from the original video. This layer, containing lighting and shadow information, is multiplied back onto the new texture, ensuring the digital fabric reacts naturally to the environment's illumination.
Results and Discussion
The method was evaluated using a mix of synthetic and real-world data. The dataset-train and dataset-eval sets consisted of synthetic 3D garments undergoing complex physical simulations in Blender and Houdini, while dataset-real included smartphone videos of models in indoor and outdoor settings.
Performance Metrics
On synthetic evaluation sets, the system achieved a Mean Squared Error (MSE) of 16.82mm, a significant improvement over standard RBF interpolation (54.39mm). Qualitatively, the system outperformed baselines like DensePose and HumanGPS, which often produce distorted, "rubbery" textures on loose garments.
Robustness and Limitations
The system demonstrates remarkable robustness to rapid motion, such as dancing or jumping. However, limitations remain. The requirement for a physical patterned garment is a barrier to entry for existing inventory. Additionally, the current model assumes matte materials and struggles with extremely fine details like thin belts or small decorative knots.
Conclusion
Pattern-guided UV recovery represents a significant step toward democratizing high-end virtual fashion production. By combining clever mathematical constraints with efficient neural architectures, this research provides a practical tool for realistic, automated video re-texturing in real-world settings. Future work aims to incorporate specular material properties and improve masking for complex garment accessories.
Summary updated