Abstract
In this paper, we present an innovative solution for the challenges of the virtual try-on task: our novel Hierarchical Cross-Attention Network (HCANet). HCANet is crafted with two primary stages: geometric matching and try-on, each playing a crucial role in delivering realistic virtual try-on outcomes. A key feature of HCANet is the incorporation of a novel Hierarchical Cross-Attention (HCA) block into both stages, enabling the effective capture of long-range correlations between individual and clothing modalities. The HCA block enhances the depth and robustness of the network. By adopting a hierarchical approach, it facilitates a nuanced representation of the interaction between the person and clothing, capturing intricate details essential for an authentic virtual try-on experience. Our experiments establish the prowess of HCANet. The results showcase its performance across both quantitative metrics and subjective evaluations of visual realism. HCANet stands out as a state-of-the-art solution, demonstrating its capability to generate virtual try-on results that excel in accuracy and realism. This marks a significant step in advancing virtual try-on technologies.
Introduction
Virtual try-on (VTON) is a complex computer vision task that goes far beyond simple image overlay. To achieve a realistic result, a model must account for how fabric warps around a specific body shape, how textures stretch over physical contours, and how garments interact with a person's unique posture.
Existing 2D image-based methods often struggle with capturing long-range global interactive correlations. Relying primarily on local convolutions or simple feature concatenation, these models frequently fail to "see" how a shoulder's position on one side of the image should affect the drape of a sleeve on the other. This limitation leads to visual artifacts such as misaligned textures, garments that appear to "float" on the body, or clothing that fails to respect the wearer's underlying geometry. HCANet addresses these issues by introducing a mechanism designed to model global relationships between the person and the garment.
Methodology
The core of the HCANet architecture is the Hierarchical Cross-Attention (HCA) block, which is integrated into a robust two-stage pipeline designed for precision and realism.
- Stage I: Geometric Matching: In this stage, the model utilizes Thin-Plate Spline (TPS) transformation to warp the in-shop clothing image to fit the specific pose and body shape of the person.
- Stage II: Try-On: The warped clothing from the first stage is fused with the person's representation to generate the final photorealistic try-on image.
The HCA Block
Unlike standard models that simply stack feature maps, the HCA block performs attention in two hierarchical steps. First, it performs Internal Person Fusion, conducting cross-attention between different person modalities (such as pose heatmaps, body shape masks, and RGB images) to create a unified representation of the individual. Second, it executes Person-Clothing Fusion, a parallel cross-attention operation between the refined person features and the clothing features. This allows the model to compute the "response" at any pixel as a weighted sum of features from the other modality, effectively aligning the fabric to the body in feature space.
Experimental Setup
The researchers evaluated HCANet using the VITON dataset, specifically the cleaned and reorganized version introduced in the CP-VTON+ study. This dataset contains approximately 19,000 pairs of front-view images of women and corresponding top clothing. After meticulous cleaning to remove inconsistencies, the dataset was split into 14,221 pairs for training and 2,032 pairs for validation and testing.
Performance and Results
HCANet achieved a new state-of-the-art (SOTA) performance, outperforming established baselines such as CP-VTON+, ACGPN, and FS-VTON across multiple metrics.
- Quantitative Excellence: The model achieved a Fréchet Inception Distance (FID) of 8.25, a significant improvement over the previous best of 8.89 (FS-VTON). It also scored 0.859 in SSIM and 0.102 in LPIPS, indicating superior structural similarity and perceptual realism.
- Qualitative Superiority: As shown in visual comparisons, HCANet excels at preserving intricate textures—such as fine stripes, logos, and lace—while producing sharp, natural boundaries where clothing meets skin.
In human evaluations, HCANet was preferred for "Image Realism" 27.2% of the time and "Clothing Consistency" 29.8% of the time, surpassing all other tested models.
Limitations and Future Outlook
While HCANet represents a major leap forward, some challenges remain. The model occasionally struggles with high-detail areas like hands and fingers due to the lack of granular keypoints in standard pose heatmaps. Additionally, extreme poses involving heavy self-occlusion (e.g., arms crossed) can still cause distortions. Future work may involve incorporating 3D body meshes or more detailed human annotations to resolve these complex geometric and lighting inconsistencies.
Summary updated