Hierarchical Cross-Attention Network for Virtual Try-On

Hao Tang, Bin Ren, Pingping Wu, Nicu Sebe

Abstract

In this paper, we present an innovative solution for the challenges of the virtual try-on task: our novel Hierarchical Cross-Attention Network (HCANet). HCANet is crafted with two primary stages: geometric matching and try-on, each playing a crucial role in delivering realistic virtual try-on outcomes. A key feature of HCANet is the incorporation of a novel Hierarchical Cross-Attention (HCA) block into both stages, enabling the effective capture of long-range correlations between individual and clothing modalities. The HCA block enhances the depth and robustness of the network. By adopting a hierarchical approach, it facilitates a nuanced representation of the interaction between the person and clothing, capturing intricate details essential for an authentic virtual try-on experience. Our experiments establish the prowess of HCANet. The results showcase its performance across both quantitative metrics and subjective evaluations of visual realism. HCANet stands out as a state-of-the-art solution, demonstrating its capability to generate virtual try-on results that excel in accuracy and realism. This marks a significant step in advancing virtual try-on technologies.

Introduction

Virtual try-on (VTON) is a complex computer vision task that goes far beyond simple image overlay. To achieve a realistic result, a model must account for how fabric warps around a specific body shape, how textures stretch over physical contours, and how garments interact with a person's unique posture.

Existing 2D image-based methods often struggle with capturing long-range global interactive correlations. Relying primarily on local convolutions or simple feature concatenation, these models frequently fail to "see" how a shoulder's position on one side of the image should affect the drape of a sleeve on the other. This limitation leads to visual artifacts such as misaligned textures, garments that appear to "float" on the body, or clothing that fails to respect the wearer's underlying geometry. HCANet addresses these issues by introducing a mechanism designed to model global relationships between the person and the garment.

Methodology

The core of the HCANet architecture is the Hierarchical Cross-Attention (HCA) block, which is integrated into a robust two-stage pipeline designed for precision and realism.

  1. Stage I: Geometric Matching: In this stage, the model utilizes Thin-Plate Spline (TPS) transformation to warp the in-shop clothing image to fit the specific pose and body shape of the person.
  2. Stage II: Try-On: The warped clothing from the first stage is fused with the person's representation to generate the final photorealistic try-on image.
The HCANet framework consists of two stages: Geometric Matching for garment warping and Try-On for final image synthesis, both utilizing the Hierarchical Cross-Attention (HCA) block.

The HCA Block

Unlike standard models that simply stack feature maps, the HCA block performs attention in two hierarchical steps. First, it performs Internal Person Fusion, conducting cross-attention between different person modalities (such as pose heatmaps, body shape masks, and RGB images) to create a unified representation of the individual. Second, it executes Person-Clothing Fusion, a parallel cross-attention operation between the refined person features and the clothing features. This allows the model to compute the "response" at any pixel as a weighted sum of features from the other modality, effectively aligning the fabric to the body in feature space.

Experimental Setup

The researchers evaluated HCANet using the VITON dataset, specifically the cleaned and reorganized version introduced in the CP-VTON+ study. This dataset contains approximately 19,000 pairs of front-view images of women and corresponding top clothing. After meticulous cleaning to remove inconsistencies, the dataset was split into 14,221 pairs for training and 2,032 pairs for validation and testing.

Performance and Results

HCANet achieved a new state-of-the-art (SOTA) performance, outperforming established baselines such as CP-VTON+, ACGPN, and FS-VTON across multiple metrics.

  • Quantitative Excellence: The model achieved a Fréchet Inception Distance (FID) of 8.25, a significant improvement over the previous best of 8.89 (FS-VTON). It also scored 0.859 in SSIM and 0.102 in LPIPS, indicating superior structural similarity and perceptual realism.
  • Qualitative Superiority: As shown in visual comparisons, HCANet excels at preserving intricate textures—such as fine stripes, logos, and lace—while producing sharp, natural boundaries where clothing meets skin.
Visual comparison of try-on results. HCANet demonstrates superior ability in preserving garment details and handling complex body-garment interactions compared to existing methods.

In human evaluations, HCANet was preferred for "Image Realism" 27.2% of the time and "Clothing Consistency" 29.8% of the time, surpassing all other tested models.

Limitations and Future Outlook

While HCANet represents a major leap forward, some challenges remain. The model occasionally struggles with high-detail areas like hands and fingers due to the lack of granular keypoints in standard pose heatmaps. Additionally, extreme poses involving heavy self-occlusion (e.g., arms crossed) can still cause distortions. Future work may involve incorporating 3D body meshes or more detailed human annotations to resolve these complex geometric and lighting inconsistencies.

Summary updated

Abstract

In this paper, we present an innovative solution for the challenges of the virtual try-on task: our novel Hierarchical Cross-Attention Network (HCANet). HCANet is crafted with two primary stages: geometric matching and try-on, each playing a crucial role in delivering realistic virtual try-on outcomes. A key feature of HCANet is the incorporation of a novel Hierarchical Cross-Attention (HCA) block into both stages, enabling the effective capture of long-range correlations between individual and clothing modalities. The HCA block enhances the depth and robustness of the network. By adopting a hierarchical approach, it facilitates a nuanced representation of the interaction between the person and clothing, capturing intricate details essential for an authentic virtual try-on experience. Our experiments establish the prowess of HCANet. The results showcase its performance across both quantitative metrics and subjective evaluations of visual realism. HCANet stands out as a state-of-the-art solution, demonstrating its capability to generate virtual try-on results that excel in accuracy and realism. This marks a significant step in advancing virtual try-on technologies.
BibTeX
@article{tang2025hierarchical,
  author = {Hao Tang and Bin Ren and Pingping Wu and Nicu Sebe},
  title = {{Hierarchical Cross-Attention Network for Virtual Try-On}},
  journal = {IEEE Trans. Multimedia},
  year = {2025},
  url = {https://openreview.net/forum?id=JV7C2Q69md},
}
Figure