Hierarchical Cross-Attention Network for Virtual Try-On

Hao Tang, Bin Ren, Pingping Wu, Nicu Sebe

TL;DR

HCANet introduces a Hierarchical Cross-Attention mechanism to the virtual try-on task, moving beyond simple image concatenation to model the complex, long-range relationships between body shape and clothing. By refining feature interactions across two stages—geometric matching and image synthesis—it achieves a new state-of-the-art FID score of 8.25, significantly improving the realism of draped textures and garment alignment.

What Problem Does This Paper Address?

Virtual try-on (VTON) isn't as simple as "copy-pasting" a shirt onto a person. To look realistic, the software must account for how fabric warps around a specific body shape, how textures stretch over contours, and how the garment interacts with the person's posture.

The authors identify a major gap in existing 2D image-based methods: they often fail to capture long-range global interactive correlations. Most models rely on local convolutions or simple feature concatenation, which can't "see" how a shoulder's position on one side of the image should affect the drape of a sleeve on the other. This leads to visual anomalies like "floating" clothes, misaligned textures, or garments that don't respect the wearer's unique geometry.

Methods

The core innovation is the Hierarchical Cross-Attention (HCA) block, which is integrated into a two-stage pipeline:

  1. Stage I: Geometric Matching: The model uses Thin-Plate Spline (TPS) transformation to warp the in-shop clothing to fit the person's pose.
  2. Stage II: Try-On: The warped clothing and the person's representation are fused to generate the final photorealistic image.

The HCA Block "Secret Sauce": Instead of just stacking feature maps, the HCA block performs attention in two hierarchical steps:

  • Internal Person Fusion: It first performs cross-attention between different "person" modalities (pose heatmaps, body shape masks, and RGB identity images). This creates a nuanced, unified representation of the individual.
  • Person-Clothing Fusion: It then performs a parallel cross-attention operation between this refined person feature and the clothing features. This allows the model to compute the "response" at any pixel as a weighted sum of features from the other modality, effectively "aligning" the fabric to the body in feature space.

Datasets

The researchers utilized the widely recognized VITON dataset (originally by Han et al.), specifically using a reorganized and cleaned version proposed in the CP-VTON+ work.

  • Scale: Approximately 19,000 pairs of front-view images of women and top clothing.
  • Curation: 16,253 pairs were meticulously cleaned to remove inconsistencies.
  • Split: 14,221 pairs for training and 2,032 for validation/testing.

Key Insights and Ideas

  • Attention over Concatenation: The "aha moment" here is that simply concatenating features (the standard approach) doesn't force the model to learn the relationship between body and cloth. Cross-attention acts as a dynamic filter, telling the model exactly which parts of the clothing should "pay attention" to which parts of the body.
  • Hierarchical Refinement: By fusing person-specific data (pose + shape) before introducing the clothing, the model establishes a more robust geometric foundation, preventing the "floating" effect seen in earlier models.
  • Global Receptive Fields: Because attention mechanisms have a global receptive field, the model can maintain consistency in patterns (like stripes or logos) even when the garment undergoes significant warping.

Key Results

HCANet outperformed several major baselines, including CP-VTON+, ACGPN, and the recent FS-VTON.

  • Quantitative:
    • FID (Fréchet Inception Distance): Achieved 8.25, a notable improvement over the previous best of 8.89 (FS-VTON) and a massive leap from CP-VTON+ (21.04). Lower FID indicates higher image quality and diversity.
  • SSIM/LPIPS: Scored 0.859 and 0.102 respectively, indicating high structural similarity and perceptual realism compared to ground truth.
  • Qualitative: The model excels at preserving intricate textures (like fine stripes or lace) and produces much sharper boundaries where the clothing meets the skin or other garments.
  • User Study: In human evaluations, HCANet was preferred for "Image Realism" 27.2% of the time and "Clothing Consistency" 29.8% of the time, beating all other tested models.

Limitations and Future Work

Despite the impressive results, the authors highlight a few "inevitable" failure cases:

  • The Hand Problem: The model sometimes distorts hands or fingers. This is because the input data (pose heatmaps) lacks detailed finger/palm keypoints, forcing the model to "guess" the hand structure.
  • Lighting Inconsistencies: There are occasionally visible artifacts (unnatural shadows or light) at the junction where the shirt meets the pants.
  • Extreme Poses: Very large transformations or heavy self-occlusion (e.g., an arm crossed over the chest) still pose a challenge for 2D-based methods.
  • Future Direction: The authors suggest that incorporating 3D data (like body meshes) or more granular human annotations could solve the hand and occlusion issues.

Summary updated

Abstract

In this article, we present an innovative solution tailored for the intricate challenges of the virtual try-on task—our novel Hierarchical Cross-Attention Network, HCANet. HCANet is meticulously crafted with two primary stages: geometric matching and try-on, each playing a crucial role in delivering realistic and visually convincing virtual try-on outcomes. A distinctive feature of HCANet is the incorporation of a novel Hierarchical Cross-Attention (HCA) block into both stages, enabling the effective capture of long-range correlations between individual and clothing modalities. The HCA block functions as a cornerstone, enhancing the depth and robustness of the network. By adopting a hierarchical approach, it facilitates a nuanced representation of the interaction between the person and clothing, capturing intricate details essential for an authentic virtual try-on experience. Our extensive set of experiments establishes the prowess of HCANet. The results showcase its cutting-edge performance across both objective quantitative metrics and subjective evaluations of visual realism. HCANet stands out as a state-of-the-art solution, demonstrating its capability to generate virtual try-on results that not only excel in accuracy but also satisfy subjective criteria of realism. This marks a significant step forward in advancing the field of virtual try-on technologies.
BibTeX
@article{tang2025hierarchical,
  author = {Hao Tang and Bin Ren and Pingping Wu and Nicu Sebe},
  title = {{Hierarchical Cross-Attention Network for Virtual Try-On}},
  journal = {IEEE Trans. Multimedia},
  year = {2025},
  url = {https://openreview.net/forum?id=JV7C2Q69md},
}

Loading related content...

Figure