Resolving Self-Occlusion Artifacts in Virtual Try-On

97 concepts76 papers3 proposals

Virtual try-on (VTON) and pose-guided person synthesis aim to map a garment or appearance onto a target human body. While simple standing poses are relatively straightforward, the task becomes exponentially harder when the subject moves. As limbs cross the torso or hands tuck into pockets, the model must resolve complex spatial interactions where one part of the body hides another. These "self-occlusions" are the primary source of visual artifacts in modern synthesis pipelines.

The Geometry of Occlusion

Most early VTON frameworks, such as VITON, rely on a coarse-to-fine strategy: first warping the garment to match the body's shape, then refining the result. However, as Han et al. observed, 2D geometric transformations struggle with extreme poses, such as arms positioned behind the back. Because 2D warping assumes a continuous surface, it cannot easily account for hidden surfaces or the way fabric should disappear and reappear behind a limb.

This lack of 3D structural information leads to two distinct failure modes, as categorized by Yang et al.:

  1. Inherent Artifacts: Portions of the original garment remain visible (ghosting) because the model fails to fully "de-occlude" the background.
  2. Acquired Artifacts: The new garment incorrectly overlaps body parts, such as a sleeve covering a hand that should be in front of the fabric.

The Parsing Trap

To handle these spatial relationships, many systems use human semantic parsing (segmentation) to define boundaries between skin and clothes. However, Ge et al. and Lin argue that these external tools are themselves vulnerable to complex poses. When a segmentation tool fails to distinguish a crossed arm from the torso, the synthesis model may "melt" the garment into the skin or erase the limb entirely.

To bypass this, the authors of S-WUTON and RMGN proposed "parser-free" methods. Instead of relying on rigid, error-prone masks, these models learn to identify clothing boundaries implicitly through distillation or regional guidance, reducing the "ghosting" artifacts common in traditional pipelines.

Preserving the Unseen

A subtle but persistent artifact occurs when a model erases personal accessories like watches, jewelry, or bags. Because these items are not part of the "target garment" input, generative models often treat them as part of the old clothing to be replaced. Kim et al. note that standard latent diffusion models often fail to distinguish between the garment and overlapping accessories, leading to their unintended erasure.

Furthermore, when a subject undergoes extreme pose changes, the model must "hallucinate" textures that were previously hidden. Ma et al. highlight that the non-rigid nature of the human body requires models to manage non-linear transformations. If a model overfits to the source image's pixel coordinates, it creates a "copy-paste" effect that fails to realistically reconstruct occluded regions in the new orientation, as noted by Lu et al..

Moving Toward 3D and Latent Correspondence

To solve these issues, the field is shifting away from explicit 2D warping toward 3D-aware and latent-space solutions:

  • 3D Priors: Guan et al. and Cao et al. argue that moving into 3D space (using models like SMPL) allows the system to maintain a complete texture map even when parts are occluded, preventing "melting" limbs.
  • Latent Correspondence: Instead of warping pixels, Kim et al. (StableVITON) learn semantic correspondences directly within the latent space of a diffusion model. This allows the model to integrate the garment more naturally around occlusions like long hair or overlapping limbs.
  • Attention Regularization: Zhou et al. introduced Leffa, which treats attention maps as learnable flow fields. By guiding the model to "look" at the correct reference regions even during large occlusions, they maintain texture consistency when garment sections are hidden and then revealed.

By treating occlusion not just as a masking problem but as a challenge of 3D spatial reasoning, these methods allow for high-fidelity synthesis even in the most complex human configurations.

Key Foundational Work

VITON: An Image-Based Virtual Try-On Network (2018) — contributed 2 concept elements to this direction.
VITON: An Image-based Virtual Try-on Network (2017) — contributed 1 concept elements to this direction.
M2E-Try On Net: Fashion from Model to Everyone (2018) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_self_occlusion_artifacts_26672bbf,
  title = {self-occlusion artifacts},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/self-occlusion-artifacts/rdr_26672bbfaac148a1b0a9eddebd057b9b},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_self_occlusion_artifacts_26672bbf_v1,
  title = {self-occlusion artifacts},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/self-occlusion-artifacts/rdr_26672bbfaac148a1b0a9eddebd057b9b/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake