3D Magic Mirror: Clothing Reconstruction from a Single Image via a Causal Perspective

Zhedong Zheng, Jiayin Zhu, Wei Ji, Yi Yang, Tat‐Seng ChuaResearch Square 2025 · DOI

Abstract

This paper presents a self-supervised 3D clothing reconstruction method that recovers the geometry and texture of human clothing from a single image. We address three primary challenges: the lack of 3D ground-truth meshes, the limitations of rigid template-based methods for non-rigid objects like dresses and handbags, and the inherent ambiguity between camera distance and object scale (the "collider" problem). We propose a causality-aware self-supervised learning method (CASL) that utilizes an explainable structural causal map (SCM) and expectation-maximization loops to disentangle camera position, shape, texture, and illumination. Extensive experiments on the ATR, Market-HQ, and CUB datasets demonstrate that our method yields high-fidelity 3D reconstructions, significantly outperforming existing baselines.

Introduction: The Inverse Problem of 3D Reconstruction

Reconstructing a 3D object from a single 2D image is a classic "inverse problem" fraught with ambiguity. In the fashion world, this is particularly difficult because most existing models rely on rigid human body templates (like SMPL) which fail to capture the flow of a dress or the structure of a handbag. Furthermore, 3D annotations for clothing are incredibly expensive and rare, making supervised learning impractical.

A core issue identified by the authors is the "collider" problem: the inherent ambiguity between a large object far away and a small object close to the camera. In standard deep learning, if the reconstruction fails, the loss function "punishes" all encoders (shape, camera, texture, illumination) equally. This prevents the model from truly learning to separate these distinct factors.

Methodology: Causality-aware Self-supervised Learning (CASL)

The proposed CASL architecture is built around an explainable Structural Causal Map (SCM) that explicitly defines how variables like the 3D prior template, shape offsets, and camera pose interact.

The CASL architecture utilizes four encoders and an Integration Module (IM) to disentangle shape, camera, texture, and illumination.

Key innovations include:

  • Two Expectation-Maximization (EM) Loops: To solve the "collider" problem, the authors use causal "interventions." The Encoder Loop fixes three variables while optimizing the fourth, ensuring the loss specifically penalizes the correct encoder. The Prototype Loop iteratively updates a category-specific "mean shape," allowing the shape encoder to focus on fine-grained instance details.
  • Integration Module (IM): This module fuses local features from the 3D template with visual features from the 2D image, helping the model better align the mesh with the actual pixels.
  • Differentiable Rendering: Since there is no 3D ground truth, the model reconstructs the 3D mesh, renders it back into a 2D image, and compares it against the original input to calculate loss.

Experimental Results

The model was evaluated on the ATR and Market-HQ fashion datasets, as well as the CUB bird dataset to demonstrate scalability to general non-rigid objects.

Qualitative comparison of 3D reconstruction and novel view synthesis against state-of-the-art methods on the ATR dataset.

On the ATR dataset, the model achieved a 80.4% MaskIoU, significantly outperforming established methods like HMR (69.7%) and ROMP (70.3%). The model also showed a major lead in FID (Fréchet Inception Distance) for novel views, meaning the 3D models look realistic even when rotated. Furthermore, the authors demonstrated "clothing exchange" results, where the texture from one person is successfully mapped onto the 3D shape reconstructed from another, proving the model truly understands the underlying geometry.

Conclusion

By treating neural network training as a causal system, the "3D Magic Mirror" prevents "compensation effects" where one encoder hides the errors of another. While the model achieves high-fidelity results, it still faces the "Janus Issue"—a common problem where the model might duplicate front-facing features on the back of the object due to symmetry assumptions. Future work aims to leverage large-scale multi-view datasets to learn better 3D priors and expand the method to even more complex real-world scenarios.

Summary updated

Abstract

<title>Abstract</title> This paper aims to study a self-supervised 3D clothing reconstruction method, which recovers the geometry shape and texture of human clothing from a single image. Compared with existing methods, we observe that three primary challenges remain:(1) 3D ground-truth meshes of clothing are usually inaccessible due to annotation difficulties and time costs; (2) Conventional template-based methods are limited to modeling non-rigid objects, e.g., handbags and dresses, which are common in fashion images;(3) The inherent ambiguity compromises the model training, such as the dilemma between a large shape with a remote camera or a small shape with a close camera. In an attempt to address the above limitations, we propose a causality-aware self-supervised learning method to adaptively reconstruct 3D non-rigid objects from 2D images without 3D annotations. In particular, to solve the inherent ambiguity among four implicit variables, i.e., camera position, shape, texture, and illumination, we introduce an explainable structural causal map (SCM) to build our model. The proposed model structure follows the spirit of the causal map, which explicitly considers the prior template in the camera estimation and shape prediction. When optimization, the causality intervention tool, i.e., two expectation-maximization loops, is deeply embedded in our algorithm to (1) disentangle four encoders and (2) facilitate the prior template. Extensive experiments on two 2D fashion benchmarks (ATR and Market-HQ) show that the proposed method could yield high-fidelity 3D reconstruction. Furthermore, we also verify the scalability of the proposed method on a fine-grained bird dataset, i.e., CUB.
BibTeX
@inproceedings{zheng2025d,
  author = {Zhedong Zheng and Jiayin Zhu and Wei Ji and Yi Yang and Tat‐Seng Chua},
  title = {{3D Magic Mirror: Clothing Reconstruction from a Single Image via a Causal Perspective}},
  booktitle = {Research Square},
  year = {2025},
  doi = {10.21203/rs.3.rs-6439497/v1},
}
Figure