Abstract
This paper presents a self-supervised 3D clothing reconstruction method that recovers the geometry and texture of human clothing from a single image. We address three primary challenges: the lack of 3D ground-truth meshes, the limitations of rigid template-based methods for non-rigid objects like dresses and handbags, and the inherent ambiguity between camera distance and object scale (the "collider" problem). We propose a causality-aware self-supervised learning method (CASL) that utilizes an explainable structural causal map (SCM) and expectation-maximization loops to disentangle camera position, shape, texture, and illumination. Extensive experiments on the ATR, Market-HQ, and CUB datasets demonstrate that our method yields high-fidelity 3D reconstructions, significantly outperforming existing baselines.
Introduction: The Inverse Problem of 3D Reconstruction
Reconstructing a 3D object from a single 2D image is a classic "inverse problem" fraught with ambiguity. In the fashion world, this is particularly difficult because most existing models rely on rigid human body templates (like SMPL) which fail to capture the flow of a dress or the structure of a handbag. Furthermore, 3D annotations for clothing are incredibly expensive and rare, making supervised learning impractical.
A core issue identified by the authors is the "collider" problem: the inherent ambiguity between a large object far away and a small object close to the camera. In standard deep learning, if the reconstruction fails, the loss function "punishes" all encoders (shape, camera, texture, illumination) equally. This prevents the model from truly learning to separate these distinct factors.
Methodology: Causality-aware Self-supervised Learning (CASL)
The proposed CASL architecture is built around an explainable Structural Causal Map (SCM) that explicitly defines how variables like the 3D prior template, shape offsets, and camera pose interact.
Key innovations include:
- Two Expectation-Maximization (EM) Loops: To solve the "collider" problem, the authors use causal "interventions." The Encoder Loop fixes three variables while optimizing the fourth, ensuring the loss specifically penalizes the correct encoder. The Prototype Loop iteratively updates a category-specific "mean shape," allowing the shape encoder to focus on fine-grained instance details.
- Integration Module (IM): This module fuses local features from the 3D template with visual features from the 2D image, helping the model better align the mesh with the actual pixels.
- Differentiable Rendering: Since there is no 3D ground truth, the model reconstructs the 3D mesh, renders it back into a 2D image, and compares it against the original input to calculate loss.
Experimental Results
The model was evaluated on the ATR and Market-HQ fashion datasets, as well as the CUB bird dataset to demonstrate scalability to general non-rigid objects.
On the ATR dataset, the model achieved a 80.4% MaskIoU, significantly outperforming established methods like HMR (69.7%) and ROMP (70.3%). The model also showed a major lead in FID (Fréchet Inception Distance) for novel views, meaning the 3D models look realistic even when rotated. Furthermore, the authors demonstrated "clothing exchange" results, where the texture from one person is successfully mapped onto the 3D shape reconstructed from another, proving the model truly understands the underlying geometry.
Conclusion
By treating neural network training as a causal system, the "3D Magic Mirror" prevents "compensation effects" where one encoder hides the errors of another. While the model achieves high-fidelity results, it still faces the "Janus Issue"—a common problem where the model might duplicate front-facing features on the back of the object due to symmetry assumptions. Future work aims to leverage large-scale multi-view datasets to learn better 3D priors and expand the method to even more complex real-world scenarios.
Summary updated