ReFu: Refine and Fuse the Unobserved View for Detail-Preserving Single-Image 3D Human Reconstruction

Gyumin Shim, Minsoo Lee, Jaegul Choo

Reconstructing a complete 3D human model from a single 2D image is a fundamental challenge in computer vision. While implicit function-based methods have advanced the field, they frequently struggle with the "unobserved view"—the side of the person not visible to the camera. This typically results in over-smoothed textures and "floating" geometry artifacts. ReFu (Refine and Fuse) addresses these issues by synthesizing a high-detail image of the unobserved view and fusing it back into the reconstruction process to generate realistic textures and patterns.

The Problem: Beyond Plausible Geometry

Current single-view reconstruction methods often suffer from two primary limitations:

  1. Over-smoothing: Because the model lacks direct visual evidence of the back, it often predicts an "average" texture. This results in a blurry, featureless appearance that lacks the wrinkles and patterns of the original clothing.
  2. Diffused Occupancy: Many models fail to establish a sharp boundary between the human surface and empty space. This leads to "cloudy" artifacts or noisy surfaces where the model is uncertain about the exact boundary of the body.

ReFu aims to move beyond "plausible but blurry" results to achieve "realistic and detailed" reconstructions.

The ReFu Methodology

ReFu introduces a three-stage "coarse-to-fine" pipeline designed to preserve high-frequency details across the entire 3D surface.

1. Initial Coarse Reconstruction

The first stage predicts a coarse 3D occupancy field and texture. To mitigate the "floating artifact" problem, ReFu utilizes occupancy-based volume rendering. By supervising the model with both 2D and 3D data, the framework encourages lower entropy in occupancy values, leading to a cleaner and more defined initial shape compared to standard implicit methods.

2. Detail Refinement via Front-to-Back Warping

The core innovation of ReFu lies in its refinement stage. The model renders a coarse image of the unobserved back view and then employs a StyleGAN2-based architecture to enhance it. To ensure visual consistency, the authors implement a front-to-back warping mechanism. This transfers high-frequency details—such as fabric patterns or specific garment textures—from the observed front view to the synthesized back view, ensuring the back of a garment matches the front.

3. Multi-View Fusion

In the final stage, the original source image and the newly refined backside image are treated as dual inputs. The model fuses features from both views to reconstruct a final 3D mesh. This approach allows the system to leverage standard pixel-aligned feature extraction on the synthesized back view, effectively "baking" the generated details into the 3D geometry rather than attempting to generate them directly in 3D space.

Experimental Evaluation

The researchers evaluated ReFu using high-quality 3D scan datasets, including THUman2.0 and Twindom, as well as real-world images from DeepFashion to test performance on "in-the-wild" photos.

  • Geometry: On THUman2.0, ReFu achieved a Point-to-Surface (P2S) error of 1.339 cm, outperforming the previous state-of-the-art method, PaMIR (1.346 cm). While the numerical gain in geometry is incremental, the visual improvement in surface sharpness is significant.
  • Perceptual Quality: The most significant gains were observed in texture and detail. ReFu achieved superior LPIPS scores (perceptual similarity) and ReID distances, indicating that the reconstructed back views are both realistic and consistent with the subject's identity.

Conclusion and Future Work

ReFu represents a significant step toward realistic single-image 3D human reconstruction. By explicitly refining the unobserved view and fusing it into the final mesh, it overcomes the traditional "backside blur." However, the method remains dependent on accurate SMPL parametric model fitting; extreme or unusual poses can still cause reconstruction failures. Additionally, the multi-stage approach is computationally intensive. Future research will focus on more memory-efficient 3D representations and streamlining the pipeline into a more efficient training process.

Summary updated

BibTeX
@inproceedings{shim2022refu,
  author = {Gyumin Shim and Minsoo Lee and Jaegul Choo},
  title = {{ReFu: Refine and Fuse the Unobserved View for Detail-Preserving Single-Image 3D Human Reconstruction}},
  booktitle = {ACM Multimedia},
  year = {2022},
  url = {https://openreview.net/forum?id=8wagrq5HDp},
}
Figure