Lightweight Explicit 3D Human Digitization via Normal Integration

Jiaxuan Liu, Jingyi Wu, Ruiyang Jing, Han Yu, Jing Liu, Liang SongSensors 2025 · DOI

TL;DR

This paper introduces a lightweight and efficient method for creating detailed 3D human models from single 2D images. By integrating Dilated Convolutions and Cross-Covariance Attention, they significantly reduce model complexity while a novel normal map loss function enhances reconstruction quality, achieving an ~80% reduction in parameters compared to existing methods.

What Problem Does This Paper Address?

Generating high-fidelity 3D human models from images is a crucial task with applications ranging from virtual reality to autonomous driving. However, state-of-the-art methods often rely on large, computationally expensive neural networks. This makes them impractical for deployment on resource-constrained devices like mobile phones or embedded systems, where inference speed and memory usage are critical. The challenge lies in balancing the need for detailed 3D reconstruction with the constraints of real-world applications.

Methods

The core of this work is a novel lightweight generative network called Lite-GN. This network is designed to efficiently generate surface normal maps, which are then used to reconstruct the 3D human mesh.

  • Lightweight Architecture: Lite-GN combines Dilated Convolutions and Cross-Covariance Attention. Dilated convolutions expand the network's receptive field without increasing the number of parameters or computational cost, allowing it to capture broader contextual information. Cross-Covariance Attention, on the other hand, focuses on inter-channel relationships, improving feature extraction efficiency. This combination is key to reducing model complexity.
  • Normal Map Generation: The process starts with estimating SMPL-X parameters from the input image to get a coarse 3D human model. A differentiable renderer then generates initial normal maps from this model. These, along with the input image, are fed into Lite-GN to produce refined front and back normal maps.
  • Surface Refinement: The generated normal maps are integrated using a depth-aware silhouette-consistent bilateral normal integration (d-BiNI) method. This process leverages prior information from the SMPL-X model to ensure alignment and consistency.
  • Novel Loss Function: A significant contribution is a new loss function specifically designed for normal maps. This loss function incorporates VGG Loss for perceptual similarity and Huber Loss for spatial accuracy, aiming to improve the geometric quality of the reconstructed surface.
  • SMPL-X Prior: The SMPL-X model (a skinned multi-person linear model) is used as a prior to guide the reconstruction, ensuring accurate pose and shape estimation, especially for complex clothing and poses.

Datasets

  • THuman2.0: Used for training the generative network. This dataset is known for its high-quality human scans and corresponding SMPL/SMPL-X parameters.
  • CAPE: Used for quantitative evaluation. This dataset is noted for its diverse pose variations and is used to assess robustness.
  • RenderPeople: Also used for quantitative evaluation, this dataset focuses on evaluating robustness to intricate clothing.

Key Insights and Ideas

  • Efficiency through Architecture: The paper demonstrates that by carefully combining Dilated Convolutions and Cross-Covariance Attention, it's possible to achieve high-quality 3D reconstruction with significantly fewer parameters, making it suitable for edge devices.
  • Normal Maps as a Bridge: Using normal maps as an intermediate representation proves effective for capturing surface details and can be efficiently generated by a lightweight network.
  • Tailored Loss for Geometry: A specialized loss function that considers both perceptual and spatial aspects of normal maps is crucial for achieving accurate geometric reconstruction.
  • SMPL-X as a Versatile Prior: Leveraging SMPL-X not only helps with pose and shape but also aids in handling complex clothing and filling in occluded areas.

Key Results

  • Parameter Reduction: The proposed method achieves approximately 80% reduction in training parameters compared to existing methods.
  • Quantitative Performance: On the CAPE dataset, the method achieves competitive results, with a Chamfer distance of 0.965, P2S of 0.930, and Normals difference of 0.0472. It significantly outperforms PIFUHD and PaMIR, and is comparable to ICON and ECON.
  • Lightweight Efficiency: The model size is substantially reduced, with a weight size of 201.6 MB, compared to over 1 GB for methods like PIFUHD and ECON. Inference speed is also improved, with a runtime of 12.8 ms.
  • Qualitative Results: The method generates detailed 3D human models, accurately reconstructing complex poses and clothing, as shown in the provided figures. User studies indicated a higher preference for the proposed method over comparative approaches.

Limitations and Future Work

  • Handling Extreme Cases: The authors acknowledge that challenges remain in handling extremely complex poses and clothing, where artifacts like missing limbs or inconsistent back-side reconstructions might still occur.
  • Future Directions: Future work will focus on improving robustness to extreme poses, refining clothing reconstruction, and enhancing overall consistency to advance monocular 3D human reconstruction for broader real-world applications.

Summary updated

Abstract

In recent years, generating 3D human models from images has gained significant attention in 3D human reconstruction. However, deploying large neural network models in practical applications remains challenging, particularly on resource-constrained edge devices. This problem is primarily because large neural network models require significantly higher computational power, which imposes greater demands on hardware capabilities and inference time. To address this issue, we can optimize the network architecture to reduce the number of model parameters, thereby alleviating the heavy reliance on hardware resources. We propose a lightweight and efficient 3D human reconstruction model that balances reconstruction accuracy and computational cost. Specifically, our model integrates Dilated Convolutions and the Cross-Covariance Attention mechanism into its architecture to construct a lightweight generative network. This design effectively captures multi-scale information while significantly reducing model complexity. Additionally, we introduce an innovative loss function tailored to the geometric properties of normal maps. This loss function provides a more accurate measure of surface reconstruction quality and enhances the overall reconstruction performance. Experimental results show that, compared with existing methods, our approach reduces the number of training parameters by approximately 80% while maintaining the generated model's quality.
BibTeX
@inproceedings{liu2025lightweight,
  author = {Jiaxuan Liu and Jingyi Wu and Ruiyang Jing and Han Yu and Jing Liu and Liang Song},
  title = {{Lightweight Explicit 3D Human Digitization via Normal Integration}},
  booktitle = {Sensors},
  year = {2025},
  doi = {10.3390/s25051513},
}
Figure