3DHR-Co: A Collaborative Test-Time Refinement Framework for In-the-Wild 3D Human-Body Reconstruction Task

Jonathan Samuel Lumentut, Kyoung Mu Lee

TL;DR

This paper introduces 3DHR-Co, a novel collaborative test-time refinement framework that significantly improves 3D human body reconstruction (3DHR) performance on challenging "in-the-wild" datasets. By combining a pre-adaptation strategy with a regeneration-based bilevel refinement, 3DHR-Co enhances existing 3DHR models without requiring architectural changes, effectively addressing the limitations of 3D supervision in real-world scenarios.

What Problem Does This Paper Address?

3D human body reconstruction (3DHR) aims to estimate a person's 3D pose and shape from images or videos. While significant progress has been made, applying these techniques to "in-the-wild" data (real-world, diverse scenes) remains a challenge. The primary hurdle is the scarcity of accurate 3D ground truth data for training in these uncontrolled environments. Existing methods often rely on 2D pose keypoints to compensate for the lack of 3D supervision. However, the authors observe that using only 2D information during test-time refinement can lead to overfitting on common 3DHR backbones, making the refinement process intractable. This paper tackles this problem by proposing a collaborative approach that leverages multiple 3DHR models to refine predictions during inference, mitigating overfitting and improving robustness.

Methods

The core of the proposed method, 3DHR-Co, is a two-stage test-time refinement strategy:

  1. Pre-adaptation Stage: This stage aims to provide a "ready-to-adapt" weight for the 3DHR model. It involves using two types of intermediate data:
  2. Intermediate Label Data: Generated from the non-perturbed output of a pre-trained 3DHR model, often using temporal information from neighboring frames.
  3. Intermediate Perturbed Data: Generated from the predicted body and camera parameters, but corrupted with Gaussian noise. This simulates partial occlusions and variations common in in-the-wild data. The goal is to train a "learner" model (fs) to mimic the "teacher" model (fo) using these intermediate data, thereby learning to handle variations without overfitting to specific frames.

  4. Regeneration-based Bilevel Test-Time Refinement: This stage further refines the pre-adapted model (fs) to produce the final refined output (fa). It employs a bilevel adaptation strategy (similar to prior work like DynaBOA) but with a regeneration mechanism. This mechanism re-initializes the model with the pre-adapted weights (fs) when a new sequence is encountered. This prevents the model from overfitting to a particular frame or sequence by ensuring it always starts from a robust, pre-adapted state that has already learned to handle variations.

The framework is designed to be plug-and-play, allowing existing 3DHR backbones (like SPIN or RSN) to be directly integrated and refined.

Datasets

The paper primarily uses the 3DPW (3D Poses in the Wild) dataset for evaluation. This dataset represents real-world scenarios with diverse poses and environments, making it a challenging benchmark for in-the-wild 3DHR. The authors also mention using Human3.6M for pre-training in prior works, which provides 3D ground truth but is more controlled.

Key Insights and Ideas

  • Overfitting is a major bottleneck: The paper highlights that simply using 2D pose information for test-time refinement on in-the-wild data can lead to overfitting, hindering performance.
  • Collaboration is key: By making different 3DHR models "collaborate" (one acting as a teacher, another as a learner), the framework can effectively transfer knowledge and improve robustness.
  • Perturbation as a training signal: Introducing noise (Gaussian corruption) to simulate real-world occlusions during pre-adaptation is crucial for teaching the model to handle variations.
  • Pre-adaptation is essential: Providing a pre-adapted model weight before the main refinement step helps avoid overfitting and preserves learned knowledge from the initial training.
  • Regeneration prevents catastrophic forgetting: Re-initializing with pre-adapted weights during refinement ensures the model doesn't overfit to specific test sequences.
  • Unlocking potential of existing models: The approach demonstrates that even common, classic 3DHR backbones can achieve state-of-the-art performance on in-the-wild data when enhanced with this collaborative refinement strategy.

Key Results

  • Significant error reduction: 3DHR-Co achieved up to -34.53 mm pose error suppression on the 3DPW benchmark when applied to classic 3DHR backbones like SPIN.
  • Improved performance on SPIN: For the SPIN backbone, the MPJPE score on 3DPW was improved from an initial 98.25 mm to 63.72 mm (with MPN as intermediate label extractor) and 63.92 mm (with TCMR).
  • Improved performance on RSN: For the RSN backbone, the MPJPE score on 3DPW was improved from an initial 98.51 mm to 64.83 mm (with MPN) and 64.82 mm (with TCMR).
  • Qualitative improvements: The method showed better handling of occluded body parts and more plausible pose estimations compared to existing methods like DynaBOA, especially on internet-sourced data.
  • Efficiency: The pre-adaptation stage is computationally efficient, taking approximately 0.15 seconds per batch (8 frames).

Limitations and Future Work

  • Dataset dependency: The performance of 3DHR-Co is dependent on the intermediate data extraction strategy. When tested on non-in-the-wild datasets like MPI and Human3.6M, the refinement performance was sub-optimal (only around -2 mm improvement). This is attributed to the homogeneous nature of these datasets, which are already well-fitted during training.
  • Noise sensitivity: While noise is beneficial, very high noise levels (σ₃) can sometimes lead to sub-optimal results, suggesting that lower noise values are preferable for datasets like 3DPW.
  • Future work: The authors suggest future studies should consider how to optimally perform 3DHR refinement across various dataset domains, particularly those with static backgrounds and homogeneous scenes.

Summary updated

Abstract

The task of 3D human-body reconstruction (3DHR), which mostly utilizes parametric pose and shape representations, has witnessed significant advances in recent years. However, the application of 3DHR techniques in handling real-world in-the-wild data, still faces limitations. Training the 3DHR model in such scenario with 3D human pose’s ground truth (GT) is non-trivial. Curating the accurate 3D human pose GT for in-the-wild scenes remains difficult due to various factors. Recent test-time refinement approaches on 3DHR task leverage 2D off-the-shelf human keypoints information to support the lack of 3D supervision on in-the-wild data. However, we observed that additional 2D supervision alone could cause overfitting issue on common 3DHR backbones, making the 3DHR test-time refinement task seem intractable. We answer this challenge by proposing a strategy that complements 3DHR test-time refinement work under a collaborative approach. Specifically, we initially apply a pre-adaptation approach that works by collaborating various 3DHR models in a single framework to directly improve their initial outputs. This approach is then further combined with the test-time adaptation work under specific settings that minimize the overfitting issue to further boost the 3DHR performance. The whole framework is termed as 3DHR-Co, and on the experiment side, the proposed work can significantly enhance the scores of common classic 3DHR backbones up to -34 mm pose error suppression, putting them among the top list on the in-the-wild benchmark data. Such achievement shows that our approach helps unveil the true potential of the common classic 3DHR backbones. Based on these findings, we further investigate various settings on the proposed framework to better elaborate the capability of our collaborative strategy in the 3DHR task.
BibTeX
@inproceedings{lumentut2023dhr,
  author = {Jonathan Samuel Lumentut and Kyoung Mu Lee},
  title = {{3DHR-Co: A Collaborative Test-Time Refinement Framework for In-the-Wild 3D Human-Body Reconstruction Task}},
  booktitle = {IEEE Access},
  year = {2023},
  url = {https://openreview.net/forum?id=BMXyvVe3fQ},
}
Figure