Introduction
Capturing the 3D motion of a single individual is a largely solved problem in computer vision. However, when humans engage in close physical contact—such as hugging, dancing, or wrestling—traditional systems often fail. These "messy" scenarios involve significant occlusion and complex overlaps that baffle standard 2D joint detectors. AvatarPose addresses these challenges by reconstructing personalized, high-fidelity neural avatars for each participant, using them as geometric priors to optimize poses based on direct pixel-level consistency.
The Challenge of Close Interaction
In scenarios involving close human interaction (CHI), 2D joint detectors—the backbone of most pose estimation systems—frequently assign joints to the wrong person or fail to detect them entirely due to occlusion. Existing multi-view methods typically attempt to triangulate these noisy 2D points into 3D space, which often results in jittery motion or physically impossible configurations, such as limbs passing through another person's torso.
While some researchers have turned to the SMPL body model as a guide, these generic "naked" mesh models lack the specific clothing and shape details necessary to accurately match the actual pixels in a video. Without accounting for the volume of clothing, collision constraints remain imprecise.
The AvatarPose Framework
AvatarPose shifts the strategy from "matching points" to "matching volumes." The pipeline consists of two primary modules operating in an alternating optimization loop.
1. Multi-Avatar Prior Learning
The system first creates a personalized "digital twin" for each individual. This is achieved using an efficient Neural Radiance Field (NeRF) variant, specifically Instant NGP, combined with a SMPL-based deformation module. To manage multiple people simultaneously, the framework employs layered volume rendering. This allows the system to render each person into a separate layer before compositing them, maintaining distinct identities even during dense physical overlap.
2. Avatar-guided Pose Optimization
Once the avatars are learned, they serve as a powerful geometric prior. Instead of minimizing the distance between sparse, error-prone detected joints, AvatarPose optimizes the pose by minimizing the photometric and silhouette difference between the rendered avatar and the actual video frames. This "pixels over points" approach provides a much denser signal, utilizing the visible texture of clothing or the contours of a person’s back to guide the fit.
3. Implicit Collision Handling
Because the avatars are represented as 3D density fields, the authors can calculate a precise "collision loss." If high-density regions of two different avatars occupy the same 3D space, the system applies a penalty. This effectively pushes the bodies apart to prevent interpenetration. Unlike mesh-to-mesh distance calculations, which are computationally expensive, using the NeRF density field is efficient and more representative of the actual clothed body shape.
Experimental Evaluation
The researchers evaluated AvatarPose on several challenging benchmarks:
- Hi4D & CHI3D: Primary benchmarks for close human interaction (e.g., hugging, jumping).
- Shelf & MultiHuman: Used to demonstrate scalability to scenarios with three or four individuals.
- SAM-Track: Utilized to generate the foreground masks (silhouettes) required for the optimization process.
Results and Key Insights
AvatarPose achieved state-of-the-art (SotA) performance across multiple metrics:
- Accuracy: On the Hi4D dataset, it achieved a Mean Per Joint Position Error (MPJPE) of 32.10mm, a significant improvement over the previous best method, 4DAssociation (41.29mm).
- Robustness: The system reached a 96.9% PCP3D (Percentage of Correct Parts), demonstrating superior reliability in identifying limb positions during complex overlaps compared to joint-based methods like MVPose (~90%).
- Physical Plausibility: Qualitative results indicate that AvatarPose successfully prevents "ghosting" and interpenetration, producing physically plausible interactions where previous methods showed limbs clipping through bodies.
The "secret sauce" of the method lies in the alternating optimization: better poses lead to cleaner avatar reconstructions, and cleaner avatars lead to more precise pose refinements. By personalizing the model to the specific clothing of the subjects, the system gains a significant advantage over generic body models.
Limitations and Future Work
Despite its success, AvatarPose faces several challenges:
- Local Minima: If the initial pose estimate is significantly incorrect (e.g., the subject is facing the wrong direction), the optimization can get stuck and fail to recover.
- Hand Modeling: The current iteration does not model intricate hand and finger movements, which are essential for fine-grained interactions like grasping.
- Computational Intensity: While Instant NGP is fast, the alternating optimization loop is more resource-intensive than simple bottom-up joint detection, making real-time performance a goal for future iterations.
Conclusion
AvatarPose represents a significant step forward in 3D pose estimation for complex human interactions. By leveraging neural avatars as dynamic, personalized priors, the system moves beyond the limitations of sparse joint detection, offering a robust solution for capturing the nuances of human physical contact.
Summary updated