Introduction
Creating realistic virtual humans requires high-fidelity hand motion capture (MoCap). However, researchers often face a "synchronous acquisition" dilemma: accurate tracking typically requires physical markers or gloves that degrade the hand's visual appearance. Conversely, markerless capture preserves appearance but often lacks tracking precision.
This paper addresses the Hand Appearance Recovery task—translating degraded images (containing markers, paint, or occlusions) into clean, bare-hand images. This is challenging due to the lack of paired data (the same pose with and without markers) and the fact that markers significantly alter the hand's silhouette and geometry, making standard image-to-image translation models like CycleGAN prone to failure.
Methodology: A Two-Stage Pipeline
The authors propose a framework that first disentangles the hand's underlying structure and then "wraps" the original appearance back onto a cleaned version.
1. Structure Disentanglement (The "Sketcher")
Instead of direct pixel translation, the model extracts a "bare hand structure" represented as a pixel-aligned normal map.
- ViT Sketcher: A Vision Transformer (ViT) is employed to model the hand structure. ViTs utilize global context, allowing them to "see through" local degradations like markers better than standard CNNs.
- Fixed-Point Property: The sketcher is trained to ensure stability; feeding its output back into itself yields the same result ($S[S(X)] = S(X)$), making the extracted structure domain-agnostic.
2. Appearance Wrapping (The "Translator")
Once a clean "sketch" is obtained, the model paints the original skin tones and textures back onto it.
- Dual Adversarial Discrimination (DAD): This novel scheme combines unsupervised result-based discrimination with supervised process-based guidance.
- The Partner Domain: To bridge the gap in unpaired real data, the authors synthetically add markers to bare hand images. This "partner domain" provides trustworthy pairs that guide the model on how to wrap appearance while ignoring structural degradations.
Datasets and Robustness
The framework was trained and tested on a diverse array of datasets to ensure real-world applicability:
- Degraded Domains: FPHAB (inertia markers), MHP (gloves), PaintedHand, and object-occluded images from Freihand and HO3D.
- Bare Hand Domains: OneHand10k, DMTL, and InterHand2.6M.
- Structure Training: A rendered dataset of 100k diverse hand poses was used to establish a strong prior for what a "clean" hand structure looks like.
Key Results and Insights
The proposed method demonstrates significant improvements over unsupervised baselines like CycleGAN and CUT.
- Quantitative Performance: The model achieved superior scores in FID (Frechet Inception Distance) and KID (Kernel Inception Distance), indicating higher image quality and better alignment with the distribution of real bare hands.
- Qualitative Recovery: As shown in the comparisons, the model successfully removes bulky MoCap markers and even "erases" occluding objects like pens, replacing them with plausible skin textures while maintaining the original pose.
- Downstream Utility: Beyond aesthetics, restoring hand appearance improved the performance of off-the-shelf pose estimators (e.g., OpenPose), proving the functional value of the recovery process for MoCap workflows.
Limitations and Future Work
While robust, the model can struggle with extreme occlusions where the underlying pose is entirely ambiguous. Additionally, the CNN-based translator is not perfectly rotation-invariant, occasionally producing artifacts for hands at highly unusual angles. Future work aims to extend this framework to multi-hand interactions and full-body appearance recovery.
Summary updated