Introduction
Gesture-to-gesture translation involves re-posing a source hand image to match a target gesture while preserving the hand's identity, including texture and skin tone. This task is critical for applications in sign language production and human-computer interaction. However, the human hand's high articulation and tendency for self-occlusion make this challenging. Previous approaches typically relied on sparse 2D joints (21 points) to guide translation. This sparse representation often fails to capture the full volume or surface of the hand, leading to blurry outputs, anatomical distortions, or "phantom" artifacts where finger identity is lost.
Methodology
To address the limitations of sparse representations, this paper introduces a model-aware framework that incorporates a 3D hand prior. The system bridges the gap between 2D joints and realistic pixels through three primary modules:
- Hand Representation Extraction: The model estimates parameters for the MANO model, a differentiable 3D statistical model of the human hand. Parameters are obtained via either fast "Direct CNN Regression" or a high-precision "Iterative Fitting" method.
- Topology Modeling: The 3D mesh surface is "unraveled" into a flattened 2D space. This creates a dense topology map where each pixel in the hand region corresponds to a specific location on the 3D mesh. By comparing source and target maps, the system calculates a precise transformation flow for accurate pixel warping.
- Gesture Synthesis: Structural information from the topology maps is integrated using Spatially-Adaptive Normalization (SPADE). An Attention Sampler adaptively extracts local characteristics from the source image based on the calculated flow, ensuring fine details like skin folds are preserved.
Experimental Results
The framework was evaluated on the Stereo Hand Tracking Benchmark (STB) and the Senz3D dataset. The transition from 21 sparse joints to a dense mesh (778 vertices and 1,538 faces) provided the necessary anatomical constraints to prevent distorted generations.
Quantitatively, the model achieved a 33.8% improvement in LPIPS on the STB dataset compared to the previous state-of-the-art (Global-Local), while also leading in SSIM and Inception Score. Qualitatively, the model produces striking visual improvements, preserving fine-grained details such as fingernails and palm textures that were previously lost. Furthermore, the use of a 3D model ensures smoother temporal transitions in video sequences, reducing the "flickering" common in 2D-joint-based methods.
Conclusion and Future Work
By leveraging 3D anatomical priors, the proposed framework significantly advances the state of gesture-to-gesture translation. While the results are high-fidelity and anatomically plausible, the iterative fitting process introduces computational overhead. Future work may focus on optimizing real-time mesh fitting and enhancing the attention mechanism to handle extreme self-occlusions more robustly.
Summary updated