Model-Aware Gesture-to-Gesture Translation

Hezhen Hu, Weilun Wang, Wengang Zhou, Weichao Zhao, Houqiang Li

Introduction

Gesture-to-gesture translation involves re-posing a source hand image to match a target gesture while preserving the hand's identity, including texture and skin tone. This task is critical for applications in sign language production and human-computer interaction. However, the human hand's high articulation and tendency for self-occlusion make this challenging. Previous approaches typically relied on sparse 2D joints (21 points) to guide translation. This sparse representation often fails to capture the full volume or surface of the hand, leading to blurry outputs, anatomical distortions, or "phantom" artifacts where finger identity is lost.

Methodology

To address the limitations of sparse representations, this paper introduces a model-aware framework that incorporates a 3D hand prior. The system bridges the gap between 2D joints and realistic pixels through three primary modules:

  1. Hand Representation Extraction: The model estimates parameters for the MANO model, a differentiable 3D statistical model of the human hand. Parameters are obtained via either fast "Direct CNN Regression" or a high-precision "Iterative Fitting" method.
  2. Topology Modeling: The 3D mesh surface is "unraveled" into a flattened 2D space. This creates a dense topology map where each pixel in the hand region corresponds to a specific location on the 3D mesh. By comparing source and target maps, the system calculates a precise transformation flow for accurate pixel warping.
  3. Gesture Synthesis: Structural information from the topology maps is integrated using Spatially-Adaptive Normalization (SPADE). An Attention Sampler adaptively extracts local characteristics from the source image based on the calculated flow, ensuring fine details like skin folds are preserved.
The proposed framework architecture, illustrating the transition from 3D hand representation to dense topology modeling and final gesture synthesis.

Experimental Results

The framework was evaluated on the Stereo Hand Tracking Benchmark (STB) and the Senz3D dataset. The transition from 21 sparse joints to a dense mesh (778 vertices and 1,538 faces) provided the necessary anatomical constraints to prevent distorted generations.

Quantitatively, the model achieved a 33.8% improvement in LPIPS on the STB dataset compared to the previous state-of-the-art (Global-Local), while also leading in SSIM and Inception Score. Qualitatively, the model produces striking visual improvements, preserving fine-grained details such as fingernails and palm textures that were previously lost. Furthermore, the use of a 3D model ensures smoother temporal transitions in video sequences, reducing the "flickering" common in 2D-joint-based methods.

Comparison of gesture translation results. The top row shows the limitations of sparse 2D skeleton-based methods, while the bottom row demonstrates the superior detail and anatomical correctness of the proposed 3D mesh-based approach.

Conclusion and Future Work

By leveraging 3D anatomical priors, the proposed framework significantly advances the state of gesture-to-gesture translation. While the results are high-fidelity and anatomically plausible, the iterative fitting process introduces computational overhead. Future work may focus on optimizing real-time mesh fitting and enhancing the attention mechanism to handle extreme self-occlusions more robustly.

Summary updated

Abstract

Hand gesture-to-gesture translation is a significant and interesting problem, which serves as a key role in many applications, such as sign language production. This task involves fine-grained structure understanding of the mapping between the source and target gestures. Current works follow a data-driven paradigm based on sparse 2D joint representation. However, given the insufficient representation capability of 2D joints, this paradigm easily leads to blurry generation results with incorrect structure. In this paper, we propose a novel model-aware gesture-to-gesture translation framework, which introduces hand prior with hand meshes as the intermediate representation. To take full advantage of the structured hand model, we first build a dense topology map aligning the image plane with the encoded embedding of the visible hand mesh. Then, a transformation flow is calculated based on the correspondence of the source and target topology map. During the generation stage, we inject the topology information into generation streams by modulating the activations in a spatially-adaptive manner. Further, we incorporate the source local characteristic to enhance the translated gesture image according to the transformation flow. Extensive experiments on two benchmark datasets have demonstrated that our method achieves new state-of-the-art performance.
BibTeX
@inproceedings{hu2021model,
  author = {Hezhen Hu and Weilun Wang and Wengang Zhou and Weichao Zhao and Houqiang Li},
  title = {{Model-Aware Gesture-to-Gesture Translation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  doi = {10.1109/cvpr46437.2021.01616},
}
Figure