LISA: Learning Implicit Shape and Appearance of Hands

Enric Corona, Tomas Hodan, Minh Vo, Francesc Moreno-Noguer, Chris Sweeney, Richard Newcombe

Abstract

Modeling human hands is a fundamental challenge in computer vision due to high articulation, frequent self-occlusion, and diverse appearances. Traditional parametric mesh models like MANO suffer from low resolution and a lack of native texture representation. We present LISA (Learning Implicit Shape and Appearance), a novel neural model that represents human hands using an articulated implicit representation. By combining Signed Distance Fields (SDFs) with a kinematic skeleton and a learned skinning mechanism, LISA captures high-fidelity geometry and appearance. Our model enables realistic animation and achieves sub-millimeter reconstruction accuracy, significantly surpassing existing mesh-based and implicit baselines.

Introduction

For years, the industry standard for hand modeling has been MANO, a parametric mesh model. While MANO provides a useful framework for hand pose estimation, it possesses two major limitations: it uses a fixed, coarse mesh that cannot capture fine geometric details, and it lacks a native method for representing color and texture. Recent implicit models, such as Neural Radiance Fields (NeRFs) or DeepSDF, offer high detail but are typically designed for static objects. Adapting these to the highly mobile and articulated structure of a hand—where the surface changes dynamically based on the skeleton's pose—is a complex mathematical task.

LISA bridges this gap by merging skeletal animation with implicit neural representations. It provides a versatile framework that can be fitted to 3D point clouds or reconstructed from single RGB images while maintaining a fully animatable structure.

Methods

LISA’s architecture is built upon three primary pillars designed to handle the complexities of hand geometry and movement.

Articulated Implicit Representation

Instead of a discrete mesh, LISA defines the hand through a Signed Distance Field (SDF). For any given 3D point, the model predicts its distance to the surface and its corresponding color. This allows for arbitrary resolution and the capture of individual-specific traits that mesh models often smooth over.

Per-Bone Decomposition and Neural Skinning

To manage articulation, LISA decomposes the hand into its constituent parts. The model utilizes $n_b$ independent Multi-Layer Perceptrons (MLPs), one for each bone in the kinematic skeleton. Each MLP predicts the SDF and color within that bone's local coordinate frame.

To merge these per-bone predictions into a cohesive hand surface, LISA employs a "Skinning MLP." This network predicts weights that blend the outputs of the individual bone MLPs. Unlike traditional Linear Blend Skinning (LBS) used in computer graphics, this is a learned process operating in continuous space. A key innovation here is the "SDF as a Guide" trick: the skinning weight MLP receives predicted per-bone SDF values as input. This provides a strong geometric hint; if a bone's SDF indicates a point is far away, the skinning weight for that bone is naturally driven toward zero.

Disentangled Latent Spaces

The model effectively separates Pose (joint angles), Shape (the identity and proportions of the hand), and Appearance (color and texture) into distinct latent vectors. This disentanglement allows for powerful applications, such as retargeting the specific shape and texture of one individual onto the motion and poses of another.

Training and Datasets

Training implicit models solely on 2D images often results in "hollow" or physically inaccurate 3D geometry. To overcome this, LISA was trained using a combination of multi-view RGB data and high-quality 3D scans:

  • InterHand2.6M: A massive multi-view RGB dataset used to learn appearance and shape from image data.
  • 3DH: A dataset comprising approximately 13,000 high-quality 3D hand scans. This was essential for training a geometric prior, ensuring the model understands 3D structure even when refined with 2D image data.
  • MANO & DeepHandMesh: These served as baselines for evaluation to demonstrate LISA's superior reconstruction capabilities.

Results and Key Insights

LISA sets a new state-of-the-art in hand modeling across several metrics:

  • Sub-Millimeter Accuracy: In 3D point cloud fitting tasks, LISA achieved a vertex-to-surface (V2S) error of 0.93mm. This significantly outperforms previous implicit models such as NASA (1.14mm) and NARF (2.19mm).
  • Generalization: The model maintains high accuracy even when tested on "unseen" hands from the MANO dataset, proving that it learns a robust representation rather than simply memorizing training samples.
  • Qualitative Improvements: Visually, LISA captures realistic palm lines, finger shapes, and skin textures that traditional mesh-based models cannot represent.

Limitations and Future Work

While LISA represents a significant leap forward, several challenges remain. The current reconstruction process involves an optimization loop that takes approximately five minutes on an Nvidia P100 GPU, which is insufficient for real-time VR/AR applications. Additionally, the model requires a rough initial pose and a foreground mask to function effectively. Finally, while LISA captures significantly more detail than MANO, extremely fine high-frequency details like skin pores or sharp wrinkles remain difficult to represent, suggesting the need for even higher-capacity networks or hybrid representations in the future.

Summary updated

Abstract

This paper proposes a do-it-all neural model of human hands, named LISA. The model can capture accurate hand shape and appearance, generalize to arbitrary hand subjects, provide dense surface correspondences, be reconstructed from images in the wild, and can be easily animated. We train LISA by minimizing the shape and appearance losses on a large set of multi-view RGB image sequences annotated with coarse 3D poses of the hand skeleton. For a 3D point in the local hand coordinates, our model predicts the color and the signed distance with respect to each hand bone independently, and then combines the perbone predictions using the predicted skinning weights. The shape, color, and pose representations are disentangled by design, enabling fine control of the selected hand parameters. We experimentally demonstrate that LISA can accurately reconstruct a dynamic hand from monocular or multi-view sequences, achieving a noticeably higher quality of reconstructed hand shapes compared to baseline approaches. Project page: https:// www.iri.upc.edu/ people/ecorona/lisa/.
BibTeX
@misc{corona2022lisa,
  author = {Enric Corona and Tomas Hodan and Minh Vo and Francesc Moreno-Noguer and Chris Sweeney and Richard Newcombe},
  title = {{LISA: Learning Implicit Shape and Appearance of Hands}},
  year = {2022},
  eprint = {2204.01695},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2204.01695},
}
Figure