TL;DR
VACO (Vertex-Aligned COntact) is a novel framework for estimating dense hand contact from a single RGB image by grounding each mesh vertex in local image features. By replacing global "shared-query" bottlenecks with vertex-specific image sampling, VACO eliminates "geometric bleeding" and sets a new state-of-the-art across five major benchmarks, including hand-object and hand-human interactions.
What Problem Does This Paper Address?
Understanding exactly where a hand touches an object or another person is critical for robotics (dexterous manipulation) and AR/VR (realistic digital interaction). However, inferring this from a single 2D image is notoriously difficult due to occlusions and the subtle visual cues of contact.
Existing methods generally fall into two camps:
- Reconstruction-based: These models first reconstruct the 3D hand and the object, then calculate contact based on geometric proximity. This is "brittle" because a small error in object reconstruction leads to completely wrong contact estimates.
- Direct Prediction: These models regress contact labels directly from pixels. While faster, most current models use a "shared-query" approach where a single global token predicts contact for the entire hand. This lacks spatial grounding, often causing "geometric bleeding"—where contact "smears" from a touching fingertip onto adjacent, non-touching fingers.
VACO addresses this by ensuring every vertex on the hand mesh "looks" at its own corresponding spot in the image to make a contact decision.
Methods
The core innovation of VACO is Vertex-Aligned Decoding. Instead of treating the hand mesh as a flat list of points, VACO uses the mesh as a "spatial scaffold."
- The Backbone: The system uses a co-trained Vision Transformer (ViT) backbone (based on WiLoR) that simultaneously reconstructs the 3D hand mesh and extracts image features.
- Feature Sampling: For each of the 778 vertices in the MANO hand model, the system calculates its 2D projection onto the image. It then samples multi-scale features from the image at that specific location.
- Transformer Decoder: These vertex-specific features are fed into a Transformer decoder. Self-attention allows the model to learn structural correlations (e.g., if four fingers are closed in a power grasp, they are likely all in contact), while cross-attention lets vertices look at the broader scene context (like the object being held).
- Robust Training: To handle imperfect hand pose estimation at inference, the model is trained using "projection noise"—intentionally jittering the ground-truth projections so the decoder learns to be robust to slight misalignments.
Datasets
VACO was trained on a massive, unified pool of 11 datasets standardized by the recent HACO framework, totaling roughly 0.6 million training frames. These include:
- Hand-Object: ARCTIC, DexYCB, HO3D, H2O3D, HOI4D, ObMan.
- Hand-Human/Bimanual: Hi4D, InterHand2.6M.
- Hand-Scene/Face: RICH, Decaf.
The model was evaluated on in-domain splits (DexYCB, Hi4D, RICH) and held-out datasets (HOT3D, HOCap) to test generalization to unseen camera views and interaction types.
Key Insights and Ideas
- Grounding is the "Missing Ingredient": The authors argue that human-like reasoning for contact requires looking at the specific "footprint" of a vertex. By anchoring contact decisions to local pixels, they drastically reduce false positives on neighboring fingers.
- The Mesh as a Scaffold: Rather than using 3D geometry to calculate proximity (which is error-prone), VACO uses geometry simply to tell the network where to look in the 2D image.
- End-to-End Synergy: Jointly training for reconstruction and contact allows the image encoder to adapt its features to pick up on hyper-local cues like skin indentations or occlusion edges that signify pressure.
Key Results
VACO significantly outperforms previous state-of-the-art (SOTA) methods:
- Massive Gains in Complex Scenes: On the Hi4D dataset (close hand-human interaction), VACO improved the F1-score by +20.7 percentage points over the previous best direct predictor (HACO).
- Generalization: Even on held-out datasets like HOT3D and HOCap, VACO saw gains of +5.5 and +2.7 pp, proving it doesn't just memorize specific dataset biases.
- Qualitative Precision: As shown in Figure 3, VACO produces much sharper contact maps, correctly identifying when only a fingertip is touching, whereas previous methods "smeared" contact across the palm or adjacent fingers.
- Downstream Impact: When used as a prior for 3D Hand-Object Interaction (HOI) reconstruction, VACO nearly halved the alignment error (Chamfer distance), leading to much more physically plausible grasps.
Limitations and Future Work
- Scaffold Dependence: The model's accuracy is ultimately bounded by the quality of the predicted hand mesh. If the pose estimator fails completely, the "scaffold" points to the wrong pixels.
- 2D Ambiguity: Because it relies on 2D projections, vertices that overlap in the image (e.g., the palm and the back of the hand) sample the same features. The model currently relies on learned embeddings to distinguish these, but explicit depth or visibility cues could help.
- Temporal Consistency: The current model processes images frame-by-frame. Incorporating temporal reasoning could further stabilize predictions in video sequences.
Summary updated