TL;DR
This paper introduces EMO-X, a novel one-stage method for efficiently estimating expressive human pose, hand gestures, and facial expressions from single images. EMO-X leverages the Mamba architecture for efficient global context modeling and introduces a new Scan-based Global-Local Decoder with a Local Bidirectional Scan mechanism to capture fine-grained local details, achieving state-of-the-art efficiency with competitive accuracy.
What Problem Does This Paper Address?
Estimating expressive human pose and shape (EHPS) from monocular images is crucial for applications like virtual reality and human-computer interaction. Existing methods often rely on multi-stage pipelines or Transformer-based architectures. Multi-stage approaches can lead to inconsistencies between different body parts, while Transformer-based methods, despite their success in capturing long-range dependencies, suffer from quadratic computational complexity in their self-attention mechanisms. This makes them inefficient, especially for multi-person scenarios where the number of tokens increases significantly. While recent advancements like Mamba offer linear complexity for global modeling, they often struggle with capturing essential local details for precise EHPS. EMO-X aims to bridge this gap by combining the efficiency of Mamba with a specialized mechanism for local refinement, addressing the trade-off between accuracy and computational cost in multi-person EHPS.
Methods
EMO-X employs an encoder-decoder architecture. The input image is first processed into human tokens. These tokens are then fed into a novel Scan-based Global-Local Decoder (SGLD). The SGLD is designed to progressively refine human tokens by integrating global context and local details.
Key components of the SGLD include:
- Global Mamba Block: Utilizes Mamba's efficient, input-dependent selective scan mechanism to model global context and capture long-range dependencies across human tokens. This is crucial for understanding inter-person relationships in multi-person scenes.
- Cross-Attention: Integrates image features with the refined human tokens to associate them with their corresponding spatial locations in the image.
- Local Mamba Block: This is where the core innovation for local refinement lies. It employs a Local Bidirectional Scan mechanism. This mechanism goes beyond simple sequential scanning by considering human tokens separately and performing a skeleton-aware scan that follows a specific joint topology. This allows for more precise modeling of human structural details, particularly for hands and faces.
The model also incorporates anchor boxes as explicit positional priors, which are continuously updated and guide the Mamba blocks, enhancing robustness in multi-person tasks.
Datasets
The paper uses the following datasets:
- Training: AGORA, BEDLAM, COCO, and UBody (for single-person datasets).
- Evaluation: AGORA, EHF, UBody, and BEDLAM.
The authors note that the EHF dataset was not included in their training set, highlighting the model's generalization capabilities.
Key Insights and Ideas
- Mamba for Global Context, Specialized Scan for Local Details: The paper successfully demonstrates that combining Mamba's efficient global modeling with a novel, skeleton-aware local scanning mechanism is key to achieving both high accuracy and efficiency in complex EHPS tasks.
- Skeleton-Aware Local Scanning is Crucial: Simply using bidirectional scanning (as in some prior vision Mamba works) is insufficient for detailed EHPS. The proposed Local Bidirectional Scan, which leverages joint topology, significantly improves the capture of fine-grained local features like hands and faces.
- Anchor Boxes as Positional Priors: The use of continuously updated anchor boxes as positional priors within the Mamba blocks proves vital for improving robustness and performance in multi-person scenarios.
- One-Stage Efficiency: By adopting a one-stage approach and leveraging efficient Mamba blocks, EMO-X drastically reduces inference time compared to existing methods, making real-time applications more feasible.
Key Results
- Efficiency: EMO-X achieves a significant reduction in inference time, requiring 69.8% less time compared to the state-of-the-art AiOS method on the AGORA dataset (124.8 ms vs. 413.2 ms).
- Accuracy: The model achieves competitive accuracy, matching the Mean Vertex Error (MVE) of AiOS (91.9 mm) while significantly reducing inference time. On the AGORA testing set, EMO-X outperforms most existing methods in terms of NMVE and NMJE, and achieves SOTA performance for pure reconstruction quality of faces and hands.
- Qualitative Results: Visualizations on AGORA and in-the-wild images demonstrate excellent reconstruction quality, particularly in challenging areas like hands and faces, where red circles in the figures highlight areas where other methods were inaccurate.
Limitations and Future Work
- Human-Human Interaction: The paper acknowledges that EMO-X does not explicitly consider human-human interaction. This limitation means the model might not generalize well to scenarios with severe occlusion or complex interaction between multiple individuals.
- Future Directions: While not explicitly detailed, the success of EMO-X suggests future work could focus on incorporating explicit interaction modeling or extending the local bidirectional scan to handle more complex scene dynamics.
Summary updated