EventHPE: Event-based 3D Human Pose Estimation

Shihao Zou, chuan guo, Xinxin Zuo, Sen Wang, Minglun Gong, Li ChengOpenReview 2022 · arXiv · OpenReview

TL;DR

This paper introduces EventHPE, a novel two-stage deep learning framework for estimating 3D human pose and shape using event camera data. By leveraging inferred optical flow from events and introducing a new flow coherence loss, EventHPE significantly improves accuracy and reduces reliance on traditional image sequences, while also releasing a large, multi-modal dataset for future research.

What Problem Does This Paper Address?

Estimating 3D human pose and shape from visual data is a long-standing challenge in computer vision, typically relying on RGB or RGB-D cameras. Event cameras, inspired by biological vision, offer a new paradigm by asynchronously capturing brightness changes, excelling at detecting motion. However, their sparse and motion-centric nature presents unique challenges for pose and shape estimation, which traditionally focuses on static postures. Existing event-based methods either only estimate 2D poses or still require a continuous stream of traditional images alongside events, limiting their potential. This paper aims to bridge this gap by developing a method that primarily uses event data for accurate 3D human pose and shape estimation.

Methods

EventHPE employs a two-stage approach:

  1. FlowNet (Unsupervised Optical Flow Inference): This stage takes event frames (aggregated from event packets) as input and uses a CNN-based encoder-decoder architecture to predict optical flow. Crucially, this stage is trained using unsupervised learning with a combination of photometric and smoothness losses, making it robust to the sparse nature of event data. This optical flow provides explicit geometric motion cues.
  2. ShapeNet (3D Pose and Shape Estimation): This stage takes a sequence of event frames and their corresponding predicted optical flows as input. A CNN extracts feature representations, which are then fed into an RNN (specifically, a GRU) to predict pose variations and global translations over time. A key novelty here is the introduction of a flow coherence loss. This loss enforces consistency between the image-based optical flow (from FlowNet) and the shape-based flow (movement of human body shape vertices), leveraging the fact that both originate from the same underlying human motion. This helps to regularize the motion estimation and mitigate discrepancies. The method assumes the initial pose and shape are known or can be extracted from the first grayscale image.

Datasets

  • DHP19 dataset: Used for evaluating the method's performance on existing event-based pose estimation data.
  • MMHPSD (Multi-Modality Human Pose and Shape Dataset): A newly curated in-house dataset. It is described as the largest event-based 3D human pose and shape dataset to date, featuring 240k frames with multi-modal data including event cameras, polarization cameras, and RGB-D cameras. This dataset is publicly released to facilitate future research.

Key Insights and Ideas

  • Optical Flow as a Bridge: The core idea is that optical flow, inferred from events, acts as a crucial intermediate representation. It bridges the gap between the sparse, motion-focused event data and the need for geometric information required for pose and shape estimation.
  • Flow Coherence Loss: This novel loss function is a significant contribution. By enforcing consistency between image-based flow and shape-based flow, it effectively regularizes the estimation process and improves accuracy, especially when dealing with dynamic human motion.
  • Reduced Reliance on Image Sequences: By effectively using event data and inferred optical flow, EventHPE significantly reduces the need for continuous grayscale image sequences, a limitation of prior event-based methods like EventCap. This makes it more suitable for scenarios where traditional cameras are unavailable or unreliable.
  • Dataset Importance: The creation and release of the MMHPSD dataset are vital. The lack of publicly available, comprehensive event-based 3D human datasets has been a bottleneck, and MMHPSD aims to address this by providing rich, multi-modal annotations.

Key Results

  • Improved Accuracy: EventHPE demonstrates significant improvements over existing methods on both the DHP19 and MMHPSD datasets. For instance, on DHP19, it achieves a reduction of over 20mm in PA-MPJPE and PEL-MPJPE compared to DHP19.
  • Robustness to Initial Estimates: Qualitative results show that EventHPE can effectively rectify inaccurate initial pose and shape estimates, a capability that is less pronounced in methods like EventCap.
  • Ablation Studies: Ablation studies highlight the importance of optical flow and the flow coherence loss. Removing optical flow or the geodesic distance loss for pose representation leads to substantial increases in joint and shape errors.

Limitations and Future Work

  • Initial Pose/Shape Requirement: A current limitation is the need for the beginning pose and shape to be provided or detected from the first grayscale image.
  • Future Work: The authors plan to address the limitation of requiring initial pose/shape information by developing methods to infer 3D shapes solely from event signals.

Summary updated

Abstract

Event camera is an emerging imaging sensor for capturing dynamics of moving objects as events, which motivates our work in estimating 3D human pose and shape from the event signals. Events, on the other hand, have their unique challenges: rather than capturing static body postures, the event signals are best at capturing local motions. This leads us to propose a two-stage deep learning approach, called EventHPE. The first-stage, FlowNet, is trained by unsupervised learning to infer optical flow from events. Both events and optical flow are closely related to human body dynamics, which are fed as input to the ShapeNet in the second stage, to estimate 3D human shapes. To mitigate the discrepancy between image-based flow (optical flow) and shape-based flow (vertices movement of human body shape), a novel flow coherence loss is introduced by exploiting the fact that both flows are originated from the identical human motion. An in-house event-based 3D human dataset is curated that comes with 3D pose and shape annotations, which is by far the largest one to our knowledge. Empirical evaluations on DHP19 dataset and our in-house dataset demonstrate the effectiveness of our approach.
BibTeX
@misc{zou2022eventhpe,
  author = {Shihao Zou and chuan guo and Xinxin Zuo and Sen Wang and Minglun Gong and Li Cheng},
  title = {{EventHPE: Event-based 3D Human Pose Estimation}},
  year = {2022},
  eprint = {2108.06819},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2108.06819},
  howpublished = {OpenReview},
}
Figure