Embodied Human Activity Recognition

Sha Hu, Yu Gong, Greg MoriWACV 2024

TL;DR

This paper introduces "Embodied Human Activity Recognition" (EHAR), a new task where an agent must intelligently move in a 3D environment to recognize human activities. They propose a reinforcement learning approach that learns a navigation policy, demonstrating that active movement significantly improves activity recognition compared to passive observation.

What Problem Does This Paper Address?

Recognizing human activities from video is a well-studied problem. However, traditional approaches often rely on static cameras with pre-determined viewpoints, which are limited in their ability to adapt to dynamic environments or capture subtle human movements. This paper addresses the challenge of recognizing human activities in 3D environments when the observer (an embodied agent, like a robot or drone) can actively move. The core problem is that initial egocentric observations might be unconstrained and insufficient for accurate recognition. The agent needs to strategically move to find better viewpoints to understand ongoing human actions, especially when those actions evolve over time and require specific perspectives (e.g., observing gait from the front rather than from above).

Methods

The authors propose a novel task called Embodied Human Activity Recognition (EHAR) and tackle it using a deep reinforcement learning framework. The agent's architecture consists of two main components:

  • Activity Recognition Network: This network (based on CTR-GCN) processes egocentric human skeleton data to predict the ongoing activity. It learns to extract features and maintain an internal "recognition state" representing its accumulated understanding.
  • Policy Network: This network learns to control the agent's movement. It takes inputs from three encoders:
  • Accumulated Recognition State Encoder: Encodes the internal state of the activity recognition network.
  • Visual Encoder: Processes the current egocentric visual observation.
  • Position Encoder: Provides spatial context about the agent's location.

These encoders inform a fused state that is fed into an actor-critic network to output movement actions. The agent's action space is discretized into 7 actions (move forward/backward, rotate left/right, tilt up/down, and hover). The training is staged: first, the recognition network is trained on static data, then it's frozen, and the policy network is trained using reinforcement learning. They use a combination of Imitation Learning (IL) and Proximal Policy Optimization (PPO) for policy training.

Datasets

The paper uses two datasets for training and evaluation:

  • ExPI (Extreme Pose Interaction): Contains motion capture data of dancers performing various collective activities.
  • AIST++: Contains motion capture data of single-person activities.

For EHAR, they construct episodes by combining human scenarios with agent starting positions. They also create a "visual pre-train dataset" by sampling viewpoints from a uniform grid to train the activity recognition network.

Key Insights and Ideas

  • Active movement is crucial for embodied activity recognition: The paper strongly suggests that an agent's ability to move and change its viewpoint is essential for accurately recognizing human activities, especially in complex 3D environments.
  • EHAR is a challenging "hard exploration" problem: The agent needs to learn not just to recognize activities but also to discover optimal viewpoints, which requires effective exploration strategies.
  • Imitation Learning (IL) aids exploration: Pre-training the policy network with IL on oracle demonstrations significantly improves performance and stability compared to training with PPO alone, likely by providing a good initial exploration strategy.
  • Synergy between perception and action: The learned state representations are shown to correlate with activity classes, indicating that the agent learns to move based on the semantic understanding of the activity.

Key Results

  • Significant performance gains: The proposed embodied agent consistently outperforms passive (No-Act) and heuristic baselines (Random, Towards, H-Rotate, V-Rotate) across various observation ratios on both ExPI and AIST++ datasets. For instance, on ExPI, their "OURS" model achieves an accuracy of 74.96% at 90% observation ratio, significantly higher than the passive agent's 61.48%.
  • Improved early recognition: The agent demonstrates better accuracy at lower observation ratios, indicating its ability to recognize activities earlier.
  • Effective movement strategies: Qualitative analysis shows the agent explores a wide range of positions and tends to adjust its elevation to match human levels, suggesting intelligent navigation.

Limitations and Future Work

  • Simulated environments: The experiments are conducted in simulated environments using motion capture data. The authors acknowledge that real-world deployment with complex scenes, obstacles, and realistic human interactions is a direction for future work.
  • End-to-end training: The current approach uses staged training (recognition network first, then policy). They suggest that end-to-end training could potentially lead to better synergy between perception and action.
  • Limited action space: The action space is discretized and limited to 3 degrees of freedom. Continuous action spaces could offer finer control.
  • Predictability variations: The paper notes that the predictability of human activities can differ between passive and embodied perception settings, with some activities remaining challenging even with active observation.

Summary updated

BibTeX
@inproceedings{hu2024embodied,
  author = {Sha Hu and Yu Gong and Greg Mori},
  title = {{Embodied Human Activity Recognition}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
}
Figure