ReHAR: Robust and Efficient Human Activity Recognition

Xin Li, Mooi Choo Chuah

Abstract

Designing a scheme that can achieve a good performance in predicting single person activities and group activities is a challenging task. In this paper, we propose a novel robust and efficient human activity recognition scheme called ReHAR, which can be used to handle single person activities and group activities prediction. First, we generate an optical flow image for each video frame. Then, both video frames and their corresponding optical flow images are fed into a Single Frame Representation Model to generate representations. Finally, an LSTM is used to predict the final activities based on the generated representations. The whole model is trained end-to-end to allow meaningful representations to be generated for the final activity recognition. We evaluate ReHAR using two well-known datasets: the NCAA Basketball Dataset and the UCFSports Action Dataset. The experimental results show that the proposed ReHAR achieves a higher activity recognition accuracy with an order of magnitude shorter computation time compared to the state-of-the-art methods.

Introduction

Human Activity Recognition (HAR) is a cornerstone of modern computer vision, with applications ranging from smart surveillance to automated sports highlights. However, two major hurdles remain in the field. First is the trade-off between complexity and efficiency; many high-performing models are computationally heavy, making them unsuitable for real-time use or mobile deployment. Second is the difficulty in capturing both individual and group dynamics. While recognizing a single person walking is relatively straightforward, identifying a "layup" or a "steal" in a crowded basketball game requires understanding the intricate relationships between multiple actors, the ball, and the environment over time.

Existing solutions often rely on fixed-length frame stacks, which fail on long sequences, or hierarchical LSTMs that are too slow for practical use. ReHAR (Robust and Efficient Human Activity Recognition) addresses these challenges by providing a framework that is both accurate across different action types and fast enough for real-world application.

Methodology

The ReHAR architecture is designed as an end-to-end deep learning framework consisting of three primary stages:

  1. Preprocessing (Dual-Stream Input): For every video frame, the model calculates Optical Flow (utilizing FlowNet 2.0). This creates a 3-channel "motion image" representing pixel movement between frames. The model processes both the original RGB frame for context and the Optical Flow image for motion.
  2. Single Frame Representation Model: These two streams are fed into separate CNNs with VGG16 backbones. Instead of standard fully connected layers, the architecture employs Global Average Pooling (GAP) and Global Max Pooling (GMP). These features are processed by a first-stage LSTM (LSTM1) to generate a compact representation of that specific moment.
  3. Activity Recognition Model: A second-stage LSTM (LSTM2) takes the sequence of representations from the first stage to predict the final activity label.
The ReHAR architecture overview, illustrating the dual-stream preprocessing, the single-frame representation model, and the final activity recognition stage.

A key innovation is the use of multi-task learning. The model is trained using a multi-task loss function (Loss1 and Loss2) that penalizes the model if intermediate single-frame representations do not align with the correct activity, forcing the network to learn highly discriminative features early in the pipeline.

Experimental Results

ReHAR was evaluated on two distinct benchmarks: the NCAA Basketball Dataset (group activity) and the UCF Sports Action Dataset (single-person actions).

Performance and Speed

ReHAR achieved a 58.9% mAP on the NCAA dataset, representing a 7.3% improvement over previous state-of-the-art attention-based models. On the UCF Sports dataset, it reached 92.8% mAP, outperforming existing methods by 6.1%.

Crucially, ReHAR is an order of magnitude faster than its competitors. It processes 10 frames in approximately 103ms using a VGG16 backbone. This efficiency makes it viable for real-time broadcast analysis, whereas other group activity models often take seconds to process a single clip.

Interpretability and Robustness

Using visualization techniques similar to Grad-CAM, the researchers demonstrated that ReHAR naturally learns to focus on the "key actor" performing the action (such as the shooter) rather than distracting elements like referees or the ball, which can be difficult to track reliably.

Visualization of model attention on the NCAA dataset, showing how the network focuses on the primary actors involved in the play.

The model also showed high robustness, successfully distinguishing between very similar actions, such as "3-pointer success" versus "2-pointer success," by effectively analyzing the player's position relative to court markings.

Conclusion

ReHAR demonstrates that explicit motion modeling via optical flow, combined with an end-to-end multi-task LSTM architecture, can significantly improve both the accuracy and speed of human activity recognition. While the model still faces challenges with extremely scarce data categories (e.g., "slam dunk failure"), future work involving more modern backbones like MobileNet or 3D Convolutional Networks (C3D) promises even greater performance gains.

Summary updated

Abstract

Designing a scheme that can achieve a good performance in predicting single person activities and group activities is a challenging task. In this paper, we propose a novel robust and efficient human activity recognition scheme called ReHAR, which can be used to handle single person activities and group activities prediction. First, we generate an optical flow image for each video frame. Then, both video frames and their corresponding optical flow images are fed into a Single Frame Representation Model to generate representations. Finally, an LSTM is used to pre- dict the final activities based on the generated representations. The whole model is trained end-to-end to allow meaningful representations to be generated for the final activity recognition. We evaluate ReHAR using two well-known datasets: the NCAA Basketball Dataset and the UCFSports Action Dataset. The experimental results show that the pro- posed ReHAR achieves a higher activity recognition accuracy with an order of magnitude shorter computation time compared to the state-of-the-art methods.
BibTeX
@misc{li2018rehar,
  author = {Xin Li and Mooi Choo Chuah},
  title = {{ReHAR: Robust and Efficient Human Activity Recognition}},
  year = {2018},
  eprint = {1802.09745},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1802.09745},
}
Figure