Real-Time Human Action Recognition on Embedded Platforms

Ruiqi Wang, Zichen Wang, Pengfei Gao, Mingzhen Li, Jae‐Hwan Jeong, Yihang Xu, Yejin Lee, Carolyn Baum, Lisa Tabor Connor, Chenyang LuarXiv 2024 · DOI · arXiv

Abstract

With advancements in computer vision and deep learning, video-based human action recognition (HAR) has become practical. However, due to the complexity of the computation pipeline, running HAR on live video streams incurs excessive delays on embedded platforms. This work tackles the real-time performance challenges of HAR with four contributions: 1) an experimental study identifying a standard Optical Flow (OF) extraction technique as the latency bottleneck in a state-of-the-art HAR pipeline, 2) an exploration of the latency-accuracy tradeoff between the standard and deep learning approaches to OF extraction, which highlights the need for a novel, efficient motion feature extractor, 3) the design of Integrated Motion Feature Extractor (IMFE), a novel single-shot neural network architecture for motion feature extraction with drastic improvement in latency, 4) the development of RT-HARE, a real-time HAR system tailored for embedded platforms. Experimental results on an Nvidia Jetson Xavier NX platform demonstrated that RT-HARE realizes real-time HAR at a video frame rate of 30 frames per second while delivering high levels of recognition accuracy.

Introduction

Human Action Recognition (HAR) is a critical technology for diverse applications, from healthcare monitoring for elderly patients to safety detection in industrial or automotive settings. While deep learning has significantly improved recognition accuracy, these models are often too computationally intensive for real-time deployment on embedded platforms.

Most high-performing HAR systems utilize a "two-stream" architecture: one stream processes static RGB images to capture spatial context, while the second stream analyzes motion through Optical Flow (OF). Researchers have found that the motion stream is the primary source of delay. Specifically, traditional OF extraction methods like TV-L1 can account for a staggering 93% of total latency. Moreover, this latency is unstable; as motion intensity in the video increases, processing time spikes, making it impossible to guarantee the consistent performance required for live video streams.

Methodology and System Design

To overcome these limitations, the authors developed RT-HARE, a framework designed for true real-time performance on resource-constrained devices.

General architecture of a two-stream action recognition pipeline, highlighting the parallel spatial and temporal feature extraction stages.

Integrated Motion Feature Extractor (IMFE)

The primary innovation of the framework is the Integrated Motion Feature Extractor (IMFE). Traditional pipelines are inefficient because they generate high-resolution, human-readable motion maps (Optical Flow) only to immediately compress them into feature vectors. IMFE bypasses this "middleman" by using a single-shot neural network architecture. It takes a batch of frames and uses correlation modules to extract motion features directly, avoiding the expensive reconstruction of pixel-level motion vectors.

The Integrated Motion Feature Extractor (IMFE) architecture, which replaces traditional optical flow computation with a single-shot batched feature encoder.

Optimization and Training

To ensure the lightweight IMFE maintains high performance, the authors used Knowledge Distillation. A high-quality (but slow) TV-L1-based model acted as a teacher, guiding the IMFE student during training to minimize the difference in their feature outputs. The entire RT-HARE pipeline, which incorporates the Long Short-Term Transformer (LSTR) for action recognition, was further optimized using Nvidia TensorRT to leverage the hardware capabilities of the Jetson Xavier NX.

Experimental Results

The system was benchmarked using the 50 Salads dataset, focusing on fine-grained activity recognition.

  • Throughput: RT-HARE achieved 30 FPS on the Nvidia Jetson Xavier NX, a 10x speedup compared to the 3 FPS managed by standard state-of-the-art pipelines.
  • Latency: End-to-end latency was reduced from 614ms to just 68.83ms.
  • Accuracy: RT-HARE maintained a respectable 63.56% accuracy, significantly outperforming other efficient alternatives like RAFT (59.24%) or RGB-only models (53.16%).

Key Insights

  • The Downsampling Paradox: A common optimization strategy is to skip frames (downsampling). However, for traditional OF algorithms, this actually increases latency. Larger gaps between frames lead to higher motion intensity, forcing the algorithm to use more pyramid levels and compute cycles to track pixels.
  • Predictability and Stability: Unlike traditional CV algorithms, neural-network-based extraction (like IMFE) provides stable execution times. This predictability is essential for real-time systems to avoid dropped frames.
  • Hardware Heterogeneity: By offloading the RGB stream to the Jetson's Deep Learning Accelerator (DLA) and the motion stream to the GPU, the researchers achieved additional latency gains, reaching a final performance of ~63ms.

Conclusion

RT-HARE represents a significant step toward practical, real-time HAR on embedded devices. By replacing traditional, computationally expensive motion extraction with the integrated IMFE architecture, the researchers achieved a 10x performance gain with minimal accuracy loss. Future work will explore closing the remaining accuracy gap and optimizing resource utilization through more advanced parallel scheduling on heterogeneous hardware.

Summary updated

Abstract

With advancements in computer vision and deep learning, video-based human action recognition (HAR) has become practical. However, due to the complexity of the computation pipeline, running HAR on live video streams incurs excessive delays on embedded platforms. This work tackles the real-time performance challenges of HAR with four contributions: 1) an experimental study identifying a standard Optical Flow (OF) extraction technique as the latency bottleneck in a state-of-the-art HAR pipeline, 2) an exploration of the latency-accuracy tradeoff between the standard and deep learning approaches to OF extraction, which highlights the need for a novel, efficient motion feature extractor, 3) the design of Integrated Motion Feature Extractor (IMFE), a novel single-shot neural network architecture for motion feature extraction with drastic improvement in latency, 4) the development of RT-HARE, a real-time HAR system tailored for embedded platforms. Experimental results on an Nvidia Jetson Xavier NX platform demonstrated that RT-HARE realizes real-time HAR at a video frame rate of 30 frames per second while delivering high levels of recognition accuracy.
BibTeX
@misc{wang2024real,
  author = {Ruiqi Wang and Zichen Wang and Pengfei Gao and Mingzhen Li and Jae‐Hwan Jeong and Yihang Xu and Yejin Lee and Carolyn Baum and Lisa Tabor Connor and Chenyang Lu},
  title = {{Real-Time Human Action Recognition on Embedded Platforms}},
  year = {2024},
  eprint = {2409.05662},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.05662},
}
Figure