A Two-Stream Method for Human Action Recognition Using Facial Action Cues

Zhimao Lai, Yan Zhang, Xiubo LiangSensors 2024 · DOI

TL;DR

This paper introduces "Face in Action" (FIA), a novel two-stream deep learning model for human action recognition that specifically targets scenarios with significant occlusion. By leveraging both facial image sequences and facial landmark data, FIA achieves robust action recognition even when full-body information is unavailable, outperforming existing methods in challenging real-world conditions.

What Problem Does This Paper Address?

Human Action Recognition (HAR) is a vital technology for applications like video surveillance and healthcare monitoring. However, most current HAR methods rely heavily on full-body information. This approach falters in real-world scenarios where occlusions are common, such as crowded public spaces or indoor surveillance where a person might be partially hidden. The paper argues that in these situations, faces often remain visible and provide crucial cues for understanding actions. Existing methods often overlook this valuable facial information, leading to a lack of robustness when dealing with occlusions. The paper aims to bridge this gap by developing a method that specifically utilizes facial cues for action recognition, particularly in occluded environments.

Methods

The core of the proposed solution is a two-stream network called "Face in Action" (FIA). This architecture processes two types of facial data:

  • RGB Stream: This stream takes facial image sequences as input. It employs a novel Fine-Spatio-Multitemporal (FSM) 3D convolution module. The FSM module is designed to capture fine-grained local facial movements by using smaller spatial receptive fields and to model broader temporal dynamics with larger temporal receptive fields. This contrasts with traditional 3D CNNs that often use uniform receptive fields.
  • Landmark Stream: This stream processes facial landmark sequences. It utilizes a Normalized Temporal Attention (NTA) module integrated within an NTA-GCN (Graph Convolutional Network) block. The NTA module is designed to enhance the detection of crucial facial keyframes by applying temporal attention, making the model more sensitive to important temporal moments in the landmark data. The NTA-GCN block then leverages these attended landmarks for action classification.

Finally, the outputs from both streams are combined using a late fusion strategy for the final action prediction.

Datasets

The authors evaluated their FIA method on two datasets:

  • NTU-FACE: This dataset was generated for the study and is based on the NTU RGB+D dataset. It specifically focuses on face image sequences and facial landmarks, designed to mimic surveillance camera environments. It includes 60 action categories and two benchmarks: cross-subject and cross-view.
  • NTU-FACE 120: An extension of NTU-FACE, this dataset includes more action categories (120 in total) and a larger number of sequences. It also features cross-subject and cross-setup benchmarks.

The paper notes that while datasets like Kinetics and UCF101 are common for HAR, they often capture actions from closer perspectives and may not fully represent surveillance-like conditions. The NTU RGB+D and NTU RGB+D 120 datasets were used by simulating occlusions.

Key Insights and Ideas

  • Face as a Robust Cue: The paper strongly advocates for the face as a reliable source of information for action recognition, especially when full-body data is occluded. This challenges the prevailing reliance on complete body posture.
  • Tailored Receptive Fields: The FSM module's design, with its emphasis on smaller spatial and larger temporal receptive fields, is a key innovation for effectively capturing subtle facial movements and their temporal evolution.
  • Temporal Attention for Keyframes: The NTA module's ability to highlight critical facial keyframes in landmark sequences is crucial for dealing with the sparse nature of facial movements in certain actions.
  • Complementarity of Streams: The two-stream approach effectively combines local facial movements (RGB stream) with global facial dynamics and temporal cues (landmark stream), leading to a more comprehensive understanding of actions.

Key Results

  • Superior Performance in Occlusion: The FIA method significantly outperforms existing SOTA methods on the NTU-FACE and NTU-FACE 120 datasets, particularly in scenarios where occlusion is a factor.
  • Outperforming Whole-Body Methods: FIA achieved higher accuracy than a whole-body method in 11 out of 60 action categories on the NTU dataset, demonstrating the effectiveness of focusing on facial cues for specific actions.
  • Specific Action Category Wins: FIA showed particular strength in recognizing actions involving subtle facial expressions and localized movements, such as "cough/sneeze," "headache," and "nausea/vomiting."
  • Efficiency: The model demonstrated a frame processing time of 35 milliseconds and achieved a frame rate of 30 FPS, indicating its potential for real-time applications.

Limitations and Future Work

  • Late Fusion: The authors acknowledge that their late fusion strategy for combining the RGB and landmark streams does not fully exploit the complementarity between the two modalities.
  • Future Work: They propose exploring deep fusion techniques to better integrate information from the two streams. Additionally, they aim to further improve the method by making it easier to recognize human actions in surveillance-like environments.

Summary updated

Abstract

Human action recognition (HAR) is a critical area in computer vision with wide-ranging applications, including video surveillance, healthcare monitoring, and abnormal behavior detection. Current HAR methods predominantly rely on full-body data, which can limit their effectiveness in real-world scenarios where occlusion is common. In such situations, the face often remains visible, providing valuable cues for action recognition. This paper introduces Face in Action (FIA), a novel two-stream method that leverages facial action cues for robust action recognition under conditions of significant occlusion. FIA consists of an RGB stream and a landmark stream. The RGB stream processes facial image sequences using a fine-spatio-multitemporal (FSM) 3D convolution module, which employs smaller spatial receptive fields to capture detailed local facial movements and larger temporal receptive fields to model broader temporal dynamics. The landmark stream processes facial landmark sequences using a normalized temporal attention (NTA) module within an NTA-GCN block, enhancing the detection of key facial frames and improving overall recognition accuracy. We validate the effectiveness of FIA using the NTU RGB+D and NTU RGB+D 120 datasets, focusing on action categories related to medical conditions. Our experiments demonstrate that FIA significantly outperforms existing methods in scenarios with extensive occlusion, highlighting its potential for practical applications in surveillance and healthcare settings.
BibTeX
@inproceedings{lai2024two,
  author = {Zhimao Lai and Yan Zhang and Xiubo Liang},
  title = {{A Two-Stream Method for Human Action Recognition Using Facial Action Cues}},
  booktitle = {Sensors},
  year = {2024},
  doi = {10.3390/s24216817},
}
Figure