Human Interaction Recognition Based on Whole-Individual Detection

Qing Ye, Haoxin Zhong, Chang Qu, Yongmei ZhangSensors 2020 · DOI

TL;DR

This paper tackles the challenge of recognizing human interactions in videos by proposing a novel "whole-individual detection" approach. By combining global features from the entire interaction scene with detailed features from individual participants, their method achieves a significant improvement in recognition accuracy, reaching 91.7% on the UT-interaction dataset.

What Problem Does This Paper Address?

Recognizing human interactions in videos is a complex task in computer vision. Existing methods struggle with several issues: the "spatial complexity" of interactions (how people are positioned and move relative to each other), the "differences in action characteristics at different time periods" (how an action evolves), and the inherent "complexity of interactive action features." These challenges limit the accuracy of current recognition systems. The authors aim to overcome these limitations by developing a more robust framework that leverages both the overall scene context and individual participant details.

Methods

The proposed method integrates several key techniques:

  • Gaussian-based Video Downsampling: To address the temporal differences in action characteristics, they employ an improved Gaussian model for downsampling video frames. This method prioritizes frames with higher distinction between actions, effectively reducing redundant information and focusing on critical temporal phases.
  • Parallel Multi-Feature Fusion Network: To handle the complexity of interactive action features, a novel convolutional neural network architecture is introduced. This network fuses features extracted by parallel Inception V3 and ResNet models. This fusion not only reduces the number of network parameters but also enhances performance and mitigates network degradation issues common in deep architectures.
  • Whole-Individual Detection: This is the core contribution. The approach splits the interaction video into two parts: the "whole video" capturing global interaction features (like relative positions and orientations) and "individual videos" focusing on the detailed actions of each person.
  • Decision-Level Fusion: The preliminary classification results from the whole video and the individual videos are combined using a probabilistic weighting strategy. This decision-level fusion aims to leverage the strengths of each perspective to produce a final, more accurate classification.

Datasets

  • UT-interaction dataset: This dataset was used for training and evaluation. It contains videos of six human-human interaction classes: shake-hands, point, hug, push, kick, and punch. The paper notes that the dataset has variations in background, scale, and illumination, posing challenges for recognition.
  • UCF101 dataset: Extended experiments were conducted on a subset of this dataset to verify the algorithm's applicability and versatility. UCF101 is a larger, more diverse dataset of realistic action videos collected from YouTube, known for its variations in camera motion, object appearance, and background clutter.

Key Insights and Ideas

  • The power of combined perspectives: The most significant insight is that treating human interactions as a single entity (whole video) or as isolated actions (individual videos) independently misses crucial information. Combining both global and individual features provides a more comprehensive understanding of the interaction.
  • Temporal focus through Gaussian downsampling: The idea of using a Gaussian distribution to guide video downsampling is clever. It acknowledges that not all frames are equally informative and that critical action phases have higher "probability density" in terms of feature distinctiveness.
  • Fusion for robustness: The parallel Inception and ResNet fusion network highlights the trend of combining powerful, pre-trained architectures to achieve better feature representation and avoid the pitfalls of simply increasing network depth.

Key Results

  • The proposed whole-individual detection method achieved an accuracy of 91.7% on the UT-interaction dataset.
  • In comparison, individual video analysis yielded lower accuracies (e.g., 83.3% for the left individual video and 75.0% for the right individual video on UT Set 1), while whole video analysis alone reached 86.1% on UT Set 1. The fusion method significantly outperformed these individual components.
  • Extended experiments on the UCF101 dataset also showed improved recognition accuracy compared to individual detection or whole video analysis alone.
  • The Gaussian-based downsampling method showed an improvement in recognition accuracy compared to equal interval sampling, particularly for actions like punching and pushing.

Limitations and Future Work

  • Algorithm Complexity: The authors acknowledge that the algorithm's complexity is relatively high and suggest that future work will focus on improving its efficiency.
  • Action Discrimination: The paper notes that certain actions, like punching and kicking, were sometimes confused with other actions (push, handshake) due to low motion discrimination. This suggests that further refinement in feature extraction or model architecture might be needed for these specific cases.
  • Generalization: While extended experiments on UCF101 were performed, further validation on even more diverse and challenging datasets could be beneficial.

Summary updated

Abstract

Human interaction recognition technology is a hot topic in the field of computer vision, and its application prospects are very extensive. At present, there are many difficulties in human interaction recognition such as the spatial complexity of human interaction, the differences in action characteristics at different time periods, and the complexity of interactive action features. The existence of these problems restricts the improvement of recognition accuracy. To investigate the differences in the action characteristics at different time periods, we propose an improved fusion time-phase feature of the Gaussian model to obtain video keyframes and remove the influence of a large amount of redundant information. Regarding the complexity of interactive action features, we propose a multi-feature fusion network algorithm based on parallel Inception and ResNet. This multi-feature fusion network not only reduces the network parameter quantity, but also improves the network performance; it alleviates the network degradation caused by the increase in network depth and obtains higher classification accuracy. For the spatial complexity of human interaction, we combined the whole video features with the individual video features, making full use of the feature information of the interactive video. A human interaction recognition algorithm based on whole-individual detection is proposed, where the whole video contains the global features of both sides of action, and the individual video contains the individual detail features of a single person. Making full use of the feature information of the whole video and individual videos is the main contribution of this paper to the field of human interaction recognition and the experimental results in the UT dataset (UT-interaction dataset) showed that the accuracy of this method was 91.7%.
BibTeX
@inproceedings{ye2020human,
  author = {Qing Ye and Haoxin Zhong and Chang Qu and Yongmei Zhang},
  title = {{Human Interaction Recognition Based on Whole-Individual Detection}},
  booktitle = {Sensors},
  year = {2020},
  doi = {10.3390/s20082346},
}
Figure