Human Behavior Recognition Based on Wavelet Moment and Regional Optical Flow

Manyi Wang, Liang ZhangAdvances 2014 · DOI

Abstract

Motion history image (MHI) and motion energy image (MEI) are effective for simple action recognition but lack velocity information. This paper proposes a method combining wavelet moments of these temporal motion descriptors with regional optical flow features to represent actions more comprehensively. Wavelet moments provide rotation, translation, and scale invariance, while regional optical flow captures velocity changes. Experiments on outdoor video sequences demonstrate the effectiveness of this approach in distinguishing actions where speed is a critical factor.

Introduction

Recognizing human behavior in computer vision often relies on temporal templates like Motion History Images (MHI) and Motion Energy Images (MEI). While these templates effectively capture the spatial extent and temporal progression of movement, they struggle to represent velocity. For instance, a "gentle touch" and a "fierce strike" may produce similar static motion templates despite being fundamentally different actions. Additionally, traditional descriptors like Hu Moments are sensitive to image noise. This research addresses these limitations by integrating wavelet-based shape descriptors with optical flow-based velocity features.

Methodology

The proposed framework utilizes a multi-stage pipeline for robust feature extraction:

  1. Foreground Extraction: A Mixture of Gaussians (MoG) model is employed for background subtraction, isolating the moving subject from environmental noise.
  2. Temporal Templates: MEI (a binary map of the total motion area) and MHI (a grayscale map where brightness represents recency) are generated to represent the action's "shape."
  3. Wavelet Moments: Cubic B-spline wavelet moments are applied to the MHI and MEI. Unlike global Hu moments, wavelets offer multi-resolution analysis, ensuring invariance to rotation, scale, and translation while maintaining robustness against noise.
  4. Regional Optical Flow: The Lucas-Kanade algorithm calculates optical flow specifically within the foreground regions to capture speed efficiently. Flow directions are binned into four 90° intervals and weighted by amplitude to create a velocity profile.
Figure 1: A Motion History Image (MHI) template representing the temporal progression of a human action.

Experimental Setup

The method was evaluated using the Weizmann Database, which contains ten distinct actions: bending, jumping jacks, jumping, vertical jumping, running, sideways jumping, skipping, walking, waving one arm, and waving both arms. The dataset includes variations in subjects' physical characteristics and clothing to test the generalizability of the recognition system.

Figure 6: A sample frame from the Weizmann dataset showing a subject performing a walking action in an outdoor environment.

Results and Discussion

The proposed method (Wavelet + Optical Flow) achieved an average recognition rate of 85.5%, significantly outperforming the traditional Hu Moment approach (71.5%). Notably, the system achieved 100% accuracy for actions like Jumping Jacks, Vertical Jumping, Walking, and Waving one arm. The lowest performance was observed for "Skipping" (55.5%), likely due to its visual similarity to running and jumping in 2D projections. The inclusion of velocity features proved crucial for distinguishing actions with similar spatial patterns but different speeds.

Conclusion

By fusing multi-resolution wavelet moments with regional optical flow, this research provides a more descriptive and noise-resilient representation of human motion. While the current four-direction binning for optical flow is computationally efficient, future work could explore finer directional granularity and testing on more complex, cluttered real-world datasets to further enhance recognition accuracy.

Summary updated

Abstract

Motion history image (MHI) and motion energy image(MEI) can recognize simple actions effectively, but they can't represent the velocity information of the actions. Sometimes in the behavior recognition, we must take the speed into account. For example, touching a person, fierce and friendly touches are two totally different movements. The paper combines the wavelet moment of temporal motion descriptors (MHI and MEI) and the speed feature based on optical flow to represent the action. Wavelet moments are rotation, translation and scale invariance. Foreground is obtained from video sequences by background subtraction. Then we use Lucas-Kanade algorithm to get the regional optical flow information, whose direction amplitude can represent the velocity changes in different direction intervals. Experiments based on video sequences outdoor scenarios carried out to verify the effectiveness of the proposed method.
BibTeX
@inproceedings{wang2014human,
  author = {Manyi Wang and Liang Zhang},
  title = {{Human Behavior Recognition Based on Wavelet Moment and Regional Optical Flow}},
  booktitle = {Advances},
  year = {2014},
  doi = {10.2991/icfcce-14.2014.18},
}

Loading related content...

Figure