Robust Features Face

Nadav Israel, Lior Wolf, Ran Barzilay, Gal Shoval

Abstract: Automatic recognition of facial gestures is becoming increasingly important as real-world AI agents become a reality. In this paper, we present an automated system that recognizes facial gestures by capturing local changes and encoding the motion into a histogram of frequencies. We evaluate the proposed method by demonstrating its effectiveness on spontaneous face action benchmarks: the FEEDTUM dataset, the Pain dataset, and the HMDB51 dataset. The results show that, compared to known methods, the new encoding methods significantly improve the recognition accuracy and the robustness of analysis for a variety of applications.

Introduction

Recognizing facial expressions is a fundamental human capability that remains a significant challenge for artificial intelligence. While AI systems have improved at identifying "posed" expressions—those exaggerated for a camera—they often struggle with "spontaneous" expressions that occur naturally. These natural expressions are characterized by subtle, high-frequency movements and local intensity shifts that generic action recognition features, such as those designed for body movements like running or jumping, often fail to capture.

This paper addresses the gap by introducing a specialized feature extraction method that focuses on the temporal "rhythm" of facial movements rather than just spatial trajectories. By encoding these movements into a Histogram of Frequencies (HOF), the system achieves high sensitivity to subtle twitches while remaining robust to changes in lighting and head pose.

Methodology

The proposed system follows a structured pipeline designed to isolate and analyze local facial changes over time.

1. Preprocessing and Grid Decomposition

The initial stage involves face detection using the Viola-Jones algorithm. To ensure consistency across different videos, the system localizes three fiducial points—the eyes and the mouth—to align and normalize the face into a standard 2D plane. Once normalized, the face is divided into a grid of equally sized cells to allow for localized feature extraction.

Facial alignment and grid decomposition process showing fiducial points and normalized face crops.

2. Haar Wavelet Filter Banks

For each cell in the grid, the system calculates the mean gray-level intensity across all frames in a video sequence, creating a temporal signal. This signal represents the intensity fluctuations in a specific facial region over time. To analyze these fluctuations, the signal is passed through a Haar wavelet tree. This decomposition process breaks the signal into different frequency bands (scales), allowing the system to capture both slow-moving expressions and rapid, subtle twitches.

The Haar wavelet tree architecture used to decompose temporal signals into multiple frequency scales.

3. Histogram of Frequencies (HOF) and Classification

The resulting wavelet coefficients are aggregated into histograms, forming the "Histogram of Frequencies." This representation is temporally invariant; it captures the presence of a specific frequency signature (like a smile or a wince) regardless of exactly when it occurs in the video. These histograms are then used as "visual words" in a Bag of Words (BoW) model. Finally, a linear Support Vector Machine (SVM) is trained using a one-versus-all strategy to classify the facial action.

Experimental Evaluation

The HOF method was evaluated on three distinct benchmarks to test its performance across various types of facial actions:

  • PainDB (UNBC-McMaster Shoulder Pain): Focused on distinguishing clinical pain from neutral expressions. The HOF method achieved 85.96% accuracy, surpassing the previous state-of-the-art MS-MIL (83.70%).
  • HMDB51 (Facial Subset): A subset of 484 videos focusing on seven actions (e.g., chewing, drinking, laughing). HOF reached 84.71%, a significant improvement over standard Dense Trajectory methods (66.74%).
  • FEEDTUM: A dataset of six basic emotions elicited by external stimuli. HOF achieved 81.12%, remaining highly competitive with established benchmarks like MilBoost.

Conclusion

The Histogram of Frequencies approach demonstrates that for facial analysis, the frequency of change in a specific area is a more robust signature than spatial paths. The method is not only accurate but also computationally efficient; processing a 256-frame video takes approximately 0.27 seconds, making it suitable for real-time applications. While individual variability in expression remains a challenge, the integration of frequency-based features provides a powerful tool for more natural and robust human-computer interaction.

Summary updated

Abstract

Automatic recognition of facial gestures is becoming increasingly important as real world AI agents become a reality. In this paper, we present an automated system that recognizes facial gestures by capturing local changes and encoding the motion into a histogram of frequencies. We evaluate the proposed method by demonstrating its effectiveness on spontaneous face action benchmarks: the FEEDTUM dataset, the Pain dataset and the HMDB51 dataset. The results show that, compared to known methods, the new encoding methods significantly improve the recognition accuracy and the robustness of analysis for a variety of applications.
BibTeX
@misc{israel2017robust,
  author = {Nadav Israel and Lior Wolf and Ran Barzilay and Gal Shoval},
  title = {{Robust Features Face}},
  year = {2017},
  eprint = {1702.01426},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1702.01426},
}
Figure