Hue Histograms to Spatiotemporal Local Features for Action Recognition

Fillipe Dias Moreira de Souza, Eduardo Valle, Guillermo Cámara Chávez, Arnaldo de Albuquerque AraújoarXiv 2011 · arXiv

Introduction

In the evolution of computer vision, action recognition systems have historically prioritized motion and shape over color. Early state-of-the-art frameworks, such as Space-Time Interest Points (STIP), relied almost exclusively on grayscale intensity to describe "where" movement occurred and "what" shapes were present. This paper introduces HueSTIP, an extension that integrates color information via saturation-weighted hue histograms, exploring whether color context can enhance recognition in complex, real-world video data.

Methodology: Beyond Grayscale

The standard STIP descriptor combines two primary components: Histogram of Oriented Gradients (HoG) for local appearance and Histogram of Optical Flow (HoF) for local motion. HueSTIP appends a third component: a Hue Histogram.

To ensure robustness, the authors implement saturation weighting. Because hue becomes unstable in desaturated or grayscale-adjacent regions, each hue sample is weighted by its saturation value. This ensures that vivid, informative colors contribute more significantly to the descriptor than muted or noisy tones. The final HueSTIP feature is a concatenated vector of HoG, HoF, and the saturation-weighted Hue Histogram, extracted from the same spatiotemporal patch. Classification is performed using a Bag-of-Features (BoF) approach with a 4,000-word visual vocabulary and a linear SVM.

Experimental Evaluation

The researchers utilized the Hollywood2 dataset, which consists of clips from 69 different movies. This dataset is notoriously challenging due to varied lighting conditions, camera movements, and cluttered backgrounds across 12 distinct action classes.

Representative frames from the Hollywood2 dataset illustrating the diversity of actions and environments.

Results and Discussion

The performance of HueSTIP proved to be highly class-dependent, outperforming the original grayscale STIP in 6 of the 12 tested categories.

Action HueSTIP (Color) STIP (Grayscale)
FightPerson 68.57% 62.86%
HugPerson 18.18% 12.12%
SitDown 45.37% 41.67%
AnswerPhone 12.5% 9.38%
Run 58.87% 57.45%
StandUp 54.11% 51.37%

Contextual Successes

Color acts as a powerful proxy for environmental context. For instance, the "AnswerPhone" class improved because indoor backgrounds often share consistent color distributions. Similarly, in "FightPerson," specific color cues helped distinguish aggressive physical contact from other interactions.

The 'HugPerson' class saw a significant relative improvement in accuracy when color information was included.

Limitations and "Color Noise"

Conversely, color can introduce noise. In classes like "DriveCar" or "GetOutCar," the extreme variety in vehicle colors across different movies confused the classifier. Furthermore, performance dropped in "Eat" (45.45% vs 57.58%) and "Kiss" (38.83% vs 49.51%). The authors suggest that the local neighborhood used for color extraction might not align well with the specific motion-color relationships required for these intimate, close-up actions.

Conclusion

HueSTIP demonstrates that color is a valuable, albeit complex, feature for action recognition. While it provides critical contextual cues for certain environments, its effectiveness is limited by the intra-class variance of object colors. Future research will focus on developing color-specific interest point detectors and exploring multi-scale color descriptors to better capture the nuances of human action.

Summary updated

BibTeX
@misc{souza2011hue,
  author = {Fillipe Dias Moreira de Souza and Eduardo Valle and Guillermo Cámara Chávez and Arnaldo de Albuquerque Araújo},
  title = {{Hue Histograms to Spatiotemporal Local Features for Action Recognition}},
  year = {2011},
  eprint = {1104.3742},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1104.3742},
}
Figure