Featureless: Bypassing feature extraction in action categorization

Silvia L. Pintea, Pascal S. M. Mettes, Jan C. van Gemert, Arnold W. M. SmeuldersICIP 2016 · DOI

Abstract

Traditional action recognition pipelines rely on computationally expensive feature extraction and codebook assignment. This paper proposes a "featureless" approach that bypasses these stages by learning a direct mapping from raw pixels to high-level video representations using a multiclass extension of the Waldboost algorithm. Evaluated on the UCF11 dataset, the method achieves a 10x speedup over traditional motion features while maintaining competitive accuracy, demonstrating that feature extraction can be treated as a learnable approximation task.

Introduction

In computer vision, the standard pipeline for video understanding typically involves a multi-step process: raw pixel input, extraction of mathematical descriptors (e.g., HOG or HOF), mapping to a "codebook" of visual words, and final classification. While modern deep learning often automates these steps through hierarchical feature extraction, the computational cost remains high, especially for motion-based features (3D) which track pixel movement over time.

Researchers from the University of Amsterdam and Delft University of Technology propose a "featureless" approach. Instead of calculating complex descriptors at test time, they suggest learning the mapping from raw grayscale pixels directly to video representations.

Comparison of the classic, CNN-based, and proposed featureless approaches to video representation.

Methodology: The Multiclass Waldboost Approach

The core innovation is treating handcrafted features not as inputs, but as ground truth labels for training. The process involves:

  1. Mapping as Learning: During training, standard descriptors (HOG/HOF) label raw image patches with their corresponding "codebook ID."
  2. Multiclass Waldboost: A boosting framework is trained to predict these IDs directly from raw pixels. Waldboost utilizes Wald’s Sequential Probability Ratio Test, allowing for "early exits." If a patch can be confidently classified (or rejected) early in the evaluation of weak learners, the process stops, saving significant computation.
  3. Featureless Execution: At test time, the system bypasses descriptor extraction and codebook lookups entirely. Raw patches are fed into the Waldboost model to generate predicted codebook IDs, which form the final representation for an SVM classifier.

Experimental Results

The approach was evaluated on the UCF11 (YouTube Action) dataset, containing 1,600 clips across 11 categories. The results highlight a significant efficiency gain without a prohibitive loss in accuracy.

Performance and efficiency comparison between Linear SVM, Adaboost, and the proposed Waldboost method.

As shown in the results, the Waldboost method achieves a Mean Average Precision (MAP) of 41%, matching Adaboost but operating at a fraction of the time (0.60s per frame vs. 4.00s). Compared to a standard Linear SVM approach, it is 25 times faster. Furthermore, when applied to 3D motion features (3D HOF), the speedup reached over 10x compared to traditional extraction methods (0.60s vs 7.50s per frame).

Interestingly, the "featureless" representation appears to capture information complementary to traditional Bag-of-Words (BoW) models. Combining the two methods resulted in a MAP of 50%, significantly outperforming the 44% baseline.

Conclusion

This research demonstrates that feature extraction can be bypassed by approximating the traditional pipeline with a fast, learnable classifier. By treating features as labels, the system achieves real-time performance levels suitable for resource-constrained applications. While current experiments rely on grayscale patches and small codebooks, the proof of concept suggests that "recognizing" features rather than calculating them is a viable path for efficient video understanding.

Summary updated

BibTeX
@inproceedings{pintea2016featureless,
  author = {Silvia L. Pintea and Pascal S. M. Mettes and Jan C. van Gemert and Arnold W. M. Smeulders},
  title = {{Featureless: Bypassing feature extraction in action categorization}},
  booktitle = {ICIP},
  year = {2016},
  doi = {10.1109/icip.2016.7532346},
}
Figure