Abstract
This paper makes two complementary contributions to event retrieval in large collections of videos. First, we propose hyper-pooling strategies that encode the frame descriptors into a representation of the video sequence in a stable manner. Our best choices compare favorably with regular pooling techniques based on k-means quantization. Second, we introduce a technique to improve the ranking. It can be interpreted either as a query expansion method or as a similarity adaptation based on the local context of the query video descriptor. Experiments on public benchmarks show that our methods are complementary and improve event retrieval results, without sacrificing efficiency.
Introduction
In large-scale video retrieval, the objective is to identify all instances of a specific event—such as a concert or a protest—within a massive dataset, given only a single query video. A common approach is to represent a video by averaging the descriptors of its constituent frames. However, this "average pooling" often discards vital information, particularly in long videos containing diverse shots.
The authors address this by exploring "hyper-pooling," a hierarchical aggregation method that groups frame descriptors into "cells" before final aggregation. A significant challenge here is the instability of standard clustering techniques like k-means in high-dimensional spaces; minor frame variations can lead to drastically different cluster assignments, introducing noise. This research proposes stable hyper-pooling strategies and a novel query expansion technique to enhance retrieval performance without requiring manual labels.
Stable Hyper-pooling
The core idea behind stable hyper-pooling is the observation that the initial components of a PCA-reduced descriptor exhibit the highest temporal stability—they change minimally between consecutive frames. Leveraging this, the authors propose two strategies to partition the descriptor space robustly:
- Partial k-means (PKM): This method performs k-means clustering using only the most stable (first few) PCA components.
- Sign of Stable Components (SSC): A simpler, training-free approach that assigns frames to cells based on the binary sign (positive or negative) of the first $m$ PCA components.
By using these stable dimensions to "hash" frames into cells, the resulting video descriptor becomes significantly more robust to natural visual fluctuations. This stability is crucial for maintaining a consistent video signature across temporal variations.
Difference of Neighborhood (DoN) Query Expansion
Standard query expansion typically involves averaging the top-ranked search results with the original query to refine the search. This paper introduces a more sophisticated approach called Difference of Neighborhood (DoN), inspired by the "Difference-of-Gaussians" operator in image processing.
The DoN method calculates the difference between two neighborhoods:
- $N_1$: A small neighborhood of the most similar results.
- $N_2$: A much larger neighborhood representing the local background context.
By subtracting the average of $N_2$ from the average of $N_1$, the system effectively "boosts" the signal of relevant results while "subtracting out" common background noise or irrelevant features shared by the local context of the query. This "sharpening" effect makes the query vector more discriminative.
Experimental Results
The proposed methods were evaluated on several benchmarks, including the EVVE event retrieval dataset and the Oxford5k image dataset.
- Performance Gains: On the EVVE dataset, the baseline Mean MVLAD (MMV) achieved a mean Average Precision (mAP) of 33.3%. Implementing SSC hyper-pooling increased this to 36.3%. The addition of DoN query expansion provided a substantial boost, reaching 44.0% mAP.
- Robustness to Clutter: The hyper-pooling approach demonstrated superior resilience when query videos were embedded within long sequences of unrelated distractor footage, outperforming simple averaging techniques.
- Generalization: The DoN technique also proved effective for static image retrieval, improving mAP on Oxford5k from 40.0% to 50.4%, indicating its broad applicability beyond video.
Conclusion
This work highlights that in high-dimensional video retrieval, stability often trumps precision. By focusing on stable PCA components for hyper-pooling and using local context to "sharpen" queries via DoN, significant improvements in unsupervised event detection are possible. While hyper-pooling is most effective for high-dimensional descriptors like VLAD, and DoN requires careful parameter tuning for neighborhood sizes, these methods offer a powerful, efficient path forward for large-scale video search.
Summary updated