Query by Activity Video in the Wild

Tao Hu, William Thong, Pascal Mettes, Cees Snoek

Introduction

In computer vision, activity retrieval—the task of providing a video clip to find similar instances—is often evaluated on curated, balanced datasets. However, real-world data is inherently "long-tailed": a few common activities like "walking" or "talking" have thousands of examples, while rare activities like "playing the bagpipes" or "archery" may have only a handful. Standard machine learning models are typically biased toward frequent classes, leading to a performance collapse when retrieving rare or novel activities. This paper introduces a novel visual-semantic embedding network designed to ensure that rare activities are not overshadowed by common ones, enabling more balanced and accurate retrieval in uncurated, "in the wild" scenarios.

Methodology: Visual-Semantic Embedding Network

The core innovation of this work is a dual-alignment architecture that moves beyond simple classification to learn a more robust embedding space. The system utilizes two primary modules to stabilize the representation of rare classes:

1. Visual Alignment Module

To decouple class representation from sample frequency, the network maintains a "Visual Bank." This bank contains prototypical representations for every activity in the dataset. Regardless of whether an activity has 1,000 training videos or just five, it occupies exactly one slot in this bank. An attention-based global alignment operator allows the model to position an input video relative to the entire landscape of activity prototypes. This forces the model to learn discriminative features for all classes, even those with limited data.

2. Semantic Alignment Module

To provide additional stability, the researchers leverage language as a fixed anchor. A "Semantic Bank" is populated with word embeddings (such as word2vec or ELMo) of the activity names. Because the semantic relationship between concepts—like "cycling" and "driving"—is constant and independent of video frequency, these anchors guide the visual features into a more logical structure. The final training objective combines standard classification with these two alignment losses, ensuring the embedding space is both discriminative and balanced.

Experimental Setup: VR-ActivityNet

To test the model's efficacy in realistic conditions, the authors introduced VR-ActivityNet, a benchmark derived from ActivityNet 1.3. The dataset was reorganized into 100 "base" classes (frequent) and 100 "novel" classes (infrequent). For the novel classes, the model is restricted to seeing only five examples during training. Furthermore, the researchers added 573 "distractor" videos—irrelevant segments—to the test set to simulate the messy reality of uncurated video collections.

Key Insights and Results

The primary metric for evaluation was the Harmonic Mean (H) of the mean Average Precision (mAP) across base and novel classes, which measures the balance of the system.

  • Performance Gains: The full model achieved an H-score of 24.16, a significant improvement over the baseline's 19.95.
  • Global Alignment: The authors found that global alignment (comparing a sample to all known activities) prevents the model from over-clustering common classes. This increased the "scatteredness" of the embedding space, pushing class prototypes further apart and making each activity more unique.
  • Versatility: Beyond standard retrieval, the method proved effective for Clip Retrieval and Moment Retrieval (locating specific activities within long, untrimmed videos).
  • Sample Efficiency: Analysis showed that the performance gap between this method and the baseline widens as more samples are added to novel classes, suggesting the architecture is superior at distilling knowledge from limited data.

Limitations and Future Directions

Despite the improvements, the model still faces challenges with fine-grained distinctions where visual contexts are similar. For instance, "brushing teeth" and "smoking a cigarette" were occasionally confused due to similar hand-to-mouth gestures and backgrounds. The authors suggest that integrating explicit object recognition—identifying the specific tool being used—could be the key to resolving these subtle ambiguities and further boosting retrieval accuracy in the wild.

Summary updated

BibTeX
@inproceedings{hu2023query,
  author = {Tao Hu and William Thong and Pascal Mettes and Cees Snoek},
  title = {{Query by Activity Video in the Wild}},
  booktitle = {ICIP},
  year = {2023},
  doi = {10.1109/icip49359.2023.10222796},
}
Figure