Cross-Class Relevance Learning for Temporal Concept Localization

Junwei Ma, Satya Krishna Gorti, Maksims Volkovs, Ilya Stanevich, Guang Wei YuarXiv 2019 · OpenReview

Abstract

We present a novel Cross-Class Relevance Learning approach for the task of temporal concept localization. Most localization architectures rely on feature extraction layers followed by a classification layer which outputs class probabilities for each segment. However, in many real-world applications classes can exhibit complex relationships that are difficult to model with this architecture. In contrast, we propose to incorporate target class and class-related features as input, and learn a pairwise binary model to predict general segment to class relevance. This facilitates learning of shared information between classes, and allows for arbitrary class-specific feature engineering. We apply this approach to the 3rd YouTube-8M Video Understanding Challenge together with other leading models, and achieve first place out of over 280 teams. In this paper we describe our approach and show some empirical results.

The Challenge of Temporal Localization

Temporal concept localization involves identifying the precise start and end times of specific objects or actions within a video. While image classification has seen massive strides, video localization remains difficult due to three primary factors:

  1. Scale: Datasets like YouTube-8M feature thousands of classes and millions of videos, making exhaustive segment-by-segment scoring computationally prohibitive.
  2. Class Relationships: Traditional models treat similar labels (e.g., "Ford Focus" and "Ford Fiesta") as independent, failing to exploit hierarchical or semantic similarities.
  3. Data Scarcity: Rare classes often lack sufficient training data, leading to overfitting in standard architectures where class-specific weights only update occasionally.

Cross-Class Relevance Learning (CCRL)

The core innovation of this work is rethinking the final classification layer. Instead of a standard architecture that outputs a fixed-length probability vector, we propose a pairwise model that treats the target class identity as an input feature.

Comparison of the traditional classification architecture vs. the proposed CCRL architecture, where class identity is treated as an input.

By feeding class features ($\omega(c)$) directly into the model, the network learns a generalized "relevance" function. This allows the system to transfer knowledge between related categories; for instance, learning what a "Car" looks like helps the model localize "Truck" segments even if truck-specific data is sparse. For the final reasoning, the authors utilized Gradient Boosting Machines (XGBoost) on top of deep features, allowing for the inclusion of non-smooth "Similarity Features" that measure distances between current segments and known class prototypes.

Scalability via Candidate Generation

To manage the computational burden of processing millions of videos, the researchers implemented a two-stage pipeline. First, a video-level ensemble (e.g., LSTMs) identifies the top-K most likely classes for a given video. The localization model then only evaluates segments for these high-probability candidates. This "Candidate Generation" (CG) step reduced the processing load by over 20x while maintaining a 99.7% recall of the relevant segments.

Experimental Results

The approach was evaluated on the YouTube-8M dataset, which contains 6.1 million videos and 237,000 human-verified segment labels. The CCRL framework, particularly when combined with Candidate Generation, demonstrated superior performance over standard deep learning baselines.

Performance comparison (mAP) of various models on the YouTube-8M localization task, showing the impact of Candidate Generation (CG).

The CCRL model achieved a mean Average Precision (mAP) of 0.8091, outperforming standard LSTMs (0.8023) and Transformers (0.7955). An ensemble of these models reached a private leaderboard score of 0.8329, securing first place in the 3rd YouTube-8M Video Understanding Challenge. Qualitative analysis further suggested that CCRL provides more consistent predictions across related clusters, such as different vehicle models, compared to traditional architectures that exhibit higher variance.

Conclusion and Future Work

By treating class identity as an input rather than a static output index, Cross-Class Relevance Learning enables more robust information sharing and flexible feature engineering. While highly effective, the pairwise nature of the model increases inference costs if many classes must be checked per segment. Future research will explore richer class descriptions beyond one-hot encodings and joint prediction architectures to further improve localization efficiency.

Summary updated

Abstract

We present a novel Cross-Class Relevance Learning approach for the task of temporal concept localization. Most localization architectures rely on feature extraction layers followed by a classification layer which outputs class probabilities for each segment. However, in many real-world applications classes can exhibit complex relationships that are difficult to model with this architecture. In contrast, we propose to incorporate target class and class-related features as input, and learn a pairwise binary model to predict general segment to class relevance. This facilitates learning of shared information between classes, and allows for arbitrary class-specific feature engineering. We apply this approach to the 3rd YouTube-8M Video Understanding Challenge together with other leading models, and achieve first place out of over 280 teams. In this paper we describe our approach and show some empirical results.
BibTeX
@misc{ma2019cross,
  author = {Junwei Ma and Satya Krishna Gorti and Maksims Volkovs and Ilya Stanevich and Guang Wei Yu},
  title = {{Cross-Class Relevance Learning for Temporal Concept Localization}},
  year = {2019},
  eprint = {1911.08548},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1911.08548},
}
Figure