Abstract
We present a novel Cross-Class Relevance Learning approach for the task of temporal concept localization. Most localization architectures rely on feature extraction layers followed by a classification layer which outputs class probabilities for each segment. However, in many real-world applications classes can exhibit complex relationships that are difficult to model with this architecture. In contrast, we propose to incorporate target class and class-related features as input, and learn a pairwise binary model to predict general segment to class relevance. This facilitates learning of shared information between classes, and allows for arbitrary class-specific feature engineering. We apply this approach to the 3rd YouTube-8M Video Understanding Challenge together with other leading models, and achieve first place out of over 280 teams. In this paper we describe our approach and show some empirical results.
The Challenge of Temporal Localization
Temporal concept localization involves identifying the precise start and end times of specific objects or actions within a video. While image classification has seen massive strides, video localization remains difficult due to three primary factors:
- Scale: Datasets like YouTube-8M feature thousands of classes and millions of videos, making exhaustive segment-by-segment scoring computationally prohibitive.
- Class Relationships: Traditional models treat similar labels (e.g., "Ford Focus" and "Ford Fiesta") as independent, failing to exploit hierarchical or semantic similarities.
- Data Scarcity: Rare classes often lack sufficient training data, leading to overfitting in standard architectures where class-specific weights only update occasionally.
Cross-Class Relevance Learning (CCRL)
The core innovation of this work is rethinking the final classification layer. Instead of a standard architecture that outputs a fixed-length probability vector, we propose a pairwise model that treats the target class identity as an input feature.
By feeding class features ($\omega(c)$) directly into the model, the network learns a generalized "relevance" function. This allows the system to transfer knowledge between related categories; for instance, learning what a "Car" looks like helps the model localize "Truck" segments even if truck-specific data is sparse. For the final reasoning, the authors utilized Gradient Boosting Machines (XGBoost) on top of deep features, allowing for the inclusion of non-smooth "Similarity Features" that measure distances between current segments and known class prototypes.
Scalability via Candidate Generation
To manage the computational burden of processing millions of videos, the researchers implemented a two-stage pipeline. First, a video-level ensemble (e.g., LSTMs) identifies the top-K most likely classes for a given video. The localization model then only evaluates segments for these high-probability candidates. This "Candidate Generation" (CG) step reduced the processing load by over 20x while maintaining a 99.7% recall of the relevant segments.
Experimental Results
The approach was evaluated on the YouTube-8M dataset, which contains 6.1 million videos and 237,000 human-verified segment labels. The CCRL framework, particularly when combined with Candidate Generation, demonstrated superior performance over standard deep learning baselines.
The CCRL model achieved a mean Average Precision (mAP) of 0.8091, outperforming standard LSTMs (0.8023) and Transformers (0.7955). An ensemble of these models reached a private leaderboard score of 0.8329, securing first place in the 3rd YouTube-8M Video Understanding Challenge. Qualitative analysis further suggested that CCRL provides more consistent predictions across related clusters, such as different vehicle models, compared to traditional architectures that exhibit higher variance.
Conclusion and Future Work
By treating class identity as an input rather than a static output index, Cross-Class Relevance Learning enables more robust information sharing and flexible feature engineering. While highly effective, the pairwise nature of the model increases inference costs if many classes must be checked per segment. Future research will explore richer class descriptions beyond one-hot encodings and joint prediction architectures to further improve localization efficiency.
Summary updated