Large-scale video semantic recognition based on consistency of segment-level and video-level predictions

Rui Wang, Yu–Gang Jiang, Jingjing Chen, Zheng Wang, Zejia WengScientia 2020 · DOI

Abstract

Segment-level video semantic recognition, which known to be an important task in video analysis, attempts to identify the semantic concepts in short video clips. Labeling video segments is difficult because there is an extremely large number of segments and there are no network tags; consequently, only a portion of the video segments are labeled. Determining how to improve the accuracy of semantic recognition of fragmented videos with limited semantic labels is a key challenge in video semantic recognition. This paper proposes a video semantic recognition algorithm based on the consistency of video- and segment-level predictions. The proposed algorithm introduces the constraint of consistency between complete video semantics and fragmentary video semantics. The proposed algorithm can be applied to filter the video segment semantic results to improve recognition accuracy. The proposed algorithm achieved 82.62% mean average precision on the video segment semantic recognition task using the large-scale video dataset YouTube-8M and ranked second in the third YouTube-8M competition.

Introduction

In the field of computer vision, video-level recognition—such as tagging a full video as "Cooking"—has reached high levels of accuracy. However, segment-level recognition, which involves identifying specific actions like "chopping onions" within a 5-second clip, remains a significant challenge. The primary obstacle is the label gap: while video-level labels are abundant due to user-generated tags, human-labeled segment data is sparse. Training models on these limited labels often results in high false-positive rates, where models predict concepts that are contextually inconsistent with the overall video (e.g., predicting "Pizza" in a "Chinese Cooking" tutorial).

Methodology

The proposed approach bridges the gap between global context and local actions through a multi-stage pipeline designed for large-scale video understanding.

Feature Extraction and Aggregation

The process begins with extracting video and audio features, which are then processed through various aggregation modules. These modules transform frame-level sequences into compact representations. The authors utilized NeXtVLAD (an efficient NetVLAD variant), GatedDBOF (Gated Deep Bag-of-Frames), and GRUs (Gated Recurrent Units) to capture diverse temporal and spatial information.

Overview of the video semantic recognition pipeline

Training Strategy and Consistency Constraint

A two-stage training process was employed:

  1. Pre-training: Models were initially trained on the massive YouTube-8M video-level dataset to learn general semantic features.
  2. Fine-tuning: These models were then refined using the smaller, human-labeled segment dataset to specialize in short-clip recognition.

The core innovation is the Consistency Constraint. By assuming a segment's semantics must be a subset of the overall video's semantics, the researchers implemented a filtering mechanism. The video-level model generates a "candidate list" of likely tags; if a segment-level prediction does not appear on this list, it is suppressed. This effectively uses global context as a noise filter to eliminate out-of-context errors.

Model Ensembling

To maximize performance, the authors ensembled 10 different architectures, including variations of ResNet, NetFV, and NeXtVLAD, using weighted averaging to combine diverse perspectives of the video data.

Experimental Results

The study utilized the YouTube-8M dataset, featuring 6.1 million videos across 3,862 classes. A subset of these includes human-verified labels for 5-second segments across 1,000 classes.

The proposed algorithm achieved a mean average precision (MAP) of 82.62% on the segment-level task, ranking 2nd out of 283 teams in the 3rd YouTube-8M competition. Ablation studies confirmed that the Consistency Constraint significantly improved performance by pruning the long tail of potential false positives. Furthermore, fine-tuning on segment-specific data yielded a nearly 6% improvement in MAP, highlighting that while features are shared, the decision boundaries for short clips are distinct from full videos.

Conclusion and Future Work

Enforcing consistency between video-level context and segment-level predictions is a powerful tool for large-scale video understanding. While highly effective, the authors suggest that future research could explore causal reasoning to model temporal flow and symbolic learning to incorporate explicit knowledge graphs. Additionally, distilling the current 10-model ensemble into a single efficient model would improve real-time applicability for consumer software.

Summary updated

Abstract

Segment-level video semantic recognition, which known to be an important task in video analysis, attempts to identify the semantic concepts in short video clips.Labeling video segments is difficult because there is an extremely large number of segments and there are no network tags; consequently, only a portion of the video segments are labeled. Determining how to improve the accuracy of semantic recognition of fragmented videos with limited semantic labels is a key challenge in video semantic recognition.This paper proposes a video semantic recognition algorithm based on the consistency of video- and segment-level predictions. The proposed algorithm introduces the constraint of consistency between complete video semantics and fragmentary video semantics. The proposed algorithm can be applied to filter the video segment semantic results to improve recognition accuracy. The proposed algorithm achieved 82.62% mean average precision on the video segment semantic recognition task using the large-scale video dataset YouTube-8M and ranked second in the third YouTube-8M competition.
BibTeX
@inproceedings{wang2020large,
  author = {Rui Wang and Yu–Gang Jiang and Jingjing Chen and Zheng Wang and Zejia Weng},
  title = {{Large-scale video semantic recognition based on consistency of segment-level and video-level predictions}},
  booktitle = {Scientia},
  year = {2020},
  doi = {10.1360/ssi-2020-0014},
}
Figure