Abstract
Understanding video advertisements requires sophisticated content structuring, moving beyond simple classification to precise scene segmentation and multi-label tagging. This paper describes a two-stage multimodal framework developed by researchers from Fudan University for the 2021 TAAC competition. The system utilizes Temporal Convolutional Networks (TCNs) for segmentation and a novel "global-local" attention mechanism to fuse visual features with speech (ASR) and on-screen text (OCR). The framework achieved 4th place in the competition with a final score of 0.2470.
Introduction
Video advertisements are ubiquitous in digital media, yet their automated understanding remains a significant challenge. Unlike standard video classification, "content structuring" for ads requires identifying exact scene boundaries (segmentation) and the specific products or themes within those scenes (tagging). The authors identify two primary hurdles: boundary ambiguity, where semantic shifts are harder to detect than simple camera shots, and underutilized text data, where on-screen text (OCR) is often noisy and speech (ASR) lacks temporal alignment.
Methodology
The proposed framework operates in two distinct stages to address these challenges.
1. Scene Segmentation
The first stage employs Temporal Convolutional Networks (TCNs) with dilated convolutions to capture long-range dependencies across frames. To improve signal quality, the team implemented Label Refinement, modifying ground-truth labels to mark both the beginning and end of scenes as transition points. This approach allows for direct end-to-end boundary prediction, avoiding the errors common in traditional two-step "shot detection and merging" methods.
2. Multi-modal Tagging
For each segmented scene, the model predicts multiple labels by aggregating visual and textual data. Visual features are processed using NeXt-SoftDBoF, a parameter-efficient variant of Soft Bag-of-Features.
The core innovation is the Global-Local Attention mechanism. By combining global ASR (accurate but temporally broad) with local OCR (scene-specific but noisy) within a BERT model, the system uses global context to "clean" and ground local text signals. The input format [CLS][OCR text][SEP][ASR text] allows the model to learn cross-modal dependencies effectively.
Experimental Results
The framework was evaluated on the 2021 TAAC video ads dataset, consisting of 10,000 annotated videos. The results highlight several key findings:
- Fine-tuning is Essential: Using raw pre-trained features resulted in poor performance (0.1565). Fine-tuning backbones on the ad dataset led to a massive accuracy jump.
- 3D Motion Signal: Incorporating a SlowFast (3D CNN) backbone significantly boosted tagging performance, indicating that motion dynamics are critical for ad categorization.
- Textual Synergy: The Global-Local Attention mechanism (Exp VI) improved scores by over 1% compared to visual-only baselines.
Conclusion
The final ensemble, utilizing ResNet101, EfficientNet-B4/B5, and SlowFast backbones, achieved a score of 0.2470. While highly effective, the authors noted that 3D features did not improve segmentation, likely due to temporal alignment issues. Future work may focus on optimizing the computational cost of such heavy ensembles for real-time production environments.
Summary updated