NewsNet: A Novel Dataset for Hierarchical Temporal Segmentation

Haoqian Wu, Keyu Chen, Haozhe Liu, Mingchen Zhuge, Bing Li, Ruizhi Qiao, Xiujun Shu, Bei Gan, Liangsheng Xu, Bo Ren, Mengmeng Xu, Wentian Zhang, Raghavendra Ramachandra, Chia-Wen Lin, Bernard Ghanem

Introduction

Temporal video segmentation is a fundamental task in computer vision, involving the partitioning of long-form video into meaningful units. While existing AI models have become proficient at identifying low-level changes, such as camera cuts or specific actions, they often struggle with the high-level semantic structures found in professional broadcasts. Most current datasets focus on a single layer of granularity, such as "shots" or "scenes," failing to capture the complex, multi-layered storytelling inherent in long-form content.

To address this gap, researchers have introduced NewsNet, a massive dataset designed to facilitate hierarchical temporal segmentation. A typical 60-minute news broadcast is not merely a sequence of shots; it is a structured hierarchy where individual events form scenes, which build into stories, which are ultimately grouped by overarching topics. NewsNet provides the scale and annotation depth necessary for models to learn how to navigate these nested semantic layers.

NewsNet features multimodal data and four levels of hierarchical annotation: Event, Scene, Story, and Topic.

The NewsNet Dataset

NewsNet is a comprehensive benchmark consisting of 1,000 videos totaling 946 hours of content. Sourced from 20 different mainstream news platforms, the dataset ensures high diversity in production styles and content.

Key features of NewsNet include:

  • Four Levels of Annotation: Every video is annotated at the Event, Scene, Story, and Topic levels.
  • Multimodal Cues: The dataset includes aligned video, audio, and textual (ASR) data, allowing models to leverage diverse signals for segmentation.
  • Scale: It significantly surpasses previous benchmarks like MovieNet or TI-News in both duration and semantic abstraction.

Methodology: Hierarchical Modeling

The researchers explored two primary modeling paradigms: Separate Modeling, where individual models are trained for each level, and Hierarchical Modeling, where a single model predicts multiple levels simultaneously. Within hierarchical modeling, they tested "Multi-Label" (one head for all levels) and "Multi-Head" (separate branches for each level) architectures.

The proposed framework utilizes multimodal feature embedding and a Hierarchical Ranking Loss to enforce logical relationships between segmentation levels.

A core technical contribution is the Hierarchical Ranking Loss. This novel loss function enforces the logical constraint that a boundary at a coarse level (e.g., Topic) must also be a boundary at all finer levels (Story, Scene, and Event). By ensuring that model confidence scores follow a non-increasing order from coarse to fine, the loss function helps the model internalize the inherent "nesting" of video segments.

To process the multimodal data, the framework extracts features using ResNet50 for visual data, Cnn14 for audio, and BERT-base for text transcripts.

Key Insights and Results

The study yielded several critical insights into the nature of video understanding:

  • The Necessity of Multimodality: Visual data alone proved insufficient for high-level segmentation. For "Topic" detection, integrating audio and text features improved the F1 score from approximately 63 to 73. In news media, what is said is often a stronger indicator of a topic shift than what is seen.
  • Structural Constraints Improve Performance: Implementing the Hierarchical Ranking Loss boosted the Multi-Head model's Topic F1 score from 70.4 to 73.2. This suggests that teaching a model the relationship between levels improves its performance on individual levels.
  • The Topic Challenge: Segmenting by topic remains the most difficult task due to long-term temporal dependencies, requiring models to maintain context over several minutes.

Conclusion

NewsNet represents a significant step forward in teaching AI to understand the structural logic of professional video. By providing a large-scale, multimodal, and hierarchically annotated dataset, the authors have created a robust foundation for future research in long-form video analysis. Future work will look toward applying these segmentation capabilities to downstream tasks such as video question answering (VQA), automated short-video creation, and more sophisticated video grounding.

Summary updated

Abstract

Temporal video segmentation is the get-to- go automatic video analysis, which decomposes a long-form video into smaller components for the following-up understanding tasks. Recent works have studied several levels of granularity to segment a video, such as shot, event, and scene. Those segmentations can help compare the semantics in the corresponding scales, but lack a wider view of larger temporal spans, especially when the video is complex and structured. Therefore, we present two abstractive levels of temporal segmentations and study their hierarchy to the existing fine-grained levels. Accordingly, we collect NewsNet, the largest news video dataset consisting of 1,000 videos in over 900 hours, associated with several tasks for hierarchical temporal video segmentation. Each news video is a collection of stories on different topics, represented as aligned audio, visual, and textual data, along with extensive frame-wise annotations in four granularities. We assert that the study on NewsNet can advance the understanding of complex structured video and benefit more areas such as short-video creation, personalized advertisement, digital instruction, and education. Our dataset and code is publicly available at https://github.com/NewsNet-Benchmark/NewsNet.
BibTeX
@inproceedings{wu2023newsnet,
  author = {Haoqian Wu and Keyu Chen and Haozhe Liu and Mingchen Zhuge and Bing Li and Ruizhi Qiao and Xiujun Shu and Bei Gan and Liangsheng Xu and Bo Ren and Mengmeng Xu and Wentian Zhang and Raghavendra Ramachandra and Chia-Wen Lin and Bernard Ghanem},
  title = {{NewsNet: A Novel Dataset for Hierarchical Temporal Segmentation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52729.2023.01028},
}
Figure