What and when to look? Temporal span proposal network for video relation detection

Sangmin Woo, Junhyug Noh, Kangil Kim

TL;DR

This paper introduces the Temporal Span Proposal Network (TSPN), a novel approach for Video Visual Relation Detection (VidVRD). TSPN efficiently identifies what objects are related and when these relations occur by first scoring the likelihood of a relation and then predicting its temporal span, achieving state-of-the-art performance and significant speedups over existing methods.

What Problem Does This Paper Address?

Understanding complex visual scenes in videos requires not just identifying objects but also comprehending their interactions and relationships over time. While significant progress has been made in image-based visual relation detection, extending this to videos presents unique challenges. These include handling dynamic object appearances and disappearances, modeling long-term temporal dependencies, and accurately pinpointing the start and end times of interactions. Existing methods for Video Visual Relation Detection (VidVRD) fall into two main categories: segment-based, which struggle with long-term context, and window-based, which can be computationally prohibitive due to exhaustive search. This paper aims to bridge this gap by proposing a more efficient and effective method that leverages global video context.

Methods

The core of the proposed solution is the Temporal Span Proposal Network (TSPN), which operates in three main stages:

  1. Object Trajectory Proposal: This initial step leverages established object detection (Faster R-CNN) and multi-object tracking (DeepSORT) techniques to identify and track objects throughout the video, generating object trajectories.
  2. Relationness Scoring: To address the combinatorial explosion of potential object pairs and relations, TSPN first scores the "relationness" of each object pair. This is achieved by fusing visual features, geometric information (bounding box coordinates), and semantic information (class probabilities) of the object pair. A Hadamard product is used for efficient fusion, and a score is computed to identify pairs that are likely to have meaningful interactions. This step effectively sparsifies the search space.
  3. Relation & Temporal Span Prediction: For the selected "pairs-of-interest" identified by the relationness scoring, TSPN then jointly predicts both the relation categories and their temporal spans (start and end times). This is done by quantizing the temporal span into discrete sectors and predicting the likelihood of each relation category within these sectors. This approach allows TSPN to leverage global video context, unlike segment-based methods.

The entire TSPN model is trained end-to-end.

Datasets

The proposed TSPN model is evaluated on two standard VidVRD datasets:

  • ImageNet-VidVRD [15]: A subset of ILSVRC2016-VID, containing 1,000 videos with annotated object categories, bounding box trajectories, and relation triplets. Relations are annotated at the segment level, which the authors link to video-level relations for training.
  • VidOR [30]: A larger-scale dataset with 10,000 videos, providing object categories, bounding box trajectories, and relation triplets. This dataset generally features longer videos compared to ImageNet-VidVRD.

Key Insights and Ideas

  • Relationness as a Filter: The crucial insight is that not all object pairs interact. By introducing a "relationness" score, TSPN effectively prunes the vast search space, focusing computational resources on the most promising interactions. This is a key differentiator from methods that consider all possible pairs.
  • Joint Prediction of "What" and "When": TSPN's ability to simultaneously predict relation categories and their temporal spans is a significant advantage. This joint modeling allows for a more holistic understanding of interactions, capturing both the nature and duration of relationships.
  • Leveraging Global Video Context: Unlike segment-based methods that are limited by short temporal windows, TSPN's architecture allows it to process the entire video context. This is vital for detecting long-term or complex relations that unfold over extended periods.
  • Efficiency through Sparsity: The combination of relationness scoring and temporal span quantization leads to significant computational efficiency, making TSPN approximately 2x faster than segment-based methods and orders of magnitude faster than window-based approaches.

Key Results

  • State-of-the-Art Performance: TSPN achieves state-of-the-art results on both ImageNet-VidVRD and VidOR benchmarks for the Visual Relation Detection (VRDet) task.
  • Significant Speedup: TSPN is shown to be approximately 2x faster than segment-based methods and orders of magnitude faster than window-based methods, making it more practical for real-world applications.
  • Ablation Studies: Experiments confirm the effectiveness of each component:
  • Combining visual, geometric, and semantic features for relationness scoring yields the best performance.
  • The relationness scoring and temporal span prediction modules are complementary and crucial for strong performance.
  • An optimal number of pairs-of-interest (p=64) and temporal sectors (k=16 or 32) were identified for balancing performance and efficiency.
  • Qualitative Examples: Visualizations demonstrate TSPN's ability to capture long-term relations (e.g., "person-past-bicycle") and more active relations (e.g., "elephant-kick-ball") that segment-based methods often miss.

Limitations and Future Work

  • Limitations of Segment-Based Comparison: The paper notes that direct comparison of total computing time with window-based methods is difficult as their source codes are not publicly available.
  • Future Work: The paper does not explicitly detail future work directions, but the success of TSPN suggests potential for extending its principles to other video understanding tasks requiring temporal reasoning. The authors also mention that the VidOR test set was not publicly available at the time of publication.

Summary updated

Abstract

Highlights•We identify the challenges of Video Relation Detection task.•We propose a new approach called Temporal Span Proposal Network (TSPN).•We propose two key modules: (i) relationness scoring module and (ii) temporal span proposal module.•We show that TSPN is not only effective but also efficient.•We conduct comprehensive ablation studies to show the efficacy of our design choices.
BibTeX
@article{woo2026what,
  author = {Sangmin Woo and Junhyug Noh and Kangil Kim},
  title = {{What and when to look? Temporal span proposal network for video relation detection}},
  journal = {Expert Systems with Applications},
  year = {2026},
  doi = {10.1016/j.eswa.2025.129503},
}
Figure