Social Fabric: Tubelet Compositions for Video Relation Detection

Shuo Chen, Zenglin Shi, Pascal Mettes, Cees G. M. Snoek

Introduction

Video Relation Detection (VRD) is the task of identifying and localizing object interactions over time, typically represented as triplets like $\langle subject-predicate-object \rangle$. While significant progress has been made in static image relation detection, the temporal dimension of video introduces unique challenges, particularly for interactions that evolve over long durations. Traditional methods often struggle to maintain context across many frames, leading to fragmented or incorrect relationship predictions.

The Challenge of Temporal Context

Most current VRD frameworks rely on a "bottom-up" strategy: dividing videos into short snippets, predicting local relations within those snippets, and then attempting to associate them over time. This approach frequently loses the "big picture." For instance, a complex interaction like a "chase" might be misidentified as simple "running" if the model only sees a one-second window. To overcome this, researchers from the University of Amsterdam proposed the "Social Fabric" framework, which considers the entire trajectory of object pairs—referred to as tubelets—from the outset.

Social Fabric Encoding (SFE)

The centerpiece of this research is the Social Fabric Encoding (SFE). Inspired by dictionary-learning techniques like NetVLAD, SFE represents any interaction between two tubelets as a weighted combination of $K$ learned "interaction primitives." These primitives act as the fundamental building blocks of social interaction, such as "approaching," "touching," or "moving away."

Decomposition of complex video relations into learned interaction primitives.

By using these primitives, the model can describe a wide variety of complex relations using a shared, interpretable vocabulary. The system operates in a streamlined two-stage network:

  1. Interaction Proposals: The model sifts through all possible tubelet pairs to identify those that are actually interacting.
  2. Predicate Prediction: For the interacting pairs, the model uses the SFE representation—incorporating motion, visual appearance (I3D features), and language embeddings—to classify the specific relationship.

Experimental Results

The "Social Fabric" approach was evaluated on two major benchmarks: ImageNet-VidVRD and the large-scale VidOR dataset. It achieved state-of-the-art performance on both, reaching a mean Average Precision (mAP) of 20.08 on ImageNet-VidVRD and 11.21 on VidOR, significantly outperforming previous snippet-based methods.

A key finding was the model's robustness to interaction length. Unlike snippet-based baselines, which show declining accuracy as interactions grow longer, the Social Fabric method's performance actually improves with duration. This demonstrates its superior ability to capture the full temporal evolution of social interactions that unfold over many seconds.

Conclusion and Future Work

By treating video relations as a "fabric" woven from basic primitives, this work provides a powerful new lens for understanding dynamic scenes. While currently reliant on external object detectors and trackers, future iterations could integrate tubelet generation directly into the pipeline to handle noisy detections more robustly. Furthermore, the interpretable nature of the learned primitives enables advanced "query-by-example" video retrieval, where users can search for specific interactions across massive databases using simple visual examples.

Summary updated

Abstract

This paper strives to classify and detect the relationship between object tubelets appearing within a video as a 〈subject-predicate-object〉 triplet. Where existing works treat object proposals or tubelets as single entities and model their relations a posteriori, we propose to classify and detect predicates for pairs of object tubelets a priori. We also propose Social Fabric: an encoding that represents a pair of object tubelets as a composition of interaction primitives. These primitives are learned over all relations, resulting in a compact representation able to localize and classify relations from the pool of co-occurring object tubelets across all timespans in a video. The encoding enables our two-stage network. In the first stage, we train Social Fabric to suggest proposals that are likely interacting. We use the Social Fabric in the second stage to simultaneously finetune and predict predicate labels for the tubelets. Experiments demonstrate the benefit of early video relation modeling, our encoding and the two-stage architecture, leading to a new state-of-the-art on two benchmarks. We also show how the encoding enables query-by-primitive-example to search for spatio-temporal video relations. Code: https://github.com/shanshuo/Social-Fabric.
BibTeX
@inproceedings{chen2021social,
  author = {Shuo Chen and Zenglin Shi and Pascal Mettes and Cees G. M. Snoek},
  title = {{Social Fabric: Tubelet Compositions for Video Relation Detection}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  doi = {10.1109/iccv48922.2021.01323},
}
Figure