Open-Vocabulary Video Scene Graph Generation via Union-aware Semantic Alignment

Ziyue Wu, Junyu Gao, Changsheng Xu

Abstract

Video Scene Graph Generation (VidSGG) plays a crucial role in various visual-language tasks by providing accessible structured visual relation knowledge. However, the requirement of annotating all categories of prevailing VidSGG methods limits their application in real-world scenarios. Despite the popular VLMs facilitating preliminary exploration of open-vocabulary VidSGG tasks, the correspondence between visual union regions and relation predicates is usually ignored. Therefore, we propose an Open-vocabulary VidSGG framework named Union-Aware Semantic Alignment Network (UASAN) to explore the alignment between visual union regions and relation predicate concepts in the same semantic space. Specifically, a visual refiner is designed to acquire open-vocabulary knowledge and the ability to bridge different modalities. To achieve better alignment, we first design a semantic-aware context encoder to achieve a comprehensive semantic interaction between object trajectories, visual union regions, and trajectory motion information to obtain semantic-aware union region representations. Then, a union-relation alignment decoder is utilized to generate the discriminative relation token for each union region for final relation prediction. Extensive experimental results on two benchmark datasets show that our UASAN achieves significant performance over existing methods, which also verifies the necessity of modeling union region-predicate alignment in the VidSGG pipeline. Code is available in supplementary material.
BibTeX
@inproceedings{wu2024open,
  author = {Ziyue Wu and Junyu Gao and Changsheng Xu},
  title = {{Open-Vocabulary Video Scene Graph Generation via Union-aware Semantic Alignment}},
  booktitle = {ACM Multimedia},
  year = {2024},
  doi = {10.1145/3664647.3681061},
}

Loading related content...

Figure