Learning Unified Video-Language Representations via Joint Modeling and Contrastive Learning for Natural Language Video Localization
Lists
Only you see this
Abstract
Natural language video localization (NLVL) aims to locate the matching span relevant to a given query sentence from an untrimmed video. This task requires not only understanding video and text but also aligning the semantics between video and language. Existing methods obtain vision-language representations via separate encoders, cross-modal interactions are not fine-grained enough, and the semantics are not fully aligned. In this paper, we address the vision-language alignment via joint modeling and contrastive learning. We propose a unified Video-Language Representation Network (UniNet), employing a transformer encoder to learn vision-language representations aligned. Simultaneously taking video and text as input, the encoder jointly learns the representations of both and captures the inter-relations between video and text. Then the representations are used by the predictor to locate the grounding video span. Besides, we train our model with contrastive learning to enhance vision-language representations in the training stage. Experiments on three benchmark datasets show that UniNet outperforms the baseline methods and adopting unified representation and contrastive learning can improve vision-language semantic alignment.
BibTeX
@inproceedings{cui2023learning,
author = {Chenhao Cui and Xinnian Liang and Shuangzhi Wu and Zhoujun Li},
title = {{Learning Unified Video-Language Representations via Joint Modeling and Contrastive Learning for Natural Language Video Localization}},
booktitle = {IJCNN},
year = {2023},
url = {https://openreview.net/forum?id=zIJv08dzpM},
}Loading related content...