Point Prompt Tuning for Temporally Language Grounding

Yawen ZengSIGIR 2022 · OpenReview

Abstract

The task of temporally language grounding (TLG) aims to locate a video moment from an untrimmed video that match a given textual query, which has attracted considerable research attention. In recent years, typical retrieval-based TLG methods are inefficient due to pre-segmented candidate moments, while localization-based TLG solutions adopt reinforcement learning resulting in unstable convergence. Therefore, how to perform TLG task efficiently and stably is a non-trivial work. Toward this end, we innovatively contribute a solution, Point Prompt Tuning (PPT), which formulates this task as a prompt-based multi-modal problem and integrates multiple sub-tasks to tuning performance. Specifically, a flexible prompt strategy is contributed to rewrite the query firstly, which contains both query, start point and end point. Thereafter, a multi-modal Transformer is adopted to fully learn the multi-modal context. Meanwhile, we design various sub-tasks to constrain the novel framework, namely matching task and localization task. Finally, the start and end points of matched video moment are straightforward predicted, simply yet stably. Extensive experiments on two real-world datasets have well verified the effectiveness of our proposed solution.
BibTeX
@inproceedings{zeng2022point,
  author = {Yawen Zeng},
  title = {{Point Prompt Tuning for Temporally Language Grounding}},
  booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
  year = {2022},
  url = {https://openreview.net/forum?id=0YYWo6YTwwb},
}

Loading related content...

Figure