Semantics-Aware Spatial-Temporal Binaries for Cross-Modal Video Retrieval

Mengshi Qi, Jie Qin, Yi Yang, Yunhong Wang, Jiebo Luo

Abstract

With the current exponential growth of video-based social networks, video retrieval using natural language is receiving ever-increasing attention. Most existing approaches tackle this task by extracting individual frame-level spatial features to represent the whole video, while ignoring visual pattern consistencies and intrinsic temporal relationships across different frames. Furthermore, the semantic correspondence between natural language queries and person-centric actions in videos has not been fully explored. To address these problems, we propose a novel binary representation learning framework, named Semantics-aware Spatial-temporal Binaries ($\text{S}^{2}$ Bin), which simultaneously considers spatial-temporal context and semantic relationships for cross-modal video retrieval. By exploiting the semantic relationships between two modalities, $\text{S}^{2}$ Bin can efficiently and effectively generate binary codes for both videos and texts. In addition, we adopt an iterative optimization scheme to learn deep encoding functions with attribute-guided stochastic training. We evaluate our model on three video datasets and the experimental results demonstrate that $\text{S}^{2}$ Bin outperforms the state-of-the-art methods in terms of various cross-modal video retrieval tasks.
BibTeX
@article{qi2021semantics,
  author = {Mengshi Qi and Jie Qin and Yi Yang and Yunhong Wang and Jiebo Luo},
  title = {{Semantics-Aware Spatial-Temporal Binaries for Cross-Modal Video Retrieval}},
  journal = {IEEE Trans. Image Process.},
  year = {2021},
  doi = {10.1109/tip.2020.3048680},
}

Loading related content...

Figure