Video Caption Based Searching Using End-to-End Dense Captioning and Sentence Embeddings

Akshay Aggarwal, Aniruddha Chauhan, Deepika Kumar, Mamta Mittal, Sudipta Roy, Tai-Hoon Kim

TL;DR

This paper proposes a novel video search method that leverages the video's actual content by generating descriptive captions and then embedding both captions and search queries into a shared semantic space. This approach aims to overcome the limitations of traditional metadata-based search, offering improved accuracy, especially for unlabeled videos.

What Problem Does This Paper Address?

Current video search engines, like those on YouTube, primarily rely on metadata such as titles, tags, and descriptions. This approach is limited because it doesn't utilize the actual visual and temporal information within the video. Consequently, users often struggle to find relevant content, especially for videos with sparse or inaccurate metadata. The core challenge is effectively encoding the dynamic events within a video and comparing them to a user's query in a meaningful way.

Methods

The proposed methodology is a two-stage process:

  1. End-to-End Dense Video Captioning: The authors employ a masked transformer-based model for dense video captioning. This model is designed to generate descriptive captions for specific events or segments within a video, rather than just a single overall description. The "end-to-end" nature means the video encoder and caption decoder are trained jointly, allowing for better feature learning and more accurate captions. The model comprises a CNN-based encoder for visual features, a proposal decoder to identify event boundaries (start and end times), and a captioning decoder to generate word sequences.
  2. Caption-Based Search with Sentence Embeddings: Once video captions are generated, the search process involves embedding both these captions and user queries into a common multidimensional vector space. Seven different sentence embedding techniques were evaluated for this purpose. The similarity between a query embedding and caption embeddings is then calculated using cosine similarity. Videos are ranked based on this similarity score, with higher similarity indicating a better match.

Datasets

  • YouCook2: This dataset was used for experimentation. It's a cooking-specific dataset containing untrimmed videos with annotations describing cooking procedures. The authors chose it due to its focus on human-involved processes and relatively smaller vocabulary size, making it suitable for initial testing.

Key Insights and Ideas

  • Content-Driven Search is Crucial: The paper strongly advocates for moving beyond metadata to leverage the actual video content for search, highlighting the limitations of current approaches.
  • Dense Captioning for Granularity: Generating "dense" captions, which describe specific events or segments, is key to capturing the nuanced content of videos, enabling more precise searches.
  • Unified Embedding Space is Powerful: The core idea of mapping both video captions and user queries into the same semantic space is fundamental to enabling direct comparison and retrieval.
  • Universal Sentence Encoder Shines: The Universal Sentence Encoder (USE) emerged as the most effective method for creating these embeddings, demonstrating its robustness across various tasks.

Key Results

  • Improved Search Accuracy: The proposed method, when integrated with traditional techniques, is shown to improve search result quality.
  • Universal Sentence Encoder Dominance: Among the seven sentence embedding techniques tested, the Universal Sentence Encoder (USE) consistently outperformed others, achieving a median percentile score of 99.51.
  • Top-1% Performance: USE achieved a 65.01% score for Top-1% performance, meaning that for 65.01% of search queries, the relevant video clip was found within the top 1% of the ranked results. This indicates high precision for the most relevant results.
  • Broad Success: Across various percentile levels (Top-1%, Top-5%, Top-10%, Top-15%, Median), USE consistently showed superior performance compared to other models.

Limitations and Future Work

  • Dataset Specificity: While YouCook2 is a good starting point, the authors acknowledge that the approach could be tested on other video captioning datasets to assess broader applicability.
  • Future Research Directions: The authors suggest exploring different video captioning models, embedding techniques, and similarity metrics to further enhance performance. They also propose creating a dedicated dataset with annotated videos, search results, and queries to facilitate future research in this area. The approach could also be extended to search for specific segments or events within longer videos.

Summary updated

Abstract

Traditionally, searching for videos on popular streaming sites like YouTube is performed by taking the keywords, titles, and descriptions that are already tagged along with the video into consideration. However, the video content is not utilized for searching of the user’s query because of the difficulty in encoding the events in a video and comparing them to the search query. One solution to tackle this problem is to encode the events in a video and then compare them to the query in the same space. A method of encoding meaning to a video could be video captioning. The captioned events in the video can be compared to the query of the user, and we can get the optimal search space for the videos. There have been many developments over the course of the past few years in modeling video-caption generators and sentence embeddings. In this paper, we exploit an end-to-end video captioning model and various sentence embedding techniques that collectively help in building the proposed video-searching method. The YouCook2 dataset was used for the experimentation. Seven sentence embedding techniques were used, out of which the Universal Sentence Encoder outperformed over all the other six, with a median percentile score of 99.51. Thus, this method of searching, when integrated with traditional methods, can help improve the quality of search results.
BibTeX
@inproceedings{aggarwal2020video,
  author = {Akshay Aggarwal and Aniruddha Chauhan and Deepika Kumar and Mamta Mittal and Sudipta Roy and Tai-Hoon Kim},
  title = {{Video Caption Based Searching Using End-to-End Dense Captioning and Sentence Embeddings}},
  booktitle = {Symmetry},
  year = {2020},
  url = {https://openreview.net/forum?id=bByTyklXA9},
}
Figure