Query-by-Emoji Video Search

Spencer Cappallo, Thomas Mensink, Cees G. M. Snoek

Abstract

This technical demo presents Emoji2Video, a query-by-emoji interface for exploring video collections. Ideogram-based video search and representation presents an opportunity for an intuitive, visual interface and concise non-textual summary of video contents, in a form factor that is ideal for small screens. The demo allows users to build search strings comprised of ideograms which are used to query a large dataset of YouTube videos. The system returns a list of the top-ranking videos for the user query along with an emoji summary of the video contents so that users may make an informed decision whether to view a video or refine their search terms. The ranking of the videos is done in a zero-shot, multi-modal manner that employs an embedding space to exploit semantic relationships between user-selected ideograms and the video's visual and textual content.
BibTeX
@inproceedings{cappallo2015query,
  author = {Spencer Cappallo and Thomas Mensink and Cees G. M. Snoek},
  title = {{Query-by-Emoji Video Search}},
  booktitle = {ACM Multimedia},
  year = {2015},
  doi = {10.1145/2733373.2807961},
}

Loading related content...

Figure