Kuaipedia: a Large-scale Multi-modal Short-video Encyclopedia

Haojie Pan, Yuzhou Zhang, Zepeng Zhai, Ruiji Fu, Ming Liu, Yangqiu Song, Zhongyuan Wang, Bing QinarXiv 2022 · DOI · arXiv

Introduction

Traditional encyclopedias like Wikipedia are highly effective at conveying "know-what" facts through text and static images. However, they often fall short when it comes to "know-how" (skills) or "know-why" (complex processes). For instance, while a text entry can describe a Shiba Inu's origin, a short video is far more effective at demonstrating how to train one or even how to draw a stylized version of one.

An example of a 'know-how' video title: 'Teach you to draw Shiba Inu buns in one minute'.

With the explosion of short-video platforms like TikTok and Kuaishou, a vast amount of procedural knowledge is being generated daily. Kuaipedia addresses the challenge of structuring this "hidden" knowledge by linking over 200 million videos to 26 million entities and their specific sub-topics, or "aspects."

Methodology: Building the Multi-modal Encyclopedia

The construction of Kuaipedia involves a three-stage pipeline designed to filter, categorize, and link video content to a structured knowledge base.

1. Knowledge Video Detection

To distinguish educational content from pure entertainment, a multi-modal BERT-based classifier is employed. This model processes multiple data streams—including video captions, cover images, OCR text (on-screen text), and ASR (speech-to-text)—to identify "knowledge-dense" clips.

2. Item-Aspect Mining

A core innovation of Kuaipedia is the use of "aspects" to bridge broad entities and specific videos. Rather than using static categories, the system mines aspects from millions of user search queries. This ensures the encyclopedia reflects real-world utility. For an item like "Shiba Inu," aspects might include "feeding," "training," or "how-to-draw."

3. Multi-modal Item-Aspect Linking

The final stage links specific videos to Item-Aspect pairs using a two-phase Recall and Ranking model. The ranking module utilizes a task-specific prompt: "Is the video about [Item]’s [Aspect]?" evaluating the relevance of visual and textual features against the knowledge tree.

A specific procedural step extracted from a knowledge video, showing detailed instructions for drawing.

Data and Scale

Kuaipedia leverages data from Kuaishou, one of China's largest short-video platforms, and uses CN-Wikipedia and CN-DBpedia as baseline entity sources. The resulting benchmark is massive, containing over 26 million items, 2.5 million aspects, and 200 million knowledge videos, making it significantly larger than existing Chinese text-based knowledge bases.

Key Results and Impact

  • Factuality in LLMs: Kuaipedia significantly reduces hallucinations in Large Language Models (LLMs). By providing multi-modal context (OCR/ASR text), researchers improved ChatGPT's accuracy on specific factual queries from 0% to 100% in targeted test cases.
  • Enhanced AI "Imagination": Feeding Kuaipedia's video frames and ASR text into text-to-image models allows AI to visualize complex actions more accurately than text prompts alone.
  • Performance Gains: The inclusion of Kuaipedia data improved F1 scores in traditional NLP tasks such as Entity Typing (from 97.43 to 98.12) and Entity Linking (from 79.04 to 81.22).

Conclusion

Kuaipedia represents a significant step forward in knowledge representation by moving beyond text to capture the procedural richness of video. By structuring "know-how" through an item-aspect framework, it provides a valuable resource for both human users and AI systems, grounding abstract concepts in visual reality.

Summary updated

Abstract

Online encyclopedias, such as Wikipedia, have been well-developed and researched in the last two decades. One can find any attributes or other information of a wiki item on a wiki page edited by a community of volunteers. However, the traditional text, images and tables can hardly express some aspects of an wiki item. For example, when we talk about ``Shiba Inu'', one may care more about ``How to feed it'' or ``How to train it not to protect its food''. Currently, short-video platforms have become a hallmark in the online world. Whether you're on TikTok, Instagram, Kuaishou, or YouTube Shorts, short-video apps have changed how we consume and create content today. Except for producing short videos for entertainment, we can find more and more authors sharing insightful knowledge widely across all walks of life. These short videos, which we call knowledge videos, can easily express any aspects (e.g. hair or how-to-feed) consumers want to know about an item (e.g. Shiba Inu), and they can be systematically analyzed and organized like an online encyclopedia. In this paper, we propose Kuaipedia, a large-scale multi-modal encyclopedia consisting of items, aspects, and short videos lined to them, which was extracted from billions of videos of Kuaishou (Kwai), a well-known short-video platform in China. We first collected items from multiple sources and mined user-centered aspects from millions of users' queries to build an item-aspect tree. Then we propose a new task called ``multi-modal item-aspect linking'' as an expansion of ``entity linking'' to link short videos into item-aspect pairs and build the whole short-video encyclopedia. Intrinsic evaluations show that our encyclopedia is of large scale and highly accurate. We also conduct sufficient extrinsic experiments to show how Kuaipedia can help fundamental applications such as entity typing and entity linking.
BibTeX
@misc{pan2022kuaipedia,
  author = {Haojie Pan and Yuzhou Zhang and Zepeng Zhai and Ruiji Fu and Ming Liu and Yangqiu Song and Zhongyuan Wang and Bing Qin},
  title = {{Kuaipedia: a Large-scale Multi-modal Short-video Encyclopedia}},
  year = {2022},
  eprint = {2211.00732},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2211.00732},
}
Figure