Pavideoge: A New Video Processing Method in Video Search Engine

Pu Yang, Jun Guo, Guang ChenarXiv 2009 · arXiv

Abstract

Traditional video search engines often struggle with the inherent complexity of video data, typically relying on sparse metadata such as titles and tags. This paper introduces Pavideoge, a novel data structure designed to treat videos more like interconnected webpages. By extracting "hidden" properties—including direct file links, real-time play counts, and a structural "Videorank"—the Pavideoge framework provides a richer context for indexing. Experimental results on a large-scale dataset from Tudou.com demonstrate a significant improvement in search precision compared to traditional keyword-matching methods.

Introduction

Searching for video content presents unique challenges that distinguish it from text-based web search. The authors identify three primary hurdles:

  1. Storage Constraints: The massive size of video files prevents crawlers from indexing full content in real-time.
  2. Structural Independence: Unlike the web's interconnected nature via hyperlinks, videos often exist as "islands" without inherent relationships.
  3. Information Opacity: Critical data is often locked within binary files, forcing search engines to rely on brief, often inaccurate, user-provided metadata.
Traditional search engines like Baidu and Sogou rely heavily on limited title and tag metadata.

As shown in Figure 1, commercial engines historically focused on simple keyword matching within titles and tags, which fails to capture the full relevance of a video to a user's query.

The Pavideoge Framework

The core of this research is the Pavideoge (Page-Video) structure, which transforms a video into a multi-dimensional metadata object. This structure is built upon four key attributes:

  • Real Link: The system simulates server-side actions to identify the direct URL of the video file, bypassing the limitations of the standard player UI.
  • Playnum (Popularity): By mimicking the requests made by video players, the system retrieves dynamic popularity data (play counts) that are typically invisible to standard crawlers.
  • Videorank: Inspired by Google’s PageRank, this algorithm treats "related video" recommendations on playpages as virtual hyperlinks. This allows the system to build a network graph and calculate the relative importance of a video within the site's ecosystem.
  • Rich Text Information: The parser extracts not just titles, but also tags, descriptions, and user comments from the entire playpage to provide a broader linguistic context.

System Architecture

The implementation of Pavideoge involves a sophisticated pipeline that integrates crawling, parsing, and ranking.

The Pavideoge system architecture, illustrating the flow from web crawling to the generation of search results.

As illustrated in Figure 3, the crawler feeds data into a repository, where the Videorank Generator and Parser work in tandem to populate the Pavideoge structure. This information is then indexed and used to generate search results based on a weighted relevance score.

Experimental Evaluation

The researchers evaluated the Pavideoge method using a dataset of 80,000 videos crawled from Tudou.com. The performance was measured against a traditional baseline that indexed only titles and tags.

The system was tuned using 60 training queries to determine the optimal "effect factors" for balancing text relevance, popularity, and Videorank. In testing across 300 queries, the Pavideoge method showed a clear advantage:

  • Average Precision Gain: The method achieved an overall improvement of approximately 4.9%.
  • Peak Performance: In specific test groups, precision increased by as much as 14.7%.

Conclusion and Future Work

Pavideoge represents a significant step toward making video content as searchable as text. By simulating player behavior and leveraging recommendation networks, the system overcomes the "island" problem of video data. Future research will focus on developing more generalized parsers for diverse video platforms and investigating dynamic weight adjustment for the various ranking factors.

Summary updated

BibTeX
@misc{yang2009pavideoge,
  author = {Pu Yang and Jun Guo and Guang Chen},
  title = {{Pavideoge: A New Video Processing Method in Video Search Engine}},
  year = {2009},
  eprint = {0909.2496},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/0909.2496},
}
Figure