Large scale content analysis engine

David Gibbon, Zhu Liu2009 · DOI

TL;DR

This paper introduces the Large Scale Content Analysis Engine (CAE), a robust and scalable system designed to automatically analyze and enrich vast amounts of video content. The CAE addresses the critical need for detailed, reliable content descriptions to improve video retrieval, repurposing, and personalization in the era of IP video.

What Problem Does This Paper Address?

The explosion of IP video content through platforms like IPTV and the web has created an unprecedented volume of material. However, finding relevant content is hampered by a lack of detailed and consistent descriptions. Manual annotation is costly and inconsistent, making automated content analysis essential. Existing retrieval technologies struggle with this challenge, leading to a need for systems that can process large-scale video archives efficiently and extract rich semantic information for various applications.

Methods

The CAE employs a flexible, multi-server architecture designed for scalability and robustness. Its core functionality is divided into several stages:

  • Content Ingestion: The system supports a wide variety of content sources, including over-the-air broadcasts (ATSC), IPTV, podcasts, and enterprise content from CMS/DAM systems. It handles diverse formats and metadata standards.
  • Media Processing: This is the heart of the CAE, encompassing a suite of computationally intensive tasks:
  • Metadata Processing: Extracting and augmenting information from sources like PSIP, EPG, and external databases.
  • Text Processing: Utilizing Optical Character Recognition (OCR) for closed captions and Large Vocabulary Automatic Speech Recognition (LVASR) for audio transcription, followed by alignment and normalization.
  • Audio Processing: Performing speech recognition and speaker segmentation/clustering to identify distinct voices.
  • Video Processing: Shot boundary detection, keyframe extraction, face detection and clustering, and concept detection using SVMs trained on low-level features.
  • Multi-modal Processing: Combining cues from audio, video, and text to derive higher-level descriptors like major cast detection and story segmentation.
  • Video Transcoding: Converting content into various formats (e.g., MP3, WMV, MPEG4) for different devices and bandwidths.
  • Scalable Processing Control: A centralized database and a client processing controller (CPC) manage the status and workflow of ingestion and processing servers, ensuring efficient load balancing and automatic reprocessing of failed content.
  • Content Description: Processed results are represented in XML, with support for standards like MPEG-7 and MediaRSS, enabling interoperability and rich semantic descriptions.
  • Client Interfaces: The CAE provides interfaces for IPTV and mobile devices, allowing for interactive browsing and search of processed content.

Datasets

The paper mentions participation in the TRECVID 2007 shot boundary detection evaluation. It also describes pilot content collections (U1, U2, U3, M, P) which are continuous acquisitions from various sources like MCE, Linux, IP multicast, selected series, and podcasts. These collections are used for evaluating the system's performance and the effectiveness of metadata normalization.

Key Insights and Ideas

  • Scalability is Paramount: The paper emphasizes the need for a scalable, distributed architecture to handle the sheer volume of IP video. The multi-server approach with a central control database is key to this.
  • Multi-modal Fusion is Crucial: Extracting meaningful insights requires combining information from different modalities (audio, video, text). This fusion enables higher-level understanding like story segmentation and major cast detection.
  • Standardization for Interoperability: Using XML and supporting standards like MPEG-7 and MediaRSS is vital for integrating the CAE with other systems and enabling broad application of the analyzed content.
  • Leveraging Existing Metadata: The CAE effectively augments existing metadata (like EPG and closed captions) with automatically extracted information, creating richer descriptions.
  • Flexibility through Profiles and Workflows: The system's ability to adapt processing based on application-specific profiles and web service-driven workflows makes it highly versatile.

Key Results

  • Shot Boundary Detection: The system achieved strong results in the TRECVID 2007 evaluation, with an overall recall of 85.5% and precision of 89.2% for shot boundary detection. Specifically, for cut boundaries, recall was 88.9% and precision was 90.4%.
  • Content Collections: The paper summarizes the size of several pilot content collections, with the largest (U2) containing 34,000 assets and 25,000 hours of video.
  • Real-time Factor: Several computationally intensive components were evaluated for their real-time factor. For example, content-based sampling had a factor of 0.80, and face detection was 0.07, indicating efficient processing.

Limitations and Future Work

  • Accuracy Variability: The paper acknowledges that the accuracy of speech recognition can vary significantly depending on the quality of the speech.
  • Metadata Consistency: While the system attempts metadata normalization, it notes that the use of genre and category names is not entirely consistent across various sources.
  • Future Work: The paper implicitly suggests future work in refining the accuracy of individual processing components, expanding the range of supported content sources and metadata standards, and developing more sophisticated multi-modal analysis techniques. The focus on scalable processing control and flexible workflows also points towards continued development in system management and application integration.

Summary updated

Abstract

The evolution of IP video systems has resulted in unprecedented access to a wide range of video material for consumers via IPTV and Web delivery. Retrieval technologies help users find relevant content, but suffer from a paucity of reliable content descriptions. In this paper, we describe the large scale content analysis engine (CAE) which is designed to facilitate content based video retrieval, as well as a number of other applications such as content repurposing, video browsing, discovering content relationships, and fine-grained content personalization. A scalable system is presented to handle video from broadcast, enterprise and web sources for applications including IPTV and mobile video. Media processing includes shot boundary detection, automatic speech recognition, face detection, clustering of shots and faces, speaker segmentation, closed caption alignment, concept detection, and transcoding. Metadata ingest, augmentation and delivery using a standards-based approach enable the CAE to serve a range of content processing applications.
BibTeX
@misc{gibbon2009large,
  author = {David Gibbon and Zhu Liu},
  title = {{Large scale content analysis engine}},
  year = {2009},
  doi = {10.1145/1631058.1631076},
}

Loading related content...

Figure