A Multidisciplinary Approach to Unlocking Television Broadcast Archives

Laura Hollink, Guus Schreiber, Bouke Huurnink, M. van Liempt, Maarten de Rijke, A.w.m. Smeulders, Johan Oomen, Annemieke de JongInterdisciplinary 2009 · DOI

TL;DR

This paper presents the MuNCH project, which tackles the challenge of accessing vast television broadcast archives by integrating multimedia analysis, language technology, and semantic technologies. By developing automatic methods for content analysis and enriching existing metadata, the project aims to significantly improve search capabilities for users.

What Problem Does This Paper Address?

Television broadcast archives, like the Netherlands Institute for Sound and Vision (NISV), hold immense cultural heritage but are notoriously difficult to access. While digitization has made content available online, the sheer volume of material and the limitations of manual annotation create a "semantic gap" – a disconnect between the raw digital data and its meaningful interpretation for users. Traditional manual cataloging is time-consuming and cannot keep pace with the influx of new content, leading to incomplete descriptions and an "annotation bias" towards professional users. This leaves the burden of search on users who may lack the specialized knowledge to navigate these complex archives effectively.

Methods

The MuNCH project employs a multidisciplinary approach, combining three key research fields:

  • Multimedia Analysis: This involves automatically detecting objects and scenes within video content. The paper details the development of "concept detectors," which are machine-learned algorithms trained to identify specific concepts (e.g., "car," "tree," "person") in video shots. This is achieved by extracting low-level visual features (color, texture, edges) and then training classifiers (Support Vector Machines) on labeled examples.
  • Language Technology: This focuses on leveraging textual information, particularly automatic speech transcriptions. The paper highlights the challenge of "temporal mismatch" between spoken words and their visual occurrence in video and proposes methods to utilize this information for search.
  • Semantic Technologies: This area is used to bridge the semantic gap by enriching existing metadata and improving query understanding. This includes:
  • Concept Selection: Developing methods to automatically select the most appropriate concept detector for a given user query, comparing text matching, ontology querying, and semantic visual querying.
  • Thesaurus Enrichment: Enhancing the NISV's in-house thesaurus (GTAA) by mapping its terms to external knowledge resources like WordNet. This allows for inferring new semantic relationships between terms, enabling query expansion and improving search recall.

Datasets

  • Netherlands Institute for Sound and Vision (NISV) Archive: This serves as the primary source of audiovisual material and existing manual annotations for the project.
  • TRECVID Benchmark Data: 400 hours of video material from NISV were provided for the TRECVID (Text REtrieval Conference Video) benchmark, which includes queries and ground truth data. This data was crucial for evaluating the developed techniques in a standardized, large-scale experimental setting.

Key Insights and Ideas

  • The Semantic Gap is Real and Pervasive: The core challenge is translating low-level visual and audio data into high-level semantic understanding that users can query.
  • Multidisciplinarity is Key: No single discipline can solve the problem of archive access. Combining multimedia analysis, language technology, and semantic technologies creates synergistic solutions that are more powerful than individual approaches.
  • User Needs Drive Research: Understanding how real users search (through logging systems and analysis of query patterns) is essential for developing effective retrieval systems.
  • Automatic Annotation is Crucial for Scale: Manual annotation is insufficient for large archives. Automatic methods, while not perfect, are necessary to provide broader and deeper access.
  • Thesaurus Enrichment Offers Significant Potential: Leveraging external semantic knowledge to enrich internal thesauri can dramatically improve search recall by expanding queries and connecting related concepts.
  • Temporal Mismatch in Speech and Video Can Be Exploited: The delay between mentioning an object in speech and its visual appearance can be a valuable signal for improving search.

Key Results

  • Concept Detectors: The project developed a growing lexicon of approximately 500 concept detectors for news broadcasts and documentaries, demonstrating the feasibility of automatic object and scene recognition.
  • Concept Selection Methods: Text matching, ontology querying, and semantic visual querying showed similar performance for a set of 24 search queries, suggesting that a combination of these methods would be beneficial for robust retrieval.
  • Thesaurus Enrichment: The enrichment of the GTAA thesaurus with inferred relations from WordNet resulted in a comparable, and in some cases improved, retrieval performance compared to using the original thesaurus alone. This suggests that external semantic resources can effectively augment local vocabularies.
  • Logging System: A query logging module was developed to monitor user behavior, providing valuable insights into user needs and search patterns, which directly informed the project's research directions.

Limitations and Future Work

  • Concept Detector Imperfection: The paper acknowledges that concept detectors are not absolute and vary in quality, with some being rarely perfect. More research is needed to improve their accuracy and generalization.
  • Scalability of Multimedia Analysis: Current multimedia analysis techniques demand significant processing power, posing a challenge for institutes with massive archives. Future work will focus on scaling these techniques for efficiency.
  • Human-Computer Interaction: The complexity of advanced search systems needs to be managed through intuitive user interfaces that match users' cognitive skills. Future work will address Human-Computer Interaction issues.
  • Peer-to-Peer and Web 2.0 Technologies: The paper notes growing interest in these technologies for filtering, recommending, and tagging video content, suggesting future research directions in this area.
  • Analysis of User Search Behavior: Further research into user search behavior, potentially using Information Foraging Theory, is planned to better support browsing and diverse search mechanisms.
  • Combining Multiple Information Channels: A key future direction is integrating information from various sources (manual annotations, automatic detections, user data) into a unified system.

Summary updated

Abstract

Audiovisual material is a vital component of the world's heritage but it remains difficult to access. With the Netherlands Institute for Sound and Vision as one of its partners, the MuNCH project aims to investigate new methods for improving access to a wide range of audiovisual documents. MuNCH brings together three research fields: multimedia analysis, language technology and semantic technologies.Within the MuNCH project, we have investigated several combinations of these fields. We have compared text matching, ontology querying, and semantic visual querying as methods to translate a multimedia query to the vocabulary of the retrieval system. In addition, we have investigated how users make such a translation, and have used this as a benchmark to create automatic methods. We have used multimedia technology to automatically detect objects and scenes as they occur in video, and made use of language technology to exploit automatic transcriptions of speech. We have enriched the Sound and Vision thesaurus that is used to annotate the TV programmes in order to provide a user with a wider range of search results.In order to verify the results of the project against real user needs, MuNCH has participated in the creation of a logging system which monitors the usage of the Sound and Vision catalogue system. Insights in the needs of real users will be used as input for all three of MuNCH's research strands.
BibTeX
@inproceedings{hollink2009multidisciplinary,
  author = {Laura Hollink and Guus Schreiber and Bouke Huurnink and M. van Liempt and Maarten de Rijke and A.W.M. Smeulders and Johan Oomen and Annemieke de Jong},
  title = {{A Multidisciplinary Approach to Unlocking Television Broadcast Archives}},
  booktitle = {Interdisciplinary},
  year = {2009},
  doi = {10.1179/174327909x441144},
}
Figure