Towards automated annotation of audio and video recordings by application of advanced web-services

Przemyslaw Lenkiewicz, Dieter van Uytvanck, Peter Wittenburg, Sebastian DrudeInterspeech 2012

TL;DR

This paper introduces the AVATecH project's suite of automated audio and video analysis tools, designed to drastically reduce the time and effort required for manual annotation of multimedia research data. By leveraging advanced web services, these tools provide semi-automated pre-annotations, enabling researchers to achieve annotation speeds up to 50% faster with high-quality results.

What Problem Does This Paper Address?

Annotating audio and video recordings for research purposes, particularly in fields like linguistics, speech recognition, and human-machine interfaces, is a crucial but incredibly time-consuming and labor-intensive task. This is due to the sheer volume of data and the complexity of human communication, which often involves subtle nuances and diverse conditions. Existing methods for automated processing are either insufficient or not widely available, creating a significant bottleneck for researchers. The AVATecH project aims to bridge this gap by developing and deploying advanced web-services that automate parts of this annotation process, making it more efficient and accessible.

Methods

The AVATecH project employs a modular approach, developing a suite of "recognizers" that can be chained together via web services to create semi-automated annotation pipelines. These recognizers fall into several categories:

  • General Recognizers: These assist with navigation and identifying relevant segments within recordings. This includes "acoustic query by example" for finding similar audio clips, "video shot/sub-shot detection and key-frame extraction" for creating visual storylines, and "camera motion detection" to highlight significant visual events.
  • Audio Recognizers: These focus on segmenting and classifying audio streams. Key functionalities include:
  • Audio Segmentation: Noise-robust segmentation into homogeneous regions, identifying boundaries between speakers or significant acoustic changes using Bayesian information criterion.
  • Speech/Non-Speech Classification: Language-independent identification of human speech segments.
  • Speaker Diarization: Language-independent clustering of identical speakers within a document.
  • Vowel and Pitch Contour Detection: Graphical specification and search for typical pitch contours.
  • Video Recognizers: These aim to segment and analyze gesturing actions. This involves:
  • Skin Color Estimation: An iterative algorithm to identify human skin regions.
  • Hands and Head Tracking: Utilizing skin color estimation to detect and track faces and hands, enabling gesture analysis based on movement speed, acceleration, and orientation.
  • Gesture Space Projection: Projecting hand positions onto a gesture space grid (McNeill [4]).

These recognizers are exposed as RESTful web services, compatible with frameworks like WebLicht and Taverna, and described using WADL and CMDI.

Datasets

The paper does not explicitly mention specific datasets used for training the recognizers. However, it describes experiments conducted on "field recordings" and "short video recordings" that represent typical conditions encountered by researchers in linguistics and gesture analysis. The experiments focus on evaluating the efficiency of the annotation process itself, rather than on achieving specific accuracy metrics on pre-defined datasets.

Key Insights and Ideas

  • Web Services as a Unifying Framework: The paper highlights the power of using web services to make complex multimedia analysis tools accessible and combinable, aligning with the UNIX philosophy of simple, powerful modules.
  • Significant Time Savings: The core idea is that semi-automation, even with the need for human correction, can dramatically reduce annotation time compared to fully manual methods.
  • Modularity for Flexibility: The suite of recognizers allows researchers to build custom annotation pipelines tailored to their specific research questions and data types.
  • Bridging the Gap for Researchers: By automating tedious tasks, AVATecH empowers researchers to focus on higher-level analysis and interpretation, accelerating scientific discovery.

Key Results

The paper presents experimental results demonstrating significant efficiency gains:

  • Audio Annotation: In an audio segmentation and speaker classification experiment, the automated scenario reduced annotation time by approximately 50% (from 1800 minutes to 780 minutes for a 40-minute recording), achieving an annotation speed increase of 0.49% (from 309 blocks/minute to 585 blocks/minute).
  • Video Annotation: For video gesture annotation, the recognizer-supported scenario resulted in an annotation time that was only 28% of the time required for fully manual annotation.

Limitations and Future Work

  • Human Correction Still Necessary: The authors acknowledge that automated annotations are "never flawless" and require verification and correction by human annotators.
  • Transferring Large Multimedia Objects: The paper notes that the feasibility of transferring large multimedia objects via web services is still under study, with CLARIN collaborating with EUDATS on pilot experiments.
  • Dependence on Web Service Availability: Users of these web services are dependent on the availability and service level offered by the hosting parties.
  • Future Work: The paper suggests further investigation into the feasibility of transferring large multimedia objects and the potential for more complex annotation scenarios.

Summary updated

BibTeX
@inproceedings{lenkiewicz2012towards,
  author = {Przemyslaw Lenkiewicz and Dieter van Uytvanck and Peter Wittenburg and Sebastian Drude},
  title = {{Towards automated annotation of audio and video recordings by application of advanced web-services}},
  booktitle = {Interspeech},
  year = {2012},
}
Figure