The ester 2 evaluation campaign for the rich transcription of French radio broadcasts

Sylvain Galliano, Guillaume Gravier, Laura ChaubardInterspeech 2009

TL;DR

The ESTER 2 campaign evaluated automatic systems for transcribing and analyzing French radio broadcasts, pushing the boundaries of speech recognition and information extraction on diverse audio content. This effort significantly advanced the state-of-the-art in French broadcast transcription and laid the groundwork for future research by creating valuable annotated resources.

What Problem Does This Paper Address?

Accurate and comprehensive transcription of spoken language, especially from broadcast media, is a significant challenge. While read speech transcription has seen progress, analyzing spontaneous speech from radio broadcasts, with its varied speaking styles, accents, and background noise, is considerably harder. Furthermore, extracting meaningful information (like named entities) from these transcripts adds another layer of complexity. The ESTER 2 campaign aimed to address this by evaluating systems on a broad range of French radio content, moving beyond simple transcription to include tasks like sound event detection and named entity recognition, and providing a standardized benchmark for progress.

Methods

The ESTER 2 evaluation campaign was structured around three main categories of tasks:

  • Segmentation (S): This involved identifying and tracking different audio events. Specific tasks included:
  • Sound Event Tracking (SES): Differentiating between speech, music, and other sounds, both in foreground and background.
  • Speaker Diarization (SRL): Segmenting audio into speaker turns and grouping segments by the same speaker, without prior knowledge of who the speakers are.
  • Speaker Tracking (SVL): (No submissions received for this task).
  • Transcription (T): This is the core task of converting audio speech into normalized orthographic text. The primary metric was Word Error Rate (WER).
  • Information Extraction (E): This focused on extracting higher-level semantic information from the transcribed text. The main task was:
  • Named Entity Detection (NE): Identifying and classifying entities like persons, locations, and organizations within the transcripts.

The campaign also explored prospective tasks, such as overlapping speech detection and transcription with contemporary data, but these were not fully implemented due to protocol definition difficulties or lack of participant interest.

Datasets

The campaign utilized several datasets:

  • Training Data:
    • ESTER 2 Corpus: 150 hours of manually transcribed French radio broadcast news (100 hours rich transcription, 50 hours rapid transcription) recorded between 1999-2003.
  • EPAC Project Data: 45 hours of semi-manually transcribed radio broadcast news data, focusing on spontaneous speech.
  • ESTER 1 Corpus: 100 hours of manually transcribed radio broadcast news from 1998-2004.
  • Le Monde Newspaper Corpus: Approximately 450 million words from articles published between 1987-2006, used for information extraction tasks.
  • Development Set: 6 hours of radio broadcast news recorded in 2007.
  • Test Set: 7 hours of radio broadcast shows recorded from January to February 2008, including news, debates, and entertainment programs from various French and French-speaking African radio stations.

Participants were allowed to use any data recorded prior to January 2008.

Key Insights and Ideas

  • Diversity is Key: The ESTER 2 campaign deliberately included a wider variety of speaking styles and accents (including strong African accents) and broadcast types (beyond just news) compared to its predecessor. This highlighted the significant performance drop when systems encounter unrepresented linguistic variations.
  • Transcription Errors Cascade: The performance of Named Entity Detection (NE) was drastically impacted by transcription errors. This underscores the critical role of accurate Automatic Speech Recognition (ASR) as a front-end for information extraction tasks.
  • Community Building: The campaign fostered a strong community of French-speaking labs interested in rich transcription, with many participants being funded by their own institutions, demonstrating the value placed on evaluation benchmarks.
  • Resource Creation: A major outcome is the creation of valuable resources, including annotation guidelines, protocols, scoring tools, and hundreds of hours of transcribed and untranscribed radio broadcast speech, which will be made publicly available.

Key Results

  • Segmentation Tasks:
    • Sound Event Tracking (SES): Systems performed well on speech detection (very low miss rates), likely due to tuning for transcription. Music detection was more challenging, especially for background music.
  • Speaker Diarization (SRL): Performance varied significantly depending on the show's characteristics (length, number of speakers, accent), with error rates ranging from 0.22% to 23.86% for the best systems.
  • Transcription Tasks (TRS): Most systems showed improvement over ESTER 1, despite the ESTER 2 test data being more challenging (more spontaneous speech, accents, background noise). Word Error Rates (WER) varied, with the best systems achieving around 10.8% on average. Performance was notably worse on broadcasts with stronger accents (e.g., Africa 1, TVME) and more spontaneous speech (e.g., debates on Inter).
  • Named Entity Detection (NE): Performance on reference transcripts was good, but transcription errors significantly degraded performance on automatic transcripts. Systems originally designed for standard texts struggled more with transcripts, while those adapted for automatic transcripts showed better robustness.

Limitations and Future Work

  • Prospective Task Challenges: Several promising prospective tasks (e.g., overlapping speech detection, sentence segmentation) could not be fully implemented due to difficulties in defining clear evaluation protocols.
  • Accent and Spontaneity Impact: The results clearly showed that system performance is highly sensitive to speaker accent and the degree of speech spontaneity, indicating a need for more robust adaptation techniques.
  • Future Directions: The authors plan to continue broadcast news transcription efforts, incorporating more information extraction tasks like topic detection and question answering. They also aim to organize future campaigns focusing on more challenging content, such as large TV streams encompassing a wider variety of program types beyond news.

Summary updated

BibTeX
@inproceedings{galliano2009ester,
  author = {Sylvain Galliano and Guillaume Gravier and Laura Chaubard},
  title = {{The ester 2 evaluation campaign for the rich transcription of French radio broadcasts}},
  booktitle = {Interspeech},
  year = {2009},
}
Figure