The broadcast narrow band speech corpus: a new resource type for large scale language recognition

Christopher Cieri, Linda Brandschain, Abby Neely, David Graff, Kevin Walker, Chris Caruso, Alvin F. Martin, Craig S. GreenbergInterspeech 2009

TL;DR

This paper introduces the Broadcast Narrow Band Speech (BNBS) corpus, a novel resource designed to facilitate large-scale language recognition research. By leveraging readily available broadcast news, the BNBS corpus offers a cost-effective alternative to traditional telephone speech corpora, enabling broader language coverage and more robust system development.

What Problem Does This Paper Address?

Developing accurate language recognition systems, particularly for a wide array of languages, has historically relied on conversational telephone speech (CTS) corpora. However, collecting CTS is expensive and time-consuming, limiting the scale and diversity of available datasets. This paper addresses the need for a more economical and scalable data source that can support the development and evaluation of language recognition technologies for a large number of languages and speakers. The authors aim to demonstrate that broadcast news, specifically narrow-band audio embedded within news programs, can serve as a viable and cost-effective alternative to CTS.

Methods

The creation of the BNBS corpus involved several key steps:

  • Source Data Collection: The primary source of data was Voice of America (VOA) broadcasts, chosen for their multilingual programming aimed at large international audiences. This approach helps minimize the correlation between signal properties and linguistic features.
  • Segmentation: Raw audio streams were processed to identify segments of speech. This involved:
  • Speech activity detection to distinguish speech from music and silence.
  • Narrow-band vs. wide-band signal classification.
  • Consolidation of these signals to identify the largest contiguous segments of narrow-band speech.
  • Discarding segments shorter than 33 seconds.
  • Extracting a precise 30-second segment from the center of each 33-second segment to ensure consistent duration and minimize transition artifacts.
  • Auditing: A crucial step involved a rigorous auditing process to ensure data quality and establish ground truth. This process included:
  • Remote Auditing Interface: A web-based interface was developed to allow remote auditors to listen to audio clips and provide judgments.
  • Auditor Training: Auditors were trained to distinguish between narrow-band and wide-band speech and to assess audio quality.
  • Judgment Criteria: Auditors evaluated segments for:
  • Presence of only speech (no music/sound effects).
  • Speech exclusively in the target language.
  • "Telephone-like" quality.
  • Single speaker presence.
  • Speaker uniqueness (not heard previously).
  • Speaker sex, dialect (native, non-native, dialect), and comments.
  • Signal quality (clarity, noise levels).
  • Language vs. Dialect Handling: The paper acknowledges the ambiguity in defining languages and dialects and opts to respect the labels provided by the broadcast sources, using the term "variety" to remain agnostic.

Datasets

The primary dataset introduced is the Broadcast Narrow Band Speech (BNBS) Corpus. This corpus is derived from Voice of America (VOA) broadcasts and is intended to support large-scale language recognition technology development and evaluation. It aims to provide a cost-effective alternative to conversational telephone speech (CTS) corpora like CALLFRIEND. The BNBS corpus is planned to support blind language recognition technology evaluations, with details about specific languages withheld for evaluation purposes. The paper mentions that the corpus is based on VOA data collected in three distinct efforts: VOA1, VOA2, and VOA3.

Key Insights and Ideas

  • Broadcast News as a Scalable Resource: The paper highlights that broadcast news is a rich and readily available source of narrow-band speech that can significantly reduce the cost and increase the scale of language recognition data collection compared to traditional methods like CTS.
  • Cost-Effectiveness: By leveraging existing broadcast infrastructure and a streamlined collection and auditing process, the BNBS corpus offers a more economical path to acquiring large multilingual speech datasets.
  • Language vs. Dialect Nuance: The authors acknowledge the inherent difficulty in definitively classifying linguistic varieties and adopt a pragmatic approach by respecting the labels provided by broadcast sources, using the term "variety" to maintain neutrality.
  • Auditing for Quality Control: The detailed auditing process, including remote access and specific judgment criteria, is crucial for ensuring the reliability of the ground truth data, which is essential for training and evaluating language recognition systems.
  • Speaker Uniqueness and Segment Length: The strategy of extracting 30-second segments from the center of longer recordings and auditing for speaker uniqueness is a cost-effective way to maximize the number of distinct speaker samples and avoid over-representation of a single speaker.

Key Results

  • The BNBS corpus has yielded adequate segments to support NIST LRE evaluations for 23 languages and 16 "out of set" languages.
  • The corpus contains over 11,000 hours of speech in nearly 70 languages.
  • Auditing resulted in 18,721 database records, with 17,985 used for analysis.
  • The average time to audit a 30-second segment was 42 seconds.
  • Auditors agreed with their prior decisions concerning the language of a segment 99.4% of the time.
  • For segments audited by two different auditors for mutually intelligible varieties, agreement rates were high (e.g., 99.5% for French auditors).

Limitations and Future Work

  • Comparability to CTS: The paper notes that clear measures of the comparability of BNBS to CTS used in previous LRE evaluations are not yet available, though early results from LRE 2009 have been encouraging.
  • Auditor Bias: While efforts were made to ensure auditor reliability, the paper acknowledges that auditors are not necessarily linguists and may not be native speakers of every dialect. Differences in auditor experience (e.g., bilingualism) can lead to variations in agreement rates for certain language pairs.
  • Future Work: The authors plan to release the VOA2 and VOA3 portions and the exposed subset of VOA1 after their use in the LRE evaluation campaign. The long-term usefulness of the BNBS corpus for language recognition technology development and evaluation is yet to be fully determined, but its potential to support a variety of Human Language Technologies (HLTs) is significant.

Summary updated

BibTeX
@inproceedings{cieri2009broadcast,
  author = {Christopher Cieri and Linda Brandschain and Abby Neely and David Graff and Kevin Walker and Chris Caruso and Alvin F. Martin and Craig S. Greenberg},
  title = {{The broadcast narrow band speech corpus: a new resource type for large scale language recognition}},
  booktitle = {Interspeech},
  year = {2009},
}
Figure