TL;DR
Researchers have created the Spotify Podcast Dataset, a massive corpus of 100,000 English-language podcasts, offering nearly 60,000 hours of audio and transcripts. This dataset addresses the lack of large-scale, diverse spoken language resources and serves as a benchmark for challenging NLP tasks like passage retrieval and summarization, opening new avenues for research in spoken language understanding.
What Problem Does This Paper Address?
The field of spoken language processing has historically relied on relatively small, often formal, and domain-specific corpora. This limits research into the nuances of natural, spontaneous speech found in everyday communication. Podcasts, a rapidly growing and diverse medium, represent a rich but understudied source of spoken language. They encompass a wide range of genres, speaking styles, and structural formats, often featuring unscripted discourse, disfluencies, and a mix of monologues and conversations. Existing datasets lack the scale and variety to adequately capture these complexities, hindering the development of robust NLP models for tasks like search and summarization in real-world spoken content.
Methods
The core contribution is the creation and release of the Spotify Podcast Dataset. This involved:
- Data Collection: Sampling 105,360 podcast episodes from the Spotify platform, published between January 2019 and March 2020.
- Filtering: Applying several criteria to ensure data quality and relevance, including language (English), episode length (under 90 minutes), and speech presence (at least 50% speech).
- Transcription: Utilizing Google's Cloud Speech-to-Text API for automatic speech recognition (ASR) to generate word-level transcripts with time alignments, speaker diarization, casing, and punctuation. While ASR errors are acknowledged (18.1% word error rate), the authors argue the transcripts are still valuable, especially when paired with the audio.
- Benchmark Tasks: Developing and evaluating two key NLP tasks on this corpus:
- Spoken Passage Retrieval: Identifying short, relevant segments within podcast episodes based on arbitrary queries.
- Summarization: Condensing podcast episodes into shorter forms, exploring both extractive and abstractive approaches.
Datasets
- Spotify Podcast Dataset: The primary contribution, comprising 100,000 podcast episodes (nearly 60,000 hours of audio) with automatically generated transcripts and metadata.
- Brass Set: A subset of 66,245 podcast descriptions filtered for quality and suitability as reference summaries for summarization tasks.
- Existing Datasets (for comparison/context):
- TIMIT, Broadcast News Corpora (for formal speech)
- ATIS, Switchboard, CALLHOME, Santa Barbara Corpus, TED Talks (for conversational/spoken data)
- CNN/DailyMail, AMI, ICSI meeting corpora (for summarization datasets)
Key Insights and Ideas
- Podcast Diversity is a Challenge and Opportunity: The sheer variety in podcast content, speaking styles, and production quality presents significant challenges for NLP tasks. However, this diversity also makes it an ideal testbed for developing more robust and generalizable models that can handle real-world spoken language.
- ASR Noise is Manageable but Impactful: While the automatic speech recognition (ASR) for the corpus has a higher word error rate than ideal, the authors argue it's low enough to be useful. However, ASR errors can significantly impact downstream tasks like named entity recognition and retrieval, requiring models to be more robust to such noise.
- Creator Descriptions are Noisy References: Using creator-provided descriptions as "gold standard" summaries for training and evaluation is problematic due to their variable quality and often non-summative intent. This necessitates careful filtering (e.g., the "Brass Set") and acknowledges that current summarization results might be lower than on cleaner datasets.
- Abstractive Summarization Shows Promise: Abstractive models (like BART-PODCASTS) generally outperform extractive methods (like TextRank) and simple baselines, suggesting they are better at generalizing over ASR errors and capturing the overall essence of an episode, even if they sometimes miss specific details.
Key Results
- Passage Retrieval: Standard retrieval models (BM25, QL) were evaluated. While no single system showed statistically significant superiority, Query Likelihood (QL) models generally performed better than BM25, and both performed better without the RM3 relevance model. The performance varied significantly across different query types.
- Summarization:
- ROUGE scores for the dataset were lower than typical news summarization benchmarks, attributed to noisy transcripts, informal language, and the use of creator descriptions as references.
- Human evaluations showed that the fine-tuned BART-PODCASTS model performed nearly as well as, and in some cases better than, the creator-provided descriptions.
- Abstractive models (BART-CNN, BART-PODCASTS) were rated significantly higher than unsupervised extractive methods (TextRank, First Minute baseline) by human assessors.
Limitations and Future Work
- Limitations:
- The dataset is currently restricted to the English language.
- The automatic speech recognition transcripts are noisy, impacting downstream task performance.
- Creator descriptions used as reference summaries are of variable quality and not always intended as summaries.
- The current evaluation of summarization models relies on these noisy creator descriptions.
- Future Work:
- Extend the dataset to include more languages.
- Leverage the audio data directly for tasks beyond transcript-based processing, such as end-to-end summarization or audio retrieval.
- Explore speaker identification and role prediction due to the multi-speaker nature of podcasts.
- Investigate opinion mining and discourse analysis, given the rich variety of topics and conversational styles.
- Develop models that consider linguistic style in addition to topical content.
- Further analysis using larger, manually annotated human label sets for summarization evaluation.
Summary updated