Modeling Language Usage and Listener Engagement in Podcasts

Sravana Reddy, Marina Lazarova, Yongze Yu, Rosie Jones

TL;DR

This paper presents the first large-scale quantitative analysis of how linguistic style and textual content influence listener engagement in podcasts. By analyzing transcripts and descriptions from over 5,000 episodes, researchers found that stylistic features alone can predict high-engagement content with 72% accuracy, while state-of-the-art NLP models like BERT reach over 80%.

What Problem Does This Paper Address?

While podcasting has exploded in popularity, advice for creators remains largely anecdotal—think "speak slowly" or "avoid filler words." The authors identify a significant gap: there is very little data-driven research connecting a podcast's linguistic style to whether listeners actually stay tuned.

A major challenge in this domain is defining "engagement." Raw stream counts (popularity) are often skewed by a host’s existing reputation, marketing budgets, or search engine placement. To isolate the effect of the content itself, the authors propose a new metric: stream rate. This is defined as the proportion of first-time listeners who continue listening for at least five minutes. By focusing on this window, the researchers aim to capture the moment a listener decides the content is worth their time, independent of the show's overall fame.

Methods

The researchers employed a multi-pronged approach to model engagement:

  1. Feature Engineering: They defined a set of explainable linguistic features, including vocabulary diversity (entropy), distinctiveness (perplexity), reading grade level (Flesch-Kincaid), sentiment/emotion (using the NRC lexicon and Google Natural Language API), and syntactic patterns (POS tags).
  2. Topic Modeling: They used Latent Dirichlet Allocation (LDA) to identify 100 topics across the corpus to control for genre-specific language norms.
  3. Predictive Modeling: The team compared several architectures:
  4. Logistic Regression using hand-crafted linguistic features and Bag-of-Ngrams.
  5. Feed-Forward Neural Networks using Paragraph Vector representations.
  6. BERT: A fine-tuned "uncased" transformer model to capture deep contextual information.
  7. Comparative Analysis: To see if these patterns were unique to podcasts, they applied the same analysis to a corpus of TED talks, using the ratio of ratings-to-views as a proxy for engagement.

Datasets

  • Spotify Podcast Dataset: A subset of a larger 100,000-episode corpus. After filtering for English-language episodes longer than 10 minutes with sufficient stream data, the final dataset consisted of 5,371 episodes.
  • TED Talks Corpus: 2,480 talks used as a secondary benchmark to identify "universal" vs. "podcast-specific" engagement features.

Key Insights and Ideas

The study yielded several findings that both confirm and challenge conventional podcasting wisdom:

  • Faster is (sometimes) better: Contrary to advice to speak slowly, high-engagement podcasts actually correlate with higher speech rates. This suggests that a brisk, energetic delivery may be more captivating than a slow, deliberate one.
  • Complexity wins: High-engagement podcasts tend to have higher reading grade levels and greater vocabulary diversity. Listeners seem to prefer intellectually stimulating content over overly simplified language.
  • The "Me and You" Factor: Analysis of predictive terms showed that first and second-person pronouns (me, you, us, our) are strong indicators of engagement, supporting the idea that podcasting is an intimate, personal medium.
  • Ads: A Mixed Bag: Large amounts of ads in the audio transcript hurt engagement. However, "ad-like" content in written descriptions (which often includes useful links and show notes) actually correlates with higher engagement in popular shows.
  • Mainstream vs. Niche: Surprisingly, high-engagement podcasts use less distinctive (more mainstream) language compared to the rest of the corpus.
This table illustrates the specific terms that distinguish high and low engagement, highlighting the preference for personal pronouns and specific conjunctions.

Key Results

The predictive power of text in podcasts is remarkably high. As shown in the table below, the BERT model achieved the best performance, but even simple linguistic features performed significantly better than chance.

Accuracy of various models in predicting high vs. low engagement. Note the strong performance of BERT and the contribution of combined description and transcript data.
  • Linguistic Features Only: Achieved 71.51% accuracy in distinguishing the top 25% from the bottom 25% of podcasts by stream rate.
  • BERT (Descriptions + Transcripts): Reached a peak accuracy of 80.54%.
  • TED Comparison: The models achieved similar accuracy (~71%) on TED talks, though some features (like reading grade level) showed opposite correlations, suggesting that "engagement" manifests differently in public speaking versus the more casual podcast format.

Limitations and Future Work

The authors are careful to note that their findings represent correlations, not causation. A high speech rate might be a symptom of an engaging host, not the direct cause of engagement.

Other limitations include:

  • Dataset Bias: The data comes exclusively from Spotify-owned or operated publishers, which may not represent the entire podcasting landscape.
  • Transcription Errors: The use of automated transcripts (with an estimated 18% word error rate) likely introduced noise, particularly for syntactic parsing.
  • Demographics: The study lacks data on listener demographics, which could influence language preferences.

Future research directions include analyzing paralinguistic features (pitch, intonation, and rhythm) and studying how linguistic features shift dynamically over the course of an episode.

Summary updated

Abstract

While there is an abundance of popular writing targeted to podcast creators on how to speak in ways that engage their listeners, there has been little data-driven analysis of podcasts that relates linguistic style with listener engagement. In this paper, we investigate how various factors -- vocabulary diversity, distinctiveness, emotion, and syntax, among others -- correlate with engagement, based on analysis of the creators' written descriptions and transcripts of the audio. We build models with different textual representations, and show that the identified features are highly predictive of engagement. Our analysis tests popular wisdom about stylistic elements in high-engagement podcasts, corroborating some aspects, and adding new perspectives on others.
BibTeX
@inproceedings{reddy2021modeling,
  author = {Sravana Reddy and Marina Lazarova and Yongze Yu and Rosie Jones},
  title = {{Modeling Language Usage and Listener Engagement in Podcasts}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}

Loading related content...

Figure