Speech vs. Transcript: Does It Matter for Human Annotators in Speech Summarization?

Roshan S. Sharma, Suwon Shon, Mark Lindsey, Hira Dhamyal, Rita Singh, Bhiksha Raj

Introduction

In the evolving field of abstractive speech summarization, researchers typically rely on "gold standard" reference summaries written by humans. However, the process of creating these references lacks standardization: some annotators listen to raw audio, while others read text transcripts. This paper investigates whether the source modality—audio versus text—significantly alters the content and quality of the resulting summaries. The findings suggest that "speech-based" summaries are generally more concise and factually consistent, whereas "transcript-based" summaries are more detailed but susceptible to inheriting transcription errors.

Experimental Methodology

To understand the impact of modality, the researchers conducted a study comparing three primary variables: source modality (audio vs. transcript), transcription quality (manual vs. ASR), and annotator expertise (experts vs. crowdsourced non-experts).

Overview of the experimental framework comparing transcript-based and speech-based summarization.

The study utilized the Interview corpus, consisting of NPR radio segments. A new benchmark of 1,002 recordings (averaging 2 minutes each) was curated. For each recording, four summaries were collected: two from experts (one speech-based, one transcript-based) and two from non-experts. Evaluation involved a mix of intrinsic metrics (length, compression), semantic metrics (BERTScore, BARTScore), factual consistency (UniEval), and qualitative assessments via LLM and human preference testing.

Key Insights: Listening vs. Reading

The research revealed distinct cognitive approaches between the two modalities:

  • Selective Listening vs. Detailed Reading: Listeners act as high-level filters, capturing salient points. Readers, conversely, tend to be more "extractive," including specific names and minor details because the text is easier to reference and copy.
  • Factual Consistency: Speech-based summaries demonstrated higher factual consistency. The inability to "copy-paste" from audio forces listeners to synthesize information in their own words, reducing hallucinations or misinterpretations of complex syntax.
  • The ASR Error Trap: Summaries derived from ASR transcripts often inherit the model's errors. As the Word Error Rate (WER) increases, the informativeness (Entity F1) of the human summary decreases.
  • Crowdsourcing Challenges: Evidence suggested that non-expert workers often utilized generative AI, making their summaries less representative of human cognitive processes and less useful for modality studies.

Key Results

The quantitative analysis highlighted significant differences between the two summary types, particularly among expert annotators.

Comparative performance metrics for expert and non-expert annotators across different source modalities.
  • Conciseness: Expert speech-based summaries were notably shorter (~57 words) than transcript-based ones (~79 words).
  • Consistency: Speech-based summaries achieved higher UniEval consistency scores (0.85 vs. 0.83).
  • Information Density: Transcript-based summaries captured more specific named entities, reflected in a higher Entity F1 score (33.63 vs. 26.21).
  • Preference: Despite the conciseness of speech summaries, both humans and LLMs generally preferred transcript-based summaries in AB testing, likely due to the higher level of detail.

Conclusion and Future Directions

The study underscores that the modality of the source material does indeed matter. Current evaluation metrics, which are heavily text-biased, may unfairly penalize speech-based summaries that capture non-textual nuances like tone or emphasis. Future work should focus on developing audio-aware metrics and better methods for detecting AI usage in human annotation tasks to ensure data integrity.

Summary updated

BibTeX
@inproceedings{sharma2024speech,
  author = {Roshan S. Sharma and Suwon Shon and Mark Lindsey and Hira Dhamyal and Rita Singh and Bhiksha Raj},
  title = {{Speech vs. Transcript: Does It Matter for Human Annotators in Speech Summarization?}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.acl-long.790},
}
Figure