Using BERT Embeddings to Model Word Importance in Conversational Transcripts for Deaf and Hard of Hearing Users

Akhter Al Amin, Saad Hassan, Cecilia Alm, Matt HuenerfauthACL 2022

TL;DR

This paper explores how to better evaluate live TV captions for Deaf and Hard of Hearing (DHH) users by modeling word importance. They found that BERT-based contextualized word embeddings correlate better with human importance judgments than traditional word2vec embeddings, leading to improved caption evaluation metrics.

What Problem Does This Paper Address?

Live TV captioning is crucial for DHH individuals, but current evaluation metrics often treat all words equally, failing to capture the nuances of how DHH users perceive caption importance. This can lead to dissatisfaction with caption quality, especially in live contexts where production time is limited. The authors aim to develop more effective caption evaluation methods by incorporating the relative importance of words, reflecting DHH users' preferences.

Methods

The core of this work involves comparing different methods for generating word embeddings and their correlation with human-annotated word importance scores.

  • Word Embeddings:
    • Word2Vec: A traditional approach used in prior work, generating static word embeddings.
  • BERT (Bidirectional Encoder Representations from Transformers): A more recent, powerful language model that generates contextualized word embeddings. This means the embedding for a word changes based on its surrounding words in a sentence.
  • Correlation Analysis: The authors calculated the correlation between semantic scores derived from these embeddings and human-annotated word importance scores.
  • Classification Models: They then used these BERT embeddings to train interpretable machine learning models (like Logistic Regression) to predict word importance categories.

Datasets

  • Switchboard Corpus: A dataset of conversational transcripts used in prior work.
  • Augmented Dataset: The authors created a new dataset by pairing BERT-based contextualized word embeddings with human-annotated word importance scores from a subset of the Switchboard corpus. This dataset augments existing resources for researchers.

Key Insights and Ideas

  • BERT's Superiority: BERT's contextualized embeddings significantly outperform word2vec in capturing human-perceived word importance. This is likely because BERT can understand how a word's meaning and significance change based on its context, which is vital for conversational speech.
  • Human-Centric Evaluation: The research directly addresses the need for caption evaluation metrics that align with the preferences of DHH users, moving beyond generic metrics like Word Error Rate (WER).
  • Practical Applications: The findings suggest that broadcasters could use BERT embeddings to automatically identify and potentially highlight important words in captions, improving readability and usability for DHH viewers.

Key Results

  • Correlation: BERT-based word importance scores showed a significantly stronger correlation (r = 0.41) with human annotations compared to word2vec (r = 0.30). A Fisher z-transformation confirmed this difference was statistically significant (z = -3.05, p < 0.001).
  • Classification Performance: Using BERT embeddings, a Logistic Regression model achieved a macro-averaged F1-score of 0.57 in classifying words into six importance categories. This demonstrates the practical utility of BERT embeddings for predicting word importance.
  • TF-IDF Comparison: Even traditional TF-IDF scores showed a lower correlation (r = 0.25) with human annotations than word2vec.

Limitations and Future Work

  • Hyperparameter Tuning: The authors acknowledge that they did not extensively tune model hyperparameters, suggesting future work could explore variations to improve prediction accuracy.
  • Bias in BERT: They plan to investigate whether BERT introduces latent biases related to disabilities, as large language models can sometimes reflect societal biases.
  • Alternative Methods: Future research could explore other methods for generating semantic scores and compare them to human annotations.
  • POS Tagging: Incorporating Part-of-Speech (POS) tags alongside contextual embeddings could further enhance word importance prediction.
  • Conversational Nuances: Future work might also consider strategies for scoring the importance of filler words common in conversational speech.
  • Broader Impact: The authors are broadly investigating how to accurately measure live TV caption quality from the DHH user perspective and plan to incorporate these findings into new weighted metrics.

Summary updated

BibTeX
@inproceedings{amin2022using,
  author = {Akhter Al Amin and Saad Hassan and Cecilia Alm and Matt Huenerfauth},
  title = {{Using BERT Embeddings to Model Word Importance in Conversational Transcripts for Deaf and Hard of Hearing Users}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure