Words Worth: Verbal Content and Hirability Impressions in YouTube Video Resumes

Skanda Muralidhar, Laurent Nguyen, Daniel Gatica-PerezEMNLP 2018

TL;DR

This paper explores how the words spoken in YouTube video resumes influence perceptions of hirability. By analyzing transcriptions of over 200 video resumes using advanced NLP techniques like Word2Vec and GloVe, the researchers found that verbal content significantly contributes to hirability impressions, even when using automatically generated transcripts.

What Problem Does This Paper Address?

First impressions are crucial in many social and professional interactions, including job interviews. While nonverbal cues have been extensively studied for their role in forming these impressions, the impact of verbal content in video resumes has received less attention. This is particularly true for "in-the-wild" real-world video resumes, which often have noisy audio and less controlled environments compared to formal interviews. The authors aim to bridge this gap by investigating how the language used in these videos relates to perceived hirability.

Methods

The study employs a two-pronged approach to represent verbal content:

  • Document-Based Representation:
    • LIWC (Linguistic Inquiry and Word Count): A lexicon-based approach that categorizes words into 70 predefined psychological and linguistic categories. This method requires no text preprocessing.
  • Doc2Vec: A neural network-based method that learns vector representations for entire documents (in this case, video transcripts). This requires preprocessing steps like lowercasing, stop word removal, and stemming.
  • Word-Based Representation:
    • Word Embeddings: Pre-trained models like Word2Vec (CBOW) and GloVe (trained on Wikipedia and a larger corpus) are used to generate vector representations for individual words.
  • Aggregation Techniques: To obtain a document-level representation from word embeddings, two aggregation methods are applied:
  • Averaging: Simple mean of word vectors.
  • TF-IDF (Term Frequency-Inverse Document Frequency): Weights word vectors based on their importance within a document and across the corpus.

These representations are then fed into regression models (Support Vector Machines and Random Forests) to predict hirability scores. The models are trained and evaluated using cross-validation.

Datasets

The study utilizes a dataset of 292 YouTube video resumes. This dataset was originally collected by Nguyen et al. (2016) and consists of noisy, real-world videos. The videos were manually annotated for demographics and hirability impressions on a 5-point Likert scale by multiple Amazon Mechanical Turkers, with inter-rater agreement assessed using the Intraclass Correlation Coefficient (ICC).

The dataset was also processed in three ways for transcription:

  1. Manual Transcription: A native English speaker transcribed the videos. Due to the cost and time involved, only the first 2 minutes of each video were transcribed for this "gold-standard."
  2. Automatic Transcription (ASR-2min): Google Speech API was used to transcribe the first 2 minutes of each video.
  3. Automatic Transcription (ASR-Full): Google Speech API was used to transcribe the entire duration of each video.

Key Insights and Ideas

  • Words Matter for Hirability: The study strongly suggests that the verbal content of video resumes is a significant predictor of hirability impressions, challenging the notion that only nonverbal cues are dominant.
  • Advanced NLP Techniques Shine: Pre-trained word embeddings like GloVe, when aggregated, outperform simpler methods like LIWC and Doc2Vec in capturing nuances related to hirability.
  • ASR is a Viable Alternative: Automatic Speech Recognition (ASR) can be used to generate transcripts that yield comparable hirability inference performance to manual transcriptions, making large-scale analysis more feasible.
  • Contextual Embeddings are Key: GloVe, particularly the version trained on a larger corpus (GloVe(S)), showed the best performance, indicating that the global co-occurrence statistics it captures are valuable for understanding hirability.
  • Full Duration Can Help: Transcribing the full video duration, rather than just the first two minutes, can lead to a slight improvement in inference performance for some models.

Key Results

  • Best Performance with Manual Transcripts: Using manually transcribed texts, GloVe(S) (Averaged) achieved the highest inference performance for Overall Hirability with an R² of 0.23.
  • ASR Performance: Automatic transcriptions (ASR-2min) yielded comparable results, with Word2Vec (TF-IDF) achieving the best performance for Overall Hirability with an R² of 0.21.
  • Impact of Duration: Using ASR-Full transcripts showed a slight improvement in inference performance for some models, with Word2Vec (Avg) achieving an R² of 0.22 for Overall Hirability.
  • Comparison to Prior Work: The reported R² values are generally higher than those found in previous studies using LIWC features or nonverbal cues for similar tasks. For instance, one prior study using LIWC reported an R² of 0.11.

Limitations and Future Work

  • Dataset Size: The authors acknowledge that their experiments would benefit from a larger dataset, especially for methods like Doc2Vec which require substantial data for accurate document representation.
  • Limited Scope: The current study focuses solely on verbal content. Future work could explore fusing verbal content with other non-textual predictors (e.g., visual cues) and personality impressions.
  • Broader Analysis: The authors plan to investigate the impact of verbal content on hirability using the complete dataset of 939 videos.
  • ASR Errors: While ASR performance is promising, the study notes that ASR errors (measured by WER) can impact performance, and some models (like Word2Vec) might be more robust to these errors than others.

Summary updated

BibTeX
@inproceedings{muralidhar2018words,
  author = {Skanda Muralidhar and Laurent Nguyen and Daniel Gatica-Perez},
  title = {{Words Worth: Verbal Content and Hirability Impressions in YouTube Video Resumes}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure