Leveraging Multimodal Behavioral Analytics for Automated Job Interview Performance Assessment and Feedback

Anumeha Agrawal, Rosa Anil George, Selvan Sunitha Ravi, Sowmya Kamath S, Anand KumarACL 2020

TL;DR

This paper introduces a multimodal framework to automatically assess job interview performance by analyzing video, audio, and text. By fusing these modalities, the system aims to provide candidates with constructive feedback on their behavioral cues and overall presentation, moving beyond traditional interview assessments.

What Problem Does This Paper Address?

Traditional job interviews, while crucial for assessing candidates, rely heavily on human interpretation, which can be subjective and prone to bias. While technical skills are often evaluated, nuanced aspects like personality, interpersonal skills, and performance under pressure are harder to quantify consistently. This paper addresses the need for an objective, data-driven approach to analyze candidate behavior during interviews, providing actionable feedback that can improve interview performance and potentially lead to better hiring decisions. The authors aim to leverage the rich information present in non-verbal cues (facial expressions, body language) and vocal characteristics, in addition to verbal content, to achieve a more comprehensive assessment.

Methods

The proposed methodology employs a multimodal analytical framework that integrates features from three distinct modalities:

  • Audio: Prosodic features such as pitch, intensity, duration, and spectral characteristics are extracted from the audio recordings. This involves analyzing time-domain, frequency-domain, and cepstral features.
  • Video: Facial landmarks (e.g., nose, mouth, eyes) are captured to analyze local changes indicative of expressions. Additionally, head pose features (pitch, roll, yaw) and a binary classification for smiling are extracted. A pre-trained convolutional neural network (LeNet) is used for smile detection.
  • Lexical: Text transcripts, obtained via Google Cloud Speech-to-Text API, are processed to extract linguistic features. This includes speaking rate, word counts, unique word counts, filler word usage, and sentiment analysis using IBM's Tone Analyzer to categorize emotions like joy, sadness, and anger.

These extracted features are then fused into a single feature vector. Feature selection techniques, including correlation analysis, Benjamini-Hochberg procedure, and K-best feature selection, are employed to identify the most relevant features for each prediction task. Finally, four machine learning classifiers – Random Forest, Support Vector Classifier (SVC), Multitask Lasso, and Multilayer Perceptron (MLP) – are trained to predict ratings on a scale of 1-7 for nine predefined labels (e.g., Eye Contact, Speaking Rate, Engaged, Pauses, Calmness, Not Stressed, Focused, Authentic, Not Awkward).

Datasets

The study utilizes the MIT interview dataset. This dataset consists of recordings of 138 mock job interviews from 69 candidates, including both pre- and post-intervention sessions. The dataset is annotated with Amazon Mechanical Turk worker scores for various labels, which are used as ground truth. The authors note that the current dataset has 169 videos, suggesting potential for expansion.

Key Insights and Ideas

  • Multimodality is Key: Combining audio, visual, and textual information provides a more comprehensive understanding of interview performance than any single modality alone. The paper demonstrates that while individual modalities offer some insights, their fusion generally leads to better prediction accuracy.
  • Feature Importance Varies: Different behavioral cues are more indicative of specific interview performance labels. For instance, eye contact and speaking rate are crucial for certain assessments, while other cues might be more relevant for evaluating calmness or authenticity.
  • Feature Selection is Crucial: With a large number of extracted features, effective feature selection is vital to avoid overfitting and improve model performance. The study explores various feature selection techniques to identify the optimal set of features for each prediction task.
  • Random Forest Dominance: The Random Forest classifier consistently outperformed other models across most of the evaluated labels, highlighting its robustness and ability to handle complex feature interactions. This is attributed to its ensemble nature, which averages predictions and cancels out bias.

Key Results

The proposed multimodal framework achieved promising results in predicting interview performance labels.

  • Overall Performance: The Random Forest classifier, when trained on the fused multimodal feature vector, outperformed other models for 8 out of 9 evaluated labels.
  • Specific Labels:
    • Speaking Rate: Achieved high accuracy (96.43%) with the Random Forest classifier.
  • Pauses: The Random Forest classifier with K-best feature selection achieved the best accuracy of 82.14%.
  • Not Stressed: Both Random Forest and SVC achieved 82.14% accuracy.
  • Eye Contact: The Random Forest classifier achieved an accuracy of 64.28% when using the family-wise error technique for feature selection.
  • Modality Combinations: Experiments with different modality combinations showed that while Audio+Video+Lexical was not always the most accurate, two-modality combinations often performed better than single modalities.

Limitations and Future Work

  • Dataset Size: The authors acknowledge that the current dataset of 169 videos is relatively small, which makes achieving very high accuracy for all labels challenging. They suggest expanding the dataset with more videos scored by human annotators.
  • Incorporating More Cues: Future work could involve incorporating additional behavioral cues, such as hand movements and body posture, to gain a more refined understanding of candidate performance.
  • Web Application Integration: The authors envision integrating the current model into a web application to provide real-time feedback to candidates, helping them train for interviews and improve their performance. This would allow for more meaningful suggestions based on their strengths and weaknesses.

Summary updated

BibTeX
@inproceedings{agrawal2020leveraging,
  author = {Anumeha Agrawal and Rosa Anil George and Selvan Sunitha Ravi and Sowmya Kamath S and Anand Kumar},
  title = {{Leveraging Multimodal Behavioral Analytics for Automated Job Interview Performance Assessment and Feedback}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
}
Figure