HireNet: A Hierarchical Attention Model for the Automatic Analysis of Asynchronous Video Job Interviews

Léo Hemamou, Ghazi Felhi, Vincent Vandenbussche, Jean-Claude Martin, Chloé ClavelAAAI 2019 · DOI

Abstract

Asynchronous video interviews (AVIs) have become a standard tool in modern recruitment, yet the sheer volume of data often exceeds the capacity for manual review. This paper introduces HireNet, a hierarchical attention-based framework designed to predict candidate hirability by analyzing multimodal data (text, audio, and video). By incorporating the context of job titles and specific interview questions, HireNet achieves state-of-the-art performance on a large-scale, real-world dataset of over 7,000 interviews.

Introduction

The rise of digital recruitment has led to an explosion of asynchronous video interviews. While efficient for candidates, recruiters are often overwhelmed by hundreds of hours of footage. Previous attempts to automate the screening process have been limited by small, artificial datasets and models that fail to account for the hierarchical and contextual nature of an interview. HireNet addresses these gaps by modeling interviews as a structured sequence of question-answer pairs influenced by the specific job role.

The HireNet Architecture

HireNet is designed to mirror the natural structure of a job interview. It employs a hierarchical encoding strategy using Gated Recurrent Units (GRUs) at two distinct levels:

  1. Low-level Encoder: Processes sequences of words or video frames within a single answer.
  2. High-level Encoder: Aggregates these individual answer representations into a comprehensive interview profile.

The "secret sauce" of HireNet is its Contextual Attention mechanism. Unlike flat models, HireNet uses the question as local context to identify salient moments in an answer and the job title as global context to weigh the importance of different questions for a specific role. This allows the model to prioritize different behaviors depending on whether a candidate is answering a technical question or a soft-skills inquiry.

The HireNet architecture featuring hierarchical encoders and contextual attention layers.

A Massive Real-World Dataset

A significant contribution of this work is the use of a proprietary corpus of 7,095 real-world job interviews for 475 sales positions. This dataset is notably larger and more realistic than those used in previous studies, which often relied on mock interviews with small samples.

Comparison of HireNet's dataset scale and context against previous works in the field.

The interviews were recorded in "in-the-wild" conditions (home environments, varying lighting, and hardware), providing a rigorous test for multimodal analysis. Labels were derived from actual recruiter decisions, such as shortlisting or inviting candidates to face-to-face interviews, ensuring the model learns from real-world hiring criteria.

Key Results and Insights

HireNet consistently outperformed non-sequential and standard bidirectional GRU baselines across all modalities. The primary metric used was the F1-score for the "hirable" class:

Modality Non-sequential Bidirectional GRU HireNet
Text 0.376 0.561 0.643
Audio 0.519 0.566 0.642
Video 0.507 0.528 0.605

Contextual Importance: The attention mechanism revealed that the importance of specific behaviors is question-dependent. For instance, audio features like prosody were more influential during job-centric questions than during soft-skill inquiries. This confirms that a "one-size-fits-all" approach to behavior analysis is insufficient for interview screening.

Interpretability: The model acts as a "highlight reel" generator, identifying key moments—such as a candidate smiling or showing signs of anxiety—that correlate with recruiter decisions. This transparency is vital for building trust in AI-assisted recruitment tools.

Conclusion and Future Work

HireNet demonstrates that hierarchical modeling and contextual attention are vital for analyzing complex human interactions like job interviews. While the video modality faced challenges due to real-world recording conditions, the overall framework provides a scalable and interpretable assistant for recruiters. Future research will focus on more sophisticated cross-modal fusion and addressing potential algorithmic biases to ensure fair and ethical automated screening.

Summary updated

BibTeX
@inproceedings{hemamou2019hirenet,
  author = {Léo Hemamou and Ghazi Felhi and Vincent Vandenbussche and Jean-Claude Martin and Chloé Clavel},
  title = {{HireNet: A Hierarchical Attention Model for the Automatic Analysis of Asynchronous Video Job Interviews}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.1609/aaai.v33i01.3301573},
}
Figure