Infant AFAR: Automated facial action recognition in infants

Itır Önal Ertuğrul, Yeojin Amy Ahn, Maneesh Bilalpur, Daniel S. Messinger, Matthew L. Speltz, Jeffrey F. CohnBehavior 2022 · DOI

TL;DR

Researchers have developed Infant AFAR, a deep-learning-based system that automatically detects facial "Action Units" (AUs) in infants. By fine-tuning models originally trained on adults, the team created a tool that significantly outperforms previous methods, providing a much-needed open-source resource for studying infant emotion and development.

What Problem Does This Paper Address?

In the world of psychology, the "gold standard" for describing facial expressions is the Facial Action Coding System (FACS). Instead of using subjective labels like "happy" or "sad," FACS identifies specific muscle movements called Action Units (e.g., AU12 is a lip corner puller). For infants, there is a specialized version called Baby FACS.

The problem? Manual coding is grueling. It requires expert training and can take hours to annotate just a few minutes of video. While automated systems exist for adults, they fail miserably when applied to babies. Infants have different facial proportions (larger eyes, smaller jaws), smoother skin with fewer wrinkles, and unique fat pads that obscure muscle movements. This paper tackles the challenge of building a reliable, automated system specifically tuned to the unique "landscape" of an infant's face.

Methods

The researchers employed a transfer learning approach using a Convolutional Neural Network (CNN) architecture called VGG-16. Their pipeline involved three distinct stages:

  1. ImageNet Pre-training: The model started with weights learned from millions of general images.
  2. Adult Training (Adult AFAR): The model was trained on two large databases of adult facial expressions (EB+ and GFT) to learn the general "logic" of facial movements.
  3. Infant Fine-tuning (Infant AFAR): Finally, the model was fine-tuned using manually annotated infant video databases.

By using the adult model as a starting point rather than training from scratch, the network could leverage its existing knowledge of facial structures while adapting to the specific textures and proportions of babies. The system also uses the ZFace module for 3D face tracking, allowing it to handle head movements without needing person-specific calibration.

Datasets

The study utilized two primary infant datasets, which provided a rigorous test of how well the model generalizes across different ages and environments:

  • MIAMI: Features 43 four-month-old infants recorded during a "Face-to-Face/Still-Face" protocol (interacting with mothers).
  • CLOCK: Features 80 thirteen-month-old infants recorded during emotion-induction tasks (like blowing bubbles or playing with toys).
  • Adult Benchmarks: The EB+ (200 subjects) and GFT (150 subjects) databases were used to train the initial "Adult AFAR" model.

Key Insights and Ideas

  • Adult Models Don't Translate: The study confirmed that models trained solely on adults perform poorly on infants. The lack of wrinkles and different facial geometry in babies means adult-trained detectors often "miss" subtle movements.
  • The Power of Fine-Tuning: One of the most important "aha" moments was that fine-tuning an adult model (Study 5) outperformed training a model from scratch on infant data alone (Study 2). The features learned from adult faces provided a robust foundation that improved infant AU detection.
  • Domain Sensitivity: Even within the "infant" category, models don't always generalize perfectly. A model trained on 4-month-olds (MIAMI) performed differently when tested on 13-month-olds (CLOCK), highlighting how much infant faces change in just a few months.
  • Open Science: Unlike many commercial tools (like Affectiva or FaceReader) which are "black boxes," the authors have made Infant AFAR available to the research community via GitHub.

Key Results

Infant AFAR set a new state-of-the-art for infant facial analysis:

  • Superior Accuracy: Infant AFAR significantly outperformed previous deep learning attempts. For example, it showed a 37% improvement in Positive Agreement (PA) for AU4 (brow lowerer) compared to the previous leading method (Hammal et al., 2017).
  • Robustness: The system achieved high Area Under the Curve (AUC) scores across core AUs, typically ranging from 0.79 to 0.88, indicating a strong ability to distinguish between the presence and absence of facial actions.
  • Versatility: While focused on four core AUs (AU4, AU6, AU12, AU20) common to both datasets, the researchers also successfully trained models for five additional AUs (AU1, AU2, AU3, AU9, AU28) using the CLOCK database.

Limitations and Future Work

The authors acknowledge a few key hurdles:

  • Limited AU Vocabulary: Because the model relies on manual "Baby FACS" labels for training, it can currently only detect a limited number of Action Units compared to adult systems.
  • Generalizability: While the model is robust, its performance on infants in very different environments (e.g., low light or extreme head angles) remains an open question.
  • Data Privacy: Because the training data consists of identifiable videos of minors, the raw datasets cannot be shared publicly, though the resulting model weights and code are available.

Future work will likely focus on expanding the number of detectable AUs and testing the system in clinical settings, such as identifying early signs of neurodevelopmental disorders or assessing infant pain.

Summary updated

Abstract

Automated detection of facial action units in infants is challenging. Infant faces have different proportions, less texture, fewer wrinkles and furrows, and unique facial actions relative to adults. For these and related reasons, action unit (AU) detectors that are trained on adult faces may generalize poorly to infant faces. To train and test AU detectors for infant faces, we trained convolutional neural networks (CNN) in adult video databases and fine-tuned these networks in two large, manually annotated, infant video databases that differ in context, head pose, illumination, video resolution, and infant age. AUs were those central to expression of positive and negative emotion. AU detectors trained in infants greatly outperformed ones trained previously in adults. Training AU detectors across infant databases afforded greater robustness to between-database differences than did training database specific AU detectors and outperformed previous state-of-the-art in infant AU detection. The resulting AU detection system, which we refer to as Infant AFAR (Automated Facial Action Recognition), is available to the research community for further testing and applications in infant emotion, social interaction, and related topics.
BibTeX
@inproceedings{ertugrul2022infant,
  author = {Itır Önal Ertuğrul and Yeojin Amy Ahn and Maneesh Bilalpur and Daniel S. Messinger and Matthew L. Speltz and Jeffrey F. Cohn},
  title = {{Infant AFAR: Automated facial action recognition in infants}},
  booktitle = {Behavior},
  year = {2022},
  doi = {10.3758/s13428-022-01863-y},
}
Figure