Expression Classification in Children Using Mean Supervised Deep Boltzmann Machine

Shruti Nagpal, Maneet Singh, Mayank Vatsa, Richa Singh, Afzel NooreWorkshops @ CVPR 2019 · OpenReview

Abstract

Automated facial expression classification has widespread application in multiple domains such as human computer interaction, health and entertainment, biometrics, and security. There are six basic facial expressions: Anger, Disgust, Fear, Happiness, Sadness, and Surprise, apart from a neutral state. Most of the research in expression classification has focused on adult face images, with no dedicated research on automating expression classification for children. To the best of our knowledge, this is the first research which presents a deep learning based expression classification approach for children. A novel supervised deep learning formulation, termed as Mean Supervised Deep Boltzmann Machine (msDBM) is proposed which classifies an input face image into one of the seven expression classes. The proposed approach has been evaluated on two child face datasets - Radboud Faces and CAFE, along with experiments on the adult face images of the Radboud Faces dataset. Experimental results and analysis reinforces the challenging nature of the task at hand, and the effectiveness of the proposed msDBM model.

Introduction

While automated facial expression recognition (FER) has matured significantly, the vast majority of research has focused on adults. Children, however, present a unique challenge: they are often more expressive with positive emotions but project highly ambiguous or "mixed" expressions for negative emotions like anger or disgust.

A child exhibiting a negative expression, which often presents high ambiguity in automated classification.

From a technical standpoint, this results in high intra-class variation (the same emotion looking very different across different children) and low inter-class variation (different emotions looking very similar). Existing models trained on adults often fail when applied to children. Solving this is crucial for applications in daycare monitoring, e-learning platforms, and assisting children with autism in understanding social cues.

Methodology

The authors propose the Mean Supervised Deep Boltzmann Machine (msDBM). To understand this, we first look at the standard Deep Boltzmann Machine (DBM), which is typically an unsupervised model that learns to represent data through layers of hidden variables.

The "secret sauce" of the msDBM is the introduction of a supervised loss function that incorporates class labels directly into the feature learning process. The loss function consists of three distinct terms:

  1. Standard RBM Loss: The traditional unsupervised component that learns a meaningful representation of the input.
  2. Intra-class Regularization: This term forces the learned features of a specific class (e.g., "Happy") to be closer to the mean feature vector of that class.
  3. Inter-class Regularization: This term pushes the learned features of a class away from the mean representations of all other classes.

By optimizing these three terms simultaneously, the model learns a feature space where expressions are naturally clustered by emotion. These features are then fed into a Random Decision Forest (RDF) for the final classification into seven categories: Anger, Disgust, Fear, Happiness, Sadness, Surprise, and Neutral.

Experimental Setup and Datasets

The study utilizes the only two major datasets containing child expression variations:

  • Radboud Faces Dataset (RaFD): Contains 8,040 images, including a subset of 10 children and 57 adults. This allows for cross-age comparisons.
  • Child Affective Facial Expression (CAFE) Dataset: A much more challenging set containing 1,192 images of 154 children aged 2–8 years. Unlike the posed images in RaFD, these children often struggled to imitate specific emotions perfectly, leading to a highly imbalanced and "noisy" dataset.
Children are typically more expressive with positive emotions like happiness, leading to better classification performance compared to negative emotions.

Results and Discussion

The msDBM demonstrated superior performance across the board. On the Radboud Faces (Children) dataset, it achieved 75.0% accuracy, outperforming standard DBMs (71.7%) and even fine-tuned VGG-Face models (67.4%). On the notoriously difficult CAFE Dataset, it achieved 48.0% accuracy, which was nearly 14% higher than other comparative models. Interestingly, on the adult portion of RaFD, the model hit 85.9%, showing that the "Mean Supervised" approach is robust across age groups but particularly helpful for the high-variance child data.

Key Insights

  • The "Adult Bias" in AI: The authors found that a commercial tool (Microsoft Cognitive Services) achieved only 17.7% accuracy on the child dataset, despite performing well on adults. This highlights a massive gap in current "off-the-shelf" AI.
  • Supervision Matters Early: Unlike standard deep learning where supervision often happens only at the final classification layer, the msDBM introduces supervision during the feature extraction phase. This helps the model ignore "identity" noise and focus on "expression" signals.
  • The Anger-Disgust Overlap: In young children (ages 2-8), the facial muscle movements for anger and disgust are nearly indistinguishable to a computer. When the researchers combined these into a single class, accuracy jumped significantly.

Conclusion

The primary limitation is the inherent difficulty of the CAFE dataset. Because children aged 2–8 often produce "blended" expressions, even a state-of-the-art model struggles to reach the high-90s accuracy seen in adult datasets. Future work should focus on even more specialized models for the 2–8 age range and perhaps explore temporal data (video) rather than static images to better capture the nuances of how a child's expression evolves. There is also a clear need for larger, more diverse datasets of children's facial expressions to reduce the "imbalance" issues noted in the CAFE set.

Summary updated

Abstract

Automated facial expression classification has widespread application in multiple domains such as human computer interaction, health and entertainment, biometrics, and security. There are six basic facial expressions: Anger, Disgust, Fear, Happiness, Sadness, and Surprise, apart from a neutral state. Most of the research in expression classification has focused on adult face images, with no dedicated research on automating expression classification for children. To the best of our knowledge, this is the first research which presents a deep learning based expression classification approach for children. A novel supervised deep learning formulation, termed as Mean Supervised Deep Boltzmann Machine (msDBM) is proposed which classifies an input face image into one of the seven expression classes. The proposed approach has been evaluated on two child face datasets - Radboud Faces and CAFE, along with experiments on the adult face images of the Radboud Faces dataset. Experimental results and analysis reinforces the challenging nature of the task at hand, and the effectiveness of the proposed msDBM model.
BibTeX
@inproceedings{nagpal2019expression,
  author = {Shruti Nagpal and Maneet Singh and Mayank Vatsa and Richa Singh and Afzel Noore},
  title = {{Expression Classification in Children Using Mean Supervised Deep Boltzmann Machine}},
  booktitle = {Workshops @ CVPR},
  year = {2019},
  url = {https://openreview.net/forum?id=QsDCSogQoGny},
}
Figure