Towards In-Vehicle Multi-Task Facial Attribute Recognition: Investigating Synthetic Data and Vision Foundation Models

Esmaeil Seraj, Walter TalamontiarXiv 2024 · DOI · arXiv

Introduction

Developing intelligent in-cabin monitoring systems—capable of detecting driver distraction, fatigue, or emotional state—is a cornerstone of modern automotive safety. However, the development of these systems is hindered by a "data drought." Collecting large-scale, high-quality facial data within vehicles is prohibitively expensive, time-consuming, and raises significant privacy concerns regarding biometric information.

High-level overview of the proposed pipeline using synthetic data to train foundation models for downstream automotive tasks.

This research investigates the viability of using synthetic data (AI-generated images) as a solution to these hurdles. The study explores how state-of-the-art vision foundation models handle the transition from synthetic training environments to real-world inference across multiple tasks simultaneously.

Methodology

The researchers developed a Multi-Task Learning (MTL) framework designed to predict three key attributes: Gaze Plane, Age, and Facial Expression.

Architectural Comparison

The study compared two prominent architectures:

  • ResNet-18: A classic convolutional neural network (CNN) valued for its efficiency and inductive bias toward local features.
  • ViT-b_16 (Vision Transformer): A modern transformer-based model that treats images as sequences of patches, offering high performance on large-scale datasets.

Adaptation and Preprocessing

To adapt these ImageNet-pre-trained models for the automotive domain, three strategies were evaluated: Linear Probing (LP), Prefix Tuning (PT), and Full Fine-Tuning (FFT). Additionally, the researchers found that focusing the model's attention on the driver's face via automated cropping significantly improved performance compared to using full-cabin images.

Detailed architecture of the multi-task learning framework, illustrating the preprocessing, patchification for ViT, and the shared backbone with modular task heads.

Datasets and Training

The models were trained on three distinct synthetic datasets:

  • SynthA: High volume (10,000 images) but low diversity.
  • SynthB: Moderate volume (~3,000 images) with high demographic diversity (race, age, accessories).
  • SynthC: Smallest volume (~1,900 images) captured in diverse non-vehicle environments.

A combined dataset of 14,919 samples was also used. For real-world validation, the UTKFace dataset served as an Out-of-Distribution (OOD) test set to measure the "sim-to-real" transferability.

Key Insights

The ResNet Advantage

Contrary to expectations, the older ResNet-18 architecture consistently outperformed the modern Vision Transformer. The authors suggest a "complexity mismatch": the high capacity of ViTs may lead them to overfit the specific artifacts of synthetic data, whereas the simpler ResNet architecture generalizes better to the underlying facial features.

Diversity Over Volume

While SynthA provided high training accuracy, it failed to generalize to real-world data (under 10% accuracy). In contrast, SynthC—the smallest and least "automotive" dataset—offered the best generalization. This indicates that environmental and demographic diversity in synthetic data is far more critical than raw image count.

Results and Conclusion

The optimal configuration was identified as a ResNet model using Full Fine-Tuning (FFT) on the Combined Dataset. On face-only validation sets, the model achieved high accuracies: 93.1% for Gaze, 95.5% for Age, and 86.0% for Facial Expression.

When tested on the real-world UTKFace dataset, the model achieved an average accuracy of approximately 50%. While this highlights a persistent "sim-to-real" gap, it confirms that models can indeed learn transferable human features from purely synthetic pixels. Future work will focus on closing this gap through advanced data augmentation and model compression for edge deployment.

Summary updated

Abstract

In the burgeoning field of intelligent transportation systems, enhancing vehicle-driver interaction through facial attribute recognition, such as facial expression, eye gaze, age, etc., is of paramount importance for safety, personalization, and overall user experience. However, the scarcity of comprehensive large-scale, real-world datasets poses a significant challenge for training robust multi-task models. Existing literature often overlooks the potential of synthetic datasets and the comparative efficacy of state-of-the-art vision foundation models in such constrained settings. This paper addresses these gaps by investigating the utility of synthetic datasets for training complex multi-task models that recognize facial attributes of passengers of a vehicle, such as gaze plane, age, and facial expression. Utilizing transfer learning techniques with both pre-trained Vision Transformer (ViT) and Residual Network (ResNet) models, we explore various training and adaptation methods to optimize performance, particularly when data availability is limited. We provide extensive post-evaluation analysis, investigating the effects of synthetic data distributions on model performance in in-distribution data and out-of-distribution inference. Our study unveils counter-intuitive findings, notably the superior performance of ResNet over ViTs in our specific multi-task context, which is attributed to the mismatch in model complexity relative to task complexity. Our results highlight the challenges and opportunities for enhancing the use of synthetic data and vision foundation models in practical applications.
BibTeX
@misc{seraj2024towards,
  author = {Esmaeil Seraj and Walter Talamonti},
  title = {{Towards In-Vehicle Multi-Task Facial Attribute Recognition: Investigating Synthetic Data and Vision Foundation Models}},
  year = {2024},
  eprint = {2403.06088},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2403.06088},
}
Figure