One Embedding to Predict Them All: Visible and Thermal Universal Face Representations for Soft Biometric Estimation via Vision Transformers

Nélida Mirabet-Herranz, Chiara Galdi, Jean-Luc DugelayWorkshops @ CVPR 2024 · DOI

Introduction

Soft biometric estimation—the prediction of human traits such as age, gender, and weight from facial imagery—is a critical component of modern biometric systems. Traditionally, these tasks have been addressed using dedicated, end-to-end deep learning models for each specific trait. However, this "one model per task" approach is computationally expensive and often fails to generalize across different environmental conditions.

This paper introduces BEFiT (Bidirectional Encoder Face representation from image Transformers), a novel framework that leverages Vision Transformers (ViTs) to extract a single, "universal" face embedding capable of simultaneous multi-task estimation across both visible and thermal spectra.

The BEFiT Architecture

The core innovation of this work lies in the transition from local-feature-focused Convolutional Neural Networks (CNNs) to global-dependency-capturing Vision Transformers.

The BEFiT pipeline: from original image patches to a universal feature vector and task-specific classification heads.

The BEFiT model is built upon the BEiT architecture. The process begins by dividing the input face image into a sequence of patches. These patches are processed by a Transformer encoder that utilizes self-attention mechanisms to understand the global structure of the face.

The model is first fine-tuned for Face Recognition (FR) on large-scale datasets like CelebA and TUFTS. Once trained, the final classification layer is removed, leaving a robust feature vector—the "universal embedding." To perform soft biometric estimation, lightweight CNN heads (simple dense layers) are attached to this fixed embedding. This allows for the prediction of age, gender, and weight without re-training the complex backbone for each individual task.

Multi-Spectral Fusion

A significant challenge in facial biometrics is performance degradation under poor lighting or partial occlusion. To address this, the authors developed a multi-spectral approach using paired datasets like the LVT Face Database.

Examples of paired visible (top) and thermal (bottom) face images from the datasets used in the study.

The system comprises two branches: BEFiT-V for visible light and BEFiT-T for thermal imagery. Thermal sensors capture heat patterns related to physiological features like blood vessel distribution and bone structure, which are often more indicative of age and body mass than surface-level visual features. The final system, BEFiT-VT, fuses the scores from both branches using a weighted average, providing a more resilient and accurate prediction than either modality alone.

Experimental Results

The framework was evaluated using several benchmarks, including the LVT Face Database and the challenging VIS-TH dataset, which contains variations in pose and expression.

  • Gender Classification: BEFiT-VT achieved a remarkable 97% accuracy, significantly outperforming standard VGG16 baselines.
  • Age Estimation: The model reached a Mean Absolute Error (MAE) of 3.69 years. This represents a substantial leap over previous state-of-the-art models like DEX, which recorded an MAE of 7.23 years on the same data.
  • Weight Estimation: While weight remains a challenging metric, the universal embedding achieved a 60% "Percentage of Acceptable Predictions" (PAP), remaining highly competitive with specialized networks.
  • Robustness: On the VIS-TH dataset, BEFiT-VT maintained its 97% gender accuracy, whereas the CNN-based baseline performance collapsed to 28% in fusion tests, proving the Transformer's superior generalization.

Conclusion

The research demonstrates that a powerful Vision Transformer backbone, initially trained for face recognition, captures sufficient semantic nuance to serve as a universal representation for multiple soft biometric tasks. Furthermore, the integration of thermal imagery provides a physiological depth that visible light cannot match, particularly for age and weight estimation. Future work will explore expanding this "one embedding" approach to additional traits like ethnicity and height.

Summary updated

BibTeX
@inproceedings{mirabetherranz2024one,
  author = {Nélida Mirabet-Herranz and Chiara Galdi and Jean-Luc Dugelay},
  title = {{One Embedding to Predict Them All: Visible and Thermal Universal Face Representations for Soft Biometric Estimation via Vision Transformers}},
  booktitle = {Workshops @ CVPR},
  year = {2024},
  doi = {10.1109/cvprw63382.2024.00157},
}
Figure