Abstract
In this article, we propose a knowledge distillation approach with two teachers for facial age estimation. Due to the nonstationary patterns of the facial-aging process, the relative order of age labels provides more reliable information than exact age values for facial age estimation. Thus, the first teacher is a novel ranking method capturing the ordinal relation among age labels. Especially, it formulates the ordinal relation learning as a task of recovering the original ordered sequences from shuffled ones. The second teacher adopts the same model as the student that treats facial age estimation as a multiclass classification task. The proposed method leverages the intermediate representations learned by the first teacher and the softened outputs of the second teacher as supervisory signals to improve the training procedure and final performance of the compact student for facial age estimation. Hence, the proposed knowledge distillation approach is capable of distilling the ordinal knowledge from the ranking model and the dark knowledge from the multiclass classification model into a compact student, which facilitates the implementation of facial age estimation on platforms with limited memory and computation resources, such as mobile and embedded devices. Extensive experiments involving several famous data sets for age estimation have demonstrated the superior performance of our proposed method over several existing state-of-the-art methods.
Introduction
Facial age estimation is a challenging task in computer vision due to the non-stationary nature of human aging. Biological changes occurring between ages 5 and 10 (primarily bone growth) differ significantly from those between ages 40 and 45 (primarily skin texture and wrinkles). Traditional approaches typically frame this as either a regression problem or a multi-class classification problem. However, regression often suffers from overfitting due to the non-linear aging process, while classification ignores the inherent ordinal relationship between age labels—for instance, age 30 is biologically closer to 31 than to 50. Furthermore, state-of-the-art models are often too large (exceeding 500MB) for deployment on resource-constrained mobile or IoT devices.
Methodology: Two-Teacher Distillation
To address these challenges, we propose a two-teacher knowledge distillation framework. This strategy captures diverse expertise to train a single, ultra-compact student model.
The framework consists of three primary components:
- The Ordinal Teacher (Ranking): This model focuses on "visual permutation." Given a shuffled sequence of facial images, it must recover the correct chronological order. This Permutation Prediction (PVP) task forces the model to learn structural aging features rather than memorizing specific age values.
- The Classification Teacher: A high-capacity model that treats age as a classification task. It provides "dark knowledge"—the softened probability distribution across all ages—helping the student understand which ages share similar visual features.
- The Compact Student: The student model is trained using a multi-part loss function. It learns from the Ordinal Teacher via "hints" (matching intermediate feature maps) and from the Classification Teacher via "soft targets."
A key insight is that ordinality is often more reliable than exactness. Since "apparent age" can differ from "chronological age," the relative order provides a more robust supervisory signal. Additionally, the "hint" mechanism allows knowledge transfer between different architectures, as the ranking teacher and classification student do not need to share the same structure.
Experimental Results
The proposed method was evaluated on several benchmarks, including MORPH2, FG-NET, MegaAge, MegaAge-Asian, and AFAD.
Key findings include:
- State-of-the-Art Accuracy: On the MORPH2 dataset, our "normal" model achieved a Mean Absolute Error (MAE) of 1.95, becoming the first to break the 2.0 barrier on this benchmark.
- Extreme Efficiency: While the "normal" version is already efficient, the "small" student model variant is remarkably compact at 0.44 MB—approximately 1,000x smaller than many bulky state-of-the-art models—while still outperforming much larger models like RankingCNN (2.2 GB).
- High Speed: With an inference time of 1.48ms for the normal model (and even faster for the small variant), the system is highly suitable for real-time applications on embedded devices.
Conclusion
This research demonstrates that distilling both ordinal relations and dark knowledge allows for the creation of highly efficient facial age estimation models. While the training process is computationally intensive due to the requirement of two large teachers, the resulting student model is optimized for deployment on low-power devices. Future work may explore applying these techniques to video sequences to leverage temporal consistency for even stronger ordinal signals.
Summary updated