Calibrated Adaptive Teacher for Domain-Adaptive Intelligent Fault Diagnosis

Florent Forest, Olga Fink

TL;DR

This paper introduces a novel method called Calibrated Adaptive Teacher (CAT) to improve the accuracy of intelligent fault diagnosis models when applied to new operating conditions (domain adaptation). By calibrating the confidence of predictions from a "teacher" model, CAT significantly boosts the quality of automatically generated training labels, leading to state-of-the-art performance and reduced errors in fault diagnosis.

What Problem Does This Paper Address?

Deep learning models excel at fault diagnosis using sensor data, but they often struggle when deployed in real-world scenarios where operating conditions differ from the training data. This "domain shift" leads to a significant drop in performance. While unsupervised domain adaptation (UDA) techniques aim to bridge this gap by using unlabeled data from the new (target) domain, a key challenge is the unreliability of predictions from these models. Specifically, models tend to be "over-confident" in their predictions, even when incorrect, which hinders the effectiveness of self-training methods that rely on these predictions to generate labels for the target domain. This poor calibration leads to error accumulation and limits overall accuracy.

Methods

The proposed method, Calibrated Adaptive Teacher (CAT), builds upon the Adaptive Teacher (AT) framework, which combines domain-adversarial training with a teacher-student architecture. The core novelty of CAT lies in introducing post-hoc calibration of the teacher network's predictions during the self-training process.

Here's a breakdown:

  • Teacher-Student Architecture: A "teacher" network generates pseudo-labels for unlabeled target data, and a "student" network is trained using these pseudo-labels along with labeled source data. The teacher's weights are an exponential moving average of the student's weights, promoting stability.
  • Domain-Adversarial Training: A domain discriminator is used to align features from the source and target domains, making them indistinguishable.
  • Calibrated Self-Training: Before generating pseudo-labels, the teacher's predictions on target data are calibrated. This means adjusting the confidence scores to better reflect the true probability of correctness. The paper explores four post-hoc calibration techniques: temperature scaling, vector scaling, matrix scaling, and calibrated predictions with covariate shift (CPCS).
  • Adaptive Confidence Threshold: To select the most reliable pseudo-labels, an adaptive threshold is used, which dynamically adjusts based on class difficulty, allowing more challenging classes to participate earlier in training.

Datasets

The primary dataset used for evaluation is the Paderborn University (PU) bearing dataset. This dataset is well-known for its challenging domain adaptation scenarios, featuring data from four different operating conditions (rotational speed, load torque, and radial force). Experiments were conducted using both time-domain and frequency-domain (fast Fourier transform) inputs.

Key Insights and Ideas

  • Calibration is Crucial for Pseudo-Label Quality: The paper strongly demonstrates that poor model calibration in the target domain is a major bottleneck for self-training UDA methods. By addressing this, CAT significantly improves the accuracy of generated pseudo-labels.
  • Temperature Scaling is Surprisingly Effective and Lightweight: Among the tested calibration methods, temperature scaling, a simple technique that uses only a single scalar parameter, proved to be the most effective and efficient. This is particularly interesting because it doesn't explicitly account for domain shift, suggesting that well-aligned features from domain adaptation are key.
  • Domain Shift Amplifies Calibration Issues: The study highlights that calibration errors are often worse in the target domain compared to the source domain, underscoring the need for domain-specific calibration strategies.
  • Post-hoc Calibration Integrated into Self-Training: Unlike previous work that focused on calibrating already trained models, CAT integrates calibration directly into the self-training loop, allowing it to improve both model performance and calibration simultaneously.

Key Results

  • State-of-the-Art Performance: CAT, particularly the CAT+TempScaling variant, achieved state-of-the-art performance on the PU benchmark.
  • Significant Accuracy Improvement: On average, CAT achieved 7.5% higher accuracy compared to Domain-Adversarial Neural Networks (DANNs) across 12 transfer tasks in the time domain.
  • Drastic Calibration Error Reduction: CAT resulted in 4 times lower calibration error (ECE) compared to DANNs.
  • Improved Pseudo-Label Accuracy: The introduction of calibration led to an approximate 10% absolute increase in pseudo-label accuracy during training.
  • Vector/Matrix Scaling Limitations: More complex calibration methods like vector and matrix scaling, which have more parameters, failed to generalize to the target domain and even degraded performance, suggesting that simpler, transferable calibration is preferred.

Limitations and Future Work

  • Limited Improvement on Easy Tasks: The paper notes that CAT's benefits are most pronounced on challenging tasks where baseline performance is low and calibration is poor. On easier tasks where models are already well-calibrated and perform well, CAT provides less significant gains.
  • No Improvement if Already Well-Calibrated: If target predictions are already well-calibrated, the method cannot be expected to improve performance.
  • Future Work - Data Augmentation: The authors suggest exploring the application of weak-strong data augmentations (common in computer vision) for time-series and spectrum data in the context of intelligent fault diagnosis. This could further enhance the robustness of the teacher and student networks.
  • Future Work - Investigating CPCS: While CPCS performed similarly to temperature scaling, further investigation into its effectiveness with different domain adaptation strategies might be warranted.

Summary updated

Abstract

Intelligent fault diagnosis (IFD) based on deep learning can achieve high accuracy from raw condition monitoring signals. However, models usually perform well on the training distribution only, and experience severe performance drops when applied to a different distribution. This is also observed in fault diagnosis, where assets are often operated in working conditions different from the ones in which the labeled data have been collected. The scenario where labeled data are available in a source domain and only unlabeled data are available in a target domain has been addressed recently by unsupervised domain adaptation (UDA) approaches for IFD. Recent methods have relied on self-training with confident pseudo-labels for the unlabeled target samples. However, the confidence-based selection of pseudo-labels is hindered by poorly calibrated uncertainty estimates in the target domain, primarily due to over-confident predictions, which limits the quality of pseudo-labels and leads to error accumulation. In this paper, we propose a novel method called Calibrated Adaptive Teacher (CAT), where we propose to calibrate the predictions of the teacher network on target samples throughout the self-training process, leveraging post hoc calibration techniques. We evaluate CAT on domain-adaptive IFD and perform extensive experiments on the Paderborn University (PU) benchmark for fault diagnosis of rolling bearings under varying operating conditions, using both time- and frequency-domain inputs. We compare four different calibration techniques within our framework, where temperature scaling is both the most effective and lightweight one. The resulting method-CAT+TempScaling-achieves state-of-the-art performance on most transfer tasks, with on average 7.5% higher accuracy and 4 times lower calibration error compared to domain-adversarial neural networks (DANNs) across the twelve PU transfer tasks.
BibTeX
@inproceedings{forest2024calibrated,
  author = {Florent Forest and Olga Fink},
  title = {{Calibrated Adaptive Teacher for Domain-Adaptive Intelligent Fault Diagnosis}},
  booktitle = {Sensors},
  year = {2024},
  doi = {10.3390/s24237539},
}
Figure