Abstract
In real-world engineering applications, "domain shift"—where the environment changes after a model is deployed—poses a significant challenge to the reliability of deep learning models. This research, presented at the NeurIPS 2021 Pre-registration Workshop, explores whether Uncertainty Quantification (UQ) can serve as an internal mechanism to detect and correct errors caused by domain shift in time-series regression without the need for retraining.
Introduction
Deep learning models often operate under the assumption that training and deployment environments are identical. However, in mechanical engineering, factors such as sensor degradation or structural changes (e.g., a loosening joint) introduce domain shifts that can lead to catastrophic failure. Researchers from Sandia National Laboratories and their collaborators investigated using a model’s predictive uncertainty as a "check engine" light to maintain performance under these shifting conditions.
Methodology
The study evaluated two primary architectures for time-series data: WaveNet and Transformers. To quantify uncertainty, the team utilized Monte Carlo (MC) Dropout, maintaining active dropout during inference to generate a distribution of predictions rather than a single point estimate.
The proposed technical pipeline consists of four stages:
- Sampling: Generating multiple MC prediction samples to create a distribution.
- Thresholding: Comparing predictive uncertainty against a baseline. The researchers found that the most effective threshold was the maximum uncertainty observed during the training phase.
- Correction: If the threshold is exceeded, the model adjusts the prediction using the mean of the samples and a standard deviation factor adjusted for the "skew" of the distribution.
- Output: Delivering the final corrected prediction.
Experimental Setup
The methodology was tested on two mechanical engineering benchmarks:
- Simulated Mass-Spring System: A controlled "toy problem" where domain shift was induced by abruptly altering the stiffness ($k$) of the spring.
- Frictional Jointed Structure: A complex, real-world engineering problem involving contact stress and linearized modal predictions, where stiffness changes again represented the domain shift.
Key Findings
The research yielded several critical insights into the behavior of UQ in engineering contexts:
- Detection Efficacy: UQ proved to be a highly reliable detector of domain shift. By baselining against the training domain, the models could accurately flag when they were operating "out of their depth."
- Thresholding Strategy: Static error thresholds (e.g., a flat 10% error) were ineffective because different parts of the training domain have naturally varying noise levels. The robust approach was defining thresholds relative to training-time maximums.
- Architectural Performance: The Transformer architecture significantly outperformed WaveNet in detection tasks, achieving a solid F1 score of 0.87.
- Error Accumulation: In structural dynamics, uncertainty tends to compound over time as errors propagate through the time-series, providing a useful proxy for tracking gradual model degradation.
Results and Discussion
While the detection of domain shift was a clear success, the subsequent "correction" phase proved more difficult. For the mass-spring system, corrections provided only modest improvements. In the more complex frictional jointed structure, the results were noisy. Ultimately, the quantitative improvements in Mean Squared Error (MSE) were not statistically significant. The corrective factors (mean and skew) were likely too conservative to overcome the magnitude of the errors induced by significant physical shifts.
Conclusion and Future Directions
This exploratory research demonstrates that while UQ is a powerful tool for identifying when a model is failing due to domain shift, simple statistical corrections may be insufficient for drastic physical changes in mechanical systems. The authors suggest that future work should focus on more aggressive recovery algorithms that can better interpret high-uncertainty signals to reconstruct the true physical state of the system.
Summary updated