DNN Explanation for Safety Analysis: an Empirical Evaluation of Clustering-based Approaches

Mohammed Oualid Attaoui, Hazem M. Fahmy, Fabrizio Pastore, Lionel C. Briand

TL;DR

This paper tackles the critical challenge of understanding why image-processing Deep Neural Networks (DNNs) fail in safety-critical applications. By empirically evaluating 99 different pipeline configurations, the authors identify a robust approach that effectively clusters failure-inducing images, enabling engineers to pinpoint root causes and improve DNN reliability.

What Problem Does This Paper Address?

Deep Neural Networks (DNNs) are increasingly used in safety-critical domains like autonomous driving, but their "black-box" nature makes it difficult to understand why they fail. When a DNN makes an incorrect prediction, especially in a safety-critical context, it's crucial to identify the underlying cause to prevent future failures. Existing methods for analyzing DNN failures often rely on specific configurations that may not be optimal. This paper aims to systematically explore and evaluate a wide range of pipeline components for analyzing image-processing DNN failures, seeking the most effective approach for root cause analysis.

Methods

The authors propose and evaluate 99 different pipelines for analyzing DNN failures. These pipelines are built by combining various components:

  • Feature Extraction: Methods to represent images in a way that's useful for clustering. This includes:
  • Heatmap-based methods: Using Layer-Wise Relevance Propagation (LRP) and the HUDD (Heatmap-based Unsupervised Debugging of DNNs) approach to identify important image regions.
  • Transfer Learning: Utilizing pre-trained models like VGG-16, ResNet-50, Inception-V3, and Xception, trained on large datasets like ImageNet, to extract features. They also explore fine-tuning these models to adapt them to specific tasks.
  • Autoencoders (AE): Unsupervised neural networks that learn compressed representations of data.
  • Dimensionality Reduction: Techniques to reduce the number of features while preserving important information. They evaluate Principal Component Analysis (PCA) and Uniform Manifold Approximation and Projection (UMAP).
  • Clustering Algorithms: Methods to group similar images together. The paper compares K-means, DBSCAN, and HDBSCAN.

The core idea is to feed failure-inducing images into these pipelines, which then generate clusters of similar images. By inspecting these clusters, engineers can identify commonalities that represent the root cause of the DNN's failure.

Datasets

The study uses six case study subjects, which are DNNs designed for automotive applications. These include:

  • Gaze detection (UnityEyes dataset)
  • Drowsiness detection (UnityEyes dataset)
  • Head pose detection (Blender dataset)
  • Unattended child detection (SVIRO dataset)
  • Steering angle prediction (SAP dataset)
  • Car position detection (CPD dataset)

For evaluation, the authors use both original test set images that cause DNN failures and synthetically modified images with injected failure scenarios (e.g., adding noise, blurriness, masks, sunglasses) to create controlled experiments.

Key Insights and Ideas

  • Fine-tuning can hurt purity: Surprisingly, fine-tuning transfer learning models (which aims to adapt them to a specific task) often led to lower purity in the generated clusters. This is likely because the fine-tuning process, when not including the failure scenarios, can cause the model to "forget" features useful for detecting those specific faults.
  • UMAP excels at separation: UMAP, a non-linear dimensionality reduction technique, consistently outperformed PCA in creating well-separated clusters, making it easier for clustering algorithms to distinguish them.
  • DBSCAN/HDBSCAN are robust: Density-based clustering algorithms like DBSCAN and HDBSCAN generally produced purer clusters and were less affected by infrequent failure scenarios compared to K-means.
  • Non-fine-tuned transfer learning is key: The best performing pipelines consistently used non-fine-tuned transfer learning models (like VGG-16, ResNet-50, Inception-V3, Xception) combined with UMAP for dimensionality reduction and DBSCAN for clustering.

Key Results

The paper systematically evaluates 99 pipelines and identifies a winning combination: a pipeline using VGG-16 as the feature extractor, no fine-tuning, UMAP for dimensionality reduction, and DBSCAN for clustering.

  • This pipeline achieved an average purity of 94.3% and coverage of 96.7% when failure scenarios were equally distributed.
  • It also performed well with infrequent failure scenarios, maintaining good purity and coverage.
  • For pre-existing failure scenarios, this pipeline achieved 100% coverage and an average purity of 87%.
  • The identified best pipeline generates clusters that are typically small, with 75% of clusters containing at most 13 images, making manual inspection feasible.

Limitations and Future Work

  • Internal Validity: The authors acknowledge that the dataset with injected faults was created with their approach in mind, but they mitigate this by using independent methods and making their data public. They also conducted a user study to validate that engineers can indeed understand root causes from the generated clusters.
  • External Validity: The study focused on automotive DNNs. While the authors believe their approach is generalizable due to the use of ImageNet-pre-trained models and standard DNN components, further validation on diverse domains would be beneficial.
  • Future Work: The authors suggest integrating their best analysis pipeline with SEDE (a technique for automatically generating similar images for retraining) to further automate the DNN improvement process. They also note that while their current work focuses on model-level testing, system-level testing of DNNs is a promising area for future research.

Summary updated

BibTeX
@article{attaoui2024dnn,
  author = {Mohammed Oualid Attaoui and Hazem M. Fahmy and Fabrizio Pastore and Lionel C. Briand},
  title = {{DNN Explanation for Safety Analysis: an Empirical Evaluation of Clustering-based Approaches}},
  journal = {ACM Trans. Softw. Eng. Methodol.},
  year = {2024},
  doi = {10.1145/3643671},
}
Figure