Latent Space Class Dispersion: Effective Test Data Quality Assessment for DNNs

Vivek Vekariya, Mojdeh Golagha, Andrea Stocco, Alexander PretschnerarXiv 2025 · DOI · arXiv

Introduction

In traditional software engineering, code coverage serves as a standard metric for evaluating test suite quality. However, in Deep Learning (DL), high accuracy on a test set does not necessarily imply a robust model; the test data may simply mirror the training distribution, failing to challenge the model's decision boundaries. To address this, researchers often employ mutation testing—injecting faults into models to see if a test set can detect them. While effective, mutation testing is computationally prohibitive, often requiring the retraining of hundreds of models. Existing alternatives like Distance-based Surprise Coverage (DSC) attempt to measure input "surprise" but are frequently slow and struggle to differentiate between meaningful edge cases and simple outliers.

Methodology: Latent Space Class Dispersion (LSCD)

To overcome these limitations, we propose Latent Space Class Dispersion (LSCD). This metric assesses test data quality by measuring the "spread" of data points within the model's latent space—specifically the last hidden layer (logit layer) before final classification.

The LSCD calculation involves three primary steps:

  1. Centroid Calculation: For each class, a centroid is established based on the training data's representations in the latent space.
  2. Dispersion Measurement: For a given test set, LSCD calculates the mean Euclidean distance of each test sample from its respective class centroid.
  3. Quality Inference: Test data clustered tightly around training centroids is considered "low quality" as it likely replicates known patterns. Conversely, higher dispersion indicates that the data explores the boundaries of the model’s logic, where failures are more probable.
Illustration of LSCD calculation in the latent space, showing centroids (C1, C2) and distances (d1, d2) for test points (x1, x2) relative to a decision boundary.

Unlike previous metrics like DSC that rely on computationally expensive nearest-neighbor searches, LSCD's use of class centroids makes it both more robust to outliers and significantly faster.

Experimental Setup

The effectiveness of LSCD was evaluated using three image classification benchmarks:

  • MNIST: Grayscale handwritten digits.
  • SVHN: Real-world house numbers from Street View.
  • GTSRB: Complex German traffic signs with varying environmental conditions.

Models included LeNet-5 for MNIST and Spatial Transformer Networks (STNs) for the more complex SVHN and GTSRB datasets. To validate LSCD's sensitivity to high-value test data, Coverage-Guided Fuzzing (CGF) was used to generate "corner cases"—valid images that are difficult for the model to classify.

Results and Discussion

LSCD demonstrates superior performance as a proxy for mutation testing and a measure of test suite effectiveness.

Correlation with Mutation Score

A critical finding is that LSCD exhibits a 0.87 correlation with the Mutation Score (MS), which measures a test set's ability to find real-world architectural and training faults. In contrast, the state-of-the-art DSC metric achieved only a 0.25 correlation. This suggests that LSCD can effectively replace expensive mutation testing cycles.

Computational Efficiency

LSCD is remarkably efficient, running 36.55 times faster than multi-threaded DSC. In our study, while DSC required nearly 3,500 minutes to process all mutants, LSCD completed the task in under 3 minutes.

Sensitivity to Corner Cases

When applied to fuzzed corner cases—where model accuracy dropped significantly (to ~1-9%)—LSCD scores spiked. This confirms its ability to quantitatively identify difficult, high-value test samples. Furthermore, using an automated validator (SelfOracle), we confirmed that 99.56% of these high-dispersion cases remained realistic images rather than random noise.

Conclusion and Future Work

Latent Space Class Dispersion provides a fast, scalable, and accurate metric for assessing DNN test data quality. By focusing on the latent space, LSCD offers a direct window into a model's generalizability and its potential failure points.

Future research will explore the application of LSCD to non-classification tasks, such as regression in autonomous driving or object detection. Additionally, we aim to integrate LSCD directly into the objective functions of fuzzing tools to guide the automated generation of diverse, high-quality test data across various architectures, including Transformers and deep ResNets.

Summary updated

Abstract

High-quality test datasets are crucial for assessing the reliability of Deep Neural Networks (DNNs). Mutation testing evaluates test dataset quality based on their ability to uncover injected faults in DNNs as measured by mutation score (MS). At the same time, its high computational cost motivates researchers to seek alternative test adequacy criteria. We propose Latent Space Class Dispersion (LSCD), a novel metric to quantify the quality of test datasets for DNNs. It measures the degree of dispersion within a test dataset as observed in the latent space of a DNN. Our empirical study shows that LSCD reveals and quantifies deficiencies in the test dataset of three popular benchmarks pertaining to image classification tasks using DNNs. Corner cases generated using automated fuzzing were found to help enhance fault detection and improve the overall quality of the original test sets calculated by MS and LSCD. Our experiments revealed a high positive correlation (0.87) between LSCD and MS, significantly higher than the one achieved by the well-studied Distance-based Surprise Coverage (0.25). These results were obtained from 129 mutants generated through pre-training mutation operators, with statistical significance and a high validity of corner cases. These observations suggest that LSCD can serve as a cost-effective alternative to expensive mutation testing, eliminating the need to generate mutant models while offering comparably valuable insights into test dataset quality for DNNs.
BibTeX
@misc{vekariya2025latent,
  author = {Vivek Vekariya and Mojdeh Golagha and Andrea Stocco and Alexander Pretschner},
  title = {{Latent Space Class Dispersion: Effective Test Data Quality Assessment for DNNs}},
  year = {2025},
  eprint = {2503.18799},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.18799},
}
Figure