Abstract
With the complexity of the network structure, uncertainty inference has become an important task to improve the classification accuracy for artificial intelligence systems. For image classification tasks, we propose a structured DropConnect (SDC) framework to model the output of a deep neural network by a Dirichlet distribution. We introduce a DropConnect strategy on weights in the fully connected layers during training. In test, we split the network into several sub-networks, and then model the Dirichlet distribution by match its moments with the mean and variance of the outputs of these sub-networks. The entropy of the estimated Dirichlet distribution is finally utilized for uncertainty inference. In this paper, this framework is implemented on LeNet5 and VGG16 models for misclassification detection and out-of-distribution detection on MNIST and CIFAR-10 datasets. Experimental results show that the performance of the proposed SDC can be comparable to other uncertainty inference methods. Furthermore, the SDC is adapted well to different network structures with certain generalization capabilities and research prospects.
Introduction
Deep Neural Networks (DNNs) are often "overconfident," assigning high probabilities to incorrect predictions or unfamiliar data. This behavior poses significant risks in safety-critical applications like autonomous driving or medical diagnostics. While existing methods like Deep Ensembles and MC-Dropout attempt to quantify uncertainty, they often suffer from high computational costs or inconsistent results due to random sampling.
This paper introduces Structured DropConnect (SDC), a framework that transforms a standard classifier into an uncertainty-aware system by partitioning model weights into distinct sub-networks during inference.
Methodology
The SDC framework operates in two main phases: training and structured inference.
1. Training with DropConnect
Unlike standard Dropout, which zeroes out neurons, SDC utilizes DropConnect during training. This technique randomly zeroes out individual weights in the fully connected (FC) layers, encouraging the model to learn more robust features.
2. Structured Inference and Dirichlet Modeling
The core innovation of SDC occurs during testing. Instead of random sampling, the weights are partitioned into $\rho$ non-overlapping groups. This creates $\rho$ deterministic sub-networks from a single trained model.
The outputs from these sub-networks are treated as samples from a Dirichlet distribution. By matching the moments (mean and variance) of these outputs, the parameters of the Dirichlet distribution are estimated. The entropy of this distribution serves as the final uncertainty metric; high entropy indicates that the sub-networks disagree, signaling potential misclassification or out-of-distribution (OOD) data.
Experimental Setup
The framework was evaluated using classic benchmarks and their corresponding OOD counterparts:
- Architectures: LeNet5 and VGG16.
- In-Distribution Datasets: MNIST and CIFAR-10.
- OOD Datasets: SVHN (for MNIST-trained models) and Tiny ImageNet (for CIFAR-10-trained models).
Results and Discussion
Performance was measured using the Area Under the Receiver Operating Characteristic curve (AUROC), where higher scores indicate better detection of errors or unfamiliar data.
Misclassification Detection
On the MNIST dataset, SDC achieved an AUROC of 97.75%, slightly outperforming MC-Dropout (97.74%) and the baseline (97.52%).
Out-of-Distribution (OOD) Detection
SDC showed significant advantages in OOD detection, particularly on the more complex CIFAR-10 dataset. As shown in the results table, SDC achieved a Max Probability AUROC of 92.52% when detecting Tiny ImageNet samples, notably outperforming MC-Dropout's 87.60%. This suggests that the structured partitioning approach provides more diverse and reliable signals for uncertainty than random dropout.
Conclusion
Structured DropConnect offers a more stable and efficient way to quantify uncertainty in deep learning models. By replacing random sampling with structured weight partitioning, it achieves competitive or superior performance in detecting errors and unfamiliar data. Future research will focus on extending SDC to convolutional and attention-based layers to support modern, state-of-the-art architectures.
Summary updated