Abstract
Deep Neural Networks (DNNs) are notoriously vulnerable to adversarial examples—inputs with imperceptible perturbations designed to cause misclassification. This paper introduces the Deep Minimax Probability Machine (DeepMPM), a hybrid architecture that replaces the standard softmax output layer with a Minimax Probability Machine (MPM). By optimizing for the worst-case misclassification probability using global data statistics, DeepMPM achieves significant robustness against adversarial attacks while maintaining high standard classification accuracy.
Introduction
While DNNs excel at pattern recognition, their reliance on local optimization through softmax classifiers makes them "brittle." Softmax focuses on maximizing the likelihood of individual labels in isolation, failing to account for global class distributions or worst-case scenarios. DeepMPM addresses this by shifting the objective from average-case accuracy to a robust statistical bound, creating a decision boundary that is resilient by design.
Architecture and Methodology
The DeepMPM framework combines the powerful feature-learning capabilities of convolutional neural networks (CNNs) with the robust guarantees of the MPM.
Key Components:
- Feature Extraction: A standard CNN backbone maps raw input into a high-dimensional feature space.
- The MPM Layer: Instead of softmax, an MPM layer identifies a decision hyperplane that minimizes the maximum possible probability of misclassification.
- Global Statistics: The MPM objective utilizes the mean and covariance of data classes, forcing the network to learn features that are stable across the entire class distribution rather than just for individual points.
- End-to-End Training: The authors derived a differentiable loss function using the Lagrangian multiplier method, enabling standard backpropagation to train the entire system end-to-end.
Experimental Evaluation
The model was evaluated on the MNIST and CIFAR-10 datasets, focusing on binary classification tasks to demonstrate the core mechanics of the MPM integration.
Standard Accuracy
DeepMPM achieves classification performance comparable to state-of-the-art CNNs on clean data, reaching approximately 99.7% accuracy on MNIST and 95% on CIFAR-10 tasks. This demonstrates that the robust objective does not come at the cost of baseline performance.
Robustness to Adversarial Attacks
Under the Fast Gradient Sign Method (FGSM) attack, DeepMPM showed a marked advantage. While standard CNN accuracy plummeted as the perturbation magnitude (epsilon) increased, DeepMPM's accuracy degraded much more slowly.
In more sophisticated evaluations using the Carlini & Wagner (L2) attack, DeepMPM maintained an average accuracy of 87.36% on MNIST, whereas the standard CNN was almost completely defeated, dropping to near 0% accuracy.
Key Insights
- Worst-Case Optimization: By minimizing the upper bound of misclassification, the model naturally finds more conservative and resilient decision boundaries.
- Adversarial Transferability: Adversarial examples generated to fool standard CNNs often fail against DeepMPM, suggesting the model learns fundamentally different, more robust features.
- Detection: To successfully fool DeepMPM, attackers must apply significantly larger perturbations, making such attacks more likely to be detected by human observers.
Conclusion and Future Work
DeepMPM provides a robust alternative to traditional deep learning architectures by embedding statistical guarantees directly into the model's output layer. Future research will focus on extending the model to native multi-class classification and optimizing the computation of covariance matrices to scale the architecture for massive datasets like ImageNet.
Summary updated