Repair of Convolutional Neural Networks using Convex Optimization: Preliminary Experiments

Dario Guidotti, Francesco LeofanteCPS Summer School, PhD Workshop 2019 · OpenReview

Abstract

Deep Neural Networks (DNNs) are highly susceptible to adversarial perturbations, yet their immense scale makes formal verification and repair computationally prohibitive. This paper presents a novel hybrid architecture that replaces the complex final layers of a Convolutional Neural Network (CNN) with a linear Support Vector Machine (lSVM). This design choice reduces the parameter space and allows the repair process to be formulated as a computationally efficient convex optimization problem. Preliminary experiments on MNIST and CIFAR10 demonstrate that this approach maintains high accuracy while enabling efficient model repair and improved robustness.

Introduction

Adversarial perturbations—small, often imperceptible changes to input data—can cause state-of-the-art CNNs to fail. While formal verification offers guarantees of robustness, the sheer size of modern networks, often exceeding hundreds of millions of parameters, renders these tools ineffective for large-scale models. This research aims to bridge the gap between high-performance deep learning and formal repairability by proposing a "repairable by design" architecture.

Methodology

The proposed approach leverages Transfer Learning (TL) to create a hybrid model that is more amenable to mathematical analysis and repair.

Hybrid Architecture

A standard pre-trained CNN is used as a fixed feature extractor. The final fully-connected layers, which typically contain the bulk of the model's parameters, are replaced with a linear Support Vector Machine (lSVM). This architectural shift significantly reduces the number of free parameters. For instance, in a CIFAR10 model, parameters were reduced from 4.7 million to approximately 65,000.

The hybrid architecture utilizes a CNN as a feature extractor followed by a linear classification head.

Convex Optimization for Repair

By employing a linear classification head, the task of adjusting weights to correct misclassifications (repair) becomes a convex optimization problem. The authors use the cvxopt library to find the minimal weight adjustments necessary to satisfy correct classification constraints for specific adversarial examples. To handle the high-dimensional feature space of datasets like CIFAR10, the Wilcoxon Signed Rank test is applied to select the most relevant features, further reducing the optimization problem to under 6,000 variables.

Experimental Results

The hybrid approach was evaluated on the MNIST and CIFAR10 datasets to test its effectiveness in maintaining accuracy and facilitating repair.

Accuracy and Parameter Reduction

Replacing deep layers with an lSVM did not degrade performance. On MNIST, the hybrid model achieved 98.86% accuracy, slightly exceeding the original CNN's 97.12%. Crucially, the optimization variables for MNIST were reduced from over 400,000 to roughly 8,000.

Robustness and Generalization

The repair process demonstrated promising generalization on MNIST; fixing a single adversarial example provided protection against 20% of other unseen adversarial samples. However, CIFAR10 presented greater challenges. While specific targeted repairs were successful, generalizing this robustness across the more complex dataset proved difficult as accuracy dropped sharply with increasing perturbation levels.

Accuracy degradation and adversarial examples for MNIST and CIFAR10 under varying perturbation levels (Epsilon).

Discussion and Future Work

The study highlights that modularity is key to making large-scale models tractable for formal analysis. A notable observation was that adversarial vulnerabilities in hybrid models differ in nature from those in standard CNNs, suggesting that architecture changes the "flavor" of the model's weaknesses.

Future research will focus on:

  • Understanding the distinct characteristics of adversarial samples in hybrid architectures.
  • Incorporating "verification-driven kernels" into the SVM training phase to embed robustness inherently.
  • Extending convex optimization-based repair techniques into the convolutional layers themselves.

Conclusion

This work demonstrates that a hybrid CNN-lSVM architecture can maintain high accuracy while significantly simplifying the model for repair and verification tasks. By framing model repair as a convex optimization problem, the authors provide a scalable path toward more robust and reliable deep learning systems.

Summary updated

BibTeX
@inproceedings{guidotti2019repair,
  author = {Dario Guidotti and Francesco Leofante},
  title = {{Repair of Convolutional Neural Networks using Convex Optimization: Preliminary Experiments}},
  booktitle = {CPS Summer School, PhD Workshop},
  year = {2019},
  url = {https://openreview.net/forum?id=Dbq2n36rbe},
}
Figure