REGLO: Provable Neural Network Repair for Global Robustness Properties

Feisi Fu, Zhilu Wang, Weichao Zhou, Yixuan Wang, Jiameng Fan, Chao Huang, Qi Zhu, Xin Chen, Wenchao Li

Abstract

Deep Neural Networks (DNNs) are increasingly deployed in safety-critical and socially sensitive domains. However, they remain fragile to adversarial perturbations and often lack formal guarantees for global properties. This paper introduces REGLO, the first framework for post-hoc repair of pre-trained DNNs to satisfy global robustness and individual fairness properties provably. REGLO identifies violating linear regions via gradient analysis and employs robust convex optimization to modify final-layer weights. Experimental results on credit, security, and vision datasets demonstrate that REGLO achieves 0% violation rates with minimal accuracy loss, significantly outperforming existing training-based methods in both speed and reliability.

Introduction

Deep Neural Networks (DNNs) are notoriously fragile. While "local robustness"—ensuring a specific input $x$ is not easily fooled—is well-studied, global robustness ensures a model is stable across an entire region of the input space. This concept is intrinsically linked to individual fairness, where a model must produce similar outputs for inputs that differ only in non-sensitive attributes (e.g., age or gender in credit scoring).

Current solutions typically rely on "robust training," which adds penalty terms to the loss function. However, training is stochastic and offers no hard guarantees of property satisfaction. Furthermore, retraining is computationally expensive and requires access to original training data, which may be unavailable. REGLO addresses these challenges as a post-hoc repair problem, directly modifying a pre-trained model's weights to guarantee property satisfaction without full retraining.

Methodology

The core of REGLO is based on a key mathematical observation: any counterexample to a global robustness property must occur in a region with a large gradient. For ReLU-based networks, which are piecewise linear, REGLO follows a three-step iterative process:

  1. Violation Search: Using Mixed-Integer Linear Programming (MILP), REGLO searches for "linear regions" where the network's gradient exceeds a safety threshold.
  2. Bound Estimation: It utilizes Interval Taylor Network Expansion (ITNE) to compute the actual output bounds for these violating regions.
  3. Robust Optimization: REGLO formulates a convex optimization problem to find the minimal change to the weights of the last hidden layer.
The REGLO framework: (1) identifying violating regions, (2) estimating bounds, and (3) performing robust optimization on the last layer.

By restricting weight changes to the final layer, REGLO avoids the "butterfly effect" where small changes in early layers flip ReLU activations downstream, potentially creating new violations. This "surgical" approach makes the repair predictable and mathematically sound.

Experimental Evaluation

The authors evaluated REGLO across several benchmarks, including the German Credit Dataset (fairness), VGRP security benchmarks (cryptojacking and spam detection), and the German Traffic Sign Recognition Benchmark (GTSRB).

Key Results

  • Guaranteed Fairness: On the German Credit dataset, REGLO reduced the violation rate (PGD-R) to 0% with less than a 0.9% drop in accuracy, significantly outperforming specialized fairness baselines like CertiFair.
  • Efficiency and Scalability: Compared to training-based approaches like VGRP, REGLO was up to 99.96% faster, completing repairs in seconds that previously took hours.
  • Robustness in Vision: On the GTSRB dataset, REGLO reduced verification bounds by up to 99% and eliminated all discovered adversarial examples while maintaining accuracy above 90%.
Performance of REGLO across repair iterations, showing high test accuracy compared to baselines and a rapid decrease in robustness violations (PGD-B).

Discussion and Conclusion

REGLO provides a sound and complete method for repairing piecewise linear networks. Its ability to transform a global search for counterexamples into a structured optimization problem allows for rapid, provable updates.

The primary limitation is the scalability of MILP for very large, high-dimensional networks. Future work includes developing statistical techniques to handle higher complexity without sacrificing completeness, and exploring multi-layer repair strategies that offer more flexibility while maintaining provable guarantees.

Summary updated

Abstract

We present REGLO, a novel methodology for repairing pretrained neural networks to satisfy global robustness and individual fairness properties. A neural network is said to be globally robust with respect to a given input region if and only if all the input points in the region are locally robust. This notion of global robustness also captures the notion of individual fairness as a special case. We prove that any counterexample to a global robustness property must exhibit a corresponding large gradient. For ReLU networks, this result allows us to efficiently identify the linear regions that violate a given global robustness property. By formulating and solving a suitable robust convex optimization problem, REGLO then computes a minimal weight change that will provably repair these violating linear regions.
BibTeX
@inproceedings{fu2024reglo,
  author = {Feisi Fu and Zhilu Wang and Weichao Zhou and Yixuan Wang and Jiameng Fan and Chao Huang and Qi Zhu and Xin Chen and Wenchao Li},
  title = {{REGLO: Provable Neural Network Repair for Global Robustness Properties}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i11.29094},
}
Figure