Abstract
Deep Neural Networks (DNNs) are increasingly deployed in safety-critical and socially sensitive domains. However, they remain fragile to adversarial perturbations and often lack formal guarantees for global properties. This paper introduces REGLO, the first framework for post-hoc repair of pre-trained DNNs to satisfy global robustness and individual fairness properties provably. REGLO identifies violating linear regions via gradient analysis and employs robust convex optimization to modify final-layer weights. Experimental results on credit, security, and vision datasets demonstrate that REGLO achieves 0% violation rates with minimal accuracy loss, significantly outperforming existing training-based methods in both speed and reliability.
Introduction
Deep Neural Networks (DNNs) are notoriously fragile. While "local robustness"—ensuring a specific input $x$ is not easily fooled—is well-studied, global robustness ensures a model is stable across an entire region of the input space. This concept is intrinsically linked to individual fairness, where a model must produce similar outputs for inputs that differ only in non-sensitive attributes (e.g., age or gender in credit scoring).
Current solutions typically rely on "robust training," which adds penalty terms to the loss function. However, training is stochastic and offers no hard guarantees of property satisfaction. Furthermore, retraining is computationally expensive and requires access to original training data, which may be unavailable. REGLO addresses these challenges as a post-hoc repair problem, directly modifying a pre-trained model's weights to guarantee property satisfaction without full retraining.
Methodology
The core of REGLO is based on a key mathematical observation: any counterexample to a global robustness property must occur in a region with a large gradient. For ReLU-based networks, which are piecewise linear, REGLO follows a three-step iterative process:
- Violation Search: Using Mixed-Integer Linear Programming (MILP), REGLO searches for "linear regions" where the network's gradient exceeds a safety threshold.
- Bound Estimation: It utilizes Interval Taylor Network Expansion (ITNE) to compute the actual output bounds for these violating regions.
- Robust Optimization: REGLO formulates a convex optimization problem to find the minimal change to the weights of the last hidden layer.
By restricting weight changes to the final layer, REGLO avoids the "butterfly effect" where small changes in early layers flip ReLU activations downstream, potentially creating new violations. This "surgical" approach makes the repair predictable and mathematically sound.
Experimental Evaluation
The authors evaluated REGLO across several benchmarks, including the German Credit Dataset (fairness), VGRP security benchmarks (cryptojacking and spam detection), and the German Traffic Sign Recognition Benchmark (GTSRB).
Key Results
- Guaranteed Fairness: On the German Credit dataset, REGLO reduced the violation rate (PGD-R) to 0% with less than a 0.9% drop in accuracy, significantly outperforming specialized fairness baselines like CertiFair.
- Efficiency and Scalability: Compared to training-based approaches like VGRP, REGLO was up to 99.96% faster, completing repairs in seconds that previously took hours.
- Robustness in Vision: On the GTSRB dataset, REGLO reduced verification bounds by up to 99% and eliminated all discovered adversarial examples while maintaining accuracy above 90%.
Discussion and Conclusion
REGLO provides a sound and complete method for repairing piecewise linear networks. Its ability to transform a global search for counterexamples into a structured optimization problem allows for rapid, provable updates.
The primary limitation is the scalability of MILP for very large, high-dimensional networks. Future work includes developing statistical techniques to handle higher complexity without sacrificing completeness, and exploring multi-layer repair strategies that offer more flexibility while maintaining provable guarantees.
Summary updated