Abstract
Generative neural networks can be used to specify continuous transformations between images via latent-space interpolation. However, certifying that all images captured by the resulting path in the image manifold satisfy a given property can be very challenging. This is because this set is highly non-convex, thwarting existing scalable robustness analysis methods, which are often based on convex relaxations. We present ApproxLine, a scalable certification method that successfully verifies non-trivial specifications involving generative models and classifiers. ApproxLine can provide both sound deterministic and probabilistic guarantees, by capturing either infinite non-convex sets of neural network activation vectors or distributions over such sets. We show that ApproxLine is practically useful and can verify interesting interpolations in the networks latent space.
Introduction: The Challenge of Visible Robustness
Traditional neural network robustness research primarily focuses on adversarial perturbations—tiny, invisible pixel nudges designed to flip a classifier's label. While critical, this narrow focus overlooks real-world safety requirements. For instance, a self-driving car's vision system must remain robust to changes in lighting, and a facial recognition system should maintain consistency as a person turns their head.
Generative models, such as Variational Autoencoders (VAEs) and Generative Adversarial Networks (GANs), allow researchers to model these "visible" semantic transformations by interpolating between points in a low-dimensional latent space. However, certifying these interpolations is notoriously difficult. A straight line in latent space typically maps to a highly "wiggly," non-convex path in the high-dimensional image space.
Methodology: Bridging the Gap with APPROXLINE
Existing verification tools often rely on convex relaxations, wrapping the set of images in a simple geometric shape like a box or a zonotope. Because the latent-to-image path is so non-convex, these "wrappers" inevitably include nonsensical, blurry images that are merely pixel-wise averages of the endpoints. This over-approximation makes it impossible to prove that every image on the path is correctly classified.
To address this, researchers from ETH Zurich introduced APPROXLINE, a framework that balances precision and scalability through several key innovations:
- Non-Convex Tracking: Instead of a single convex shape, APPROXLINE represents activation sets as a list of line segments. As these segments pass through non-linear layers (like ReLU), they are split into smaller segments to maintain exactness.
- Adaptive Relaxation: To prevent an exponential explosion of segments, the algorithm employs a "Relaxation Heuristic." It identifies small or adjacent segments and merges them into interval boxes, allowing a customizable trade-off between precision and speed.
- Probabilistic Abstract Interpretation: APPROXLINE can compute a guaranteed lower bound on the probability that a property holds. For example, it can mathematically guarantee that at least 85% of images on an interpolation path will be classified with a specific attribute, rather than relying on statistical sampling.
- Implementation: Built on the DiffAI framework, APPROXLINE leverages GPU parallelization to handle large-scale networks.
Key Insights: Overcoming Convexity
The core challenge in certifying generative models is that the convex hull is a poor approximation of the image manifold.
As shown in the figure above, traditional verifiers are forced to consider "average" images that do not exist on the actual interpolation path. APPROXLINE avoids this by tracking the non-convex path more closely. The authors also note that the number of segments produced can be viewed as the "generative resolution" of the model; more complex models require more sophisticated tracking.
Experimental Results
The framework was evaluated on two primary benchmarks: CelebA (64x64 color images of celebrities) and MNIST.
- Scalability: APPROXLINE successfully analyzed networks with nearly 100,000 neurons, a scale where previous exact methods would fail due to memory or time constraints.
- Precision: On MNIST, while standard convex methods (Zonotope/DeepZono) failed to provide meaningful bounds, APPROXLINE achieved tight bounds in roughly half the time of exact methods.
- Real-World Applications:
- Attribute Consistency: In a head-rotation test via latent interpolation, 32 out of 40 attributes remained entirely robust.
- Attribute Independence: The tool verified that adding a "mustache" vector did not change "is bald" or "young" classifications for 30 out of 40 attributes, providing a lower bound of 83.5% for robustness probability.
- Model Comparison: The results indicated that Cycle-consistent Autoencoders (CycleAEs) generally produce more robust and consistent interpolations than standard VAEs.
Limitations and Future Work
While powerful, APPROXLINE's performance depends on hyperparameters that control the relaxation heuristic, which may require tuning for new architectures. Additionally, the probabilistic bounds currently work best with linear constraints. Future work aims to apply this to more complex distributions and explore "provably consistent" training processes for generative models.
Summary updated