Learning to Generate Inversion-Resistant Model Explanations

Hoyong Jeong, Suyoung Lee, Sung Ju Hwang, Sooel SonNeurIPS 2022 · OpenReview

TL;DR

GNIME (Generative Noise Injector for Model Explanations) is a novel defense framework designed to prevent "Model Inversion" attacks, where adversaries reconstruct private training data (like faces) from AI explanations. By using a minimax game to inject subtle, optimized noise, GNIME successfully thwarts these privacy-leaking attacks while keeping the explanations useful and interpretable for human users.

What Problem Does This Paper Address?

As Deep Neural Networks (DNNs) are deployed in sensitive fields like medicine and facial recognition, there is a growing demand for Explainable AI (XAI). These systems often provide "feature attribution maps"—heatmaps showing which parts of an input image (e.g., the eyes or a specific medical marker) led to a certain prediction.

However, these explanations are a double-edged sword. Recent research has shown that they facilitate Model Inversion (MI) attacks. An adversary can take a model's prediction and its explanation to reconstruct a high-fidelity version of the original input image. This is a massive privacy risk: an attacker could potentially recover the faces of individuals in a private training set just by querying a public API. Until now, research has focused on how to perform these attacks, but very little work has been done on how to defend against them without breaking the usefulness of the explanations themselves.

Methods

The authors introduce GNIME, which treats defense as a competitive "two-player" game, similar to the logic behind Generative Adversarial Networks (GANs):

Overview of the GNIME framework training and deployment phases.
  1. The Inversion (INV) Network: This player acts as the "attacker," trying its best to reconstruct the original image from a given explanation.
  2. The Noise Generator (NG) Network: This player acts as the "defender." Its goal is to inject the smallest possible amount of noise into the explanation to make the INV network fail.

The two networks are trained together in a minimax game. The NG learns to identify and suppress "inversion-critical" features—the specific bits of data in an explanation that are most helpful for reconstruction—while leaving "explanation-critical" features intact.

Once training is complete, the target model remains unchanged (making this great for legacy systems). Only the Noise Generator is deployed to "scrub" explanations before they are released to the user. The system also includes a tunable parameter that allows service providers to choose their preferred balance between high privacy and high explanation fidelity.

Datasets

The researchers evaluated GNIME across four diverse image datasets:

  • CelebA: 30,000 celebrity face images (the primary benchmark for privacy/facial reconstruction).
  • MNIST: 70,000 images of handwritten digits.
  • CIFAR-10: 60,000 images across 10 object categories.
  • ImageNet-100: A high-resolution subset of the famous ImageNet dataset.
  • FaceScrub: Used as an "auxiliary" dataset to simulate a more realistic, practical adversary who doesn't have access to the exact original training data.

Key Insights and Ideas

  • Optimized vs. Random Noise: Simply adding Gaussian noise to an explanation is ineffective; it either fails to stop the attacker or makes the explanation unreadable for humans. GNIME’s "adversarial" noise is surgical, targeting only the features that aid reconstruction.
  • The "Upper Bound" of Privacy: The authors use "Prediction-only MI" (reconstructing an image using only the classification result, no explanation) as the gold standard for privacy. GNIME successfully pushes the privacy level of an XAI-enabled system back down toward this baseline.
  • Preserving Functionality: A key "aha moment" is that the features humans need to understand a model's decision are often different from the high-frequency pixel data an algorithm needs to reconstruct a face. GNIME exploits this gap.
  • Transferability: The noise generated to fool one specific inversion network during training generalizes well, successfully thwarting different types of inversion attacks used by independent adversaries.

Key Results

  • Massive Attack Degradation: In facial recognition tasks, GNIME decreased the "transferable classification accuracy" (a measure of how recognizable a reconstructed face is) by up to 84.8%.
  • Visual Privacy: Qualitative results show that while an unprotected attack can recover clear facial expressions and accessories (like glasses), an attack against GNIME results in blurry, generic images that protect the individual's identity.
Visual comparison of reconstructed images with and without GNIME.
  • High Utility: Despite the noise, the explanations remained highly accurate. Using the DeePSiM metric (which measures perceptual similarity), the authors showed that the "noised" explanations still highlight the correct regions of interest, maintaining their value for debugging and transparency.
  • Robustness: The defense proved effective across multiple XAI methods, including Grad, Grad-CAM, and Layer-wise Relevance Propagation (LRP).

Limitations and Future Work

  • Model Extraction Attacks: The paper notes that if an attacker can't get an explanation, they might try to build a "surrogate" model first to generate their own explanations. While GNIME doesn't stop the creation of a surrogate, it makes the resulting explanations less useful for inversion.
  • Retraining Requirements: While GNIME doesn't require retraining the target model, it does require training the NG and INV networks on a dataset similar to the target data.
  • Black-Box Assumption: The defense assumes a black-box adversary (common in Machine Learning as a Service). Future work could explore how GNIME holds up against a "white-box" adversary who has full access to the target model's internal weights.

Summary updated

Abstract

The wide adoption of deep neural networks (DNNs) in mission-critical applications has spurred the need for interpretable models that provide explanations of the model's decisions. Unfortunately, previous studies have demonstrated that model explanations facilitate information leakage, rendering DNN models vulnerable to model inversion attacks. These attacks enable the adversary to reconstruct original images based on model explanations, thus leaking privacy-sensitive features. To this end, we present Generative Noise Injector for Model Explanations (GNIME), a novel defense framework that perturbs model explanations to minimize the risk of model inversion attacks while preserving the interpretabilities of the generated explanations. Specifically, we formulate the defense training as a two-player minimax game between the inversion attack network on the one hand, which aims to invert model explanations, and the noise generator network on the other, which aims to inject perturbations to tamper with model inversion attacks. We demonstrate that GNIME significantly decreases the information leakage in model explanations, decreasing transferable classification accuracy in facial recognition models by up to 84.8% while preserving the original functionality of model explanations.
BibTeX
@inproceedings{jeong2022learning,
  author = {Hoyong Jeong and Suyoung Lee and Sung Ju Hwang and Sooel Son},
  title = {{Learning to Generate Inversion-Resistant Model Explanations}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=wuIOg3fbgx},
}
Figure