Ensemble Attention Distillation for Privacy-Preserving Federated Learning

Xuan Gong, Abhishek Sharma, Srikrishna Karanam, Ziyan Wu, Terrence Chen, David S. Doermann, Arun Innanje

Introduction

Federated Learning (FL) aims to train a central AI model using data spread across many decentralized nodes—such as hospitals or mobile devices—without moving that data to a central server. However, standard FL faces three major hurdles: privacy leakage from shared gradients, communication bottlenecks due to iterative updates, and data heterogeneity (non-IID data) across nodes.

Comparison of standard parameter-sharing FL versus the proposed privacy-preserving FL framework.

To address these, the authors propose FedAD (Federated Attention Distillation), a "one-shot" framework that trains a central model without sharing private data, gradients, or model parameters. By distilling both predictions (logits) and structural reasoning (attention maps) from local models using unlabeled public data, it achieves state-of-the-art performance while drastically reducing communication costs and privacy risks.

The FedAD Framework

The core of FedAD is a one-shot distillation framework. Instead of updating the model iteratively, local nodes train their models independently on private data. A central server then uses an unlabeled public dataset to "interrogate" these local models and extract their collective knowledge.

The FedAD pipeline: local nodes generate logits and attention maps from public data, which are then ensembled and distilled into a central model.

The Distillation Pipeline

  1. Local Training: Each node trains its own model on its private data.
  2. Public Data Inference: The central server sends unlabeled public images to the local nodes.
  3. Knowledge Extraction: Local nodes return Logits (final predictions) and Attention Maps (generated via Grad-CAM). The attention maps show which parts of an image the model focused on—the "why" behind the prediction.
  4. Ensemble Attention Distillation: The central model is trained to match the ensemble of local predictions and reasoning processes.

The "Secret Sauce": Attention Bound Constraint

Standard distillation typically averages predictions. FedAD introduces a novel constraint based on the Intersection and Union of attention maps:

  • Consensus (Intersection): The central model is encouraged to focus on regions where all local models agree (e.g., the actual tumor in a medical scan).
  • Diversity (Union): The central model is penalized if it focuses on regions that none of the local experts found relevant, while still allowing it to learn from the unique "expertise" of individual nodes.

Experimental Evaluation

The authors evaluated FedAD across diverse tasks, including natural image classification (CIFAR-10/100), medical image classification (NIH ChestX-ray14), and semantic segmentation (Cityscapes, BraTS 2018).

Key Results

  • Superior Accuracy: On CIFAR-10 with high data heterogeneity ($\alpha=0.1$), FedAD achieved 73.11% accuracy, outperforming FedDF (71.36%) and FedAvg (68.37%).
  • Bandwidth Efficiency: FedAD required only 10.1 GB of bandwidth compared to 29.4 GB for FedDF—a nearly 3x reduction while achieving higher accuracy.
  • Cross-Domain Robustness: In medical imaging, FedAD successfully trained a model for 14 chest pathologies by distilling from nodes that only saw subsets of those classes, proving the method's robustness to domain shifts.

Conclusion

FedAD represents a significant step forward in privacy-preserving machine learning. By leveraging structural knowledge through attention distillation, it overcomes the limitations of traditional parameter-sharing FL. While it currently requires a public dataset as a transfer medium, future work may explore using Generative Adversarial Networks (GANs) to synthesize distillation data, potentially removing this dependency entirely.

Summary updated

Abstract

We consider the problem of Federated Learning (FL) where numerous decentralized computational nodes collaborate with each other to train a centralized machine learning model without explicitly sharing their local data samples. Such decentralized training naturally leads to issues of imbalanced or differing data distributions among the local models and challenges in fusing them into a central model. Existing FL methods deal with these issues by either sharing local parameters or fusing models via online distillation. However, such a design leads to multiple rounds of inter-node communication resulting in substantial band-width consumption, while also increasing the risk of data leakage and consequent privacy issues. To address these problems, we propose a new distillation-based FL frame-work that can preserve privacy by design, while also consuming substantially less network communication resources when compared to the current methods. Our framework engages in inter-node communication using only publicly available and approved datasets, thereby giving explicit privacy control to the user. To distill knowledge among the various local models, our framework involves a novel ensemble distillation algorithm that uses both final prediction as well as model attention. This algorithm explicitly considers the diversity among various local nodes while also seeking consensus among them. This results in a comprehensive technique to distill knowledge from various decentralized nodes. We demonstrate the various aspects and the associated benefits of our FL framework through extensive experiments that produce state-of-the-art results on both classification and segmentation tasks on natural and medical images.
BibTeX
@inproceedings{gong2021ensemble,
  author = {Xuan Gong and Abhishek Sharma and Srikrishna Karanam and Ziyan Wu and Terrence Chen and David S. Doermann and Arun Innanje},
  title = {{Ensemble Attention Distillation for Privacy-Preserving Federated Learning}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  doi = {10.1109/iccv48922.2021.01480},
}
Figure