Towards Interpretable Federated Learning

Anran Li, Rui Liu, Ming Hu, Luu Anh Tuan, Han Yu

Abstract

Federated learning (FL) enables multiple data owners to build machine learning models collaboratively without exposing their private local data. In order for FL to achieve widespread adoption, it is important to balance the need for performance, privacy-preservation and interpretability, especially in mission critical applications such as finance and healthcare. Thus, interpretable federated learning (IFL) has become an emerging topic of research attracting significant interest from the academia and the industry alike. Its interdisciplinary nature can be challenging for new researchers to pick up. In this paper, we bridge this gap by providing (to the best of our knowledge) the first survey on IFL. We propose a unique IFL taxonomy which covers relevant works enabling FL models to explain the prediction results, support model debugging, and provide insights into the contributions made by individual data owners or data samples, which in turn, is crucial for allocating rewards fairly to motivate active and reliable participation in FL. We conduct comprehensive analysis of the representative IFL approaches, the commonly adopted performance evaluation metrics, and promising directions towards building versatile IFL techniques.

Introduction: The Transparency Gap in Federated Learning

Federated Learning (FL) allows multiple parties, such as hospitals or financial institutions, to train a shared machine learning model without exchanging their raw, private data. While this paradigm significantly enhances privacy, it creates a "transparency gap." Because the data remains local and invisible to the central server and other participants, it is difficult to understand why a model makes specific predictions, which participants contribute the most valuable data, or how to debug the system when it fails.

Standard Explainable AI (XAI) methods typically require direct access to training data, which violates the core premise of FL. Consequently, Interpretable Federated Learning (IFL) must navigate a complex three-way trade-off between predictive performance, privacy preservation, and interpretability.

A Taxonomy of IFL Techniques

To organize the growing landscape of IFL, this survey proposes a taxonomy that categorizes techniques into five functional stages of the FL pipeline:

  1. Interpretable Client Selection: Identifying participants with high-quality data by monitoring model deviations or using Influence Functions to calculate a client's impact on the final model.
  2. Interpretable Sample Selection: Enabling clients to identify local data points most relevant to the global task using gradient norms or loss-based importance scoring.
  3. Interpretable Feature Selection: Determining which specific variables (e.g., medical symptoms) drive model decisions using model-agnostic (e.g., Gini impurity) or model-specific (e.g., LASSO) methods.
  4. Interpretable Model Optimization: Developing inherently transparent models, such as Federated Decision Trees, or applying sparsity constraints to highlight meaningful parameters.
  5. Interpretable Contribution Evaluation: Quantifying participant value using Shapley Values (SV) from cooperative game theory, often employing approximation methods like Truncated Monte Carlo to manage computational complexity.
The proposed taxonomy of Interpretable Federated Learning (IFL) techniques, categorized by functional stages.

Key Insights and Trade-offs

Interpretability in FL is not a monolithic requirement; it varies by stakeholder. The FL Server requires interpretability for system debugging and security (e.g., detecting malicious actors), while FL Clients need transparency to understand selection criteria and reward fairness.

Furthermore, there is an inherent tension between privacy and transparency. Detailed explanations can inadvertently leak information about private training data. To mitigate this, IFL is often integrated with privacy-preserving techniques like Differential Privacy (DP) or Homomorphic Encryption (HE) to remain secure.

Comparison of IFL techniques across stakeholders, threat models, privacy protection targets, and effectiveness metrics.

Evaluation Metrics and Results

The survey highlights several critical metrics for assessing IFL:

  • Shapley Value-based methods are the gold standard for fair reward allocation, despite their high computational cost.
  • Post-interpretation performance measures how much an explanation helps improve model accuracy or robustness in subsequent training rounds.
  • Faithfulness evaluates whether the identified "important" features are truly critical to the model's performance; removing them should result in a significant performance drop.

Future Directions

The authors identify several "open frontiers" for the research community:

  • Mimic Learning: Approximating complex "black-box" FL models with simpler, inherently interpretable "student" models.
  • Robustness to Noise: Developing IFL techniques that can handle "label noise" in real-world datasets without sacrificing transparency.
  • Complex Threat Models: Ensuring interpretability remains robust even when participants are actively malicious or colluding.
  • Standardized Benchmarks: Establishing standardized metrics for interpretability to allow for objective comparisons between different IFL approaches.

Summary updated

Abstract

Federated learning (FL) enables multiple data owners to build machine learning models collaboratively without exposing their private local data. In order for FL to achieve widespread adoption, it is important to balance the need for performance, privacy-preservation and interpretability, especially in mission critical applications such as finance and healthcare. Thus, interpretable federated learning (IFL) has become an emerging topic of research attracting significant interest from the academia and the industry alike. Its interdisciplinary nature can be challenging for new researchers to pick up. In this paper, we bridge this gap by providing (to the best of our knowledge) the first survey on IFL. We propose a unique IFL taxonomy which covers relevant works enabling FL models to explain the prediction results, support model debugging, and provide insights into the contributions made by individual data owners or data samples, which in turn, is crucial for allocating rewards fairly to motivate active and reliable participation in FL. We conduct comprehensive analysis of the representative IFL approaches, the commonly adopted performance evaluation metrics, and promising directions towards building versatile IFL techniques.
BibTeX
@misc{li2023towards,
  author = {Anran Li and Rui Liu and Ming Hu and Luu Anh Tuan and Han Yu},
  title = {{Towards Interpretable Federated Learning}},
  year = {2023},
  eprint = {2302.13473},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2302.13473},
}
Figure