Knowledge-Injected Federated Learning

Zhenan Fan, Zirui Zhou, Jian Pei, Michael P. Friedlander, Jiajie Hu, Chengliang Li, Yong ZhangarXiv 2022 · DOI · arXiv

Introduction

Standard Federated Learning (FL) has emerged as a powerful paradigm for training machine learning models across decentralized data silos while preserving privacy. However, traditional FL frameworks typically treat clients as passive "data containers." In many high-stakes industrial and medical applications, participants like factories or hospitals possess significant domain knowledge—human expertise, physical laws, or specialized craftsmanship—that is not fully captured in their local datasets.

This paper introduces Knowledge-Injected Federated Learning (FLwKM), a framework that allows decentralized clients to enhance a global model with their own local expert knowledge. By integrating "point-prediction" and "range-constraint" knowledge, FLwKM addresses the dual challenges of data scarcity and unreasonable model predictions that often plague purely data-driven approaches.

The FLwKM Framework

The core challenge in knowledge-injected FL is incorporating heterogeneous, proprietary knowledge without compromising privacy or requiring a complete overhaul of existing FL optimization algorithms. The proposed framework achieves this by treating local knowledge as a transformation of the global model rather than a modification of the training objective.

The architecture of Knowledge-Injected Federated Learning, where clients utilize both local data and expert knowledge to update a global server model.

Knowledge-Based Models (KMs)

FLwKM defines two primary types of local knowledge:

  1. Prediction-type (P-KM): Local expert models that provide specific point estimates based on domain expertise.
  2. Range-type (R-KM): A set of hard constraints or "guardrails" that define the set of physically or logically possible outcomes (e.g., a chemical process result that must fall within a specific safety range).

Function-Transformation Mapping

The innovation lies in the Function-Transformation Mapping ($\mathcal{T}$). Instead of altering the global server model directly, each client $m$ applies a transformation to the global model $f(\theta; \cdot)$ to create a personalized local model $f^m$. This mapping allows the global model to evolve in a function space that respects local constraints while benefiting from the aggregate data of all participants.

Visualization of the function-transformation mapping $\mathcal{T}$, showing how a global model is mapped to personalized local models in the function space.

Methodology and Implementation

The transformation formula used by each client is defined as: $$f^m = (1 - \lambda) \cdot \text{softmax}(f(x) + \infty \cdot (g_r(x) - \mathbf{1}^k)) + \lambda \cdot g_p(x)$$

In this equation, $\lambda$ (Trust Level) is a hyperparameter balancing the global data-driven model against local expert predictions. The $\infty$ term acts as a hard mask; if the global model predicts a class that the local R-KM deems impossible, that prediction is effectively zeroed out before the final softmax operation. Crucially, because this transformation occurs locally and only gradients are shared, the specific proprietary formulas of the local KMs remain private.

Experimental Evaluation

The authors evaluated FLwKM across three distinct scenarios:

  1. Industrial Coking Process: A real-world dataset from coal-mixing industries in China where data collection is expensive and domain expertise is vital.
  2. Forest Cover Type (Covtype): A classic UCI dataset for predicting forest cover from cartographic variables.
  3. Fashion MNIST (FMNIST): A standard benchmark for image classification.

Key Results

  • Constraint Satisfaction: In the industrial case study, FLwKM reduced the Percentage of Violation (POV)—the frequency with which the model breaks expert rules—from as high as 12% in standard FL to 0%.
  • Performance Gains: On the Covtype dataset, accuracy for specific clients jumped from 65% to 71%. The gains were most pronounced in "data-scarce" environments (e.g., using only 1% of available training data), demonstrating that expert knowledge can effectively compensate for missing data.
  • The "U-Shape" Effect: The impact of the trust level $\lambda$ follows a U-shape; optimal performance requires a careful balance between global data and local expertise. Too much reliance on either can lead to sub-optimal results.

Conclusion and Future Work

FLwKM provides a robust, algorithm-agnostic way to inject domain expertise into federated learning. It ensures that models remain physically and logically consistent while improving accuracy in data-poor environments. Future research directions include automating the tuning of the trust level $\lambda$ through bilevel optimization and extending the framework to handle potentially faulty or malicious local knowledge.

Summary updated

Abstract

Federated learning is an emerging technique for training models from decentralized data sets. In many applications, data owners participating in the federated learning system hold not only the data but also a set of domain knowledge. Such knowledge includes human know-how and craftsmanship that can be extremely helpful to the federated learning task. In this work, we propose a federated learning framework that allows the injection of participants' domain knowledge, where the key idea is to refine the global model with knowledge locally. The scenario we consider is motivated by a real industry-level application, and we demonstrate the effectiveness of our approach to this application.
BibTeX
@misc{fan2022knowledge,
  author = {Zhenan Fan and Zirui Zhou and Jian Pei and Michael P. Friedlander and Jiajie Hu and Chengliang Li and Yong Zhang},
  title = {{Knowledge-Injected Federated Learning}},
  year = {2022},
  eprint = {2208.07530},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2208.07530},
}
Figure