Federated Learning for Face Recognition with Gradient Correction

Yifan Niu, Weihong Deng

Abstract

With increasing appealing to privacy issues in face recognition, federated learning has emerged as one of the most prevalent approaches to study the unconstrained face recognition problem with private decentralized data. However, conventional decentralized federated algorithm sharing whole parameters of networks among clients suffers from privacy leakage in face recognition scene. In this work, we introduce a framework, FedGC, to tackle federated learning for face recognition and guarantees higher privacy. We explore a novel idea of correcting gradients from the perspective of backward propagation and propose a softmax-based regularizer to correct gradients of class embeddings by precisely injecting a cross-client gradient term. Theoretically, we show that FedGC constitutes a valid loss function similar to standard softmax. Extensive experiments have been conducted to validate the superiority of FedGC which can match the performance of conventional centralized methods utilizing full training dataset on several popular benchmark datasets.

Introduction

Face recognition (FR) systems require massive, diverse datasets to achieve high accuracy. However, centralizing such sensitive biometric data raises significant privacy concerns. Federated Learning (FL) offers a promising alternative by training models on local devices without moving raw data.

Despite its potential, standard FL algorithms like FedAvg face two major hurdles in FR:

  1. Privacy Leakage: Sharing the final classification layer (class embeddings) can allow adversaries to reconstruct high-fidelity face images via attacks like DeepInversion.
  2. Non-IID Performance Gap: Face data is typically Non-Identically and Independently Distributed (Non-IID). Each client holds a unique set of identities, meaning local models lack the "global perspective" to distinguish their local faces from those held by other clients. This leads to "client drift," where the global feature space becomes cluttered and overlapping.

The FedGC Framework

To address these challenges, we propose FedGC (Federated Gradient Correction). The framework is built upon a baseline of Federated Learning with Private Embedding (FedPE), where each client keeps its final classification layer private, and only the feature-extracting "backbone" is synchronized.

Overview of the FedGC framework, illustrating the interaction between clients (local training with private embeddings) and the server (aggregation and gradient correction).

As shown in the framework overview, FedGC ensures that identity-specific information never leaves the client. However, because clients do not see each other's classes, the model lacks the inter-client discriminative power found in centralized training. FedGC solves this by introducing a Softmax-based Regularizer on the server side. This regularizer injects a "cross-client gradient term" during backpropagation, effectively simulating a global softmax loss and ensuring that features across different clients remain separable.

Gradient Correction Mechanism

The core innovation of FedGC is the correction of local gradients to align them with the global objective. In standard private-embedding FL, local updates only focus on within-client compactness. FedGC provides the missing "push" to keep identities on different clients distinct.

Visualization of the gradient correction mechanism. (a) Without correction, client updates drift from the ideal SGD path. (b) With correction, the update is steered back towards the global optimum.

This mechanism acts as a hard-sample miner, providing stronger corrections for embeddings that are too similar across clients. Mathematically, this approach approximates standard global softmax training without the need to share private identity data.

Experimental Results

We evaluated FedGC using the CASIA-WebFace dataset (10,000 subjects, 500,000 images) partitioned across 36 clients. The model was tested on several benchmarks, including LFW, MegaFace, and IJB-C.

  • Matching Centralized Performance: On the LFW dataset, FedGC achieved 98.40% accuracy, nearly matching the centralized Softmax baseline (99.84%) and significantly outperforming standard private-embedding FL (FedPE) at 94.77%.
  • MegaFace Superiority: On the MegaFace benchmark (TPR @ 1e-6 FPR), FedGC reached 69.87%, nearly doubling the performance of the FedPE baseline (36.83%).
  • Robustness: FedGC demonstrated superior performance in scenarios with unbalanced data distributions, which is the most common real-world federated setting.

Conclusion

FedGC successfully bridges the gap between privacy and performance in federated face recognition. By keeping class embeddings private and utilizing a novel gradient correction term, it achieves near-centralized accuracy while protecting against reconstruction attacks. Future work will explore scaling the framework to thousands of edge devices and investigating performance in scenarios with overlapping identities across clients.

Summary updated

Abstract

With increasing appealing to privacy issues in face recognition, federated learning has emerged as one of the most prevalent approaches to study the unconstrained face recognition problem with private decentralized data. However, conventional decentralized federated algorithm sharing whole parameters of networks among clients suffers from privacy leakage in face recognition scene. In this work, we introduce a framework, FedGC, to tackle federated learning for face recognition and guarantees higher privacy. We explore a novel idea of correcting gradients from the perspective of backward propagation and propose a softmax-based regularizer to correct gradients of class embeddings by precisely injecting a cross-client gradient term. Theoretically, we show that FedGC constitutes a valid loss function similar to standard softmax. Extensive experiments have been conducted to validate the superiority of FedGC which can match the performance of conventional centralized methods utilizing full training dataset on several popular benchmark datasets.
BibTeX
@inproceedings{niu2022federated,
  author = {Yifan Niu and Weihong Deng},
  title = {{Federated Learning for Face Recognition with Gradient Correction}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  url = {https://openreview.net/forum?id=G5G7bfALIyw},
}
Figure