Abstract: With increasing concerns regarding privacy in face recognition, federated learning has emerged as a prevalent approach for studying unconstrained face recognition with private decentralized data. However, conventional decentralized federated algorithms that share all network parameters among clients suffer from privacy leakage. In this work, we introduce FedGC, a framework that tackles federated learning for face recognition while guaranteeing higher privacy. We explore a novel method of correcting gradients from the perspective of backward propagation and propose a softmax-based regularizer to correct the gradients of class embeddings by precisely injecting a cross-client gradient term. Theoretically, we demonstrate that FedGC constitutes a valid loss function similar to standard softmax. Extensive experiments validate the superiority of FedGC, which matches the performance of conventional centralized methods on several popular benchmark datasets.
Introduction: The Privacy-Performance Trade-off
Face recognition (FR) models typically require massive, diverse datasets to achieve high accuracy. However, strict privacy regulations like GDPR make centralizing facial data increasingly difficult. Federated Learning (FL) offers a potential solution by training models on decentralized data without moving it.
The standard FL approach, FedAvg, shares all model parameters, including the final classification layer. In FR, this layer contains "class embeddings"—digital templates of faces. Sharing these embeddings creates a significant security risk, as attackers can use techniques like DeepInversion to reconstruct high-fidelity images of private users from these templates.
To mitigate this, researchers previously proposed FedPE (Private Embeddings), where each client keeps its classification layer local. However, this leads to the "Orthogonality Trap": without a global view, the model learns to distinguish faces within a single client but fails to separate them across different clients. This results in overlapping feature spaces and a massive drop in accuracy compared to centralized training.
The FedGC Framework
The authors propose FedGC (Federated Gradient Correction) to bridge the gap between local privacy and global discriminative power. The framework consists of two primary components: private local optimization and server-side gradient correction.
In FedGC, each client keeps its final classification layer private, performing local updates using standard FR loss functions such as ArcFace or CosFace. When the server receives backbone updates, it does not simply average them. Instead, it applies a Softmax-based Regularizer to the class embeddings. This regularizer is mathematically designed to mimic the behavior of a global softmax function by calculating cross-client similarities and "injecting" a gradient term that forces the model to push embeddings from different clients away from each other.
Gradient Correction Mechanism
The core innovation of FedGC is viewing regularization through the lens of backpropagation. By manually calculating the missing global information, the server can "fix" the local gradients. This correction ensures that the decentralized model updates in a direction consistent with a model trained on a single, massive centralized dataset.
This softmax regularizer also acts as a natural hard-sample miner. It provides stronger gradients for embeddings that are "dangerously close" to each other across client boundaries, focusing optimization where it is most needed to maintain discriminative power.
Experimental Evaluation
The researchers simulated a federated environment by splitting the CASIA-WebFace dataset (approx. 10,000 subjects, 500,000 images) into 36 distinct clients. The resulting models were tested on several industry-standard benchmarks, including LFW, MegaFace, and IJB-C.
Key Results:
- LFW Accuracy: FedGC achieved 98.40%, nearly matching the centralized baseline (99.84%) and significantly outperforming the FedPE baseline (94.77%).
- MegaFace (TPR @ 1e-6 FPR): In this large-scale challenge, FedGC reached 69.87%, whereas the FedPE baseline collapsed to 36.83%.
- IJB-C Verification: Using ArcFace, FedGC reached 78.13%, demonstrating performance remarkably close to centralized training.
Conclusion and Future Work
FedGC successfully addresses the conflict between privacy and performance in federated face recognition. By keeping sensitive embeddings local and correcting gradients on the server, it achieves near-centralized accuracy without compromising identity privacy.
Future work will focus on scaling the framework to "cross-device" scenarios involving millions of clients with sparse data. Additionally, further research is needed to analyze the accuracy-privacy trade-offs when integrating Differential Privacy (DP) noise into the gradient correction process to further harden the system against sophisticated attacks.
Summary updated