Abstract
Federated Learning (FL) aims to train models collaboratively while preserving data privacy. However, multi-label classification in FL faces a critical failure mode when clients only possess positive data for a single class—a scenario common in biometric verification and personalized user models. This "only positive labels" setting leads to "embedding collapse," where all class representations converge to a single point, rendering the model non-discriminative. We introduce FedALC (Federated Averaging by exploring Label Correlations), a framework that leverages label correlations to regularize the embedding space. By using privacy-preserving hashing to detect co-occurrences and applying a correlation-aware geometric regularizer, FedALC prevents collapse and improves discriminative power. Our variant, FedALC-fixed, further optimizes communication by using static embeddings. Extensive evaluations on image and extreme multi-label text datasets demonstrate that FedALC significantly outperforms existing baselines.
Introduction: The Challenge of Single-Positive Label FL
Standard FL assumes data diversity across clients. In reality, a client (e.g., a mobile phone) might only contain images of its owner ("Person" label) and no images of other people or objects. In a multi-label setting, training on only positive instances causes the model to pull instance embeddings toward the target class without any negative pressure to push them away from others. This leads to a useless model where all outputs are identical. Previous solutions like FedAwS attempted to force class embeddings apart uniformly, but this ignores the inherent semantic relationships between labels (e.g., "Car" and "Truck" should naturally be closer in embedding space than "Car" and "Cloud").
The FedALC Framework
FedALC addresses this by incorporating label correlations into the server-side regularization. The architecture involves a multi-step process between the server and edge clients to ensure privacy while building a semantically meaningful model.
The process involves:
- Privacy-Preserving Hashing: Clients compute irreversible hashes (e.g., SHA-256) of their data. If two clients with different labels produce the same hash, the server infers a label co-occurrence without ever seeing the raw private data.
- Correlation-Aware Regularization: The server calculates weights based on these co-occurrences. It then applies a geometric regularizer that pushes class embeddings apart, with the force modulated by the correlation weight—highly correlated labels are pushed less aggressively.
Optimization and Efficiency
FedALC uses positive class embeddings as "surrogates" for private instance embeddings on the server side. This allows the server to perform the necessary contrastive "pushing" that clients cannot do locally due to their lack of negative samples.
To handle "Extreme Multi-label" (XML) scenarios with hundreds of thousands of classes, we propose FedALC-fixed. In this variant, the server pre-calculates and fixes the class embedding matrix based on label correlations before the FL process begins. This eliminates the need for frequent, bandwidth-heavy updates of the embedding layer, significantly reducing communication overhead and privacy leakage risks.
Experimental Results
FedALC was tested on diverse image datasets (COCO, VOC, FLAIR) and XML text datasets (Amazon670K, WikiLSHTC).
- Image Tasks: On VOC 2012, FedALC achieved an 8.17% relative improvement in Mean Average Precision (MAP) over the state-of-the-art FedAwS.
- Text Tasks: In the Bibtex dataset, it saw a 19.3% boost in Precision@1 (P@1).
- Communication Efficiency: FedALC-fixed proved particularly effective in high-dimensional label spaces, reducing communication costs while maintaining or exceeding the performance of dynamic embedding methods.
Conclusion
FedALC provides a robust solution for multi-label FL under extreme data imbalance. By acknowledging that labels are semantically linked rather than independent entities, it creates a more realistic and effective embedding space. Future work will focus on refining the surrogate embedding strategy to further minimize information loss during server-side optimization.
Summary updated