TL;DR
This paper introduces FedALC, a novel federated learning framework designed for multi-label classification where clients only possess positive data for a single class. By using a privacy-preserving hashing strategy to map label correlations and applying a weighted geometric regularizer on the server, FedALC prevents "embedding collapse" and significantly outperforms existing methods in both accuracy and communication efficiency.
What Problem Does This Paper Address?
In standard Federated Learning (FL), we assume clients have diverse datasets. However, in many real-world scenarios—like user verification using face, voice, or fingerprints—a client might only have data for their own "class" (themselves) and no data for others.
When training a multi-label model in this "only positive labels" setting, a major issue arises: embedding collapse. Because a client only sees their own positive instances, the model only learns to pull instance embeddings toward that specific class embedding. Without negative examples to "push" other classes away, all class embeddings eventually converge to a single point, making the model useless for discrimination.
While previous work (like FedAwS) tried to fix this by forcing all class embeddings to stay apart, it treated all labels as equally different. This is unrealistic—labels like "Desktop" and "Desk" are naturally correlated and should have similar embeddings, whereas "Desktop" and "Aircraft" should not.
Methods
The authors propose FedALC (Federated Averaging by exploring Label Correlations), which operates through two main innovations:
- Privacy-Preserving Label Collection: To understand which labels are related without seeing private data, clients use irreversible hash functions (like SHA-256) to map their instances. The server collects these hash codes and the associated labels. If two different clients (representing different labels) upload the same hash code, the server knows those two labels co-occur in the same instance.
- Correlation-Aware Regularization: The server uses these co-occurrence statistics to calculate a "correlation weight." It then applies a geometric regularizer that pushes class embeddings apart, but the "push" is weaker for highly correlated labels. This ensures that the embedding space reflects actual semantic relationships.
The authors also introduce FedALC-fixed, a variant where the server pre-calculates a high-quality, fixed class embedding matrix based on label correlations before the FL process begins. This reduces the need to transmit large embedding matrices back and forth, saving bandwidth and reducing privacy leakage risks.
Datasets
The researchers evaluated FedALC across two distinct domains:
- Multi-label Image Datasets: COCO, VOC 2012, VOC 2007, and FLAIR.
- Extreme Multi-label Text Datasets: EURLex, Bibtex, AmazonCat, WikiLSHTC, and Amazon670K (some containing over 670,000 labels).
Key Insights and Ideas
- The "Surrogate" Strategy: Since the server doesn't have access to the actual instance embeddings (which are private), it uses the positive class embeddings as a surrogate to calculate negative losses. This allows the server to perform the "pushing" part of the contrastive learning that the clients cannot do.
- Semantic vs. Statistical Correlation: The paper highlights that labels have both statistical connections (appearing in the same image) and semantic connections (hierarchical relationships like "car" and "vehicle"). FedALC captures these through the frequency of co-occurrence.
- Efficiency of Fixed Embeddings: In "Extreme" multi-label settings with thousands of classes, transmitting the embedding matrix every round is a bottleneck. FedALC-fixed proves that a well-initialized, static embedding space can actually perform better than one that is constantly being updated in a "tug-of-war" between client and server.
Key Results
FedALC consistently outperformed the state-of-the-art baseline, FedAwS:
- Image Tasks: On the VOC 2012 dataset, FedALC achieved a relative improvement of 8.17% in Mean Average Precision (MAP).
- Text Tasks: On the Bibtex dataset, it achieved a massive 19.3% improvement in Precision@1 (P@1).
- Communication: FedALC-fixed significantly reduced the volume of data transmitted. For the FLAIR task (1,628 labels), it reduced server-to-client data transmission compared to dynamic methods because the class embeddings were only sent once.
Limitations and Future Work
The authors acknowledge a few areas for improvement:
- Information Loss: Using a single positive class embedding as a surrogate for all instances of that class on the server side might lead to some information loss compared to having the actual instance embeddings.
- Surrogate Refinement: Future work could involve finding better "representative" embeddings for instances to further accelerate the learning process.
- Scale: While they tested on datasets with 670k labels, they aim to verify the method on even more massive-scale "extreme" datasets in the future.
Summary updated