TL;DR
This paper introduces Active-Passive Federated Learning (APFed), a novel framework for vertically partitioned multi-view data that addresses the critical issue of unpredictable client availability during model inference. APFed allows a single "active" client to perform inference independently after training, overcoming the limitations of traditional federated learning that require continuous collaboration.
What Problem Does This Paper Address?
Federated Learning (FL) is a powerful technique for training machine learning models on decentralized data while preserving privacy. Vertical Federated Learning (VFL) specifically tackles scenarios where different features of the same data samples are distributed across different clients. This is common in real-world applications where organizations hold distinct data attributes (e.g., a bank with credit history, a platform with purchase data).
However, existing VFL methods typically require all participating clients to collaborate not only during model training but also during inference. This "train together, infer together" strategy poses a significant problem: model inference often needs to be maintained for extended periods (months or years), during which client availability can become unpredictable due to contract cancellations, organizational changes, or network issues. Such unreliability can render the entire learned model useless. This paper addresses this critical gap by proposing a framework that decouples training from inference, allowing for independent inference.
Methods
The core innovation is the Active-Passive Federated Learning (APFed) framework. In APFed, clients are divided into two roles:
- Active Client: This client initiates the learning task, possesses the complete machine learning model (including the task-related module), and is responsible for performing inference independently.
- Passive Clients: These clients act as assistants during the training phase. They contribute by generating latent representations of their data and computing gradients, which are then sent back to the active client.
The APFed framework operates in three steps:
- Forward Pass: The active client generates latent representations of its data and shares them with passive clients. Passive clients then compute their respective losses using these representations and their own data.
- Back-propagation: Passive clients compute gradients with respect to the shared latent representations and send them back to the active client. The active client aggregates these gradients to update its own model parameters and those of the passive clients.
- Inference: Once training is complete, the active client possesses the fully trained model and can perform inference on new data samples independently, without any further collaboration from passive clients.
To demonstrate the framework's practicality, the authors instantiate it with two methods for passive clients:
- APFed-R (Reconstruction Loss): Passive clients use their data to reconstruct the latent representations received from the active client.
- APFed-C (Contrastive Loss): Passive clients learn representations that maximize similarity between positive pairs (data from the same sample across views) and minimize similarity with negative pairs (data from different samples).
Datasets
The experiments were conducted on four popular image datasets:
- MNIST
- Fashion MNIST (FMNIST)
- CIFAR10
- CIFAR100
The authors employed a common strategy in VFL research where images are split horizontally into multiple "views" to simulate vertically partitioned multi-view data. The labels were always deployed on the active client.
Key Insights and Ideas
- Decoupling Inference from Collaboration: The most significant insight is the successful decoupling of model inference from continuous client collaboration. This is crucial for real-world deployment where client availability is not guaranteed over long periods.
- Active Client as the Sole Inferer: The APFed framework empowers the active client to be the sole entity responsible for inference, making the system robust to passive client dropouts or unavailability post-training.
- Passive Clients as "Blackboxes": The framework allows passive clients to be treated as "blackboxes" by the active client. This enhances their willingness to participate, as they only need to provide gradients and don't need to reveal their internal model architectures or data processing methods.
- Flexibility of Passive Client Losses: The APFed framework is highly flexible, allowing passive clients to use various loss functions (e.g., reconstruction, contrastive, or even multi-view clustering losses) tailored to their specific data and requirements.
Key Results
- Superior Performance: Both APFed-R and APFed-C significantly outperformed traditional VFL baselines (TVFL-0, TVFL-A, TVFL-R) across all datasets and settings. These baselines, which simulate inference failures due to unpredictable collaboration, showed drastically lower accuracies, highlighting the vulnerability of existing VFL methods.
- Outperforming Single-View Learning: APFed methods also generally outperformed Single-View Learning (SingleVL), which trains and infers on a single client without any collaboration. This demonstrates that incorporating information from passive clients during training, even if they are unavailable for inference, is beneficial.
- Trade-off Parameter Effectiveness: The hyperparameter $\lambda$ (trade-off between active and passive client losses) was studied. Setting $\lambda=0$ effectively reduced APFed to SingleVL, confirming that passive client contributions are valuable. Increasing $\lambda$ generally improved accuracy, suggesting that a balanced contribution from passive clients is beneficial.
- APFed-C vs. APFed-R: APFed-C generally achieved better results than APFed-R, especially on CIFAR datasets. The authors suggest this might be due to the maturity of encoder architectures for contrastive learning compared to the more challenging reconstruction task for certain data types (like audio). However, APFed-C can be less efficient with large batch sizes due to its contrastive loss computation.
Limitations and Future Work
- Task Specificity: The current experiments are limited to image classification. The authors acknowledge that validating APFed on other tasks like multi-model object detection, tracking, and anomaly detection is necessary.
- Quantifying Passive Client Contributions: The paper did not explicitly quantify the contribution of passive clients. Developing mechanisms to measure and potentially reward these contributions could further incentivize participation.
- Efficiency of Contrastive Loss: While APFed-C shows strong performance, its contrastive loss can be computationally expensive with large batch sizes. Future work might explore more efficient contrastive learning methods or alternative passive client loss functions.
Summary updated