TL;DR
This paper introduces Patch SplitNN, a novel privacy-preserving neural network architecture for vision tasks. By splitting images into patches and processing them with multiple "upper" models before combining their outputs, Patch SplitNN significantly reduces the risk of sensitive data leakage to a central server while maintaining competitive accuracy with standard models like ResNet18.
What Problem Does This Paper Address?
As machine learning models become more integrated into various applications, the potential for sensitive data leakage during training is a growing concern. While distributed learning methods like Federated Learning and SplitNN offer solutions by keeping raw data on client devices, they often face challenges: either the central server still receives potentially sensitive intermediate information (gradients or features), or the computational burden on clients is prohibitively high, especially for complex tasks like deep neural network training. Furthermore, even with privacy-preserving techniques, ensuring that the model's accuracy doesn't suffer significantly is crucial. This paper aims to bridge this gap by proposing a method that enhances privacy and reduces client-side computational cost without compromising model performance.
Methods
The core of Patch SplitNN is inspired by SplitNN but introduces a key innovation: instead of sending intermediate features from a single split point, the input image is divided into multiple patches. Each patch is then processed by a separate "upper" neural network model. These upper models are designed to be relatively shallow, reducing the computational load on the client. The outputs from all upper models are then concatenated and fed into a single "lower" model, typically residing on the central server.
To further enhance privacy, the authors propose Patch SplitNN+, which incorporates an AdaptationNet. This additional network is trained alongside the upper models and helps to mitigate performance degradation that can occur when a large number of upper models are used (as each upper model receives less data). AdaptationNet is only active during training, ensuring it doesn't add computational overhead during inference.
The patching strategy itself is also a crucial component. Patches are made small enough to obscure individual privacy-sensitive information, and the overlap between patches is controlled to potentially improve model performance by allowing convolutions to process boundaries.
Datasets
The proposed Patch SplitNN and Patch SplitNN+ models were evaluated on the following standard image classification datasets:
- CIFAR-10: A small dataset with 10 classes, commonly used for benchmarking.
- CIFAR-100: A more challenging version of CIFAR-10 with 100 classes.
- PatchCamelyon (PCam): A larger dataset for binary classification, focusing on identifying metastatic tissue in histopathology images.
No new datasets were introduced in this paper.
Key Insights and Ideas
- Patching as a Privacy Mechanism: The fundamental idea that splitting an image into small, non-identifiable patches can inherently protect privacy is a key insight. Even if a patch is leaked, it doesn't reveal the entire sensitive image.
- Decentralized Processing with Parallel Upper Models: Distributing the processing of image patches across multiple "upper" models on the client side significantly reduces the amount of information sent to the central server and lowers the computational burden per client.
- AdaptationNet for Scalability: The introduction of AdaptationNet is crucial for addressing the performance drop associated with increasing the number of upper models. This shows a practical way to scale the privacy-preserving approach.
- Patch Overlap as Data Augmentation: The authors found that the process of splitting and concatenating patches acts as a form of data augmentation, potentially improving model accuracy.
- Patch Dropping for Enhanced Privacy: The experimental finding that dropping patches (randomly zero-filling them) can sometimes improve accuracy, while also enhancing privacy, is a surprising and valuable insight.
Key Results
- Competitive Performance: Patch SplitNN and Patch SplitNN+ achieve accuracy comparable to a standard ResNet18 model on CIFAR-10, CIFAR-100, and PCam datasets.
- Patch SplitNN+ Outperforms: In most scenarios, Patch SplitNN+ (with AdaptationNet) shows improved performance over both the standard Patch SplitNN and ResNet18, especially when using a larger number of upper models.
- Reduced Privacy Leakage: The architecture inherently reduces privacy leakage by not sending raw images or full intermediate features to the central server.
- Patch Dropping Benefits: For some datasets (e.g., PCam), dropping patches can lead to improved accuracy, suggesting it's a viable technique for both privacy enhancement and performance optimization. For instance, in PCam with M_upper_num = 1, dropping patches improved accuracy.
- Computational Cost Reduction: By processing smaller patches with shallower upper models, the computational load on the client side is reduced compared to training a full model.
Limitations and Future Work
- Client-Side Computational Load: While reduced, the client still needs to perform computations for multiple upper models. The paper focuses on reducing this compared to full backpropagation in SplitNN, but it's still a consideration.
- Performance Degradation with Many Upper Models: Without AdaptationNet, performance can degrade as the number of upper models increases. AdaptationNet mitigates this, but it's an area of ongoing research.
- Generalization to Other Tasks: The current work focuses on image classification. The authors suggest that extending Patch SplitNN to other vision tasks like face recognition, object detection, and action recognition is a promising direction for future work.
- Further Privacy Enhancements: The paper mentions the possibility of deploying multiple lower models for even greater privacy preservation.
- Epoch Training for Smaller Datasets: For datasets like CIFAR-100, longer training epochs might be necessary to achieve optimal performance, especially when patch dropping is involved.
Summary updated