TL;DR
This paper introduces a novel hybrid deep learning architecture that splits a neural network between a user's mobile device and the cloud. This allows for privacy-preserving analytics by processing sensitive raw data locally and only sending essential features to the cloud, achieving a better trade-off between utility, privacy, and performance.
What Problem Does This Paper Address?
The proliferation of Internet of Things (IoT) devices, like smartphones and smart home gadgets, has led to massive data collection for machine learning-powered services. While these services offer convenience (e.g., personalized recommendations, health monitoring), they raise significant privacy concerns. The core issue is that service providers can potentially infer sensitive information beyond the intended purpose of the data. For instance, an emotion detection service might also infer a user's identity or habits. Traditional approaches like performing all processing on the device are often too resource-intensive for complex models, while purely cloud-based solutions risk exposing raw, sensitive data. This paper tackles the challenge of enabling sophisticated analytics on mobile devices while rigorously protecting user privacy.
Methods
The authors propose a hybrid user-cloud framework that decouples a pre-trained deep neural network (DNN). The initial layers of the DNN, acting as a feature extractor, are deployed on the user's device. This module processes the raw data and extracts an "exclusive feature" that contains only the information necessary for the primary task. This exclusive feature is then sent to the cloud, where the remaining layers of the DNN (the classifier) perform the final analysis.
To ensure the exclusive feature is privacy-preserving, they introduce Siamese fine-tuning. This technique trains the feature extractor to map data points belonging to the same class (e.g., same gender) very close together in the feature space, while pushing apart points from different classes. This makes it difficult for an adversary to infer sensitive attributes (like identity) from the feature. They also explore dimensionality reduction (using PCA) and noise addition as complementary techniques to further obscure sensitive information. The combination of these methods is termed advanced embedding.
To evaluate privacy, they propose a privacy measure based on the rank of the true class's likelihood among all possible sensitive classes, and use transfer learning to assess how well a secondary task (e.g., face recognition) can be performed on the extracted features. Deep visualization is used to qualitatively assess the removal of sensitive information from image data.
Datasets
The paper evaluates their framework on two main tasks:
- Gender Classification: They used the Wiki part of the IMDB-Wiki dataset (62,359 images) for training and the Labeled Face in the Wild (LFW) dataset (13,233 images) for testing.
- Activity Recognition: They used the MotionSense dataset, which contains accelerometer and gyroscope data from an iPhone 6s, collected from 24 participants performing various activities.
Key Insights and Ideas
- Decoupling DNNs for Privacy: The core idea of splitting a DNN into a local feature extractor and a cloud classifier is a powerful way to balance computational load and privacy.
- Siamese Fine-Tuning is Key: This technique is crucial for making the extracted features "privacy-preserving" by forcing similar sensitive attributes into tight clusters, making them harder to discern.
- Higher Layers = More Privacy (but with a trade-off): Using intermediate layers closer to the output of the DNN as the feature extractor generally leads to better privacy, as these layers learn more task-specific features. However, this also increases the computational burden on the device.
- Advanced Embedding Offers the Best Trade-off: Combining Siamese fine-tuning with dimensionality reduction and noise addition provides the most robust privacy protection without significantly degrading the accuracy of the main task.
- Mobile Feasibility: The proposed hybrid approach significantly reduces the computational and memory footprint on mobile devices compared to running the full model locally, making it practical for real-world applications.
Key Results
- Accuracy: The Siamese embedding method achieved comparable accuracy to simple embedding for the main tasks (gender classification and activity recognition), indicating that privacy enhancements do not necessarily come at a significant utility cost. For gender classification, the accuracy of face recognition (a secondary task) was reduced to as low as 2.3% when using advanced embedding with higher intermediate layers.
- Privacy: The advanced embedding method, particularly when using higher intermediate layers, demonstrated superior privacy preservation, as evidenced by significantly lower accuracy in secondary inference tasks like face recognition.
- Performance: On a modern smartphone (iPhone Xs), the hybrid approach showed substantial reductions in memory usage, loading time, and running time compared to an on-premise solution, especially for larger models like VGG-16 used in gender classification. For example, memory usage for gender classification was significantly lower with the hybrid approach.
Limitations and Future Work
- Scope: The current framework is primarily designed for classification tasks. The authors suggest extending it to other supervised or unsupervised learning problems, such as regression.
- Recurrent Networks: The current work focuses on feed-forward neural networks (CNNs, MLPs). Future work could explore supporting recurrent neural networks for time-series data like speech or video.
- Hyperparameter Optimization: The current approach to balancing privacy, accuracy, and complexity is described as "naive," suggesting that hyperparameter optimization could yield even better results.
- Machine Learning as a Service (MLaaS): The authors plan to extend their work to a MLaaS scenario where users can contribute data for training new models in a privacy-preserving manner.
Summary updated