Positive-Unlabeled Compression on the Cloud

Yixing Xu, Yunhe Wang, Hanting Chen, Kai Han, Chunjing Xu, Dacheng Tao, Chang Xu

Deep learning models are notoriously "over-parameterized," often containing far more filters and weights than strictly necessary for their given tasks. To deploy these massive models on resource-constrained edge devices like mobile phones or autonomous vehicles, practitioners rely on compression techniques such as pruning or knowledge distillation.

However, there is a significant catch: to compress a model without destroying its accuracy, you almost always need access to the original training data (e.g., the 120GB ImageNet dataset) to fine-tune the "svelte" version. For a user wanting to leverage a cloud-based compression service, uploading 120GB over a standard internet connection creates a massive data transmission bottleneck, largely negating the convenience of the cloud.

To solve this, researchers have introduced a novel framework that asks a provocative question: Can we compress a model using only a tiny sliver of the original data and whatever random images the cloud already has lying around?

The Proposed Solution: A Two-Stage Pipeline

By combining a small fraction of original data with the cloud's own massive unlabeled data pools, the authors achieve performance comparable to full-data training while reducing data transmission requirements by over 90%. They accomplish this through a two-stage pipeline consisting of a Positive-Unlabeled (PU) Classifier and Robust Knowledge Distillation (RKD).

The proposed cloud-based compression pipeline using PU learning and RKD.

1. The PU Classifier (Data Selection)

Instead of uploading the entire dataset, the user uploads the pre-trained "teacher" model and a small portion (e.g., 10%) of the original data. The cloud treats this 10% as "Positive" data and its own massive internal datasets (like Flickr or ImageNet) as "Unlabeled" data.

The system then trains a binary Positive-Unlabeled (PU) classifier to scour the unlabeled pool and identify images that look like they belong to the original training set. To make this classifier effective, the authors employ an attention-based multi-scale feature extractor. Because the very last layer of a Convolutional Neural Network (CNN) is often too specialized for general data searching, this extractor pulls information from different layers of the network. This ensures the search is based on both general shapes and specific features, turning a standard training problem into a highly effective retrieval problem.

2. Robust Knowledge Distillation (RKD)

Once the cloud has "found" enough relevant data, it uses it to train a smaller "student" model. However, this newly mined data is often noisy and imbalanced (e.g., the search might return 1,000 images of dogs but only 10 of airplanes).

To handle this, the authors introduce Robust Knowledge Distillation (RKD). RKD weights the loss function based on category frequency to counteract the imbalance. Furthermore, it uses the teacher network's soft outputs (probabilities) as a guide to handle noise. If the teacher is confused by a found image, the student gives it less weight, effectively using the teacher's confidence as a stabilizer.

Key Results and Performance

The researchers evaluated their method across three primary scenarios, using a tiny original set and a massive unlabeled cloud pool:

  • CIFAR-10 (Original) with ImageNet (Unlabeled)
  • ImageNet (Original) with Flickr1M (Unlabeled)
  • MNIST (Original) with EMNIST (Unlabeled)

The results demonstrate that you do not need the full original dataset to achieve high-quality compression.

Performance comparison on CIFAR-10, showing PU-s1 outperforming manual data selection baselines.
  • Superiority over Baselines: On the CIFAR-10 dataset, the PU-based selection method achieved 93.75% accuracy. This actually outperformed a baseline that used manually selected data (93.44%), suggesting the PU classifier is better at finding high-quality, diverse training examples than simple keyword searches.
  • Massive Efficiency: On ImageNet, the proposed method used only 8% of the original data to produce a student model that achieved a top-5 accuracy of 86.00%. This nearly matches the 88.76% achieved by training with the entire dataset, proving the viability of the cloud as a "data goldmine."
  • Robustness: The method significantly outperformed other "data-free" or "few-shot" distillation methods (like FitNet or FSKD) on the MNIST/EMNIST benchmarks, especially when the number of labeled samples was extremely low.

Limitations and Future Directions

While highly effective, the framework does have limitations. Its success relies heavily on the domain gap—the cloud's unlabeled pool must contain data somewhat related to the user's task. If a user uploads a model for a highly specialized domain (e.g., rare medical conditions or specific satellite imagery) that doesn't exist in general pools like Flickr, the PU classifier may struggle to find useful data. Additionally, the algorithm currently requires an estimate of the "class prior" (the proportion of positive samples expected in the unlabeled pool), which adds a layer of complexity.

Future work will likely explore more sophisticated ways to generate synthetic data when the cloud's unlabeled pool is insufficient, potentially by combining this PU learning framework with Generative Adversarial Networks (GANs). For now, this approach represents a massive leap forward in making cloud-based model compression practical, efficient, and accessible.

Summary updated

Abstract

Many attempts have been done to extend the great success of convolutional neural networks (CNNs) achieved on high-end GPU servers to portable devices such as smart phones. Providing compression and acceleration service of deep learning models on the cloud is therefore of significance and is attractive for end users. However, existing network compression and acceleration approaches usually fine-tuning the svelte model by requesting the entire original training data (e.g. ImageNet), which could be more cumbersome than the network itself and cannot be easily uploaded to the cloud. In this paper, we present a novel positive-unlabeled (PU) setting for addressing this problem. In practice, only a small portion of the original training set is required as positive examples and more useful training examples can be obtained from the massive unlabeled data on the cloud through a PU classifier with an attention based multi-scale feature extractor. We further introduce a robust knowledge distillation (RKD) scheme to deal with the class imbalance problem of these newly augmented training examples. The superiority of the proposed method is verified through experiments conducted on the benchmark models and datasets. We can use only 8% of uniformly selected data from the ImageNet to obtain an efficient model with comparable performance to the baseline ResNet-34.
BibTeX
@inproceedings{xu2019positive,
  author = {Yixing Xu and Yunhe Wang and Hanting Chen and Kai Han and Chunjing Xu and Dacheng Tao and Chang Xu},
  title = {{Positive-Unlabeled Compression on the Cloud}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
}
Figure