Enabling On-Device Self-Supervised Contrastive Learning with Selective Data Contrast

Yawen Wu, Zhepeng Wang, Dewen Zeng, Yiyu Shi, Jingtong Hu

Introduction

Deep learning models are typically static once deployed. However, edge devices like autonomous robots or drones often encounter novel environments where continuous learning is essential. Traditional on-device learning faces two primary challenges: Non-IID data streams, where sensor data is highly correlated and repetitive, and resource constraints, including limited memory and power.

While self-supervised contrastive learning is effective for unlabeled data, it usually requires large, diverse batches—a luxury edge devices lack. This paper proposes a framework to enable efficient on-device learning by selectively filtering streaming data into a small memory buffer using a novel "Contrast Score."

The Selective Data Contrast Framework

The proposed system operates in two stages: self-supervised representation learning followed by classifier tuning with minimal labeled data.

Overview of the two-stage framework: Stage 1 focuses on self-supervised representation learning using a data buffer, while Stage 2 trains a classifier with few labels.

1. Selective Data Contrast

Instead of storing all incoming frames, the device maintains a small buffer $B$ (the size of a single mini-batch). For each new image $I$, the system calculates a Contrast Score to determine its informational value.

  • Scoring Mechanism: The system generates a deterministic transformation (e.g., a horizontal flip) of the image. It then measures the cosine similarity between the original and flipped versions in the latent space.
  • Selection Logic: A high score indicates the model perceives the original and its flip as significantly different, meaning the underlying features haven't been fully learned. These "high-value" images replace less informative ones in the buffer.

2. Lazy Scoring and Efficiency

To minimize computational overhead, the authors introduce Lazy Scoring. Instead of re-evaluating every image in the buffer at every step, scores are updated periodically (every $T$ iterations). Since model weights evolve slowly, this reduces CPU/GPU load by up to 98% without sacrificing accuracy.

Experimental Results

The framework was evaluated on standard benchmarks including CIFAR-10/100, SVHN, and ImageNet subsets, simulating real-world streaming environments.

Comparison of accuracy improvements using Contrast Scoring versus standard replacement strategies (Random, FIFO, etc.) at 1% and 10% labeling ratios.

Performance Gains

  • Accuracy: On CIFAR-10 with only 1% labeled data, the method achieved 28.36% higher accuracy than standard supervised learning.
  • Speed: The Contrast Scoring approach reached target accuracy levels 2.67x faster than random replacement baselines.
  • Robustness: As shown in the results, the method consistently outperforms state-of-the-art data selection techniques like Selective-Backprop and K-Center by 10-14% across different labeling ratios.

Key Insights

  • Gradient Correlation: The authors mathematically prove that images with high Contrast Scores correspond to larger gradients, leading to more efficient weight updates.
  • Self-Supervision Advantage: Leveraging the 99% unlabeled data through self-supervision proves far more effective for edge devices than traditional supervised learning on the limited 1% labeled subset. The self-supervised phase effectively "squeezes" more information out of the raw data stream.

Conclusion

By intelligently filtering data streams based on model uncertainty (Contrast Scoring), this framework enables deep learning models to adapt to new environments on-device without the need for massive storage or constant human labeling. Future work may explore more complex augmentations and applying this selective logic to other modalities like video or audio streams.

Summary updated

Abstract

After a model is deployed on edge devices, it is desirable for these devices to learn from unlabeled data to continuously improve accuracy. Contrastive learning has demonstrated its great potential in learning from unlabeled data. However, the online input data are usually none independent and identically distributed (non-iid) and edge devices’ storages are usually too limited to store enough representative data from different data classes. We propose a framework to automatically select the most representative data from the unlabeled input stream, which only requires a small data buffer for dynamic learning. Experiments show that accuracy and learning speed are greatly improved.
BibTeX
@inproceedings{wu2021enabling,
  author = {Yawen Wu and Zhepeng Wang and Dewen Zeng and Yiyu Shi and Jingtong Hu},
  title = {{Enabling On-Device Self-Supervised Contrastive Learning with Selective Data Contrast}},
  booktitle = {DAC},
  year = {2021},
  url = {https://openreview.net/forum?id=q2mni3Qm-y},
}
Figure