TL;DR
Contrastive self-supervised learning (CL) yields powerful data representations but is notoriously resource-hungry during training. This paper provides a systematic energy-centric benchmark of four popular CL frameworks across various low-data regimes and lightweight architectures. The findings reveal surprising counter-intuitions: SimCLR is actually the most energy-efficient framework, and larger backbones like ResNet-18 often outperform compact alternatives in real-world energy-to-accuracy efficiency.
What Problem Does This Paper Address?
Self-supervised contrastive learning (CL) has revolutionized computer vision by allowing models to learn robust representations from unlabeled data. However, standard CL training pipelines are massive compute hogs, typically requiring large batch sizes, heavy backbones (like ResNet-50), and massive datasets.
Moving CL training directly to resource-constrained edge and IoT devices is highly desirable for user privacy and local adaptation. Unfortunately, we lack a clear understanding of the actual energy, memory, and data trade-offs involved. Most edge AI research focuses purely on inference or theoretical metrics like FLOPs. The authors address this gap by profiling the actual hardware energy consumption of training CL frameworks on edge-friendly architectures.
Methods
The authors evaluate four prominent CL frameworks: SimCLR, MoCo, SimSiam, and Barlow Twins (illustrated in Figure 1).
They benchmark these frameworks across six lightweight neural network backbones: ResNet-18, EfficientNet (B0, B1, B2), SqueezeNet, and MobileNet.
Crucially, instead of relying on theoretical FLOPs or parameter counts—which do not scale linearly with real-world power draw—they directly measure computational energy consumption (CPU, GPU, and RAM) using the CodeCarbon library. They also evaluate representation quality using two metrics:
- Alignment: How close positive pairs (augmented views of the same image) are in the latent space.
- Variability Collapse Index (VCI): A metric indicating representation diversity and transferability to downstream tasks (lower is better).
Datasets
The experiments are conducted using the standard CIFAR-10 dataset. To simulate low-data edge environments, the authors randomly sample subsets of the dataset ranging from 2%, 5%, 10%, 20%, 40%, 60%, 80%, to 100%, while maintaining the original class distributions.
Key Insights and Ideas
- SimCLR is Surprisingly Green: Despite its reputation for needing massive batch sizes in data centers, SimCLR consumed the lowest energy across various data regimes. Because it avoids extra memory structures like MoCo's momentum queue or Barlow Twins' massive cross-correlation matrices, it is highly RAM and CPU efficient.
- Parameter Count $\neq$ Energy Consumption: Counter-intuitively, ResNet-18 (the largest model tested by parameter count) achieved the best accuracy and the most favorable energy-to-accuracy ratio.
- Compact Models Can Be Inefficient: EfficientNet variants and MobileNet actually drew more energy than ResNet-18. This is due to hardware-level inefficiencies; depthwise separable convolutions and fragmented operations are poorly optimized for GPU execution compared to standard residual blocks.
- Data Augmentation is "Energy-Neutral": Adding more data augmentation strategies significantly improves downstream accuracy but incurs almost zero additional energy cost during training.
Key Results
- Data-Energy Trade-off: Reducing training data to 20% slashes energy consumption by ~83% across frameworks. However, accuracy degrades gracefully for SimCLR and MoCo (~21% drop) compared to catastrophic drops for Barlow Twins (~43.6%) and SimSiam (~51.3%), as shown in Figure 2.
- Framework Breakdown: Barlow Twins is the most expensive framework due to its high-dimensional projector and intensive matrix operations, drawing significantly more GPU and RAM energy (Figure 3).
- SqueezeNet Caveat: SqueezeNet was the most energy-efficient backbone overall but suffered from severe representation collapse (high VCI score), meaning its learned features were structurally poor.
- Augmentation Robustness: MoCo demonstrated the highest robustness to minimal data augmentations, maintaining relatively high accuracy even when only a single transformation type was applied (Figure 4).
Limitations and Future Work
- Software-based Energy Estimation: Energy profiling relied entirely on the
CodeCarbonlibrary rather than direct physical hardware measurements (e.g., power meters), which might miss specific hardware variations. - Single Dataset Focus: The evaluation was restricted to CIFAR-10; testing on diverse domains and larger edge-centric datasets remains open.
- Decentralized Environments: Future work needs to explore how these frameworks handle non-IID (heterogeneous) data distributions, label imbalances, and client selection biases typical of federated learning setups at the edge.
Summary updated