Contrastive Self-Supervised Learning at the Edge: An Energy Perspective

Fernanda Famá, Roberto M. Pinheiro Pereira, Charalampos Kalalas, Paolo Dini, Lorena Qendro, Fahim Kawsar, Mohammad Malekzadeh

TL;DR

Contrastive self-supervised learning (CL) yields powerful data representations but is notoriously resource-hungry during training. This paper provides a systematic energy-centric benchmark of four popular CL frameworks across various low-data regimes and lightweight architectures. The findings reveal surprising counter-intuitions: SimCLR is actually the most energy-efficient framework, and larger backbones like ResNet-18 often outperform compact alternatives in real-world energy-to-accuracy efficiency.

What Problem Does This Paper Address?

Self-supervised contrastive learning (CL) has revolutionized computer vision by allowing models to learn robust representations from unlabeled data. However, standard CL training pipelines are massive compute hogs, typically requiring large batch sizes, heavy backbones (like ResNet-50), and massive datasets.

Moving CL training directly to resource-constrained edge and IoT devices is highly desirable for user privacy and local adaptation. Unfortunately, we lack a clear understanding of the actual energy, memory, and data trade-offs involved. Most edge AI research focuses purely on inference or theoretical metrics like FLOPs. The authors address this gap by profiling the actual hardware energy consumption of training CL frameworks on edge-friendly architectures.

Methods

The authors evaluate four prominent CL frameworks: SimCLR, MoCo, SimSiam, and Barlow Twins (illustrated in Figure 1).

Illustration of the key operating principles for key representative CL frameworks.

They benchmark these frameworks across six lightweight neural network backbones: ResNet-18, EfficientNet (B0, B1, B2), SqueezeNet, and MobileNet.

Crucially, instead of relying on theoretical FLOPs or parameter counts—which do not scale linearly with real-world power draw—they directly measure computational energy consumption (CPU, GPU, and RAM) using the CodeCarbon library. They also evaluate representation quality using two metrics:

  • Alignment: How close positive pairs (augmented views of the same image) are in the latent space.
  • Variability Collapse Index (VCI): A metric indicating representation diversity and transferability to downstream tasks (lower is better).

Datasets

The experiments are conducted using the standard CIFAR-10 dataset. To simulate low-data edge environments, the authors randomly sample subsets of the dataset ranging from 2%, 5%, 10%, 20%, 40%, 60%, 80%, to 100%, while maintaining the original class distributions.

Key Insights and Ideas

  • SimCLR is Surprisingly Green: Despite its reputation for needing massive batch sizes in data centers, SimCLR consumed the lowest energy across various data regimes. Because it avoids extra memory structures like MoCo's momentum queue or Barlow Twins' massive cross-correlation matrices, it is highly RAM and CPU efficient.
  • Parameter Count $\neq$ Energy Consumption: Counter-intuitively, ResNet-18 (the largest model tested by parameter count) achieved the best accuracy and the most favorable energy-to-accuracy ratio.
  • Compact Models Can Be Inefficient: EfficientNet variants and MobileNet actually drew more energy than ResNet-18. This is due to hardware-level inefficiencies; depthwise separable convolutions and fragmented operations are poorly optimized for GPU execution compared to standard residual blocks.
  • Data Augmentation is "Energy-Neutral": Adding more data augmentation strategies significantly improves downstream accuracy but incurs almost zero additional energy cost during training.

Key Results

  • Data-Energy Trade-off: Reducing training data to 20% slashes energy consumption by ~83% across frameworks. However, accuracy degrades gracefully for SimCLR and MoCo (~21% drop) compared to catastrophic drops for Barlow Twins (~43.6%) and SimSiam (~51.3%), as shown in Figure 2.
Energy consumption and accuracy trade-offs across different data sizes.
  • Framework Breakdown: Barlow Twins is the most expensive framework due to its high-dimensional projector and intensive matrix operations, drawing significantly more GPU and RAM energy (Figure 3).
Energy consumption components for SimCLR, MoCo, and Barlow Twins.
  • SqueezeNet Caveat: SqueezeNet was the most energy-efficient backbone overall but suffered from severe representation collapse (high VCI score), meaning its learned features were structurally poor.
  • Augmentation Robustness: MoCo demonstrated the highest robustness to minimal data augmentations, maintaining relatively high accuracy even when only a single transformation type was applied (Figure 4).
Impact of transformations on energy consumption and accuracy.

Limitations and Future Work

  • Software-based Energy Estimation: Energy profiling relied entirely on the CodeCarbon library rather than direct physical hardware measurements (e.g., power meters), which might miss specific hardware variations.
  • Single Dataset Focus: The evaluation was restricted to CIFAR-10; testing on diverse domains and larger edge-centric datasets remains open.
  • Decentralized Environments: Future work needs to explore how these frameworks handle non-IID (heterogeneous) data distributions, label imbalances, and client selection biases typical of federated learning setups at the edge.

Summary updated

Abstract

While contrastive learning (CL) shows considerable promise in self-supervised representation learning, its deployment on resource-constrained devices remains largely underexplored. The substantial computational demands required for training conventional CL frameworks pose a set of challenges, particularly in terms of energy consumption, data availability, and memory usage. We conduct an evaluation of four widely used CL frameworks: SimCLR, MoCo, SimSiam, and Barlow Twins. We focus on the practical feasibility of these CL frameworks for edge and fog deployment, and introduce a systematic benchmarking strategy that includes energy profiling and reduced training data conditions. Our findings reveal that SimCLR, contrary to its perceived computational cost, demonstrates the lowest energy consumption across various data regimes. Finally, we also extend our analysis by evaluating lightweight neural architectures when paired with CL frameworks. Our study aims to provide insights into the resource implications of deploying CL in edge/fog environments with limited processing capabilities and opens several research directions for its future optimization.
BibTeX
@misc{fama2025contrastive,
  author = {Fernanda Famá and Roberto M. Pinheiro Pereira and Charalampos Kalalas and Paolo Dini and Lorena Qendro and Fahim Kawsar and Mohammad Malekzadeh},
  title = {{Contrastive Self-Supervised Learning at the Edge: An Energy Perspective}},
  year = {2025},
  eprint = {2510.08374},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.08374},
}
Figure