Contrastive Self-Supervised Learning at the Edge: An Energy Perspective

Fernanda Famá, Roberto Pereira, Charalampos Kalalas, P. Dini, Lorena Qendro, Fahim Kawsar, Mohammad Malekzadeh

Introduction

Contrastive Learning (CL) has revolutionized the field of self-supervised learning by enabling models to extract meaningful representations from unlabeled data. By teaching models to maximize similarity between different views of the same image (positive pairs) while minimizing similarity with other images (negative samples), CL has achieved performance levels comparable to supervised learning. However, these advancements come with a high computational price tag. Standard CL pipelines typically demand massive datasets, large batch sizes, and high-end GPU clusters, making them notoriously resource-intensive.

This paper addresses a critical gap in the literature: the feasibility of deploying CL on resource-constrained "Edge" or "Fog" devices. Unlike data centers, edge devices—such as IoT sensors, wearable health monitors, and remote surveillance cameras—operate under strict battery limits, have minimal memory, and often lack high-speed connectivity. This study provides a much-needed "energy reality check," benchmarking popular CL frameworks to understand the practical trade-offs between energy consumption, data availability, and the quality of learned representations.

Methodology

The researchers conducted a systematic benchmarking study of four major CL frameworks, each representing a different approach to self-supervision:

  • SimCLR & MoCo: These are "instance discrimination" methods. SimCLR relies on large batches to provide sufficient negative samples, while MoCo uses a momentum-based queue to maintain a large set of negatives without requiring massive batches.
  • SimSiam & Barlow Twins: These methods aim to eliminate the need for negative samples. SimSiam uses a stop-gradient operation and a prediction head to prevent collapse, while Barlow Twins applies a redundancy-reduction objective to the cross-correlation matrix of twin embeddings.
Architectural comparison of the four evaluated Contrastive Learning frameworks: (a) SimCLR, (b) MoCo, (c) SimSiam, and (d) Barlow Twins.

The frameworks were evaluated across six diverse neural network backbones: the standard ResNet-18 and several "lightweight" designs including EfficientNet (B0, B1, B2), SqueezeNet, and MobileNet. To move beyond simple accuracy metrics, the authors employed:

  1. Energy Profiling: Using the CodeCarbon library to track power consumption across the CPU, GPU, and RAM during the training process.
  2. Representation Quality Metrics: They utilized Alignment (measuring how closely the model maps similar samples in latent space) and the Variability Collapse Index (VCI) to determine if the model was learning diverse, useful features or simply collapsing into a trivial state.
  3. Data Scarcity Simulation: To mimic the limited data environments of edge devices, the models were trained on varying subsets of the CIFAR-10 dataset, ranging from 100% down to just 2%.

Key Insights and Analysis

The SimCLR Energy Efficiency King

A surprising finding of the study is that SimCLR is the most energy-efficient framework among those tested. Despite its reputation for requiring large batches, its architectural simplicity—lacking the complex memory queues of MoCo or the high-dimensional matrix operations of Barlow Twins—results in significantly lower RAM and CPU overhead. This makes it a strong candidate for edge environments where memory bandwidth is a bottleneck.

The "Lightweight" Paradox

The study reveals a counter-intuitive "Lightweight Paradox": models designed with fewer parameters are not always more energy-efficient. Specifically, EfficientNet variants (B0-B2) consumed more energy than the larger ResNet-18. This discrepancy is attributed to "fragmented operations," such as depthwise separable convolutions. While these operations reduce the total number of floating-point operations (FLOPs), they are often poorly optimized for current GPU hardware compared to standard convolutions, leading to longer training times and higher cumulative power draw.

Robustness and Model Collapse

ResNet-18 emerged as the superior "all-rounder," offering the best balance between energy consumption, accuracy, and representation quality. In contrast, SqueezeNet, while being the most energy-efficient architecture, showed a high propensity for "representation collapse," failing to learn meaningful class distinctions. Furthermore, the study found that Barlow Twins and SimSiam are particularly fragile under data scarcity; their performance plummeted when training data was reduced below 40%, whereas SimCLR and MoCo remained relatively robust.

Key Results

  • Energy vs. Data Trade-off: Reducing the training data to 20% of the original set resulted in an 83% reduction in energy consumption, but at the cost of a 21% drop in accuracy for the more robust frameworks.
  • Framework Costs: Barlow Twins was the most "expensive" framework to train, largely due to the computational intensity of calculating cross-correlation matrices for high-dimensional projection heads.
  • Hardware Optimization: The results highlight that FLOPs and parameter counts are insufficient proxies for energy efficiency; actual hardware execution time and memory access patterns are the primary drivers of power consumption.

Limitations and Future Work

The authors identify several avenues for future research. First, the energy profiles were generated on a specific Intel Xeon/NVIDIA setup; performance may vary on specialized edge hardware like ARM-based processors or dedicated AI accelerators (NPUs). Second, while ResNet-18 was well-optimized, other models might benefit from more extensive hyperparameter tuning. Finally, future studies should move toward more complex, non-IID datasets that better reflect the decentralized nature of real-world edge computing and federated learning scenarios.

Summary updated

Abstract

While contrastive learning (CL) shows considerable promise in self-supervised representation learning, its deployment on resource-constrained devices remains largely underexplored. The substantial computational demands required for training conventional CL frameworks pose a set of challenges, particularly in terms of energy consumption, data availability, and memory usage. We conduct an evaluation of four widely used CL frameworks: SimCLR, MoCo, SimSiam, and Barlow Twins. We focus on the practical feasibility of these CL frameworks for edge and fog deployment, and introduce a systematic benchmarking strategy that includes energy profiling and reduced training data conditions. Our findings reveal that SimCLR, contrary to its perceived computational cost, demonstrates the lowest energy consumption across various data regimes. Finally, we also extend our analysis by evaluating lightweight neural architectures when paired with CL frameworks. Our study aims to provide insights into the resource implications of deploying CL in edge/fog environments with limited processing capabilities and opens several research directions for its future optimization.
BibTeX
@misc{fama2025contrastive,
  author = {Fernanda Famá and Roberto Pereira and Charalampos Kalalas and P. Dini and Lorena Qendro and Fahim Kawsar and Mohammad Malekzadeh},
  title = {{Contrastive Self-Supervised Learning at the Edge: An Energy Perspective}},
  year = {2025},
  eprint = {2510.08374},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.08374},
}
Figure