Sustainable Federated Fine-Tuning with LoRA: Environmental Impact and Performance Footprint

Pedro Augusto Londe Ribeiro, Rodrigo Moreira, Larissa Ferreira Rodrigues Moreira, Joberto S. B. Martins, Tereza C. Carvalho, Flávio de Oliveira SilvaarXiv 2026 · arXiv

TL;DR

This paper quantifies the environmental "hidden costs" of federated Large Language Model (LLM) fine-tuning. By evaluating LoRA (Low-Rank Adaptation) across different GPUs, the researchers discovered that the intensity of local optimization (the number of training steps) is a far greater driver of energy consumption and carbon emissions than the size of the model itself, often with diminishing returns on accuracy.

What Problem Does This Paper Address?

As Large Language Models become ubiquitous, their energy appetite is skyrocketing, with data center demand projected to double by 2030. Federated Learning (FL) combined with Parameter-Efficient Fine-Tuning (PEFT) like LoRA is often touted as a "lighter" way to adapt models while preserving privacy.

However, most research in this area focuses on accuracy and communication efficiency, leaving the actual energy and carbon footprint of these processes largely unmeasured—especially when running on the diverse, heterogeneous hardware typical of real-world federated networks. This paper seeks to fill that gap by asking: Where does the energy go during federated fine-tuning, and how can we reduce it without losing performance?

Methods

The authors employed a systematic $2^2$ full factorial design to isolate the effects of two main variables:

  1. Model Size: Comparing SmolLM-135M and SmolLM-360M.
  2. Local Optimization Intensity: Comparing 10 vs. 100 local steps per round.

They ran these configurations across three distinct GPU families: NVIDIA A40, RTX 6000 Ada, and A100. To track the environmental impact, they used CodeCarbon for energy and CO2-equivalent ($CO_2e$) estimation, alongside GPU telemetry from the NVIDIA Management Library (NVML).

The experimental workflow, illustrated in Figure 1, spans from testbed provisioning to a final "Allocation of Variation" analysis to see which factor contributed most to the energy bill.

Fig. 1: Experimental Methodology Workflow.

Datasets

The models were fine-tuned and evaluated on the HDFS log dataset, a standard benchmark for system log analytics and anomaly detection. The task involved predicting the next token in tokenized log sequences, with performance measured via Top-k accuracy and F1-score.

Key Insights and Ideas

  • Steps > Size: In the tested ranges, local steps accounted for a staggering 84.8% to 91.2% of the variation in energy and emissions. Model size contributed less than 10%. This suggests that if you want to go "green," reducing your local training budget is much more effective than just picking a slightly smaller model.
  • The Efficiency "Sweet Spot": On the A40 and RTX 6000 Ada GPUs, the 10-step configurations achieved nearly the same accuracy as the 100-step ones but used 6–8 times less energy. This highlights a massive area of waste where extra computation provides negligible gains.
  • Hardware Efficiency Paradox: While the A100 has a high thermal design power, it proved to be significantly more energy-efficient for this specific federated task compared to the A40, completing rounds faster and consuming less total energy per unit of work.
  • Accuracy per kWh: The authors advocate for "Accuracy per kWh" as a vital metric for sustainable AI, moving beyond just "Accuracy vs. Parameters."

Key Results

The quantitative findings underscore the dramatic impact of local step counts on the environment:

  • Energy Consumption: For the 360M model, jumping from 10 to 100 steps increased energy use from ~0.86 kWh to ~6.92 kWh on the A40 GPU.
  • Carbon Footprint: On the A40, $CO_2e$ emissions for the 360M model reached 3.11 kg with 100 steps, compared to just 0.38 kg with 10 steps.
  • Performance: Figure 2 (labeled as Fig. 4 in the paper) visualizes these trade-offs, showing how energy consumption scales sharply with steps across all hardware.
Fig. 2: Energy consumption and CO2-equivalent emissions by device and configuration.

Figure 3 (labeled as Fig. 5 in the paper) further breaks down the "influence" of each factor, clearly showing the dominance of Factor B (Local Steps).

Fig. 3: Factor contributions to each recorded response.

Limitations and Future Work

The authors acknowledge several constraints:

  • Scope: The study focused on one model family (SmolLM) and one dataset (HDFS). Results might vary for different architectures or more complex language tasks.
  • Communication Costs: The energy used for transmitting model updates between clients and the server was not measured, which is a significant component of real-world FL.
  • Static Budgets: The study used fixed step counts. Future work could explore adaptive local budgets that dynamically reduce steps when the model begins to converge, potentially saving even more energy.
  • Threshold Selection: The anomaly detection evaluation used an optimistic threshold selection; more robust, independent validation is needed for production settings.

Summary updated

Abstract

Federated Learning (FL) with Parameter-Efficient Fine-Tuning (PEFT) enables efficient adaptation of Large Language Models (LLMs), yet its energy and carbon costs across hardware remain poorly understood. We evaluated federated Low-Rank Adaptation (LoRA) fine-tuning on NVIDIA A40, RTX 6000 Ada, and A100 GPUs using a $2^2$ full factorial design with 135M/360M models and 10/100 local steps. CodeCarbon tracked energy and estimated CO$_2$-equivalent emissions. Descriptive allocation of variation assigned 84.8-91.2% of energy and emissions variation to local steps within these ranges. On A40 and RTX 6000 Ada, configurations with fewer steps achieved comparable accuracy with roughly 6-8-fold lower energy consumption and higher accuracy per kWh. These findings support smaller local budgets for energy-efficient federated fine-tuning within the evaluated setting.
BibTeX
@misc{ribeiro2026sustainable,
  author = {Pedro Augusto Londe Ribeiro and Rodrigo Moreira and Larissa Ferreira Rodrigues Moreira and Joberto S. B. Martins and Tereza C. Carvalho and Flávio de Oliveira Silva},
  title = {{Sustainable Federated Fine-Tuning with LoRA: Environmental Impact and Performance Footprint}},
  year = {2026},
  eprint = {2611.16856},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.16856},
}
Figure