Paying for Bad Starts: The Energy Cost of Initialization Sensitivity in Low-Rank Fine-Tuning

Most of the energy spent on fine-tuning goes to runs that are thrown away, not to the final model. Practitioners repeat each fine-tuning job across random seeds and learning-rate grids because the outcome depends on where the optimizer starts. Measurement work shows this repetition adds up: across tasks, seeds and hyperparameter sweeps, aggregate fine-tuning energy quickly overtakes the one-time cost of pre-training (Energy and Carbon Considerations of Fine-Tuning BERT). Separately, the optimization literature has produced initializations for low-rank adapters that make training better behaved. LoRA, short for low-rank adaptation, freezes the pretrained weights and learns a small update written as the product of two thin matrices, $\Delta W = BA$. Choices about how to start $A$ and $B$ include which factor is set to zero, an SVD-based start that absorbs quantization error, and a start built from curvature estimates on target data. Those papers report accuracy and iteration counts. None measures energy, and none asks whether a better start lets you run fewer runs.

This note proposes to measure the sustainability value of an initialization in terms of the whole search a practitioner actually performs. The hypothesis is that structured initializations save energy mainly by making outcomes less sensitive to seed and learning rate, so that a smaller sweep reliably finds a good model. Faster convergence within a single run is expected to be the smaller effect. The note also proposes a nearly free check at step zero that predicts which configurations will fail, before any training energy is spent on them.

Core Claim

The quantity that matters for carbon is the expected energy needed to obtain, with high probability, a model close to the best attainable quality. Energy per run is the wrong target. We define

$$E^{\star}(\varepsilon, p) = E_{\text{init}} + \min_{S} \; \mathbb{E}\big[\textstyle\sum_{r \in S} E_r\big] \quad \text{s.t.} \quad \Pr\big[\max_{r \in S} \text{acc}_r \ge \text{acc}^{\text{best}} - \varepsilon\big] \ge p .$$

Here $S$ is a sweep strategy (a set of seeds and learning rates, possibly with early stopping), $E_r$ is the wall-calibrated energy of run $r$, $E_{\text{init}}$ is the one-time cost of computing the initialization, and $\text{acc}^{\text{best}}$ is the best accuracy observed in an exhaustive reference sweep.

We claim three things. First, for 4-bit and especially 2-bit quantized base models, structured initializations reduce $E^{\star}$ far more than they reduce single-run energy. They shrink the spread of outcomes across seeds and widen the range of learning rates that converge, so the minimizing sweep $S$ becomes small. Second, the advantage grows as bit-width drops. This is the fine-tuning analogue of the shrinking basin of attraction under ill-conditioning proved for factored low-rank problems in Provable accelerated gradient method for nonconvex low rank optimization. Third, the gap between the initialized model and the full-precision model at step zero, together with the norm of the first gradient, predicts which configurations will diverge or stall. That prediction lets a sweep discard them before training. This is new because existing initialization papers report accuracy and iterations only, and existing energy papers treat initialization as fixed.

Prior work and how this differs

Three initialization papers supply the methods being compared, and each leaves the energy question open.

The Impact of Initialization on LoRA Finetuning Dynamics proves something about two ways of starting the adapter at $\Delta W=0$. Setting $B=0$ with random $A$ tolerates a learning rate scaling of order $n^{-1/2}$ in model width $n$. The opposite choice, random $B$ with $A=0$, forces order $n^{-1}$. That result is a direct statement about how wide the stable learning-rate window is, which is exactly what controls sweep cost. The paper never measures sweep cost or energy, and its analysis does not consider quantized base models.

LoftQ alternates between quantizing the weights and fitting a low-rank SVD to the remaining error, so the adapters start non-zero and the quantized model starts close to full precision. It reports the regime where zero-initialized adapters fail. Its SVD cost is treated as negligible and never weighed against savings.

LoRA-DA builds the initialization from Fisher-information statistics estimated on target data, which capture how sharply the loss curves in each direction. It reports about 6% extra time but no energy and no seed-to-seed variance.

The measurement side supplies the protocol. Energy and Carbon Considerations of Fine-Tuning BERT shows two things we rely on. Software trackers undercount by a power-loss factor of 1.06x to 1.09x relative to wall meters. Fine-tuning energy also scales with tokens processed, and distillation breaks even only after 86 MNLI runs. We borrow that break-even logic for $E_{\text{init}}$.

Two systems papers constrain the scope. Reducing Energy Bloat in Large Model Training cuts idle-power waste inside each iteration and takes the number of iterations as given. Distributed LLM Pretraining During Renewable Curtailment Windows moves work in time to periods of surplus renewable power. Both act on how a fixed amount of training is executed, whereas this proposal acts on how much training a practitioner must launch, so the two lines compose.

Finally, the brittleness result in Deep Learning Warm Starts for Trajectory Optimization on the International Space Station is a warning. There, a learned warm start that falls outside its training distribution performs worse than a cold start. A good initialization is therefore not guaranteed to save energy everywhere, and our evaluation must include tasks where the curvature estimates are poor.

Research Program

We will build a measurement harness that logs per-run energy with CodeCarbon, scaled by the wall-meter loss factor from Wang et al. We will validate the scaling on one node with a physical meter.

For each combination of model, bit-width and initialization, we will run a dense reference grid of learning rates crossed with ten seeds. We will log the full training curve, the initialization energy $E_{\text{init}}$, and several cheap step-zero diagnostics:

  • the relative error between the quantized-plus-adapter weights and the full-precision weights;
  • the initial loss on a small held-out batch;
  • the first-step gradient norm, split by factor $A$ and factor $B$.

From the dense grid we will compute $E^{\star}(\varepsilon,p)$ exactly for each sweep strategy, using bootstrap resampling over seeds. This is possible because every candidate sweep is a subset of runs already performed.

We will then fit a simple classifier, a logistic model on the step-zero diagnostics, that predicts whether a configuration ends within $\varepsilon$ of the best result. We will evaluate it with leave-one-task-out validation, so that it is never tuned on the task it screens.

The analytic component asks whether the measured width of the stable learning-rate window follows the $n^{-1/2}$ versus $n^{-1}$ separation of Hayou et al. as model width varies. It also asks how that width contracts with bit-width. This links a geometric quantity, basin size, to a sweep-size quantity that can be priced in joules.

Baselines

The naive reference is standard QLoRA practice: zero-initialized adapters with the default random-$B$ convention of many libraries, swept over the same learning-rate grid and seeds with no early stopping. It defines the energy a typical practitioner spends.

The strongest zero-cost competitor is the $B=0$, random-$A$ initialization from The Impact of Initialization on LoRA Finetuning Dynamics. On MNLI with RoBERTa-Large it reports 90.69% accuracy versus 89.47% for the opposite choice. Because it adds no $E_{\text{init}}$, it is the method any costlier initialization must beat on $E^{\star}$, not merely on accuracy.

The strongest method in the quantized regime is LoftQ. On Llama-2-7b at 2 bits on WikiText-2 it reports perplexity 7.85 where QLoRA exceeds 10; on MNLI at 2 bits it reports 88.0% at five alternating steps versus 79.9% for the zero-step version. For data-aware initialization, LoRA-DA reports 55.0% on GSM8K with LLaMA 2-7B and 84.3% average on the commonsense suite.

No reported energy result was found on Lacuna for any LoRA initialization method, so all energy comparisons are new measurements.

Every method gets the identical learning-rate grid, seeds, token budget, hardware, and early-stopping policy, and its own $E_{\text{init}}$ is charged in full. A method "beats" a baseline if it achieves lower $E^{\star}$ at $\varepsilon$ equal to one accuracy point (or 0.2 perplexity) and $p=0.9$, with bootstrap confidence intervals that do not overlap.

The ablation that isolates our key ingredient applies the step-zero screen to each initialization separately. This tells us whether the savings come from the initialization itself, from screening, or from both.

Why not just…?

Why not just use the zero-cost $B=0$ initialization and stop there? It may well be enough at 16-bit and 4-bit precision. The theory says it already widens the learning-rate window, and the dense grid will show whether LoftQ's SVD cost or LoRA-DA's Fisher cost is ever repaid there. We expect the answer to change at 2 bits, where LoftQ reports that zero-initialized adapters diverge. If the zero-cost start matches the others on $E^{\star}$ at all bit-widths, we will report that costly initializations are not justified on energy grounds. That would be a useful negative result.

Why not just prune bad runs early with successive halving? This is the real threat. Successive halving launches many configurations, trains each briefly, and keeps the best fraction. It already cuts the cost of variance without touching initialization. We will run it as a baseline on top of every initialization. The proposal's claim survives only if one of two things holds. Either structured initializations still lower $E^{\star}$ under successive halving, because fewer configurations need launching. Or the step-zero screen, which costs no training steps, discards failing configurations as accurately as successive halving does after its first cut. If successive halving on the naive initialization matches everything, the claim fails.

Why not just decouple the learning rates of $A$ and $B$? Separate rates for the two factors address the same imbalance as the $B=0$ start, but they add a hyperparameter axis that the sweep must search. We will include a decoupled-rate configuration and charge its larger grid to $E^{\star}$. This tests directly whether the extra tuning energy outweighs any accuracy gain.

Why not just rely on schedulers like Perseus? Perseus reduces energy per iteration and takes the number of runs as given. Its savings therefore multiply ours rather than compete with them, and we treat it as orthogonal rather than as a baseline.

Proposed Experiments

The main study fine-tunes Llama-2-7b at 16, 4 and 2 bits on WikiText-2 and GSM8K, and RoBERTa-Large on MNLI, RTE and SST2. It compares the naive start, the $B=0$ start, LoftQ, and LoRA-DA. For each method it reports single-run energy, seed variance, stable learning-rate width, $E_{\text{init}}$, the number of fine-tuning runs needed for $E_{\text{init}}$ to break even, and $E^{\star}$.

To probe the brittleness warning from the space-robotics work, LoRA-DA's Fisher statistics will be estimated on one task and used to initialize fine-tuning on another. This tests whether a mismatched data-aware start costs more energy than a zero-cost one.

The decisive checks are three. The ratio of $E^{\star}$ for the best structured initialization to $E^{\star}$ for the $B=0$ start must fall below one and keep falling as bit-width drops. That gap must persist under successive halving. The leave-one-task-out step-zero screen must remove at least half of the failing configurations while losing essentially none of the good ones. If the first check fails, initialization research has no energy case beyond the free $B=0$ start. If only the third succeeds, the contribution narrows to a cheap screening rule, which is still directly usable in sustainable fine-tuning practice.