Most of the carbon cost of a GPU is spent before it is ever switched on: mining, chip fabrication, the server chassis and power supplies around it. When a data center runs on clean power, this manufacturing share ("embodied carbon") can dominate the lifecycle footprint. The largest lever an operator has over it is when to retire hardware and buy new hardware. This proposal asks whether the choice of training algorithm moves that lever.
The test case is physics-informed neural networks (PINNs), networks trained to satisfy a differential equation. They are a natural test because they need very precise optimization. Plain gradient methods like Adam stall on them, and a second phase using curvature information, such as L-BFGS (a quasi-Newton method that approximates the loss surface's curvature from recent gradients), is needed to reach useful accuracy. That second phase is memory-hungry and precision-sensitive, which is exactly the kind of demand that pushes labs toward the newest, most carbon-expensive accelerators. We will measure, on old and new GPUs, the full carbon cost of reaching a fixed accuracy, including the tuning runs. We will also build a memory-budgeted hybrid optimizer meant to make legacy hardware sufficient.
Core Claim
The claim is narrow and testable. For precision-bound scientific training, the optimizer decides whether a legacy GPU can reach the target error at all. Its curvature phase is the part that runs into memory and numerical-precision limits. So the provisioning question "do we need new silicon for this workload class?" has a different answer depending on whether the hybrid is designed for a fixed memory budget.
We expect a memory-budgeted Adam-to-second-order hybrid running on retained legacy GPUs to match newer GPUs on carbon-to-target. We measure carbon-to-target as kg CO2e to reach a fixed relative L2 error, including the handover-point search. This should hold for low-to-moderate job volumes, and we will locate the job-volume break-even point at which buying new hardware wins.
The novelty is not "carbon-aware switching." We deliberately do not put embodied carbon into runtime optimizer decisions. The Sunk Carbon Fallacy studied a 14-million-job cluster trace and showed that charging amortized embodied carbon at runtime steers work onto inefficient old machines and raises net emissions. That paper constrains our design: runtime choices are scored on marginal operational carbon only, and embodied carbon enters only in the procurement and retirement analysis, where the paper says it belongs.
Why This Gap Is Real
The optimizer literature reports wall-clock speed and step counts and nothing else. MKOR cuts BERT-Large pretraining from 8 to 3.1 hours with a second-order-then-first-order hybrid. FUSE formalizes iteration-, gradient- and loss-based handover rules. Auto-PINN shows the Adam-to-L-BFGS "changing point" is among the most sensitive PINN hyperparameters, yet still needs hundreds of search evaluations. None of these papers measures energy, hardware generation, or the carbon of the search itself.
The carbon literature has the opposite blind spot.
- LLMCarbon and Hugging Carbon model training as FLOPs times hardware efficiency. They ignore the failed runs and sweeps, and assume fixed five-year lifetimes.
- FUEL shows that for modest serving throughput, an older L40 beats an H100 on lifecycle carbon, but only for inference, where the algorithm is fixed.
Our work supplies the training-side counterpart to FUEL. In training, the algorithm determines whether the older device is even feasible.
Three papers motivate the precision requirement:
- Challenges in Training PINNs shows Adam hitting a floor on ill-conditioned PDE losses, with Adam+L-BFGS reaching 14.2x smaller error on convection.
- The same paper shows its Nyström-preconditioned Newton-CG (NNCG) cutting loss by more than 10x where Adam+L-BFGS stalls, at more than 300x the per-iteration cost.
- A Space-Time Neural Network for stress evolution needs hybrid refinement to reach below 1.22% error.
Can PINNs beat FEM? constrains the scope. Finite element solvers are 5–6 orders of magnitude faster for one-off solves, while trained PINN surrogates are 2–3 orders faster at mass inference. Our claims therefore apply only to the many-query surrogate setting.
Research Program
Memory-budgeted hybrid. The method contribution is an Adam-to-second-order hybrid whose curvature phase is sized to a given GPU's memory rather than assumed to have whatever it needs. Three settings are made explicit functions of available VRAM and of FP32 versus FP64 throughput:
- the L-BFGS history length;
- the batch size used during the curvature phase, since line searches break under minibatch noise and practitioners usually fall back to full batches;
- the Nyström rank used to precondition Newton-CG.
Two efficiency ideas borrowed from other work are tested as optional components. The first is MKOR's rank-1 Sherman-Morrison inverse updates, which keep the curvature cost quadratic rather than cubic. The second is the damping and automatic step size of AdamQLR, which suggests much of second-order benefit comes from stabilizing heuristics that are cheap in memory.
Handover rule. The switch from Adam uses FUSE's loss-plateau criterion. We count the cost of finding the right threshold rather than hiding it.
Accounting. Accounting has two modes.
- Marginal mode answers "which optimizer should I run on the machines I already have?" It counts measured device and node energy times hourly grid intensity, nothing else.
- Provisioning mode answers "should this workload class justify new hardware?" It adds amortized embodied carbon:
$$ C_{\text{prov}} = \sum_{r \in R} \left( E_r \, I_r + \frac{M_h}{L_h \, u_h}\, t_r \right), $$
The symbols are defined as follows:
- $R$ is every run needed to reach the target, including tuning and failed runs.
- $E_r$ is the measured energy of run $r$.
- $I_r$ is grid carbon intensity during the run.
- $M_h$ is the whole-server embodied carbon of hardware generation $h$, taken from SCARIF so that peripherals are included.
- $L_h$ is service lifetime, varied rather than fixed at five years.
- $u_h$ is fleet utilization.
- $t_r$ is occupied server-hours.
We draw $M_h$, $I_r$, and fab parameters from distributions, following PCAM. PCAM shows that deterministic estimators can err by 23% on embodied carbon. Our provisioning conclusions are therefore reported as probabilities that one configuration beats another, not point estimates.
Break-even. The output is a break-even surface. For each PDE and hardware pair, we find how many surrogate trainings per year make buying newer GPUs carbon-favorable over keeping the old ones. We then ask how far that point shifts between plain Adam+L-BFGS and the memory-budgeted hybrid.
Baselines
Adam alone (the do-nothing reference). This is plain Adam with a cosine schedule. It is what most practitioners would run, and it isolates the value of any second-order phase. It gets the same tuning budget as every other method: a fixed number of trials, counted inside $R$. On convection, Challenges in Training PINNs reports Adam's error is 14.2x worse than Adam+L-BFGS. We expect Adam to fail the target on several PDEs on every hardware generation, in which case its carbon-to-target is undefined. We report that outcome as such rather than as a number.
Standard Adam+L-BFGS on new hardware. This is the hybrid as the PINN literature runs it, in full batch and FP64, on the newest GPU available (H100-class). It is the strongest competitor because it is what a well-resourced lab would actually do. Hugging Carbon reports roughly 56% lower emissions per unit compute for H100 versus A100. The new-hardware route therefore starts with a real efficiency advantage that our legacy configuration must overcome. Its changing point is tuned with an Auto-PINN-style decoupled search, with the search runs counted. No reported carbon-to-target result was found on Lacuna for any hybrid PINN optimizer, so this baseline's carbon figures are produced in this study.
NNCG as the accuracy ceiling. NNCG is the strongest existing optimizer for the hardest cases, reported at more than 10x lower loss than Adam+L-BFGS. It is run on both hardware generations to show where full-strength curvature is required.
Ablations. Three ablations isolate the key ingredient.
- The memory-budgeted hybrid with the second phase replaced by more Adam steps at equal energy. This tests whether curvature or just more compute is doing the work.
- The hybrid with FP32 versus FP64 in the curvature phase. This tests whether legacy GPUs' weak double precision is the real barrier.
- The hybrid with and without AdamQLR damping, which separates the stabilizing heuristics from the curvature itself.
What counts as winning. The proposal wins if the legacy-hardware hybrid reaches the same target error as the new-hardware baseline with provisioning carbon lower in a majority of the uncertainty draws, at a stated job volume. It must also not lose in marginal mode on the hardware it runs on.
Why not just…?
Why not just buy the newest GPUs, since they are more efficient per FLOP? This is a real threat, and at high job volumes it is probably correct. Hugging Carbon's per-compute advantage for H100 is large. On the other side, FUEL found the opposite ranking for low-throughput serving, because manufacturing carbon cannot be recovered when the device sits idle. Which effect wins for training depends on job volume. The break-even surface is designed to settle exactly this, and we will report the volume above which buying wins, even if it is low.
Why not just use a finite element solver? For a single solve, it wins by 5–6 orders of magnitude according to Can PINNs beat FEM?, and we concede this. We restrict claims to surrogates queried many times, where the same paper finds PINNs 2–3 orders faster at inference. We also report the query count at which a PINN's training carbon is repaid relative to repeated FEM solves on the same legacy CPU nodes.
Why not just run Adam longer, or tune its schedule harder? This is plausible on smooth PDEs, which is why the equal-energy ablation exists. If extra Adam steps reach the target on legacy hardware as cheaply as the hybrid, the optimizer contribution collapses and we will say so. The prior evidence from the PINN loss-landscape study is that Adam stalls at a non-zero gradient floor on stiff problems. AdamQLR's results suggest heavily tuned first-order baselines buy their gains with sweep runs that our accounting charges for.
Why not just power-cap the GPUs? Capping reduces power but stretches runtime, and it does nothing about an optimizer stuck on a plateau. It is complementary rather than competing. We run every configuration at stock settings and at one fixed cap, to show whether capping changes the ranking.
Proposed Experiments
Benchmarks. We use the PDE suite from the PINN loss-landscape and Auto-PINN papers: convection, reaction, wave, Burgers, heat, and Allen-Cahn. Each target is a fixed relative L2 error taken from the hybrid baseline's achieved accuracy on new hardware. We add the electromigration stress problem as an engineering case with a published 1.22% accuracy bar.
Hardware. We use three generations: an L40/V100-class legacy GPU, A100, and H100. Energy is logged from device and node power counters. Embodied carbon is computed with SCARIF-style whole-server estimates under PCAM-style distributions, and lifetimes are swept from three to seven years.
First decisive check: feasibility. Does the memory-budgeted hybrid reach every target on legacy hardware where full-batch FP64 L-BFGS either runs out of memory or degrades? If it does not, the main claim fails.
Second decisive check: provisioning carbon. At a realistic yearly volume, is the probability that legacy-plus-hybrid has lower provisioning carbon than new-plus-standard-hybrid above one half? We report the volume at which this flips.
Third decisive check: marginal carbon. On each fixed hardware generation, does the hybrid beat Adam alone on marginal carbon-to-target, with tuning runs included? This is the check that respects the sunk-carbon boundary. It tells operators what to run on the hardware they already own, independent of any embodied argument.
We also report the share of total carbon spent on changing-point search. This puts a first number on the "dark matter" that LLMCarbon and Hugging Carbon leave out.
References
- The Sunk Carbon Fallacy: Rethinking Carbon Footprint Metrics for Effective Carbon-Aware Scheduling
- MKOR: Momentum-Enabled Kronecker-Factor-Based Optimizer Using Rank-1 Updates
- FUSE: First-Order and Second-Order Unified Synthesis in Stochastic Optimization
- Auto-PINN: Understanding and Optimizing Physics-Informed Neural Architecture
- LLMCarbon: Modeling the End-to-End Carbon Footprint of Large Language Models
- Hugging Carbon: Quantifying the Training Carbon Emissions of AI Models at Scale
- Unveiling Environmental Impacts of Large Language Model Serving: A Functional Unit View
- Challenges in Training PINNs: A Loss Landscape Perspective
- A Space-Time Neural Network for Analysis of Stress Evolution Under DC Current Stressing
- Can physics-informed neural networks beat the finite element method?
- Adam through a Second-Order Lens
- SCARIF: Towards Carbon Modeling of Cloud Servers with Accelerators
- Unveiling the Uncertainty in Embodied and Operational Carbon of Large AI Models through a Probabilistic Carbon Accounting Model