Choosing Quantized Code LLMs by File Size and Energy per Solved Task

Aditi PatodiyaarXiv 2026 · arXiv

TL;DR

This paper challenges the "larger model, lower precision" heuristic for local code LLMs by introducing energy per solved task as a key metric. Through extensive benchmarking on a MacBook Air M2, the author demonstrates that jumping from a 3B (4-bit) to a 7B (2-bit) model to fit a 3GB file budget doubles energy consumption for zero gain in coding accuracy.

What Problem Does This Paper Address?

Developers running local code assistants (via tools like Ollama) face a multi-dimensional optimization problem: they must balance model accuracy against limited disk space and the energy cost of inference (which impacts battery life and thermal throttling).

A common rule of thumb in the LLM community is that a larger model at a lower quantization (e.g., 7B parameters at 2-bit) will generally outperform a smaller model at higher precision (e.g., 3B parameters at 4-bit) when both fit the same memory footprint. However, most benchmarks focus solely on accuracy or "energy per token." This paper asks: What is the energy cost of actually getting a correct answer? By pricing models in "Joules per solved task," the author provides a more pragmatic guide for developers choosing models based on file size budgets.

Methods

The researcher evaluated 24 GGUF-formatted files of the Qwen2.5-Coder-Instruct family across four parameter sizes (0.5B, 1.5B, 3B, and 7B) and various quantization levels (from q2_K up to fp16).

  • Hardware: A fanless MacBook Air M2 (16GB RAM).
  • Inference: Ollama (using the llama.cpp runner on Metal).
  • Energy Measurement: The author used the macmon tool to sample CPU, GPU, and memory power rails at 1Hz. They calculated both "marginal rail power" (power above idle) and "whole-system power."
  • Metric: The primary metric is Energy per Solved Task (J/solve), calculated by dividing the total marginal energy consumed during a benchmark run by the number of tasks successfully passed.
  • Statistical Rigor: The study used exact McNemar tests for paired correctness comparisons and 95% bootstrap intervals for energy metrics.

Datasets

The models were evaluated on EvalPlus, a rigorous extension of standard benchmarks:

  • HumanEval+: 164 tasks.
  • MBPP+: 378 tasks.
  • Total: 542 Python coding tasks. EvalPlus is preferred over the base versions because it uses significantly more test cases to catch "wrong but lucky" solutions.

Key Insights and Ideas

  • The "Anchor Step" Failure: The most significant finding is the failure of the larger-model rule at the 3GB budget. While a 7B-q2_K model fits a 3GB budget, it provides no statistically significant accuracy gain over the 2GB 3B-q4_K_M model, yet it consumes 2.03x the energy on the power rails.
  • The 4-bit Sweet Spot: Across multiple sizes (1.5B and 3B), the q4_K_M quantization (roughly 5 bits per weight) consistently sat on the "correctness and energy frontier." It offers near-FP16 accuracy with much lower energy and storage costs.
  • Quantization Tags are Misleading: The paper notes that a tag like q4 doesn't strictly mean 4 bits. For example, the 0.5B-q3_K_M file actually stores 5.66 effective bits per weight (BPW). Developers should look at actual file size and BPW rather than just the recipe name.
  • Energy vs. Accuracy Frontier: Figure 1 visualizes the "Pareto frontier" of models. A model is on the frontier if no other model is both more accurate and more energy-efficient.

![Fig.

  1. Pooled pass rate against energy per solved task. The step line represents the efficiency frontier.](/figures/art_3c52530baee24e629056160eb6c4012c)

Key Results

The paper provides a concrete selection guide based on Ollama's listed file sizes:

  • 1 GB Budget: 1.5B-q4_K_M is the clear winner (58.3% pass rate, ~39 J/solve).
  • 2 GB Budget: 3B-q4_K_M is the best pick (67.5% pass rate, ~69 J/solve).
  • 3 GB Budget: Stay with the 2 GB pick. The 7B-q2_K model (the only 7B that fits) is a "trap"—it solves roughly the same number of tasks but costs 141 J/solve.
  • 8 GB Budget: 7B-q6_K is recommended if accuracy is the priority (74.4% pass rate), though it is significantly more "expensive" in energy (~183 J/solve) than the 3B models.

The study also found that fp16 models are almost never worth the energy cost for local deployment, as they stay far off the efficiency frontier compared to high-bit quants like q8_0 or q6_K.

Limitations and Future Work

  • Model Diversity: The study is limited to the Qwen2.5-Coder family. While highly performant, results might differ for other architectures (e.g., Llama 3 or Mistral).
  • Language Scope: Evaluation was limited to Python. Energy efficiency might vary for languages that require longer context or different reasoning patterns.
  • Hardware Specifics: Results are specific to Apple Silicon (M2). The energy trade-offs between CPU and GPU might shift on NVIDIA-based laptops or different architectures like Intel/AMD.
  • Decoding Strategy: The study used greedy decoding (temperature 0). Future work could explore how different sampling strategies or "Chain of Thought" prompting affect the energy-per-solve equation.

Summary updated

Abstract

A developer running a code assistant on a laptop downloads one model file, and local inference moves the energy of every request onto that machine. A common rule says a larger model at lower precision tends to beat a smaller one at higher precision. We ran 24 GGUF files of Qwen2.5-Coder at four sizes on a fanless MacBook Air M2 and priced each in energy per solved task on 542 EvalPlus tasks. The result is a selection guide indexed by Ollama's listed file size, with an evidence label on each pick. At 1 and 2 GB it picks the q4_K_M files of the 1.5B and 3B models, both on the correctness and energy frontier under every accounting. At 3 GB as listed, the larger-model rule picks 7B-q2_K, the only measured 7B file that fits, and there it does not pay. That file brings no significant gain in solves over the 2 GB pick, 3B-q4_K_M, and the 90% CI limits any 3B-q4_K_M deficit to 2.4 points. Yet 7B-q2_K needs 2.03 times the energy per solved task on CPU, GPU and memory rails, or 1.66 to 1.81 times on whole-system power. On HumanEval+, five alternating session pairs give a ratio of 1.925 against 1.956 in the main grid. At 8 GB, 7B-q6_K buys a significant gain over the 2 GB pick at a higher energy price. From 1.5B up, q4_K_M trails the highest precision measured at its size by less than 3 points (90% CI). We release per-task results, power logs and the analysis script.
BibTeX
@misc{patodiya2026choosing,
  author = {Aditi Patodiya},
  title = {{Choosing Quantized Code LLMs by File Size and Energy per Solved Task}},
  year = {2026},
  eprint = {2611.17605},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.17605},
}
Figure