TL;DR
This paper challenges the "larger model, lower precision" heuristic for local code LLMs by introducing energy per solved task as a key metric. Through extensive benchmarking on a MacBook Air M2, the author demonstrates that jumping from a 3B (4-bit) to a 7B (2-bit) model to fit a 3GB file budget doubles energy consumption for zero gain in coding accuracy.
What Problem Does This Paper Address?
Developers running local code assistants (via tools like Ollama) face a multi-dimensional optimization problem: they must balance model accuracy against limited disk space and the energy cost of inference (which impacts battery life and thermal throttling).
A common rule of thumb in the LLM community is that a larger model at a lower quantization (e.g., 7B parameters at 2-bit) will generally outperform a smaller model at higher precision (e.g., 3B parameters at 4-bit) when both fit the same memory footprint. However, most benchmarks focus solely on accuracy or "energy per token." This paper asks: What is the energy cost of actually getting a correct answer? By pricing models in "Joules per solved task," the author provides a more pragmatic guide for developers choosing models based on file size budgets.
Methods
The researcher evaluated 24 GGUF-formatted files of the Qwen2.5-Coder-Instruct family across four parameter sizes (0.5B, 1.5B, 3B, and 7B) and various quantization levels (from q2_K up to fp16).
- Hardware: A fanless MacBook Air M2 (16GB RAM).
- Inference: Ollama (using the
llama.cpprunner on Metal). - Energy Measurement: The author used the
macmontool to sample CPU, GPU, and memory power rails at 1Hz. They calculated both "marginal rail power" (power above idle) and "whole-system power." - Metric: The primary metric is Energy per Solved Task (J/solve), calculated by dividing the total marginal energy consumed during a benchmark run by the number of tasks successfully passed.
- Statistical Rigor: The study used exact McNemar tests for paired correctness comparisons and 95% bootstrap intervals for energy metrics.
Datasets
The models were evaluated on EvalPlus, a rigorous extension of standard benchmarks:
- HumanEval+: 164 tasks.
- MBPP+: 378 tasks.
- Total: 542 Python coding tasks. EvalPlus is preferred over the base versions because it uses significantly more test cases to catch "wrong but lucky" solutions.
Key Insights and Ideas
- The "Anchor Step" Failure: The most significant finding is the failure of the larger-model rule at the 3GB budget. While a 7B-q2_K model fits a 3GB budget, it provides no statistically significant accuracy gain over the 2GB 3B-q4_K_M model, yet it consumes 2.03x the energy on the power rails.
- The 4-bit Sweet Spot: Across multiple sizes (1.5B and 3B), the
q4_K_Mquantization (roughly 5 bits per weight) consistently sat on the "correctness and energy frontier." It offers near-FP16 accuracy with much lower energy and storage costs. - Quantization Tags are Misleading: The paper notes that a tag like
q4doesn't strictly mean 4 bits. For example, the 0.5B-q3_K_M file actually stores 5.66 effective bits per weight (BPW). Developers should look at actual file size and BPW rather than just the recipe name. - Energy vs. Accuracy Frontier: Figure 1 visualizes the "Pareto frontier" of models. A model is on the frontier if no other model is both more accurate and more energy-efficient.

Key Results
The paper provides a concrete selection guide based on Ollama's listed file sizes:
- 1 GB Budget: 1.5B-q4_K_M is the clear winner (58.3% pass rate, ~39 J/solve).
- 2 GB Budget: 3B-q4_K_M is the best pick (67.5% pass rate, ~69 J/solve).
- 3 GB Budget: Stay with the 2 GB pick. The 7B-q2_K model (the only 7B that fits) is a "trap"—it solves roughly the same number of tasks but costs 141 J/solve.
- 8 GB Budget: 7B-q6_K is recommended if accuracy is the priority (74.4% pass rate), though it is significantly more "expensive" in energy (~183 J/solve) than the 3B models.
The study also found that fp16 models are almost never worth the energy cost for local deployment, as they stay far off the efficiency frontier compared to high-bit quants like q8_0 or q6_K.
Limitations and Future Work
- Model Diversity: The study is limited to the Qwen2.5-Coder family. While highly performant, results might differ for other architectures (e.g., Llama 3 or Mistral).
- Language Scope: Evaluation was limited to Python. Energy efficiency might vary for languages that require longer context or different reasoning patterns.
- Hardware Specifics: Results are specific to Apple Silicon (M2). The energy trade-offs between CPU and GPU might shift on NVIDIA-based laptops or different architectures like Intel/AMD.
- Decoding Strategy: The study used greedy decoding (temperature 0). Future work could explore how different sampling strategies or "Chain of Thought" prompting affect the energy-per-solve equation.
Summary updated