Hardware-Independent Metrics for Algorithmic Efficiency

This direction is merged into another direction: FLOPs-latency discrepancy. Its last version stays readable and citable here.

Version 1 of 114 papers20 conceptsfrozen 2026-10-04

In the pursuit of state-of-the-art performance, researchers often report "wall-clock time" or "training days" as a proxy for efficiency. However, these metrics are deeply entangled with specific hardware, low-level library optimizations, and even the temperature of a server room. To truly understand whether an algorithm is better than its predecessor, we need to measure the intrinsic work it performs, independent of the machine running it.

The Problem with Wall-Clock Time

Measuring efficiency in seconds or hours creates a moving target. As Jonas Geiping and Tom Goldstein point out in Cramming: Training a Language Model on a single GPU in one day, defining efficiency by fixed time intervals makes results dependent on specific hardware generations. A model that takes 24 hours to train today might take 2 hours next year on a newer GPU, even if the algorithm hasn't improved at all.

This issue isn't new. As early as 1980, Vincent J. Digricoli argued in First Experiments with Rue Automated Deduction that evaluating automated deduction systems based on CPU time is inaccurate. Because hardware and programming language optimizations vary so wildly, raw execution speed masks the true logical complexity of the search strategy.

Counting Floating-Point Operations

The most common hardware-agnostic alternative is counting Floating-Point Operations (FLOPs). By measuring the total mathematical work required, researchers can isolate the algorithm's complexity from the system's memory latency or parallelization overhead.

In Green AI, Roy Schwartz and colleagues advocate for FLOPs as a primary metric to ensure AI research remains inclusive. When efficiency is measured in FLOPs, a lab with a single consumer GPU can compete fairly with a lab using a massive cluster, as the "cost" of the model is quantified by the work done rather than the speed of the hardware.

This approach is being adopted across diverse fields:

Domain-Specific Units of Work

While FLOPs are useful for neural networks, other domains require different "units of work" to capture algorithmic efficiency. The goal remains the same: find a metric that represents the bottleneck of the logical search space.

In probabilistic programming, Razvan Ranca demonstrates in Slice Sampling for Probabilistic Programming that the number of likelihood evaluations is a superior metric for comparing inference engines. Similarly, Måns Magnusson and the team behind posteriordb: Testing, Benchmarking and Developing Bayesian Inference Algorithms use the number of gradient or log-density evaluations to ensure benchmarks are reproducible across different software implementations like Stan or PyMC.

For discrete logic problems, such as Boolean Satisfiability (SAT), Qun Zhao and Menghui Yang argue that we should look at how effectively a heuristic navigates the search space—for example, by counting learned clauses—rather than how many seconds the solver ran.

The shift toward hardware-independent metrics allows for a more rigorous study of the Pareto Frontier: the optimal trade-off between accuracy and computational cost.

Tianxiang Sun and colleagues introduced the ELUE (Efficient Language Understanding Evaluation) benchmark in Towards Efficient NLP: A Standard Evaluation and A Strong Baseline. By plotting FLOPs against accuracy, they provide a standardized way to identify models like ElasticBERT that can dynamically adjust their depth to save computation.

As Eva Memmel and Kim Batselier argue in their work on Tensor Networks, prioritizing these intrinsic metrics is essential for "Green AI." It moves the field away from a "Red AI" arms race—where results are bought with massive compute—and toward a scientific discipline where algorithmic ingenuity is measured by how much it can achieve with a single unit of mathematical work.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_hardware_independent_metrics_for_algorit_8162fc0d,
  title = {Hardware-Independent Metrics for Algorithmic Efficiency},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/hardware-independent-metrics-for-algorithmic-efficiency/rdr_8162fc0d9c1343508697a60668edf08b},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_hardware_independent_metrics_for_algorit_8162fc0d_v1,
  title = {Hardware-Independent Metrics for Algorithmic Efficiency},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/hardware-independent-metrics-for-algorithmic-efficiency/rdr_8162fc0d9c1343508697a60668edf08b/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}