Throughput per Dollar Optimization for Heterogeneous Workloads

73 concepts56 papers1 proposal

For decades, computing efficiency was measured in purely technical terms: minimizing execution time, maximizing throughput, or squeezing out more floating-point operations per second (FLOPs). But in modern warehouse-scale computing and cloud environments, raw performance metrics hide a critical flaw: they completely ignore the financial cost of the underlying hardware.

As Neil Thompson et al. demonstrate in A Model for Estimating the Economic Costs of Computer Vision Systems That Use Deep Learning, technical scaling laws do not directly correlate with economic efficiency. To build sustainable systems, engineers must pivot from optimizing for raw speed to maximizing throughput-per-dollar. This paradigm shift requires a deep understanding of why high-end hardware often yields diminishing economic returns, how to mathematically model financial trade-offs, and how to exploit hardware heterogeneity to achieve cost-effective performance.

The Fallacy of Raw Performance

It is tempting to assume that the fastest hardware is always the best choice. However, specialized accelerators like GPUs and FPGAs carry steep price premiums and architectural overheads that can destroy their economic viability under specific workloads.

Consider the problem of data transfer. In machine learning inference, a GPU can process batches of data incredibly quickly. But as Hong Guan et al. show in A Comparison of End-to-End Decision Forest Inference Pipelines, when dealing with smaller datasets or simpler models, the time and resource overhead required to transfer data across the PCIe bus to the GPU can completely wipe out its computational advantages. In these scenarios, highly optimized CPU-based solutions become significantly more cost-effective.

Furthermore, hardware efficiency is highly dependent on utilization. Sergiy Popovych et al. demonstrate this in PZnet: Efficient 3D ConvNet Inference on Manycore CPUs. When a neural network architecture features fewer feature maps or smaller workloads that fail to fully saturate a GPU's massive parallel capacity, the expensive accelerator sits partially idle. By aggressively optimizing memory and fusing layers on manycore CPUs, the authors show that CPU inference can outpace cloud GPUs in throughput-per-dollar for these specific configurations.

Traditional View:  [Faster Hardware] ---> [Better System]
Economic View:     [High Hardware Cost] / [Underutilized Capacity] ---> [Poor Value]

Even when workloads scale up, upgrading to the latest hardware generation isn't always economically justified. In an empirical evaluation of database management systems, Young-Kyoon Suh et al. find in An Experimental Study across GPU DBMSes toward Cost-Effective Analytical Processing that the performance gains of cutting-edge GPUs often scale sub-linearly relative to their market cost. Consequently, older or mid-range hardware configurations frequently deliver a superior performance-per-dollar ratio for analytical database tasks.

Modeling the Economic Frontier

To move away from guesswork, researchers have developed frameworks that treat financial cost as a first-class constraint alongside latency and throughput.

One approach is to embed hardware costs directly into model evaluation metrics. For instance, when evaluating large language models, focusing purely on accuracy benchmarks can lead to over-engineered, prohibitively expensive deployments. To counter this, Sumuk Shashidhar et al. propose the PeRFICS metric in Democratizing LLMs: An Exploration of Cost-Performance Trade-offs in Self-Refined Open-Source Models. PeRFICS explicitly balances a model's performance improvements against its computational and memory hardware costs, revealing that mid-sized models often provide the optimal economic sweet spot.

On a macro scale, this relationship can be formalized as "compute investment efficiency." As modeled by Konstantin F. Pilz et al. in Increased Compute Efficiency and the Diffusion of AI Capabilities, the total capability achievable per unit of currency is a direct product of hardware price-performance and algorithmic efficiency.

In cloud environments, this modeling allows for automated, budget-conscious infrastructure provisioning. Adam Barker introduces this transparency in Plug and Play Bench: Simplifying Big Data Benchmarking Using Containers, correlating workload execution times directly with cloud provider pricing models to map out the exact point of diminishing returns for cluster scaling. Similarly, Bo Wang et al. in Managing Deadline-constrained Bag-of-Tasks Jobs on Hybrid Clouds use binary nonlinear programming to select virtual machine instances based on their computational efficiency relative to price, maximizing economic throughput under strict deadlines.

Exploiting Heterogeneity and Dynamic Selection

Once we can model cost-performance, we can design software systems that actively exploit hardware heterogeneity to minimize operational expenses. Instead of relying on a uniform pool of identical, high-end instances, modern frameworks dynamically distribute tasks across a diverse mix of hardware.

The inference serving system proposed in RIBBON: cost-effective and qos-aware deep learning model inference using a diverse pool of cloud computing instances exemplifies this strategy. Rather than routing all deep learning queries to expensive GPU instances, the system mixes different tiers of hardware (such as cheap CPUs and premium GPUs). By utilizing Bayesian Optimization, it identifies the cheapest combination of instances capable of satisfying strict Quality of Service (QoS) latency targets, evaluating instances by their cost-per-query rather than raw throughput.

Workload Queue ---> [ Scheduler / Optimizer ] ---> High-Saturated Tasks ---> [ Premium GPU ]
                                                 ---> Low-Saturated Tasks  ---> [ Cheap CPU ]

This philosophy also applies to cluster-wide scheduling. Deepak Narayanan demonstrates in Heterogeneity-Aware Cluster Scheduling Policies for Deep Learning Workloads that different deep learning models exhibit vastly different performance scaling across different GPU generations. By formulating scheduling as a mathematical optimization problem that accounts for these varying cost-performance ratios, workloads can be dynamically assigned to older, cheaper hardware if the performance jump on a newer chip doesn't justify its financial premium.

Ultimately, optimizing for throughput-per-dollar changes the engineering objective. It acknowledges that the "best" system is not the one that runs the fastest in a vacuum, but the one that delivers the required performance at the lowest possible financial threshold.

Key Foundational Work

A scalable IPv6 route lookup scheme via dynamic variable-stride bitmap compression and path compression (2006) — contributed 1 concept elements to this direction.
Merrimac: Supercomputing with Streams (2003) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_cost_performance_optimization_7532c42e,
  title = {cost-performance optimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/cost-performance-optimization/rdr_7532c42e416f4adfb89d7a062ea4e029},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_cost_performance_optimization_7532c42e_v1,
  title = {cost-performance optimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/cost-performance-optimization/rdr_7532c42e416f4adfb89d7a062ea4e029/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake