ACSeL: Adaptive Configuration Search for Hardware-Constrained LLM Inference

Bokun Wang, Hung-Yueh Chiang, Diana MarculescuarXiv 2026 · arXiv

TL;DR

ACSeL is a new framework designed to optimize Large Language Model (LLM) inference on resource-constrained hardware like edge devices. By using lightweight "surrogate" models to predict both generation quality and hardware latency, it navigates a massive search space of model sizes and mixed-precision quantization recipes to find the best-performing configuration that strictly satisfies memory and latency limits.

What Problem Does This Paper Address?

Deploying LLMs on the "edge"—think workstations, vehicles, or wearable medical monitors—is a different beast than data-center serving. In a data center, you usually optimize for maximum throughput. On the edge, you face hard physical limits: finite DRAM, limited memory bandwidth, and strict "Service-Level Objectives" (SLOs) for responsiveness, such as Time-to-First-Token (TTFT) and Time-per-Output-Token (TPOT).

To meet these constraints, engineers have many "knobs" to turn: model size, batch size, and complex mixed-precision quantization (where different layers of the model use different bit-widths). The resulting search space is combinatorially explosive—often exceeding $10^7$ possible configurations. Exhaustively testing every option is impossible, and traditional latency models often fail to predict performance accurately on specific hardware backends.

Methods

The authors propose ACSeL (Adaptive Configuration Search for Hardware-Constrained LLM Inference), which relies on three core innovations to find the "Goldilocks" configuration:

  1. Relative Model Quality Degradation (RMQD): Instead of running slow, expensive benchmarks (like MMLU) for every candidate, ACSeL uses RMQD. This metric calculates the KL divergence between a candidate model's output distribution and a massive "oracle" model (e.g., Llama-3.1 405B) over a small calibration set. It’s a continuous, inexpensive proxy that correlates strongly with actual task performance.
  2. Few-shot Median Calibration (FMC): Existing latency predictors (like AIConfigurator) often struggle with specific hardware quirks. FMC takes a few (tens) of end-to-end latency measurements on the target device and uses robust log-space median regression to "calibrate" the theoretical latency estimates, significantly improving prediction accuracy.
  3. BandPareto (BP) Search: This algorithm efficiently explores layerwise mixed-precision. It uses a "soft screening" phase with a very cheap latency approximation to filter out obviously bad candidates, then applies the more expensive calibrated latency gate only to the promising "Pareto frontier" of configurations.
Overview of the ACSeL framework showing the search algorithm guided by quality and latency surrogates.

Datasets

The researchers evaluated ACSeL using:

  • Model Families: Llama-3.x and Qwen3 (sizes up to 8B).
  • Oracle Models: Llama-3.1 405B and Qwen3 235B-A22B.
  • Calibration/Evaluation: 24 samples from each of the 17 benchmarks in the NeMo Evaluator.
  • Hardware Platforms: NVIDIA RTX A6000 (workstation) and NVIDIA Jetson Thor (edge platform).
  • Latency Dataset: The authors collected a new dataset of over 17,000 latency measurements across different backends (vLLM, SGLang, TensorRT-LLM) and quantization formats (int8, int4, fp8, nvfp4).

Key Insights and Ideas

  • The "Headroom" of Mixed-Precision: Uniform quantization (applying the same bit-width to the whole model) offers very few operating points. Mixed-precision "bands" fill the gaps, allowing for configurations that achieve higher quality for the same latency budget.
Quality-latency headrooms showing how non-uniform (banded) configurations provide better trade-offs than uniform ones.
  • RMQD as a Reliable Proxy: The paper demonstrates that RMQD has a strong Spearman correlation ($\rho = -0.72$ to $-0.85$) with standard benchmarks like MMLU and GSM8K. Crucially, computing RMQD is ~91x faster than direct benchmark evaluation.
Correlation between RMQD and accuracy across MMLU, HumanEval, and GSM8K benchmarks.
  • Calibration is Mandatory: Raw analytical latency models are often too optimistic or too pessimistic. The "few-shot" calibration (FMC) was the difference between a search algorithm that frequently failed to meet hardware constraints and one that consistently succeeded.

Key Results

  • Feasibility Boost: On the A6000, using FMC improved the "answer rate" (finding a configuration predicted to be feasible) from 48% to over 89%.
  • Quality Optimization: The BandPareto algorithm combined with FMC achieved the lowest quality gap compared to the theoretical best-measured configurations in the dataset.
  • Latency Accuracy: FMC reduced the median absolute percentage error (MdAPE) for TTFT prediction from 113% (uncalibrated) to roughly 30% across various backends.

Limitations and Future Work

  • Band Constraints: The current search assumes a maximum of three contiguous "bands" for mixed-precision. While effective, more granular per-layer optimization might yield even better results if the search cost can be managed.
  • Backend Support: Some advanced quantization formats (like nvfp4) are only supported on specific newer hardware/backends, limiting the search space on older platforms.
  • Future Directions: The authors suggest exploring "agentic LLMs"—using small models as intelligent search agents—to navigate even more complex deployment spaces in the future.

Summary updated

Abstract

Serving large language models (LLMs) under tight memory and latency service-level objectives (SLOs), including on edge devices, requires jointly selecting the model size, batch size, quantization algorithm, and mixed-precision quantization recipes. Exhaustive search over this combinatorially large space is infeasible. Unlike data-center serving, which often prioritizes throughput or generation speed, resource-constrained deployments typically set memory and latency budgets as constraints and then seek the configuration with the highest generation quality among feasible ones. To this end, we propose ACSeL, a framework that identifies configurations predicted to satisfy deployment constraints while achieving high predicted generation quality. To enable efficient quality estimation, we use Relative Model Quality Degradation (RMQD), a proxy based on the divergence between the candidate and reference models' next-token distributions over a small calibration corpus. Across the evaluated model sizes and precisions, RMQD is inexpensive to evaluate and strongly correlated with downstream benchmark performance. ACSeL combines (i) a lightweight quality surrogate that predicts RMQD directly from a configuration; (ii) Few-shot Median Calibration (FMC), which calibrates operator-profile-based latency estimates using a small number of end-to-end profiles; and (iii) a structured search algorithm BandPareto for exploring layerwise mixed-precision configurations. Experiments on an NVIDIA RTX A6000 workstation and an NVIDIA Jetson Thor edge platform show that FMC substantially improves deployment feasibility, while BandPareto + FMC achieves the highest feasibility and the lowest RMQD gaps to the best measured configurations in the evaluation set.
BibTeX
@misc{wang2026acsel,
  author = {Bokun Wang and Hung-Yueh Chiang and Diana Marculescu},
  title = {{ACSeL: Adaptive Configuration Search for Hardware-Constrained LLM Inference}},
  year = {2026},
  eprint = {2611.07570},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.07570},
}
Figure