TL;DR
ACSeL is a new framework designed to optimize Large Language Model (LLM) inference on resource-constrained hardware like edge devices. By using lightweight "surrogate" models to predict both generation quality and hardware latency, it navigates a massive search space of model sizes and mixed-precision quantization recipes to find the best-performing configuration that strictly satisfies memory and latency limits.
What Problem Does This Paper Address?
Deploying LLMs on the "edge"—think workstations, vehicles, or wearable medical monitors—is a different beast than data-center serving. In a data center, you usually optimize for maximum throughput. On the edge, you face hard physical limits: finite DRAM, limited memory bandwidth, and strict "Service-Level Objectives" (SLOs) for responsiveness, such as Time-to-First-Token (TTFT) and Time-per-Output-Token (TPOT).
To meet these constraints, engineers have many "knobs" to turn: model size, batch size, and complex mixed-precision quantization (where different layers of the model use different bit-widths). The resulting search space is combinatorially explosive—often exceeding $10^7$ possible configurations. Exhaustively testing every option is impossible, and traditional latency models often fail to predict performance accurately on specific hardware backends.
Methods
The authors propose ACSeL (Adaptive Configuration Search for Hardware-Constrained LLM Inference), which relies on three core innovations to find the "Goldilocks" configuration:
- Relative Model Quality Degradation (RMQD): Instead of running slow, expensive benchmarks (like MMLU) for every candidate, ACSeL uses RMQD. This metric calculates the KL divergence between a candidate model's output distribution and a massive "oracle" model (e.g., Llama-3.1 405B) over a small calibration set. It’s a continuous, inexpensive proxy that correlates strongly with actual task performance.
- Few-shot Median Calibration (FMC): Existing latency predictors (like AIConfigurator) often struggle with specific hardware quirks. FMC takes a few (tens) of end-to-end latency measurements on the target device and uses robust log-space median regression to "calibrate" the theoretical latency estimates, significantly improving prediction accuracy.
- BandPareto (BP) Search: This algorithm efficiently explores layerwise mixed-precision. It uses a "soft screening" phase with a very cheap latency approximation to filter out obviously bad candidates, then applies the more expensive calibrated latency gate only to the promising "Pareto frontier" of configurations.
Datasets
The researchers evaluated ACSeL using:
- Model Families: Llama-3.x and Qwen3 (sizes up to 8B).
- Oracle Models: Llama-3.1 405B and Qwen3 235B-A22B.
- Calibration/Evaluation: 24 samples from each of the 17 benchmarks in the NeMo Evaluator.
- Hardware Platforms: NVIDIA RTX A6000 (workstation) and NVIDIA Jetson Thor (edge platform).
- Latency Dataset: The authors collected a new dataset of over 17,000 latency measurements across different backends (vLLM, SGLang, TensorRT-LLM) and quantization formats (int8, int4, fp8, nvfp4).
Key Insights and Ideas
- The "Headroom" of Mixed-Precision: Uniform quantization (applying the same bit-width to the whole model) offers very few operating points. Mixed-precision "bands" fill the gaps, allowing for configurations that achieve higher quality for the same latency budget.
- RMQD as a Reliable Proxy: The paper demonstrates that RMQD has a strong Spearman correlation ($\rho = -0.72$ to $-0.85$) with standard benchmarks like MMLU and GSM8K. Crucially, computing RMQD is ~91x faster than direct benchmark evaluation.
- Calibration is Mandatory: Raw analytical latency models are often too optimistic or too pessimistic. The "few-shot" calibration (FMC) was the difference between a search algorithm that frequently failed to meet hardware constraints and one that consistently succeeded.
Key Results
- Feasibility Boost: On the A6000, using FMC improved the "answer rate" (finding a configuration predicted to be feasible) from 48% to over 89%.
- Quality Optimization: The BandPareto algorithm combined with FMC achieved the lowest quality gap compared to the theoretical best-measured configurations in the dataset.
- Latency Accuracy: FMC reduced the median absolute percentage error (MdAPE) for TTFT prediction from 113% (uncalibrated) to roughly 30% across various backends.
Limitations and Future Work
- Band Constraints: The current search assumes a maximum of three contiguous "bands" for mixed-precision. While effective, more granular per-layer optimization might yield even better results if the search cost can be managed.
- Backend Support: Some advanced quantization formats (like nvfp4) are only supported on specific newer hardware/backends, limiting the search space on older platforms.
- Future Directions: The authors suggest exploring "agentic LLMs"—using small models as intelligent search agents—to navigate even more complex deployment spaces in the future.
Summary updated