Reducing the Memory and Latency of Language Model Inference

1,510 concepts1,365 papers38 proposals

Large Language Models (LLMs) have transformed natural language processing, but deploying them in production reveals a stark physical limitation: they are incredibly expensive to run.

During autoregressive inference, an LLM generates text token by token. For every single token generated, the entire matrix of model weights must be loaded from High Bandwidth Memory (HBM) into the GPU's cache (SRAM) to perform matrix-vector multiplications. Because the computational cost of processing a single token is low relative to the massive size of the weights being moved, LLM inference is fundamentally memory-bandwidth bound.

To make these models viable for real-world applications, resource-constrained edge devices, and low-latency software systems, researchers are attacking this bottleneck from two primary angles: compressing the model's physical footprint and dynamically optimizing how queries are routed.

Compressing the Weights: Quantization and Pruning

The most direct way to alleviate the memory bandwidth bottleneck is to reduce the size of the weights themselves. If a model's parameters can be shrunk from 16-bit floating-point numbers (FP16) down to 4-bit or 8-bit integers, the amount of data transferred from memory drops by a factor of two to four, instantly accelerating inference.

However, uniform or naive compression introduces a critical problem: it degrades the model's accuracy. Not all weights in a neural network are created equal. Certain parameters, activation pathways, or task-specific sub-networks are highly sensitive to precision loss.

To address this, Hanqi Xiao et al. introduced Task-Circuit Quantization (TACQ) in Task-Circuit Quantization: Leveraging Knowledge Localization and Interpretability for Compression. Instead of compressing the entire model uniformly, the authors use mechanistic interpretability to identify localized "circuits"—the specific sub-graphs of weights responsible for executing a given task. By preserving these critical circuits at higher precision while aggressively quantizing the rest of the network, they achieve ultra-low bit-widths (2–3 bits) with minimal accuracy loss.

Naive Quantization:
[All Weights] ---> Uniform Bit-Reduction ---> Loss of Critical Nuances

Circuit-Aware Quantization (TACQ):
[Model Weights] ---> Identify Task Circuits ---> Keep Circuits High-Precision
                                           ---> Quantize Remaining Weights

Another approach is to let the model's final performance guide the compression process. In GuidedQuant: Large Language Model Quantization via Exploiting End Loss Guidance, Jinuk Kim et al. proposed using the model's end loss to determine quantization parameters. By evaluating how weight alterations impact the final output error rather than just minimizing local rounding errors layer by layer, the quantization process directly protects the model's macroscopic reasoning capabilities.

When removing weights entirely through structured pruning, a similar philosophy applies. Jun Kong et al. developed AdaPruner in Sample-aware Adaptive Structured Pruning for Large Language Models, which adaptively optimizes both calibration data and importance metrics. This ensures that structural pruning removes only redundant attention heads or channels without accidentally severing vital representational pathways.

Structural Simplification via Low-Rank Decomposition

Rather than simply rounding or deleting weights, low-rank decomposition treats weight matrices as highly redundant linear transformations that can be mathematically factored into smaller matrices.

In RETHINKING COMPRESSION: REDUCED ORDER MODELLING OF LATENT FEATURES IN LARGE LANGUAGE MODELS, Arnav Chavan et al. demonstrated LLM-ROM, a training-free compression method that applies reduced-order modeling to latent features. By identifying the lower-dimensional subspaces where the model's hidden states actually live, they compress layers without requiring expensive GPU-driven retraining.

However, low-rank decomposition often struggles with "activation outliers"—extreme values that appear in specific dimensions of LLM hidden states and dominate the network's behavior. To resolve this, Zhihang Yuan et al. introduced ASVD in ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Models. ASVD scales the weight matrices based on activation variance before performing Singular Value Decomposition (SVD). This effectively insulates the low-rank approximation from being distorted by outlier activations, allowing up to a 30% reduction in model size and a 50% reduction in Key-Value (KV) cache memory.

Adaptive Query Routing and Caching

An orthogonal strategy to model compression is changing how queries are processed. In a standard deployment, a simple query (e.g., "What is the capital of France?") consumes the exact same amount of computational power as a complex reasoning query because both pass through every parameter of the giant model.

Adaptive routing systems break this symmetry by matching query complexity to the appropriate computational resource.

                /---> Simple Query ---> Symbolic/Cached Pathway ---> Low Cost
Incoming Query
                \---> Complex Query ---> Full Neural LLM Pathway ---> High Cost

For Retrieval-Augmented Generation (RAG) systems, Safayat Bin Hakim et al. introduced SymRAG in SymRAG: Efficient Neuro-Symbolic Retrieval Through Adaptive Query Routing. SymRAG dynamically routes incoming requests through symbolic, neural, or hybrid processing pathways based on the query's characteristics and current system load, preventing heavy neural models from being wasted on straightforward lookups.

To mitigate the high financial costs of commercial LLM APIs, Ilias Stogiannidis et al. proposed the Online Cost-aware Teacher-Student (OCaTS) framework in Cache me if you Can: an Online Cost-aware Teacher-Student framework to Reduce the Calls to Large Language Models. OCaTS operates as an intelligent triage system: it intercepts queries, checks an evolving cache, and attempts to answer using a lightweight, locally hosted "student" model trained on the fly. The expensive "teacher" LLM API is queried only when the student model's internal reliability metrics indicate that the query requires higher-order reasoning.

Bypassing Alignment Overhead with Weight Arithmetic

The computational expense of LLMs is not confined to inference; aligning models to follow instructions or speak new languages via Reinforcement Learning from Human Feedback (RLHF) requires massive hardware clusters.

Hung-yi Lee et al. introduced a remarkably efficient alternative using weight arithmetic in Chat Vector: A Simple Approach to Equip LLMs With New Language Chat Capabilities. By subtracting the weights of a base model from its chat-aligned counterpart, the authors isolate a directional "chat vector" in parameter space. Adding this vector directly to a completely different base model transfers instruction-following capabilities instantly, bypassing the need for expensive alignment training altogether.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_computational_cost_of_llms_e906099f,
  title = {computational cost of LLMs},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/computational-cost-of-llms/rdr_e906099f27df41df8bb58994f8fb6d7e},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_computational_cost_of_llms_e906099f_v1,
  title = {computational cost of LLMs},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/computational-cost-of-llms/rdr_e906099f27df41df8bb58994f8fb6d7e/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake