Efficient Execution of Large Language Models at the Edge

31 concepts25 papers1 proposal

Large Language Models (LLMs) are traditionally designed under the assumption of unconstrained cloud infrastructure: massive clusters of high-bandwidth memory (HBM) and parallel tensor processing units. When shifted to edge environments—such as smartphones, autonomous vehicles, and IoT nodes—this assumption collapses. As detailed in a comprehensive survey by Yupeng Zheng et al. in A Review on Edge Large Language Models: Design, Execution, and Applications, there is a stark and growing disparity between the exponential computational complexity of state-of-the-art models and the hardware capabilities of edge devices.

Running LLMs locally is highly desirable for privacy, latency, and offline reliability. However, edge hardware faces strict thermal envelopes, limited battery capacity, and a lack of server-grade GPUs. To bridge this gap, researchers are rethinking the entire execution stack—modifying attention mechanisms, redesigning runtime engines, pipelining local training, and distilling reasoning into lightweight architectures.

Alleviating the Memory and Attention Bottleneck

The primary bottleneck during LLM generation on edge hardware is not just raw compute, but memory bandwidth. In standard autoregressive generation, every single token generation step requires loading the entire model's weights from RAM to the processor. Simultaneously, the system must maintain a Key-Value (KV) cache of all previous tokens in the sequence to avoid recomputing attention matrices. This KV cache grows linearly with sequence length, quickly exhausting the limited RAM of edge devices.

To mitigate this, Sushant Mehta et al. introduce an architectural optimization in Latent Multi-Head Attention for Small Language Models. Instead of storing full-sized keys and values for every attention head, their approach combines Latent Multi-Head Attention (MLA) with Rotary Positional Embeddings (RoPE). MLA works by low-rank compressing the keys and values into a compact, shared latent vector during generation. By caching only this compressed latent representation rather than individual head allocations, the authors achieve a 45% reduction in KV-cache memory footprint and a 1.4x inference speedup, making long-context generation viable on small hardware.

Standard Multi-Head Attention (MHA):
[Token] ---> [Key Head 1, 2, ... N]  \---> Cached separately (High Memory)
        ---> [Value Head 1, 2, ... N] /

Latent Multi-Head Attention (MLA):
[Token] ---> [Shared Low-Rank Latent Vector] ---> Cached efficiently (Low Memory)

Complementing architectural changes, weight quantization compresses the precision of model parameters (e.g., from 16-bit floating-point to 4-bit integers). However, uniform quantization often degrades model accuracy because attention layers exhibit highly non-linear activation outliers that are sensitive to precision loss.

To solve this, Ziyi Guan et al. developed APTQ in APTQ: Attention-aware Post-Training Mixed-Precision Quantization for Large Language Models. Instead of treating all weights equally, APTQ uses the Hessian trace of the attention mechanism to mathematically measure how sensitive the overall network loss is to specific parameters. It then allocates mixed-precision bitwidths intelligently—preserving higher precision for critical attention weights while aggressively compressing less sensitive parameters.

Similarly, Jason Kong et al. present TinyAgent in TinyAgent: Quantization-aware Model Compression and Adaptation for On-device LLM Agent Deployment. TinyAgent pairs activation-aware weight quantization (AWQ) with a layer-dropping technique called TrimLLM. By systematically removing non-essential layers that contribute minimally to downstream task accuracy, the framework shrinks the model's footprint to fit within the strict memory bounds required for on-device IoT agents.

Rethinking the Execution Stack via Relational Databases

Standard edge deployment strategies typically rely on specialized machine learning runtimes (such as ONNX Runtime or TFLite). While effective, these frameworks still introduce substantial software overhead and often struggle to optimize memory layout on commodity hardware lacking dedicated neural processing units (NPUs).

An alternative paradigm is proposed by Rihan Hai et al. in TranSQL + : Serving Large Language Models with SQL on Low-Resource Hardware. Rather than introducing a new ML library, the authors translate LLM matrix computations directly into pure SQL queries executed by relational database engines.

This approach works because modern relational databases (like SQLite or DuckDB) are already ubiquitously deployed and highly optimized for low-resource hardware. They possess mature, decades-refined mechanisms for memory-mapping, out-of-core execution (processing data that exceeds RAM size), and row- or column-parallel aggregations. By storing LLM weights as database tables and executing token generation via relational joins and mathematical aggregations, the system achieves high memory efficiency and speedups on everyday hardware without relying on external ML libraries or GPUs.

On-Device Customization and Multi-Tenancy

Edge deployment is rarely static; models often need to be fine-tuned locally on user data to preserve privacy, or multiple specialized models must run concurrently on a single device. However, standard backpropagation requires storing intermediate activations for the entire network, causing immediate out-of-memory (OOM) crashes on edge hardware.

To enable local adaptation, Liang Li 0021 et al. introduce RingAda in RingAda: Pipelining Large Model Fine-Tuning on Edge Devices with Scheduled Layer Unfreezing. Instead of updating all layers simultaneously, RingAda uses a scheduled layer unfreezing strategy across a pipelined, ring-topology training setup. By unfreezing and training parameter-efficient adapters (like LoRA) layer-by-layer over time, the framework caps the peak memory usage. Only a small fraction of the model's gradients and activations reside in memory at any single moment, allowing successful fine-tuning on resource-constrained edge nodes.

Standard Fine-Tuning:
[All Layers Unfrozen] ---> Activations stored for ALL layers simultaneously ---> Memory Spike (OOM)

RingAda Scheduled Unfreezing:
[Step 1: Layer 1 Unfrozen] ---> [Step 2: Layer 2 Unfrozen] ---> [Step 3: Layer 3 Unfrozen]
*Activations are only stored for the active layer, keeping memory flat and predictable.*

When multiple personalized or task-specific models must coexist on the same device, keeping full copies of each model is impossible. Zhencai Shen et al. address this multi-tenant challenge in EdgeLoRA: An Efficient Multi-Tenant LLM Serving System on Edge Devices. EdgeLoRA maintains a single, frozen base model in memory and dynamically swaps or batches lightweight Low-Rank Adaptation (LoRA) matrices corresponding to different users or tasks. Through intelligent memory management and batched LoRA inference, the system eliminates the overhead of context-switching between separate models, boosting throughput on multi-tenant edge devices.

Distillation and Semantic Filtering for Downstream Tasks

When an application demands real-time performance that even an optimized LLM cannot achieve, the alternative is to distill the large model's intelligence into a smaller, task-specific architecture, or use the model to filter data before it enters the network.

In autonomous driving, where millisecond-level latency is mandatory, Haicheng Liao et al. present CoT-Drive in CoT-Drive: Efficient Motion Forecasting for Autonomous Driving With LLMs and Chain-of-Thought Prompting. Instead of running a massive LLM directly in the vehicle, the authors use a cloud-based LLM with Chain-of-Thought (CoT) prompting to generate rich semantic annotations and scene-understanding data. This advanced reasoning is then distilled into a lightweight, edge-compatible forecasting model, capturing the LLM's intelligence without its computational burden.

A similar distillation strategy applies to safety guardrails. Running a secondary "guard" model alongside a primary LLM to filter unsafe content doubles the computational cost. To bypass this, Motasem Alfarra et al. introduce TV-DiSP in Distilling Safe LLM Systems via Soft Prompts. The authors distill the safety knowledge of a heavy guard model directly into learned "soft prompts"—continuous vector embeddings prepended to the input. This allows the edge LLM to enforce safety constraints internally with virtually zero additional inference overhead.

Finally, this efficiency paradigm shifts the focus from optimizing compute to optimizing wireless network transmission. In semantic communication frameworks, such as those proposed by Guangyuan Liu et al. in Context-Aware Semantic Communication for the Wireless Networks and Yaping Sun et al. in Generative Semantic Communication: Architectures, Technologies, and Applications, edge-deployed LLMs act as intelligent filters.

Instead of transmitting raw, high-bandwidth data bits over congested wireless channels, an on-device LLM uses an "agentic context gate" or a Mixture of Experts (MoE) architecture to extract and transmit only the core "meaning" or semantic features of the data. The receiver then uses a generative model to reconstruct the full information. By leveraging optimized local edge compute to drop non-essential data before transmission, these systems drastically reduce bandwidth consumption, making AI-driven communication viable even in highly constrained wireless environments.

Key Foundational Work

CLONE: Customizing LLMs for Efficient Latency-Aware Inference at the Edge (2025) — contributed 3 concept elements to this direction.
A Review on Edge Large Language Models: Design, Execution, and Applications (2025) — contributed 3 concept elements to this direction.
Generative Semantic Communication: Architectures, Technologies, and Applications (2025) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_edge_llm_deployment_challenges_819c0f38,
  title = {edge LLM deployment challenges},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/edge-llm-deployment-challenges/rdr_819c0f38f3f94426846047591a976734},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_edge_llm_deployment_challenges_819c0f38_v1,
  title = {edge LLM deployment challenges},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/edge-llm-deployment-challenges/rdr_819c0f38f3f94426846047591a976734/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake