TENET: An Efficient Sparsity-Aware LUT-Centric Architecture for Ternary LLM Inference On Edge

Zhirui Huang, Rui Ma, Shijie Cao, Ran Shu, Ian Wang, Ting Cao, Chixiao Chen, Yongqiang XiongarXiv 2025 · arXiv · OpenReview

Abstract

Ternary quantization has emerged as a powerful technique for reducing both computational and memory footprint of large language models (LLM), enabling efficient real-time inference deployment without significantly compromising model accuracy. Conventional LLM inference platforms (e.g GPUs) cannot capitalize on its benefits, as they (i) lack native support for ternary arithmetic and memory specialization and (ii) remain severely under-utilized in low-batch, real-time scenarios. In this work, we propose TENET, a sparse-aware LUT-centric architecture that co-optimizes algorithm, compute, and memory for ternary LLM inference. To maximize the efficiency of Ternary Linear layer, TENET introduces a Sparse Ternary LUT (STL) core that optimizes ternary mixed-precision GEMM using a symmetric precompute lookup table. It also features Dynamic Activation N:M Sparsity to exploit the sparsity within the activation of each token. Additionally, we propose a LUT-based 64B:80B ternary weight decompression module to fully exploit the memory efficiency of ternary values. At the system level, we design a heterogeneous TENET accelerator with full programmability that integrates STL cores with high-precision cores. An associated Linear-Projection-aware Sparse Attention dataflow is introduced to optimize memory access and hardware utilization. We implement TENET accelerator prototype on both FPGA and ASIC platforms. Experiments across various model sizes and workloads demonstrate that TENET-FPGA and TENET-ASIC improve energy efficiency by 4.3$\times$ and 21.1$\times$, respectively, compared to the A100 GPU. Furthermore, TENET-ASIC achieves a 2.7$\times$ average speedup compared to the A100 GPU in end-to-end inference latency.

Introduction

While Large Language Models (LLMs) are moving from data centers to "the edge" (phones, robots, and IoT), they face a massive wall: edge devices have strict power and memory limits. Ternary quantization—where weights are restricted to $\{-1, 0, 1\}$—is a promising solution because it drastically reduces model size and replaces expensive multiplications with simple additions.

However, current hardware (CPUs and GPUs) cannot actually run ternary models efficiently. They typically "de-quantize" these weights back into higher-precision formats to perform calculations, wasting the very efficiency gains ternary quantization provides. Furthermore, existing systems struggle with the "memory-bound" nature of LLM decoding and the quadratic complexity of the attention mechanism as sequence lengths grow.

Comparison of real vs. ideal energy efficiency for ternary models on standard hardware.

As shown in Figure 2, there is a significant gap (up to 10.6x on GPUs) between the "Ideal" energy efficiency of ternary models and what is "Real" on current general-purpose hardware. TENET is designed to bridge this gap.

The TENET Architecture

The authors propose TENET, a "LUT-centric" (Lookup Table) architecture that replaces traditional multipliers and large adder trees with specialized logic:

Overview of the TENET block architecture featuring Ternary Linear layers.
  1. Sparse Ternary LUT (STL) Core: Instead of standard arithmetic, it uses a "mirror-halved" precompute table. Because ternary weights are symmetric ($\{-1, 0, 1\}$), the hardware only needs to store half the possible results, significantly saving area and power.
  2. Dynamic Activation N:M Sparsity (DAS): The system identifies and skips "unimportant" activations on-the-fly using a TopK approach. This reduces the computational load by roughly 50% with minimal impact on model accuracy.
  3. Ternary Weight Decompression (TWD): Theoretically, a ternary value needs only 1.58 bits. Standard hardware often rounds this up to 2 bits (INT2), wasting space. TENET uses a 64B:80B packing scheme (representing 5 weights in 8 bits) to achieve 1.6 bits per weight, nearly hitting the theoretical limit.
  4. LPSA Dataflow: The "Linear-Projection-aware Sparse Attention" dataflow fuses the QKV (Query, Key, Value) projections with the attention mechanism. This allows the hardware to reuse data on-chip, cutting down on expensive trips to external DRAM.
  5. Heterogeneous Design: The chip combines these specialized STL cores for ternary math with high-precision cores for tasks that can't be quantized (like Softmax or LayerNorm).

Key Insights and Evaluation

The researchers evaluated TENET using several prominent LLM architectures (BitNet-1.3B, BitNet-3B, Llama-7B) across seven NLP benchmarks. A central concept introduced is the Intelligence Per Joule (IPJ) metric. Instead of just looking at "tokens per second," IPJ measures how much "correctness" (1/perplexity) a model generates per unit of energy. This highlights why ternary models are superior for edge use.

Key Results

  • Energy Efficiency: Compared to an NVIDIA A100 GPU, the TENET-ASIC is 21.1x more energy-efficient. Even the FPGA prototype is 4.3x more efficient.
  • Speedup: TENET-ASIC achieves a 2.7x average speedup in end-to-end latency over an optimized A100 GPU and a staggering 27.9x speedup over a high-end Intel CPU running standard ternary software (bitnet.cpp).
  • Memory Savings: The combination of DAS and LPSA reduced end-to-end memory access by 80.3%.
  • Accuracy Trade-off: Enabling 50% sparsity (DAS) only increased model perplexity by a tiny margin (e.g., from 11.27 to 11.32 on BitNet-1.3B), making it a highly viable trade-off for massive speed gains.

Conclusion

TENET demonstrates that co-optimizing hardware with ternary quantization and dynamic sparsity can unlock massive efficiency gains for edge LLM deployment. While the benefits of dynamic sparsity currently rely on models being fine-tuned with specific masks, the architecture's LUT-centric approach and 1.6-bit weight decompression set a new standard for low-bit accelerators. Future work will explore scaling this approach to larger batch sizes and emerging low-bit formats.

Summary updated

Abstract

Ternary quantization has emerged as a powerful technique for reducing both computational and memory footprint of large language models (LLM), enabling efficient real-time inference deployment without significantly compromising model accuracy. Conventional LLM inference platforms (e.g GPUs) cannot capitalize on its benefits, as they (i) lack native support for ternary arithmetic and memory specialization and (ii) remain severely under-utilized in low-batch, real-time scenarios. In this work, we propose TENET, a sparse-aware LUT-centric architecture that co-optimizes algorithm, compute, and memory for ternary LLM inference. To maximize the efficiency of Ternary Linear layer, TENET introduces a Sparse Ternary LUT (STL) core that optimizes ternary mixed-precision GEMM using a symmetric precompute lookup table. It also features Dynamic Activation N:M Sparsity to exploit the sparsity within the activation of each token. Additionally, we propose a LUT-based 64B:80B ternary weight decompression module to fully exploit the memory efficiency of ternary values. At the system level, we design a heterogeneous TENET accelerator with full programmability that integrates STL cores with high-precision cores. An associated Linear-Projection-aware Sparse Attention dataflow is introduced to optimize memory access and hardware utilization. We implement TENET accelerator prototype on both FPGA and ASIC platforms. Experiments across various model sizes and workloads demonstrate that TENET-FPGA and TENET-ASIC improve energy efficiency by 4.3$\times$ and 21.1$\times$, respectively, compared to the A100 GPU. Furthermore, TENET-ASIC achieves a 2.7$\times$ average speedup compared to the A100 GPU in end-to-end inference latency.
BibTeX
@misc{huang2025tenet,
  author = {Zhirui Huang and Rui Ma and Shijie Cao and Ran Shu and Ian Wang and Ting Cao and Chixiao Chen and Yongqiang Xiong},
  title = {{TENET: An Efficient Sparsity-Aware LUT-Centric Architecture for Ternary LLM Inference On Edge}},
  year = {2025},
  eprint = {2509.13765},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.13765},
}
Figure