Leveraging ASIC AI Chips for Homomorphic Encryption

Jianming Tong, Tianhao Huang, Leo de Castro, Anirudh Itagi, Jingtian Dang, Anupam Golder, Asra Ali, Jevin Jiang, Arvind, G. Edward Suh, Tushar Krishna

Abstract

Cloud-based services are making the outsourcing of sensitive client data increasingly common. Although homomorphic encryption (HE) offers strong privacy guarantee, it requires substantially more resources than computing on plaintext, often leading to unacceptably large latencies in getting the results. HE accelerators have emerged to mitigate this latency issue, but with the high cost of ASICs. In this paper we show that HE primitives can be converted to AI operators and accelerated on existing ASIC AI accelerators, like TPUs, which are already widely deployed in the cloud. Adapting such accelerators for HE requires (1) supporting modular multiplication, (2) high-precision arithmetic in software, and (3) efficient mapping on matrix engines. We introduce the CROSS compiler (1) to adopt Barrett reduction to provide modular reduction support using multiplier and adder, (2) Basis Aligned Transformation (BAT) to convert high-precision multiplication as low-precision matrix-vector multiplication, (3) Matrix Aligned Transformation (MAT) to covert vectorized modular operation with reduction into matrix multiplication that can be efficiently processed on 2D spatial matrix engine. Our evaluation of CROSS on a Google TPUv4 demonstrates significant performance improvements, with up to 161x and 5x speedup compared to the previous work on many-core CPUs and V100.

Introduction: The Privacy "Tax"

Homomorphic Encryption (HE) is often hailed as the "holy grail" of data privacy, enabling servers to process encrypted data without ever seeing the underlying information. However, this privacy comes with a massive computational "tax"—processing encrypted data is typically 1000× slower than plaintext. While custom HE ASICs have been proposed, they are expensive and lack the widespread availability of cloud-based AI accelerators.

System overview of CROSS enabling HE-based privacy-preserving AI on untrusted cloud accelerators.

AI accelerators like Google’s TPUs are highly efficient and already deployed at scale, but they face a fundamental architectural mismatch when running HE:

  1. Arithmetic Mismatch: HE requires high-precision (e.g., 32-bit) modular integer math, whereas TPUs are optimized for low-precision (8-bit) matrix multiplication. Running HE on a TPU usually leaves its most powerful component, the Matrix Unit (MXU), idle.
  2. Memory Mismatch: HE algorithms like the Number Theoretic Transform (NTT) require fine-grained data shuffling, which clashes with the coarse-grained, "blocky" memory movement design of TPUs.

The CROSS Framework

The CROSS compiler bridges this gap by mathematically transforming HE primitives into operators that "look like" AI workloads.

Comparison of GPU vs. TPU architectures and the transformations (BAT and MAT) used by CROSS to resolve mismatches.

Basis-Aligned Transformation (BAT)

BAT addresses the arithmetic mismatch through a mathematical "folding" technique. It decomposes high-precision 32-bit modular multiplications into a series of 8-bit operations. By pre-calculating modular reductions for known parameters (like twiddle factors) and folding them into a dense 8-bit matrix, BAT allows the TPU’s high-throughput MXU to perform the heavy lifting that was previously restricted to slower vector units.

Matrix-Aligned Transformation (MAT)

MAT solves the memory mismatch by moving data shuffling from runtime to compile-time. Since data movement patterns in HE kernels are deterministic, CROSS embeds required transposes and permutations directly into the parameter matrices. This creates "layout-invariant" kernels that produce correctly ordered outputs without costly runtime shuffle instructions.

Key Results

The framework was evaluated using standard privacy-preserving machine learning (PPML) workloads, including MNIST CNN inference and Logistic Regression.

  • Throughput: CROSS on a Google TPU v6e achieved a 1.43× improvement in NTT throughput over the previous state-of-the-art (WarpDrive on NVIDIA A100).
  • Energy Efficiency: CROSS demonstrated significantly higher throughput per watt compared to existing platforms: 451× over CPUs (OpenFHE) and 7.81× over GPUs (WarpDrive).
  • Real-world Performance: For MNIST inference, CROSS achieved a 10× speedup over the Orion framework while maintaining 98% accuracy.

Conclusion and Future Work

CROSS demonstrates that with the right software stack, commodity AI hardware can effectively compete with custom HE ASICs, narrowing the gap to within 3–33×. Future work includes "fusing" sequential operators to reduce HBM write-backs and developing better support for complex automorphisms (slot-wise rotations) that currently result in random memory access patterns.

The kernel-level codes are open-sourced at https://github.com/google/jaxite.git.

Summary updated

Abstract

Cloud-based services are making the outsourcing of sensitive client data increasingly common. Although homomorphic encryption (HE) offers strong privacy guarantee, it requires substantially more resources than computing on plaintext, often leading to unacceptably large latencies in getting the results. HE accelerators have emerged to mitigate this latency issue, but with the high cost of ASICs. In this paper we show that HE primitives can be converted to AI operators and accelerated on existing ASIC AI accelerators, like TPUs, which are already widely deployed in the cloud. Adapting such accelerators for HE requires (1) supporting modular multiplication, (2) high-precision arithmetic in software, and (3) efficient mapping on matrix engines. We introduce the CROSS compiler (1) to adopt Barrett reduction to provide modular reduction support using multiplier and adder, (2) Basis Aligned Transformation (BAT) to convert high-precision multiplication as low-precision matrix-vector multiplication, (3) Matrix Aligned Transformation (MAT) to covert vectorized modular operation with reduction into matrix multiplication that can be efficiently processed on 2D spatial matrix engine. Our evaluation of CROSS on a Google TPUv4 demonstrates significant performance improvements, with up to 161x and 5x speedup compared to the previous work on many-core CPUs and V100. The kernel-level codes are open-sourced at https://github.com/google/jaxite.git.
BibTeX
@inproceedings{tong2026leveraging,
  author = {Jianming Tong and Tianhao Huang and Leo de Castro and Anirudh Itagi and Jingtian Dang and Anupam Golder and Asra Ali and Jevin Jiang and Arvind and G. Edward Suh and Tushar Krishna},
  title = {{Leveraging ASIC AI Chips for Homomorphic Encryption}},
  booktitle = {HPCA},
  year = {2026},
  doi = {10.1109/hpca68181.2026.11408507},
}
Figure