Architecture Protocol Co-Design for Secure Neural Inference

18 concepts13 papers1 proposal

Private Inference (PI) represents a major paradigm shift in secure computation: it allows a client to send encrypted data to a server hosting a machine learning model, obtaining an inference result without revealing their sensitive data to the server, and without the server revealing its proprietary model weights to the client. This is typically achieved using cryptographic primitives like Homomorphic Encryption (HE) and Secure Multi-Party Computation (SMPC), such as Garbled Circuits (GC).

However, deploying standard deep learning models directly into these cryptographic protocols results in prohibitive computational and communication overhead. As noted in an end-to-end characterization study, traditional neural network architectures are fundamentally incompatible with efficient secure computation because they contain a high density of operations that trigger expensive cryptographic evaluations (Characterizing and Optimizing End-to-End Systems for Private Inference).

To make private inference practical, researchers have turned to architecture-protocol co-design: rethinking the structural components of neural networks so that they natively align with the mathematical strengths and constraints of cryptographic protocols.

The Cryptographic Cost Inversion

In plaintext execution on standard hardware (CPUs and GPUs), non-linear operations like the Rectified Linear Unit (ReLU) activation function ($f(x) = \max(0, x)$) or Max Pooling are virtually free, requiring only a simple comparison or bit-shift. Conversely, high-dimensional linear convolutions and matrix multiplications consume the vast majority of the compute budget.

When executing under cryptographic constraints, this cost structure is completely inverted:

  • Linear Operations are "Cheap": In Homomorphic Encryption schemes, adding two encrypted tensors or multiplying an encrypted tensor by unencrypted model weights can be performed directly on the ciphertext without interactive communication.
  • Non-Linear Comparisons are "Expensive": Computing a step function like ReLU or finding the maximum value in a pooling window requires evaluating complex boolean circuits via Garbled Circuits or computing high-degree polynomial approximations. This introduces massive communication latency and bandwidth bottlenecks between the client and the server.

As Bo Feng et al. observe, standard neural network designs prioritize mathematical traditions that are fundamentally incompatible with efficient cryptographic execution (CRYPTOGRU: Low Latency Privacy-Preserving Text Analysis With GRU). Minimizing these non-linear hurdles requires replacing traditional model layers with cryptographically friendly alternatives.

Rethinking the Building Blocks: Activations and Pooling

The most direct way to optimize an architecture for cryptography is to swap out comparison-based operations for smooth, low-order polynomial functions that can be computed using simple homomorphic multiplications.

Replacing ReLU with Polynomials

Instead of forcing a cryptographic protocol to evaluate the sharp bend of a ReLU function, co-designed networks use low-degree polynomials. For instance, Jiajie Li et al. introduced xMLP, an architecture that replaces ReLU with an exclusive square activation ($x^2$), demonstrating that rethinking the model structure to avoid "information compounding" can yield massive speedups on GPUs while maintaining high accuracy (xMLP: Revolutionizing Private Inference with Exclusive Square Activation).

Similarly, Hongwu Peng et al. utilized trainable second-order polynomials within their PASNet framework to bypass the need for costly secure comparisons (PASNet: Polynomial Architecture Search Framework for Two-party Computation-based Secure Neural Network Deployment), while Jaiyoung Park et al. leveraged a specialized quadratic activation function called AESPA to accelerate ResNet architectures (Unlocking Private Computation at Scale: The Acceleration of Homomorphic Encryption).

Swapping Max Pooling for Average Pooling

In visual models like UNET or CNNs, downsampling is traditionally performed via Max Pooling. However, finding the maximum value among a set of encrypted numbers requires a cascade of secure comparisons. In BUNET, Song Bian 0001 et al. demonstrate that replacing max pooling with average pooling completely eliminates the need for secure comparisons (BUNET: Blind Medical Image Segmentation Based on Secure UNET). Because average pooling relies purely on additions and scalar divisions, it can be computed entirely within the linear domain of homomorphic encryption, drastically reducing latency.

While manually swapping layers can provide immediate relief, finding the optimal balance between cryptographic latency and model accuracy is highly complex. Early efforts relied on manual tuning of layer parameters like kernel sizes and strides to satisfy cryptographic constraints (Francisco Vargas et al., Efficient Representations for Privacy-Preserving Inference). To scale this process, researchers have turned to automated Neural Architecture Search (NAS).

By treating cryptographic protocols as first-class design constraints, NAS frameworks can navigate the non-linear cost structures of secure inference. For example, Song Bian 0001 et al. developed NASS, a framework that performs a simultaneous search over neural network topology, bit-width quantization, and cryptographic settings (NASS: Optimizing Secure Inference via Neural Architecture Search). This shift from adapting cryptography to existing models to designing models for specific protocols ensures that the resulting topologies are inherently efficient for secure computation.

Standard NAS:       Topology Optimization ──> High Accuracy (Plaintext Optimized)
                                                                │
                                                       Wrapped in Crypto (Slow)
                                                                v
Co-Design NAS:      [Topology + Quantization + Crypto Parameters] ──> High Accuracy + Low Latency

As highlighted by Nathalie Baracaldo and Alina Oprea, these specialized architectural optimizations allow for the execution of private inference with significantly reduced latency by strategically minimizing the use of non-linear operations across the model topology (Machine Learning Security and Privacy). Furthermore, Qian Lou et al. demonstrated that deep reinforcement learning can be used to automatically select layer-wise homomorphic encryption parameters, exploiting the inherent error and noise tolerance of deep learning models to slash latency without sacrificing accuracy (AutoPrivacy: Automated Layer-wise Parameter Selection for Secure Neural Network Inference).

The Full-Stack View: Hardware, Compilers, and Crypto

Rethinking model topology is only one facet of the solution; true efficiency gains require a holistic stack that bridges the gap between machine learning frameworks, cryptographic schemes, and physical hardware.

┌─────────────────────────────────────────────────────────┐
│               Neural Network Architecture               │
│       (Polynomial Activations, Average Pooling)         │
├─────────────────────────────────────────────────────────┤
│                    Compiler Layer                       │
│     (Tiled Tensors, Parameter Selection, NTT Mapping)   │
├─────────────────────────────────────────────────────────┤
│               Cryptographic Protocol                    │
│             (Homomorphic Encryption, SMPC)              │
├─────────────────────────────────────────────────────────┤
│               Underlying Hardware Fabric                │
│             (Memory Hierarchies, FPGAs, GPUs)           │
└─────────────────────────────────────────────────────────┘

Explicitly modeling device architecture constraints during compilation enables massive speedups across neural network layers. Huili Chen et al. developed AHEC, an end-to-end compiler framework that models target device constraints like threads, registers, and memory hierarchies to guide the optimization of approximate homomorphic encryption schemes (AHEC: End-to-end Compiler Framework for Privacy-preserving Machine Learning Acceleration).

Other works focus on optimizing the mathematical representations within the hardware itself:

Ultimately, as emphasized by Jaiyoung Park et al., isolated optimizations at a single layer fail to resolve the compounding bottlenecks of encrypted computation; scaling private inference requires a unified alignment where the cryptographic scheme, the neural network architecture, and the underlying hardware memory layout work in tandem (Unlocking Private Computation at Scale: The Acceleration of Homomorphic Encryption).

Key Foundational Work

BUNET: Blind Medical Image Segmentation Based on Secure UNET (2020) — contributed 3 concept elements to this direction.
ENSEI: Efficient Secure Inference via Frequency-Domain Homomorphic Convolution for Privacy-Preserving Visual Recognition (2020) — contributed 3 concept elements to this direction.
NASS: Optimizing Secure Inference via Neural Architecture Search (2020) — contributed 3 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_cryptography_unfriendly_architecture_93b1ddec,
  title = {cryptography-unfriendly architecture},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/cryptography-unfriendly-architecture/rdr_93b1ddec602f47268671e5f675cd2ca5},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_cryptography_unfriendly_architecture_93b1ddec_v1,
  title = {cryptography-unfriendly architecture},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/cryptography-unfriendly-architecture/rdr_93b1ddec602f47268671e5f675cd2ca5/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake