Privacy-preserving machine learning (PPML) allows a server to perform inference on a user's encrypted data without ever seeing the underlying information. However, this comes with a massive "tax." In Homomorphic Encryption (HE), every multiplication increases the noise within a ciphertext. If the noise grows too large, the data becomes undecryptable. To prevent this, researchers must either use expensive "bootstrapping" operations to reset the noise or limit the "multiplicative depth" of the neural network.
The emerging solution across recent research is Operator Fusion: mathematically collapsing multiple neural network layers into a single computational step. By treating the network not as a sequence of discrete layers but as a unified mathematical function, researchers can drastically reduce the depth and communication overhead of private inference.
Collapsing the Linear Chain
In a standard neural network, a convolution is often followed by batch normalization and a pooling layer. In the plaintext world, these are three distinct steps. In the encrypted world, Ran Gilad-Bachrach, Kristin Lauter, and their colleagues demonstrated in CryptoNets that these linear operations can be mathematically merged. Since convolutions, mean pooling, and batch normalization are all linear transformations, they can be pre-computed into a single weight matrix.
This fusion is critical because it reduces the "multiplicative depth"—the number of sequential multiplications—of the circuit. Francisco Vargas and Stephen Cummins showed in Efficient Representations for Privacy-Preserving Inference that integrating these steps into a single homomorphic operation minimizes noise accumulation, allowing for faster execution on simpler hardware. Similarly, Eunsang Lee and Joon-Woo Lee applied this to deep ResNets in Low-Complexity Deep Convolutional Neural Networks, fusing convolution and batch normalization to delay the need for expensive bootstrapping.
Fusing Non-Linearities via Polynomials and Lookups
The real challenge for fusion is the non-linear activation function (like ReLU). Standard HE cannot compute non-linearities directly; they must be approximated by polynomials.
Recent work has moved toward "fine-grained operator integration," where these polynomial activations are fused directly into the preceding linear layers. In LinGCN, Hongwu Peng and Ran Ran fused polynomial activations and normalization layers into convolutions, achieving a 14.2× speedup by eliminating the boundaries between these operations. Qian Lou and Lei Jiang took this further in HEMET by using a greedy search to merge mathematical coefficients from sequential operations into a single set of constants, reducing latency by 60%.
For quantized models, Kui Ren and Bingsheng Zhang proposed a different paradigm in A New PPML Paradigm for Quantized Models. Instead of arithmetic circuits, they treat fused layers (Conv + Activation + Pooling) as a single private lookup table. This allows a complex sequence of layers to be evaluated for the communication cost of a single query.
Maintaining Data Layouts in Transformers
In complex architectures like Transformers, the bottleneck isn't just the math—it's the data movement. Between layers, ciphertexts often need to be "reshuffled" or rotated to match the input format of the next operation.
Linhan Yang and Wangchen Dai addressed this in ARION by fusing linear and non-linear operations into a continuous sequence. This fusion ensures a consistent "data packing" format throughout the Transformer block, eliminating the need for costly ciphertext rotations.
Similarly, Nandan Kumar Jha and Brandon Reagen demonstrated in their work on Private Transformer Inference that removing non-linearities entirely in certain blocks allows for the fusion of consecutive linear layers. This "structural linearization" trades a small amount of model expressivity for a massive reduction in the bootstrapping operations that typically cripple FHE performance.
Why Fusion Works: The Intuition
The intuition behind operator fusion in PPML is twofold:
- Noise Management: In HE, every "layer boundary" is a potential source of noise growth. By merging layers, we reduce the number of sequential multiplications, keeping the noise low enough to avoid the "reloading" cost of bootstrapping.
- Communication Efficiency: In Secure Multi-Party Computation (MPC), every non-linear operation requires a round of communication between parties. Fusing operations—as seen in Orca by Neha Jawalkar and Rahul Sharma—reduces the number of keys and messages that must be exchanged, turning a "chatty" protocol into a streamlined one.
By treating the neural network as a single, compressible mathematical expression rather than a series of isolated steps, these researchers are making private inference move from theoretical possibility to practical reality.