Abstract
Homomorphic Encryption (HE) enables secure computation on encrypted data but suffers from significant latency overhead, particularly due to the "bootstrapping" operations required by standard skip connections in deep neural networks. This paper presents a novel architectural approach to realize skip connections efficiently for HE. By combining Dirac parameterization for short-term dependencies and shared-source skip connections for long-term dependencies, we reduce the number of expensive cryptographic operations. Our variant achieves a 1.3x speedup in CPU time on a ResNet50 architecture using the CIFAR-10 dataset while maintaining high accuracy (91.08%), demonstrating a practical path toward scalable secure inference.
Introduction
Homomorphic Encryption (HE) is often considered the "holy grail" of privacy-preserving machine learning. It allows a cloud server to perform computations, such as image classification, on data while it remains encrypted, ensuring the server never accesses raw information. However, HE is notoriously slow—often orders of magnitude slower than computing on unencrypted "cleartext" data.
A major bottleneck in modern Deep Neural Networks (DNNs), like ResNet, is the skip connection. In standard deep learning, these connections (which add the input of a layer to its output) are computationally "free" and essential for training deep models. In the context of HE, they are incredibly expensive. HE ciphertexts track their noise levels via a "multiplication chain index" (CIdx). When adding two encrypted values with different indices—a frequent occurrence with skip connections—the system must perform a "Bootstrap" operation. Bootstrapping is the most computationally taxing part of HE; reducing these operations is critical to making encrypted AI practical.
Methodology
To address these bottlenecks, we modified a standard ResNet50 to be "HE-friendly." This involved replacing non-polynomial layers (such as MaxPool and ReLU) with polynomial approximations (AVGPooling and polynomial activations), as HE natively supports only additions and multiplications. We then introduced two key architectural innovations to handle skip connections efficiently:
- Dirac Parameterization: Instead of using physical skip connections for short-term dependencies, we utilized Dirac parameterization. This mathematical technique parameterizes convolutional layers to behave like identity connections during training. During the inference phase, these connections effectively "collapse" into the weights of the convolutional layers, providing ResNet-like stability without any HE overhead.
- Shared-Source Skip Connections: Rather than connecting every block to its immediate successor, we implemented "long-term" connections originating from the network's initial layers. Because the first layer has a very low depth (low CIdx), adding it to later, deeper layers is significantly cheaper in terms of cryptographic noise management compared to adding two deeply processed ciphertexts.
Experimental Evaluation
We evaluated our approach using the CIFAR-10 dataset, a standard computer vision benchmark consisting of 60,000 images across 10 classes. We compared our modified "variant" against a standard reference HE-friendly ResNet50, a version with Dirac parameters but no skip connections, and a version with no skip connections at all.
The core advantage of our design is the "freshness" of the initial input. By reusing the same cryptographically "fresh" source for multiple skip connections, we avoid the compounding complexity that occurs when adding two noisy, deeply processed ciphertexts. Furthermore, the latency and bootstrapping requirements in this architecture become largely independent of the number of connections, facilitating better scaling for larger models.
Results and Discussion
The experimental results demonstrate that our variant achieves the speed of a simplified network while retaining the accuracy of a complex one.
- Computational Efficiency: We achieved a 1.3x improvement in overall CPU computing power compared to the reference HE-friendly ResNet50.
- Bootstrap Reduction: The number of costly bootstrap operations was reduced from 2,568 in the reference model to 1,888 in our variant (a 1.36x reduction).
- Accuracy Preservation: Our variant maintained a test accuracy of 91.08%, nearly identical to the reference model's 91.46%, and significantly higher than the 88.68% achieved by simply removing skip connections.
Conclusion
This research demonstrates that architectural awareness of HE constraints can lead to significant performance gains without sacrificing model performance. While the results on CIFAR-10 are promising, future work will involve testing on larger datasets like ImageNet to verify scalability. The use of shared-source connections and Dirac parameterization paves the way for "HE-aware" designs that could eventually make encrypted cloud inference a standard industry practice.
Summary updated