Abstract
In security-sensitive fields, data should be encrypted to protect against unauthorized access and maintain confidentiality throughout processing. However, traditional networks like ViTs and CNNs return different results when processing original data versus its encrypted form, meaning that they require data to be decrypted, posing a security risk by exposing sensitive information. One solution for this issue is using polynomial networks, including state-of-the-art Multilinear Operator Networks, which return the same outputs given the real data and their encrypted forms under Leveled Fully Homomorphic Encryption. Nevertheless, these models are susceptible to catastrophic forgetting in incremental learning settings. Thus, this paper will present a new low-rank adaptation method combined with the Gradient Projection Memory mechanism to minimize the issue. Our proposal is compatible with Leveled Fully Homomorphic Encryption while achieving a sharp improvement in performance compared to existing models.
Introduction
In domains such as healthcare and finance, data privacy is paramount. Leveled Fully Homomorphic Encryption (LFHE) provides a pathway for secure computation by allowing operations directly on encrypted data. However, LFHE is restricted to addition and multiplication, rendering standard neural architectures (e.g., CNNs, ViTs) that rely on non-linear activations like ReLU incompatible. While Multilinear Operator Networks (MONet) offer a polynomial-based alternative that adheres to LFHE constraints, they struggle with catastrophic forgetting during incremental learning—the tendency to lose previously acquired knowledge when learning new tasks.
This paper introduces IPoLoRA (Incremental Polynomial Low-Rank Adaptation), a framework designed to enable MONets to learn tasks sequentially without compromising security or forgetting old information.
The IPoLoRA Framework
The IPoLoRA approach modifies the underlying MONet architecture to support efficient, security-preserving incremental updates through three core components.
MONet Architecture Overview
The base MONet architecture is built hierarchically, utilizing "Mu-Layers" grouped into "Poly-Blocks" and "Basic Blocks." This structure allows the network to capture complex multilinear relationships using only polynomial operations, ensuring compatibility with the mathematical constraints of LFHE.
Polynomial Low-Rank Adaptation (PoLoRA)
Inspired by Low-Rank Adaptation (LoRA) in large language models, the authors propose PoLoRA. This method introduces lightweight, trainable low-rank matrices ($A$ and $B$) into the network's channel projections. By keeping the original weights frozen and only training these small matrices, the model can adapt to new tasks with minimal parameter overhead. Crucially, PoLoRA uses only multiplication and addition, maintaining full compatibility with LFHE.
Two variants are explored: PoLoRA-C (integrated directly into channel projections) and PoLoRA-M (a more parameter-efficient version integrated within the Mu-Layer itself).
Gradient Projection Memory (GPM) and Branching
To combat forgetting, the down-projection matrix ($A$) is initialized using Gradient Projection Memory (GPM). By ensuring the row space of $A$ is orthogonal to the gradients of all previous tasks, updates for new tasks occur in the "null space" of prior knowledge, preventing interference. During inference, a task-agnostic branching strategy concatenates these learned PoLoRA branches, allowing the model to perform a single forward pass without requiring a task ID.
Evaluation and Results
The researchers evaluated IPoLoRA on three standard benchmarks: CIFAR-100, ImageNet-R, and PACS.
Key Performance Metrics
- Accuracy Gains: IPoLoRA-C and IPoLoRA-M outperformed standard baselines by at least 17% in average accuracy. On the ImageNet-R 10-task benchmark, IPoLoRA-C achieved nearly 22% higher accuracy than simple classifier fine-tuning.
- Efficiency: The method is highly efficient; for instance, PoLoRA-M adds as little as 0.61% additional parameters on CIFAR-100 while maintaining high performance.
- Task-Agnostic Inference: The branching strategy proved robust, maintaining over 94% accuracy in certain settings even without task-specific identifiers at test time.
Conclusion
IPoLoRA successfully addresses the challenge of incremental learning in security-preserving environments. By leveraging low-rank adaptation and orthogonal gradient projections, it effectively mitigates catastrophic forgetting in polynomial networks. While the method shows sensitivity to the GPM scaling factor and faces potential branch growth challenges as tasks scale, it represents a significant step forward for secure, lifelong machine learning.
Summary updated