Introduction
The transition of Large Language Models (LLMs) from centralized cloud environments to edge devices, such as smartphones, introduces significant security vulnerabilities. Unlike cloud-based APIs that shield model weights, edge deployment exposes these weights directly on the user's hardware. An attacker with physical or root access can extract the model's architecture and parameters, enabling "foundational capability stealing"—a process where a partially recovered model is fine-tuned to replicate proprietary performance at a fraction of the original training cost.
Existing defenses, primarily based on Trusted Execution Environments (TEEs), face a severe efficiency-security trade-off. Running full models within a TEE can result in a 50x slowdown, while hybrid TEE-GPU approaches often suffer from prohibitive communication overhead due to frequent data shuffling.
The CoreGuard Framework
CoreGuard addresses these challenges through a novel "lock-and-key" mechanism that combines weight permutations with a propagation protocol. The framework consists of three main components:
- Model Locking: Prior to deployment, CoreGuard "locks" the LLM by applying row permutations to the weight matrices of linear layers. This transformation renders the model mathematically dysfunctional in its raw state.
- Propagation Protocol: To avoid the bottleneck of constant TEE interaction, CoreGuard allows the "authorized" state to propagate through the network. By permuting layer outputs to match the requirements of subsequent layers, the model effectively re-authorizes itself during GPU execution.
- Lightweight Authorization: The TEE is utilized only once per inference for initial authorization. It employs a One-Time Pad (OTP) and positional obfuscation to protect the permutation matrix ($\pi$) from reverse-engineering.
Experimental Evaluation
The efficacy of CoreGuard was evaluated across four diverse tasks: mathematical reasoning (GSM8k), code generation (Spider), medical question answering (PubMedQA), and general reading comprehension (SQuAD). These tasks were tested on four representative LLMs: Qwen2-0.5B, Gemma2-2B, ChatGLM3-6B, and LLaMA3-8B.
Key Results and Security Analysis
CoreGuard demonstrates near-perfect security with negligible performance impact. Key findings include:
- Minimal Overhead: The execution overhead is less than 0.00117% across all models, outperforming previous partial-parameter defenses by orders of magnitude.
- Robust Security: In model-stealing simulations, CoreGuard reduces attacker accuracy to levels comparable to a "Black-Box" scenario, where the attacker has no weight access.
- Mathematical Hardness: The security of the permutation matrix is grounded in the NP-hardness of the Matrix-Learning With Errors (Matrix-LWE) problem.
- Zero Accuracy Loss: Once authorized, the locked model performs identically to the original, with negligible fluctuations due to standard floating-point rounding.
Conclusion
CoreGuard provides a highly efficient and secure solution for deploying proprietary LLMs on edge devices. By leveraging the "middle-out" strategy—placing TEE authorization in the middle of the model—it forces attackers to recover a significant portion of the model to regain functionality. While future work may address side-channel attacks and GPU-based TEE optimizations, CoreGuard currently offers a practical path for safeguarding foundational AI capabilities in the wild.
Summary updated