RAPTOR: Role-Aware Private Training for Mixture-of-Experts

Duc Dm, Khai Le-Duc, Nguyen Do, Minh Son Hoang, Florent Draye, Thai Hoang, Hoang Phuong Dam, Jiarui Liu, Chris Ngo, Terry Jingchen Zhang, Anh Le Duc Tran, Nhat Do Minh, Minh Ngoc Le, My T. Thai, Ran Xu, Silvio Savarese, Mona Diab, Bernhard Schölkopf, Zhijing Jin, Huy L. Nguyen, Daeyoung KimarXiv 2026 · arXiv

TL;DR

Standard differential privacy (DP) algorithms degrade the performance of sparse Mixture-of-Experts (MoE) models because they treat the entire network as a single dense block. This paper introduces RAPTOR (Role-Aware Private Training), a framework that treats shared layers and sparse experts differently during optimization. By leveraging parallel privacy composition across expert groups, RAPTOR updates all experts at the privacy cost of updating just one, yielding substantial accuracy gains under strict privacy budgets.

What Problem Does This Paper Address?

Sparse MoE architectures scale model capacity efficiently by routing data tokens to a subset of specialized "experts" while passing all data through shared dense layers. However, when enforcing Differential Privacy (DP) via standard methods like DP-SGD or DP-Adam, this structural asymmetry is completely ignored.

Monolithic DP training applies a single clipping bound, noise scale, and normalization denominator across the entire model. The authors identify three severe structural failures resulting from this mismatch:

  1. Clipping Interference: Gradients from dense blocks dominate the total gradient norm, causing the sparse expert gradients to be aggressively suppressed during global clipping.
  2. Update-Scale Dilution: Normalizing expert gradients by the full batch size (instead of the number of tokens actually routed to that expert) dilutes sparse updates toward zero.
  3. Load-Dependent Signal Degradation: Because privacy noise is fixed across all experts, low-load experts (those receiving fewer tokens) suffer from a critically low signal-to-noise ratio (SNR).
Figure 1: Applying differential privacy to sparse MoE is not straightforward: treating the model as dense leads to three structural failures - clipping interference, update-scale dilution, and load-dependent signal degradation. Our role-aware method is designed to address these mismatches.

Methods

RAPTOR redesigns the DP mechanism to align with the shared/expert structure of MoEs using a two-stream alternating optimization framework:

  • Separate Clipping and Noise: The shared stream updates dense components using standard full-batch clipping, while the expert stream applies expert-specific clipping bounds ($C_e$) and noise scales ($\sigma_e$).
  • Public Expected-Owner Denominator: To avoid leaking private routing decisions, RAPTOR normalizes expert updates using a data-independent public denominator ($D_e = B/E$, where $B$ is batch size and $E$ is the number of experts) rather than the private, realized token counts.
  • Parallel Composition via Expert Ownership: RAPTOR deterministically assigns each record to a single "owner expert" based on gate-weighted plurality. Because the data records are partitioned into disjoint groups, the privacy costs of updating all experts within a layer compose in parallel rather than sequentially.
  • Entropy-Based Layer Selection: RAPTOR uses a small, non-private public corpus to measure routing entropy across candidate layers, selecting the most balanced layer to privately update.
Overview of RAPTOR framework

Datasets

The authors evaluated RAPTOR across several models and benchmarks:

  • Models: Google's Switch Transformer (Switch-Base-8), OLMoE-1B-7B, and the vision-language model DeepSeek-VL2-Tiny.
  • Evaluation Tasks: Four GLUE tasks (SST-2, MNLI, QNLI, QQP) for text, and ScienceQA for multimodal evaluation.
  • Public Calibration Corpora: IMDb reviews (SST-2), ANLI R2 pairs (MNLI), SQuAD contexts (QNLI), and PAWS-Wiki sentence pairs (QQP).

Key Insights and Ideas

  • The Parallel Composition Loophole: The most elegant insight of the paper is that updating all experts within a layer costs no more privacy budget than updating a single expert, provided the data partitioning is deterministic and data-independent. This completely bypasses the standard sequential composition penalty that degrades dense models.
  • Router Freezing is Key: By freezing the pretrained router prior to DP training, the framework stabilizes the data partitions. This allows the model to maintain expert specialization without requiring complex, data-dependent partition accounting.
  • Data-Independent Normalization: Using the expected count ($B/E$) keeps the mechanism entirely privacy-safe while remaining unbiased under uniform routing conditions.

Key Results

RAPTOR consistently outperformed monolithic DP baselines across all tested privacy budgets ($\varepsilon \in \{1, 4, 8\}$), with the most pronounced gains occurring at the tightest privacy budgets:

  • Switch Transformer: RAPTOR achieved accuracy improvements ranging from +0.23 to +3.24 points over Monolithic DP-Adam LoRA across the GLUE tasks.
  • OLMoE-1B-7B: On this highly sparse 64-expert model, RAPTOR demonstrated massive gains under high-noise regimes ($\varepsilon = 1$), outperforming the baseline by up to +4.50 points on QNLI.
  • Computational Efficiency: RAPTOR actually reduced peak memory usage compared to standard LoRA (e.g., 112.16 GB vs. 130.74 GB on a B200 GPU) because the expert-stream clipping operates on smaller, partitioned subsets of data rather than the full batch simultaneously.

Limitations and Future Work

  • Frozen Router: Fixing the expert ownership maps requires freezing the router parameters. This prevents the routing mechanism from adapting or optimizing alongside the expert weights during private fine-tuning.
  • Suboptimal Under Heavy Imbalance: While the public expected denominator ($B/E$) is privacy-safe, it can over- or under-scale expert updates if the routing distribution is highly skewed. Developing adaptive, private normalization techniques is left to future work.
  • Outlier Amplification: RAPTOR's residual objective up-weights low-confidence records. While mathematically sound, this risks amplifying the influence of mislabeled data or extreme outliers within small expert groups.

Summary updated

Abstract

Differentially private (DP) fine-tuning methods treat sparse Mixture-of-Experts (MoE) models as a single dense block, ignoring that shared layers see all data while experts only see routed records. We identify and formally characterize three resulting failure modes: global clipping suppresses expert gradients, batch-level normalization dilutes sparse expert updates, and fixed privacy noise degrades signal-to-noise ratio on low-load experts. We introduce RAPTOR - a Role-Aware Private Training framework, which alternates shared and expert optimization and targets each failure directly, using expert-specific clipping and noise together with a public expected-owner denominator and a count-independent update schedule that avoids conditioning on private, realized expert counts. We prove the resulting mechanism satisfies $(\varepsilon,δ)$-DP: because each record is assigned to exactly one owner expert, per-expert mechanisms within a layer compose in parallel, so updating all $E$ experts costs no more, in privacy terms, than updating one, with shared and expert streams composing sequentially across training. We further derive a bias-variance decomposition of the public-denominator estimator showing its bias grows predictably with routing imbalance, yielding a privacy-free rule for selecting which layer to protect from routing entropy measured on a small public corpus. Experiments on Switch Transformer and OLMoE fine-tuning across GLUE tasks, and on DeepSeek-VL2-Tiny, show consistent gains over standard DP baselines across several privacy levels ($\varepsilon$), with the largest margins typically at the tightest budgets. Code and models are publicly available: https://github.com/leduckhai/RAPTOR
BibTeX
@misc{dm2026raptor,
  author = {Duc Dm and Khai Le-Duc and Nguyen Do and Minh Son Hoang and Florent Draye and Thai Hoang and Hoang Phuong Dam and Jiarui Liu and Chris Ngo and Terry Jingchen Zhang and Anh Le Duc Tran and Nhat Do Minh and Minh Ngoc Le and My T. Thai and Ran Xu and Silvio Savarese and Mona Diab and Bernhard Schölkopf and Zhijing Jin and Huy L. Nguyen and Daeyoung Kim},
  title = {{RAPTOR: Role-Aware Private Training for Mixture-of-Experts}},
  year = {2026},
  eprint = {2609.05770},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2609.05770},
}
Figure