MASCing: Configurable Mixture-of-Experts Behavior via Activation Steering Masks

Jona te Lintelo, Lichao Wu, Marina Krček, Sengim Karayalçin, Stjepan PicekarXiv 2026 · arXiv

TL;DR

MASCing (MoE Activation Steering Configuration) is a lightweight, training-free framework that dynamically reconfigures the safety behavior of Mixture-of-Experts (MoE) language models. By using an LSTM surrogate model to optimize sparse steering masks for routing gates, it can either tighten defenses against multi-turn jailbreaks or safely relax constraints for domain-specific compliance—all without altering model weights or adding inference latency.

What Problem Does This Paper Address?

Mixture-of-Experts (MoE) architectures have become the design of choice for scaling Large Language Models (LLMs), powering models from Mixtral to DeepSeek. By routing each token to a sparse subset of specialized "experts," MoEs drastically cut down inference costs. However, this sparse activation paradigm introduces a unique safety vulnerability: an adversary can manipulate prompts to steer tokens away from safety-aligned experts, bypassing guardrails.

When safety policies change or new jailbreak vectors emerge, developers traditionally face a hard choice: undergo expensive full-parameter fine-tuning or completely retrain the model. The AI safety community has lacked a lightweight, post-deployment mechanism to flexibly and rapidly manage MoE safety behaviors across diverse, dynamic deployment scenarios.

Methods

MASCing introduces a three-phase, training-free pipeline to safely manipulate MoE routing decisions:

  1. Sequential Modeling of Behavior: Instead of analyzing discrete top-$k$ expert selections, MASCing extracts continuous, unnormalized routing logits across all layers and tokens. It trains a lightweight LSTM surrogate model to map these sequential routing patterns to specific downstream behaviors (e.g., refusal vs. compliance).
  2. Steering Mask Creation: Using the trained LSTM as a differentiable proxy, MASCing optimizes a dense steering matrix to drive the model toward a target behavior. It then applies $L_1$ regularization and symmetric magnitude pruning to isolate a sparse, minimal subset of critical expert interventions.
  3. Steering Mask Application: At inference time, this static steering mask is injected directly into the MoE routing gates via pre-forward hooks. Crucially, it applies a soft continuous bias ($\alpha$) to the pre-routing logits rather than hard-coding expert activation, preserving the router's inherent context-awareness.

Datasets

The authors evaluated MASCing across seven open-source MoE models using logit datasets derived from several existing benchmarks:

  • Multi-Turn Jailbreak Defense: Built using AdvBench (to capture malicious instruction refusal) and the Multi-Turn Human Jailbreaks (MHJ) dataset (to capture successful multi-turn adversarial interactions).
  • Adult-Content Generation: Built using EroticaAnalysis (to capture over-refusal of adult requests) and Facebook's NaturalReasoning (to capture compliant, benign behavior).
  • Utility Benchmarks: MMLU and GSM8K were used to verify that steering does not degrade the model's general knowledge or multi-step reasoning capabilities.

Key Insights and Ideas

  • Logits Over Discrete Choices: Continuous routing logits contain rich, latent safety information. Prior defense methods like SteerMoE only look at final top-$k$ expert selections, creating a "blind spot" where an expert ranking just below the threshold ($k+1$) is completely ignored.
  • Soft Biasing vs. Hard Overrides: Soft-biasing logits allows the MoE router to maintain its proportional weighting and semantic nuance. Hard-coding expert selections (setting logits to $+\infty$) disrupts the internal representations, leading to incoherent outputs and a sharp drop in language utility.
  • Defense Complexity: Defending against multi-turn jailbreaks requires altering a much more complex, distributed reasoning pathway across the network depth compared to domain-specific policy relaxation (like adult content), which can be achieved with sparser steering.

Key Results

  • Jailbreak Mitigation: MASCing improved the average multi-turn jailbreak defense success rate from 52.5% to 83.9% across seven open-source MoE models, significantly outperforming the state-of-the-art SteerMoE baseline (58.4%).
  • Controlled Policy Relaxation: For adult-content generation, MASCing successfully bypassed over-refusal, raising the compliance success rate from 52.6% to 82.0% on average.
Table 3: Success rates for adult-content generation before and after MASCing. Success is defined as the percentage of prompts that lead to the generation of adult content.
  • High Efficiency: The LSTM surrogate trains in under 5 minutes on a single NVIDIA H100 GPU. At inference time, the operational overhead is virtually non-existent, adding negligible latency.
  • Preserved Utility: The intervention caused only a modest average decline of 4.1% across MMLU and GSM8K benchmarks, proving that MASCing avoids catastrophic forgetting or logical collapse.

Limitations and Future Work

Despite its effectiveness, MASCing has a few notable limitations:

  • Surrogate Modeling Limits: The LSTM surrogate might struggle to accurately map highly non-linear routing dynamics in extraordinarily deep MoE architectures.
  • Inherent Capability Assumption: MASCing only steers routing; it cannot synthesize safe behaviors if the underlying experts completely lack alignment data or are fundamentally poisoned.
  • Static Masks: The optimized steering masks are static during inference, making them potentially less resilient against sophisticated, out-of-distribution, zero-day jailbreaks.

Future Directions: The authors suggest extending MASCing to employ dynamic, input-dependent steering masks that adapt on a per-prompt or per-token basis in real-time, as well as exploring task-agnostic domains like on-the-fly adaptation for specialized medical or legal tasks.

Summary updated

Abstract

Mixture-of-Experts (MoE) architectures in Large Language Models (LLMs) have significantly reduced inference costs through sparse activation. However, this sparse activation paradigm also introduces new safety challenges. Since only a subset of experts is engaged for each input, model behavior becomes coupled to routing decisions, yielding a difficult-to-control mechanism that can vary across safety-relevant scenarios. At the same time, adapting model behavior through full fine-tuning or retraining is costly, especially when developers need to rapidly configure the same model for different safety objectives. We present MASCing (MoE Activation Steering Configuration), the first framework that enables flexible reconfiguration of MoE behavior across diverse safety scenarios without retraining. MASCing uses an LSTM-based surrogate model to capture cross-layer routing dependencies and map routing logits to downstream behaviors. It then optimizes a steering matrix to identify behavior-relevant expert circuits and, at inference time, applies steering masks to the routing gates to override expert selection. This enables targeted enhancement or suppression of specific behaviors while preserving general language utility. To demonstrate its reconfigurability, we apply MASCing to two different safety-related objectives and observe consistent gains with negligible overhead across seven open-source MoE models. For multi-turn jailbreak defense, it improves the average defense success rate from 52.5% to 83.9%, with gains of up to 89.2%. For adult-content generation, MASCing enables models to comply with such requests that would otherwise be refused, increasing the average generation success rate from 52.6% to 82.0%, with gains of up to 93.0%. These results establish MASCing as a practical, lightweight, and flexible framework for scenario-specific safety reconfiguration in MoE models.
BibTeX
@misc{lintelo2026mascing,
  author = {Jona te Lintelo and Lichao Wu and Marina Krček and Sengim Karayalçin and Stjepan Picek},
  title = {{MASCing: Configurable Mixture-of-Experts Behavior via Activation Steering Masks}},
  year = {2026},
  eprint = {2604.27818},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2604.27818},
}

Loading related content...

Figure