Preference training such as Direct Preference Optimization (DPO), a method that fine-tunes a language model on pairs of preferred and rejected answers, does not make a model less toxic by silencing a few "toxic neurons." Neuron-level analyses show it works through many small, coordinated shifts. Some toxic-aligned neurons fire less, and a separate population of anti-toxic neurons is pushed toward promoting safe continuations. Current inference-time steering methods, which nudge a model's internal activations while it generates text, ignore this structure. They add one vector, or run one feedback loop on one error signal. We propose to build steering from swappable parts: a representation basis (neuron groups, sparse-autoencoder features, attention heads), a controller (constant, conditional, feedback), and a gate. Within that design we test one specific claim. Controlling the suppression and promotion populations as two separately regulated channels gives a better toxicity-versus-fluency tradeoff than either a single pooled feedback loop or static group edits.
The modular design is the method, not the contribution. Steering papers in this area each report on a different model with a different recipe, so no one can currently say whether a gain comes from the choice of representation or from the choice of controller. A shared interface in which the basis, controller and gate are independent components lets us run a factorial comparison under one tuning budget. It also lets us test the two-channel hypothesis cleanly, and lets us drop in sparse-autoencoder features where a prior continuous-steering method said it could not.
Core Claim
Two papers found that DPO's toxicity reduction in GPT-2 medium decomposes into opposing neuron groups: Ablation is Not Enough to Emulate DPO and Toxic Neurons Aren't Enough to Explain DPO. The groups are defined relative to a linear toxicity probe trained on Jigsaw comments. Toxic-aligned neurons becoming less positive account for 31.8% of the reduction. Anti-toxic-aligned neurons becoming less negative account for 37.3%. Together these two groups explain 69.1%, while the 128 most toxic neurons alone explain only 4.9%. These papers motivate the proposal.
How Does DPO Reduce Toxicity? then showed that manually shifting these groups toward post-DPO levels beats DPO itself. It reports a toxicity score of 0.045 versus DPO's 0.241 on Llama-3.1-8B. This method is open-loop: it applies fixed, corpus-averaged shifts regardless of the input.
Activation Steering with a Feedback Controller takes a different route. It shows that plain vector addition behaves like a proportional-only controller, which leaves a persistent residual error and overshoots when the strength is raised. It fixes this with a PID controller (proportional, integral and derivative terms) applied layer by layer, but on a single monolithic error. That paper supplies our controller machinery.
Our claim is that each group should get its own feedback loop, with its own target and gains. The suppression channel's error is the toxic-aligned group's projected contribution minus its reference level. The promotion channel's error is defined the same way for the anti-toxic group. Concretely, at layer $l$ and token $t$, write $e_{g,l}$ for the gap between group $g$'s current summed probe-projected activation and its reference. The correction added to that group's neurons alone is
$$u_{g,l} = K_p^{g} e_{g,l} + K_i^{g} \sum_{l' \le l} e_{g,l'} + K_d^{g}(e_{g,l} - e_{g,l-1}),$$
where the $K^g$ are per-channel gains. The reference comes from the DPO model's group activations on a calibration set.
The mechanistic prediction that makes this worth testing is about fluency. We expect the fluency collapse seen at high steering strength to come mainly from over-driving suppression, while promotion supplies safe alternatives. If so, the two channels want very different gains. Pooling them forces a compromise that sits below the achievable tradeoff frontier.
Research Program
We first build the harness. It is a small library in which a steering method is assembled from three components with fixed interfaces. The basis maps residual or MLP activations to a set of channels. The controller maps per-channel errors to corrections. The gate decides per token whether to act. We borrow two design principles from the modular-systems literature. The shared four-stage schema of OmniEvaluator cut $N \times M$ integrations to $O(N+M)$ by standardizing intermediate data between stages. The fit/predict superclass of the Universal Causal Evaluation Engine separated data generation from estimators. We follow DeepDIVA in logging commit hashes and seeds. That paper also warns that cuDNN nondeterminism remains, so we report variance over seeds rather than claim bitwise reproduction. We make no claim that the software itself is new.
We then implement the existing methods as configurations of the harness. Constant vector addition is one channel with a constant controller and no gate. PID steering is one channel with a PID controller. Group activation editing is four neuron-group channels with a constant controller. Conditional steering in the style of CAST, which applies a behavior vector only when the hidden state's cosine similarity to a condition vector exceeds a threshold, is a gate. Head-wise fusion from DeStein, which weights per-attention-head detox vectors by probe accuracy, is a head basis. Because each is reproduced inside one codebase, differences between them become attributable.
Finally, we swap in a sparse-autoencoder basis. A sparse autoencoder learns to rewrite dense activations as a large set of sparsely active, more interpretable features. We split such features into toxic-aligned and anti-toxic-aligned sets by the sign of their decoder direction's projection on the probe. Breaking Bad Tokens steered such features with constant or per-token additions and saw incoherence at strength ≥1.5 in GPT-2 Small. It also found that widening the dictionary from 16K to 65K features raised toxicity by 0.062 on average, because concepts split into multiple features. ODESteer states that its multi-step formulation, which treats steering as a differential equation solved over small steps, cannot easily incorporate such features. Our two-channel controller aggregates errors over a feature set rather than over single features. We therefore predict it will be less sensitive to feature splitting, and the 16K-versus-65K comparison tests this directly.
Baselines
The naive reference point is the unsteered base model. Alongside it we run the base model with DPO applied, using the same preference data the analysis papers used. DPO sets the bar that inference-time methods claim to beat. Both are evaluated with identical prompts, sampling settings and Perspective API scoring.
The strongest existing competitor is static group activation editing from How Does DPO Reduce Toxicity?. It uses exactly the same neuron groups we use and differs only in being open-loop and corpus-averaged. Its reported result is a toxicity score of 0.045 against DPO's 0.241 on Llama-3.1-8B, with α=0.01 and β=0.8.
The second strongest is single-channel PID steering from Activation Steering with a Feedback Controller. It reports an 8.2× toxicity reduction over the original Gemma-2B, and a 94.85% defense rate on AdvBench with Llama-3.1-8B against 93.26% for the best prior baseline. It shares our controller but not our channel structure.
We also run DeStein on GPT2-large, where it reports 0.203 expected maximum toxicity, 0.061 toxicity probability and a perplexity of 37.809. We run sparse-autoencoder constant steering from Breaking Bad Tokens on Gemma-2-2B.
Fairness is enforced in four ways. All methods are rerun inside the harness rather than compared with their quoted numbers, since those come from different models and protocols. Every method gets the same calibration data (Jigsaw for probes, the same DPO reference activations where needed). Every method gets the same number of hyperparameter configurations searched on a held-out slice of RealToxicityPrompts, so that our extra per-channel gains do not buy an advantage through a larger search. Every method is scored on the full toxicity–perplexity curve rather than at one operating point.
The key ablation is pooled PID. It uses the same neurons and the same PID controller with the same total search budget, but a single error summed across both groups and one set of gains. It isolates exactly the claim that separate channels matter. A second ablation keeps the two channels but uses constant controllers, which isolates the feedback component.
Beating the baselines means the following. The two-channel controller must dominate static group editing and pooled PID on the toxicity-versus-perplexity frontier. At matched Wikitext-2 perplexity it should have lower expected maximum toxicity, with the gap exceeding seed-to-seed variation, on at least two of the three model families. It must also leave LM Harness accuracy statistically unchanged.
Why not just…?
Why not just use static group activation editing, which already beats DPO? This is the real threat. It is possible that fixed shifts are already near the frontier and feedback adds nothing. The evidence against it is indirect. The static edit uses averaged post-DPO activations, and the analysis papers themselves flag averaged rather than input-conditioned shifts as a limitation. Separately, Activation Steering with a Feedback Controller shows open-loop addition carries a residual error under input-dependent disturbances. The two-channel-constant ablation versus the two-channel-PID configuration settles this directly. If they tie across the frontier, and also on the stress subset where input variation is largest, we will report that feedback is unnecessary once the basis is right. That would itself be a useful attribution result.
Why not just run single-direction PID steering and tune it harder? The pooled-PID ablation is designed to answer this with an equal search budget. The motivation is that the two groups contribute comparably (31.8% and 37.3%) but through opposite sign changes. A single error can drive one to its target while pushing the other past its target. If pooled PID matches two-channel PID, the channel decomposition is not doing work, and we will say so.
Why not just fine-tune with LoRA or DPO? Fine-tuning needs preference pairs and backward passes. How Does DPO Reduce Toxicity? already reports training-free edits outperforming DPO. Steering MoE LLMs via Expert (De)Activation shows weight-based alignment in mixture-of-experts models can be routed around, dropping safety to 0% under jailbreaks combined with expert steering. That model family differs from ours, so we treat it as motivation rather than proof. We include DPO as a baseline point on the frontier rather than argue it away.
Why not just ablate the toxic neurons? This is ruled out by prior work. Ablation of the top neurons recovers only 4.9–31.8% of DPO's effect, and ablating more than 1,000 neurons significantly raises perplexity (Ablation is Not Enough to Emulate DPO). We include it only as a sanity-check point.
Proposed Experiments
Models. We use GPT-2 medium, where the group decomposition was measured; Gemma-2B and Gemma-2-2B, where PID and sparse-autoencoder steering were reported; GPT2-large for DeStein; and Llama-3.1-8B for the static-edit comparison.
Toxicity and fluency. Toxicity is measured on RealToxicityPrompts, both the full non-toxic split and the 1,199-prompt challenging subset. The metrics are expected maximum toxicity and toxicity probability over 25 samples, scored by Perspective API with the standard 0.5 threshold. Fluency is Wikitext-2 perplexity.
Capability retention. We use the seven LM Harness tasks from Breaking Bad Tokens plus MMLU.
Transfer to refusal. As a second behavior, we use AdvBench defense rate and a harmless-prompt refusal rate, since CAST showed that unconditioned refusal vectors pushed harmless refusal to 91.8%. For refusal we define the two channels as refusal-promoting and compliance-promoting populations, found with the same probe-sign procedure. This tests whether the decomposition generalizes beyond toxicity.
Decisive check 1: channel attribution. We compare pooled PID against two-channel PID at matched perplexity. This is the experiment that validates or kills the core claim.
Decisive check 2: the gain-asymmetry prediction. Under per-channel tuning, the best suppression gain should be lower than the best promotion gain. Driving the suppression channel alone to high strength should reproduce the fluency collapse, and driving the promotion channel alone should not. If the tuned gains come out symmetric, the mechanistic story is wrong even if the method works.
Decisive check 3: the feedback component. We compare two-channel constant against two-channel PID on the challenging subset.
Decisive check 4: the sparse-autoencoder swap. We run the two-channel controller over 16K- versus 65K-feature autoencoders on Gemma-2-2B. It should close most of the reported 0.062 toxicity gap that per-feature steering suffers.
Reporting. All results are reported as frontiers with seed intervals. Every configuration is released as a harness recipe, so that each quoted baseline number can be regenerated from one command.
References
- Ablation is Not Enough to Emulate DPO: How Neuron Dynamics Drive Toxicity Reduction
- Toxic Neurons Aren’t Enough to Explain DPO: A Mechanistic Analysis for Toxicity Reduction
- How Does DPO Reduce Toxicity? A Mechanistic Neuron-Level Analysis
- Activation Steering with a Feedback Controller
- A Composable Evaluation System for Reproducible Omni-Modal Foundation Model Evaluation
- Universal Causal Evaluation Engine: An API for empirically evaluating causal inference models
- DeepDIVA: A Highly-Functional Python Framework for Reproducible Experiments
- Programming Refusal with Conditional Activation Steering
- DeStein: Navigating Detoxification of Language Models via Universal Steering Pairs and Head-wise Activation Fusion
- Breaking Bad Tokens: Detoxification of LLMs Using Sparse Autoencoders
- ODESteer: A Unified ODE-Based Steering Framework for LLM Alignment
- Steering MoE LLMs via Expert (De)Activation