Fine-grained activation steering changes a language model's behavior by nudging a handful of individual numbers inside its hidden layers during the forward pass, with no change to the weights. The method that best supports this approach, AUSteer, scores each hidden dimension on its own and steers the top few. That is a univariate ranking. The feature-selection literature has long shown that ranking candidates independently picks redundant ones and misses those that only help in combination. We propose to choose steering units the way good feature selection chooses inputs: one at a time, each scored by how much it still helps given the units already being steered. Our hypothesis is that this reaches the same accuracy with fewer units and causes less collateral damage to unrelated abilities.
The computational obstacle is the same one that makes conditional information gain expensive everywhere. Measuring "gain given what is already chosen" exactly would take one intervention per candidate per step, which comes to about 131,000 candidates per step for a 32-layer, 4,096-wide model. Our approach removes this cost by re-measuring each unit's first-order effect after the already-selected units have been applied. This needs one backward pass per selection step instead of one forward pass per candidate. The project will build this selector, test whether internal units actually interact enough for conditioning to matter, and compare it on AUSteer's own benchmarks.
Core Claim
The proposal treats the three source directions unequally. Internal-state steering is the application. Combinatorial feature selection supplies the question: should units be chosen jointly or independently? The cost of conditional information gain sets the constraint the method has to meet.
Fine-Grained Activation Steering: Steering Less, Achieving More motivates the work. It shows that dimensions within one transformer block push toward contradictory outputs, so steering a whole block causes interference. It introduces "activation momentum," a contrastive-pair score that a first-order Taylor argument ties to each unit's effect on the output logits. Steering at most 100 of 4,096 units this way beat block-level methods. Its momentum score, however, is computed once on the unsteered model, and every unit is ranked in isolation.
Variational Information Maximization for Feature Selection shows, for input features, what that design choice costs. Filters that score features independently rest on mutually inconsistent independence assumptions. A greedy sum of conditional gains is the principled alternative, and it is provably optimal when the dependencies form a tree. Our claim is that the same logic applies inside the network. Conditionally re-scored greedy selection of units should beat independent momentum ranking whenever the top-ranked units are redundant, meaning they encode the same direction, or synergistic, meaning they help only together. If neither holds, the two methods should coincide, and showing that would itself be a useful negative result.
Method
Let $L(S)$ be the mean cross-entropy of the desired answers on a calibration set when the units in set $S$ are steered with AUSteer's adaptive scaling, $\hat{x}_i = x_i + \gamma_i x_i$. We use cross-entropy because Estimating Conditional Mutual Information for Dynamic Feature Selection proved a useful fact: under cross-entropy, the expected drop in loss from adding a variable equals that variable's conditional mutual information with the target. We borrow that identity to give "gain given $S$" an information-theoretic reading, here applied to interventions rather than observations.
The conditional gain of adding unit $i$ is approximated by re-linearizing at the currently steered model:
$$\Delta(i \mid S) \approx -\gamma_i x_i \frac{\partial L}{\partial x_i}\Big|_{S},$$
where the gradient is taken with the steering on $S$ already active. AUSteer evaluates this gradient at $S = \emptyset$ for every unit. We re-evaluate it after each addition, adding $b$ units per step until a budget of $k$ is reached. Setting $b = k$ recovers AUSteer exactly, so the batch size is a single knob that isolates conditioning as the ingredient under test.
Each step costs one forward and one backward pass over the calibration set. The full selection therefore costs $k/b$ backward passes. Exact greedy conditional search would cost $k \cdot D$ forward passes, where $D$ is the number of candidate units.
We will also compute pairwise interaction scores among AUSteer's top-ranked units. We compare the measured gain of each pair with the sum of the two single-unit gains, in the spirit of the synergy matrix in Sensor Selection for Active Information Fusion. This tests directly whether internal units are redundant, additive, or synergistic.
Research Program
The first stage is diagnostic. On LLaMA2-7B, using AUSteer's contrastive pairs, we measure how far the momentum ranking moves after the top 10, 25, and 50 units are steered. We also measure the distribution of pairwise interaction scores. If the rankings barely change and interactions are near zero, conditioning has nothing to fix. We would report that internal units behave as approximately independent steering handles, which is evidence that univariate scoring is adequate, and we would stop there.
The second stage compares selectors at matched unit budgets of $k \in \{4, 10, 25, 50, 100\}$. The third stage measures off-target effects, our second claimed benefit. These are the fragility problems the steering literature flags, such as loss of fluency and corruption of unrelated abilities. We will measure them as perplexity on held-out general text and as accuracy on the AUSteer tasks the steering vector was not built for.
Baselines
The do-nothing reference point is the unsteered model on each task. The coarse reference point is block-level contrastive activation addition, which modifies all 4,096 units of a layer, run at AUSteer's reported layer settings.
The strongest existing method is AUSteer from Fine-Grained Activation Steering: Steering Less, Achieving More. It reports a +1.85% gain in five-task average accuracy over SADI on LLaMA2-7B while modifying at most 100 activations, and win-rate gains of up to 8.5% over the vanilla model on BPO. The file contains no absolute per-task accuracies, so we will rerun AUSteer ourselves. The rerun will use the same contrastive pairs, the same calibration set, the same scaling rule and $\gamma$ search grid, and the same unit budget as our method.
Beating AUSteer means one of two things, each holding across three seeds of calibration-set sampling:
- matching its five-task average with at most half as many units, or
- exceeding its average at an equal budget while keeping held-out perplexity no worse.
The key ablation is the batch-size sweep from $b = 1$ to $b = k$. If gains do not shrink steadily as $b$ grows toward $k$, conditioning is not the cause of the improvement.
We add two joint selectors that a reviewer would reasonably prefer:
- L1-regularized linear probe. A probe trained on activations to separate positive from negative prompts, with its top-$k$ weights steered.
- Learned gate mask. A sparse mask over units, relaxed to be continuous with the Concrete distribution as in Learning to Maximize Mutual Information for Dynamic Feature Selection, and trained directly on $L$ with a sparsity penalty that yields $k$ units.
Both receive the same contrastive data. Both also get a compute budget equal to or greater than ours, measured in backward passes.
We also run exact greedy conditional selection: true interventions over a pool of the top 512 units by momentum. This is an upper reference that checks how faithful the linearized score is.
Why not just…?
Why not just run exact greedy selection with real interventions? For the full candidate set it costs about 131,000 forward passes per step, which is infeasible. Over a pre-filtered pool of 512 units it is affordable and is included above. If prefiltered exact greedy matches our linearized selector, the conditional-selection claim still stands, and the re-linearization becomes merely a cost saving. We will report that outcome plainly.
Why not just fit a sparse probe on activations? A lasso does handle redundancy jointly. However, it selects units that predict the contrast, which is a correlational criterion, while steering needs units whose change moves the output. Work on hallucination probing, such as LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinations, shows that internal signals are highly readable, but readable is not the same as steerable. The probe baseline tests this directly. If the probe-chosen units steer as well as ours, our causal re-scoring adds nothing.
Why not just learn a sparse gate mask end to end? This is the most serious threat, because it optimizes the true joint objective. Its risks are overfitting a few thousand gates to a small contrastive set, and sensitivity to how the sparsity penalty is tuned. We give it at least our compute budget. The decisive evidence is accuracy on held-out tasks and perplexity, not accuracy on the calibration set.
Why not just steer more AUSteer units? If a larger univariate budget closes the gap, then our method's value is limited to using fewer units and causing less off-target damage. The budget sweep shows this directly: we compare the full accuracy-versus-units curves, not single points.
Proposed Experiments and Decisive Checks
All main experiments use LLaMA2-7B on AUSteer's five-task commonsense and math suite and on BPO open-ended win-rates. A replication on one of the other model families AUSteer reports, such as Gemma2 or Qwen3, tests whether the findings generalize across models.
To check the method where the answer is known, we will train a small transformer on a parity-style task, modelled on the Cube and Syn benchmarks used in Stochastic Encodings for Active Feature Acquisition. We then plant a synergistic group of units by construction: an intervention on those units changes the output only when all members are steered together. The conditional selector should recover the group, and univariate momentum should not.
There are three decisive checks:
- whether the momentum ranking changes materially after partial steering (the diagnostic),
- whether accuracy at a fixed unit budget falls monotonically as $b \to k$ (the ablation), and
- whether held-out perplexity damage is lower at a matched accuracy.
Positive results on all three would establish conditional selection as the right default for fine-grained steering. A flat diagnostic would establish the opposite, which is equally publishable.
References
- Fine-Grained Activation Steering: Steering Less, Achieving More
- Variational Information Maximization for Feature Selection
- Estimating Conditional Mutual Information for Dynamic Feature Selection
- Sensor Selection for Active Information Fusion
- Learning to Maximize Mutual Information for Dynamic Feature Selection
- LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinations
- Stochastic Encodings for Active Feature Acquisition