RL Post-Training in Diffusion Transformers Is Activation Steering

Tuna Han Salih Meral, Kaan Oktay, Alec Helbling, Atakan Kara, Duen Horng Chau, Pinar YanardagarXiv 2026 · arXiv

TL;DR

Reinforcement learning (RL) post-training is the secret sauce that makes modern diffusion models excel at text rendering and visual quality. This paper reveals a startling truth: most of these gains don't come from complex, prompt-specific weight updates, but from a simple, constant "nudge" to the model's internal activations. By adding a fixed set of steering vectors—using 65x fewer parameters than a standard LoRA—researchers recovered over 90% of the performance gains seen in state-of-the-art RL-tuned models.

What Problem Does This Paper Address?

When we use RL to fine-tune models like Stable Diffusion 3.5 or FLUX, the models get significantly better at following complex prompts, such as "an astronaut holding a sign that says 'Deep Learning'." Because these tasks are inherently dependent on the input prompt, researchers naturally assumed that the model was learning complex, input-dependent transformations.

The authors of this paper asked a fundamental question: Is this input-dependence actually necessary? They wanted to isolate how much of the RL gain is due to the model learning new, complex logic versus simply being "steered" toward capabilities it already possessed but wasn't prioritizing.

Methods

The researchers replaced the standard approach of updating model weights (like LoRA) with a technique called Activation Steering.

Instead of changing the model's knowledge, they added a fixed steering vector ($s$) to the outputs of specific transformer blocks within a frozen base model. Crucially, this vector is input-independent: the exact same vector is added regardless of the prompt, the specific tokens, or the random seed.

The steering vectors add the same vector to every token of every prompt, whereas RL-tuned weights produce input-dependent changes.

To find these vectors, they used two primary methods:

  1. Velocity Matching: They "fit" the vectors to match the output "velocity" (the direction the model moves the image during the denoising process) of existing, publicly available RL-tuned models. This is essentially a regression task that doesn't require a reward function.
  2. Direct RL: They trained the vectors from scratch using a reward function (like PickScore for human preference or OCR accuracy) while keeping the rest of the model frozen.
Fitting the steering vectors to match the velocity predictions of an RL-tuned checkpoint.

Datasets

The study evaluated eight different RL-tuned models across image and video domains, including:

  • SD3.5-M: Evaluated on PickScore (human preference), OCR (text rendering), and GenEval (compositional accuracy).
  • FLUX.1-dev: Evaluated on PickScore and HPSv2.1.
  • Wan2.1 & Wan2.2: Evaluated on video benchmarks like VBench and HPSv3.

Key Insights and Ideas

  • The "Nudge" is Enough: The most surprising finding is that a constant shift—which contains zero information about the prompt—can lead to prompt-specific improvements. For example, the same vector helps the model render "Deep Learning" in one prompt and "Tacos Save Lives" in another.
  • The Frozen Model Does the Work: How does a constant vector produce different results? The authors show that the frozen transformer layers (specifically the attention mechanism) mix the constant steering vector with the prompt-dependent activations. The model's existing internal logic "decides" how to apply the nudge based on the context.
  • Steering vs. Learning: This suggests that RL post-training acts more like a "mode seeker" than a "skill learner." It isn't teaching the model how to render text; it's steering the model to use the text-rendering pathways it already developed during its massive pre-training phase.
  • Efficiency: On SD3.5-M, these steering vectors use only 0.29M parameters, compared to 18.8M for a rank-32 LoRA. This 65x reduction in parameters still captures the lion's share of the performance.

Key Results

The results across various models and tasks were remarkably consistent:

  • Reward Recovery: The constant steering vectors recovered 90.0% to 96.3% of the reward gains of six different RL-tuned image models.
  • Video Performance: In video models, the steering vectors recovered between 89.8% and 111.9% of the gains.
  • Direct Training: When trained directly with RL, the steering vectors alone actually exceeded the gains of traditional weight-based RL on SD3.5-M for both OCR and human preference tasks.
Qualitative comparison showing that the steered base model (Base + s) produces results nearly identical to the full RL-tuned model.

Limitations and Future Work

While activation steering is powerful for RL-style "alignment," it has clear limits:

  • New Knowledge: Steering fails when the goal is to teach the model something entirely new, such as a specific person's face (e.g., DreamBooth). Supervised fine-tuning for new subjects still requires actual weight updates.
  • Scope of RL: The study focused on policy-gradient RL methods. It remains to be seen if other alignment techniques, like DPO (Direct Preference Optimization), rely on the same mechanism.
  • Video Generalization: While the results on video were strong, they were limited to a smaller set of models compared to the image domain.

The authors suggest that future RL research should investigate whether current methods can learn new input-dependent transformations, or if we are currently just scratching the surface of what pre-trained models already know.

Summary updated

Abstract

Reinforcement learning (RL) post-training substantially improves the visual quality, text rendering, and compositional accuracy of diffusion transformers. Because capabilities such as rendering requested text or composing multiple objects are inherently prompt-dependent, a natural hypothesis is that improving them requires learning complex, input-dependent transformations. To isolate how much of these gains an input-independent shift explains, we probe each RL-tuned model with a deliberately simple parameterization: a fixed set of steering vectors added to the block outputs of the frozen base model. On SD3.5-M, these vectors hold 65x fewer parameters than the RL-tuned LoRA (0.29M against 18.8M). We fit them to match the velocity predictions of public RL-tuned checkpoints, with no reward in the objective. Although the vectors are identical for every prompt, token, and sampling seed, they recover 90.0-96.3% of the reward gains of six RL-tuned image models, and 89.8% and 111.9% on two RL-tuned video models. Initialized at zero and trained with online RL, the vectors alone exceed the reward gains of weight-based RL on text rendering (OCR) and human preference (PickScore) on SD3.5-M. A constant shift thus suffices for most of the gain of policy-gradient RL post-training: without new input-dependent transformations, it steers the pretrained generator toward what it can already compute.
BibTeX
@misc{meral2026rl,
  author = {Tuna Han Salih Meral and Kaan Oktay and Alec Helbling and Atakan Kara and Duen Horng Chau and Pinar Yanardag},
  title = {{RL Post-Training in Diffusion Transformers Is Activation Steering}},
  year = {2026},
  eprint = {2611.26627},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.26627},
}
Figure