TL;DR
Reinforcement learning (RL) post-training is the secret sauce that makes modern diffusion models excel at text rendering and visual quality. This paper reveals a startling truth: most of these gains don't come from complex, prompt-specific weight updates, but from a simple, constant "nudge" to the model's internal activations. By adding a fixed set of steering vectors—using 65x fewer parameters than a standard LoRA—researchers recovered over 90% of the performance gains seen in state-of-the-art RL-tuned models.
What Problem Does This Paper Address?
When we use RL to fine-tune models like Stable Diffusion 3.5 or FLUX, the models get significantly better at following complex prompts, such as "an astronaut holding a sign that says 'Deep Learning'." Because these tasks are inherently dependent on the input prompt, researchers naturally assumed that the model was learning complex, input-dependent transformations.
The authors of this paper asked a fundamental question: Is this input-dependence actually necessary? They wanted to isolate how much of the RL gain is due to the model learning new, complex logic versus simply being "steered" toward capabilities it already possessed but wasn't prioritizing.
Methods
The researchers replaced the standard approach of updating model weights (like LoRA) with a technique called Activation Steering.
Instead of changing the model's knowledge, they added a fixed steering vector ($s$) to the outputs of specific transformer blocks within a frozen base model. Crucially, this vector is input-independent: the exact same vector is added regardless of the prompt, the specific tokens, or the random seed.
To find these vectors, they used two primary methods:
- Velocity Matching: They "fit" the vectors to match the output "velocity" (the direction the model moves the image during the denoising process) of existing, publicly available RL-tuned models. This is essentially a regression task that doesn't require a reward function.
- Direct RL: They trained the vectors from scratch using a reward function (like PickScore for human preference or OCR accuracy) while keeping the rest of the model frozen.
Datasets
The study evaluated eight different RL-tuned models across image and video domains, including:
- SD3.5-M: Evaluated on PickScore (human preference), OCR (text rendering), and GenEval (compositional accuracy).
- FLUX.1-dev: Evaluated on PickScore and HPSv2.1.
- Wan2.1 & Wan2.2: Evaluated on video benchmarks like VBench and HPSv3.
Key Insights and Ideas
- The "Nudge" is Enough: The most surprising finding is that a constant shift—which contains zero information about the prompt—can lead to prompt-specific improvements. For example, the same vector helps the model render "Deep Learning" in one prompt and "Tacos Save Lives" in another.
- The Frozen Model Does the Work: How does a constant vector produce different results? The authors show that the frozen transformer layers (specifically the attention mechanism) mix the constant steering vector with the prompt-dependent activations. The model's existing internal logic "decides" how to apply the nudge based on the context.
- Steering vs. Learning: This suggests that RL post-training acts more like a "mode seeker" than a "skill learner." It isn't teaching the model how to render text; it's steering the model to use the text-rendering pathways it already developed during its massive pre-training phase.
- Efficiency: On SD3.5-M, these steering vectors use only 0.29M parameters, compared to 18.8M for a rank-32 LoRA. This 65x reduction in parameters still captures the lion's share of the performance.
Key Results
The results across various models and tasks were remarkably consistent:
- Reward Recovery: The constant steering vectors recovered 90.0% to 96.3% of the reward gains of six different RL-tuned image models.
- Video Performance: In video models, the steering vectors recovered between 89.8% and 111.9% of the gains.
- Direct Training: When trained directly with RL, the steering vectors alone actually exceeded the gains of traditional weight-based RL on SD3.5-M for both OCR and human preference tasks.
Limitations and Future Work
While activation steering is powerful for RL-style "alignment," it has clear limits:
- New Knowledge: Steering fails when the goal is to teach the model something entirely new, such as a specific person's face (e.g., DreamBooth). Supervised fine-tuning for new subjects still requires actual weight updates.
- Scope of RL: The study focused on policy-gradient RL methods. It remains to be seen if other alignment techniques, like DPO (Direct Preference Optimization), rely on the same mechanism.
- Video Generalization: While the results on video were strong, they were limited to a smaller set of models compared to the image domain.
The authors suggest that future RL research should investigate whether current methods can learn new input-dependent transformations, or if we are currently just scratching the surface of what pre-trained models already know.
Summary updated