Few-Shot Visual Concept Extraction for Steering Diffusion Transformers

Nabyl Quignon, Antitza DantchevaarXiv 2026 · arXiv

TL;DR

Researchers have developed a training-free, few-shot method to extract "concept directions" directly from the internal activations of Diffusion Transformers (DiTs). By comparing just five pairs of positive and negative prompts and using spatial masking to isolate local attributes, they can steer image generation with fine-grained, compositional control—like changing a car's color or a person's expression—without any auxiliary model training or fine-tuning.

What Problem Does This Paper Address?

While Diffusion Transformers (like FLUX.1) produce stunning images, controlling them precisely remains a "blunt instrument" problem. If you want to change a specific attribute—say, making a specific object "metallic" or a face "older"—you usually have three choices:

  1. Prompt Engineering: Often unreliable for fine-grained or localized changes.
  2. Auxiliary Training: Training LoRAs or "Concept Sliders," which is computationally expensive and requires curated datasets for every single concept.
  3. Text-Encoder Steering: Manipulating the text embeddings, which often fails to capture visual nuances that are better represented in the image-side activations of the model.

The authors address the challenge of fine-grained visual control without the overhead of training. They ask: Can we find the "steering wheel" for a concept just by looking at how the model's internal state changes when that concept is present versus when it is absent?

Methods

The core of the paper is a framework for Few-Shot Visual Concept Extraction.

1. Difference-of-Means (DoM) Estimator

The researchers use a surprisingly simple statistical tool: the difference-of-means. They take a few (usually 5) pairs of prompts—one "positive" (e.g., "a red car") and one "negative" (e.g., "a gray car"). They run these through the model and record the "residuals" (the updates added to the image tokens) at a specific intermediate layer. By subtracting the average negative residual from the average positive residual, they isolate a vector that represents the concept of "redness."

2. Masked Difference-of-Means

The "aha!" moment of the paper is that global averaging doesn't work for local concepts. If you average the "red car" signal over the entire image, the background tokens (trees, sky, road) dilute the signal. To fix this, they introduce Masked DoM. They use a segmentation model (SAM 3) to identify exactly which tokens correspond to the target object. They then calculate the concept direction only from those tokens. This prevents the "red" signal from being washed out by the "non-car" parts of the image.

3. Inference-Time Steering

Once they have this "concept vector," they can inject it back into the model during a new generation. By multiplying the vector by a scaling factor ($\alpha$), they can continuously dial the effect up or down. Because these vectors are decoupled from the rest of the image, they can apply multiple vectors to different regions of the same image (e.g., make the left car blue and the right car green).

Datasets

The method is training-free, so it doesn't require a training dataset in the traditional sense. For evaluation and extraction, the authors used:

  • Backbone Models: FLUX.1-schnell and FLUX.1-dev.
  • Evaluation Suite: 23 distinct concepts categorized into Global (e.g., sunset, anime style), Object (e.g., wooden, metallic, red), and Face (e.g., smile, beard, glasses).
  • Prompt Pairs: Generated using GPT-4 to ensure variety in the few-shot examples.

Key Insights and Ideas

  • Spatial Distribution of Concepts: The authors observed that scene-level concepts (lighting, weather) are distributed across all image tokens, while local concepts (textures, facial features) are highly concentrated. This justifies the need for the masked approach.
  • The "Sweet Spot" Layer: Through ablation studies, they found that Layer 18 (the last dual-stream block in FLUX) is the most effective place to extract concept directions. Earlier layers are too noisy, and later layers are too specialized.
  • Timestep Matters: The best concept signals are found at the very beginning of the denoising process (Timestep 0). This suggests that the model decides on the high-level visual attributes of a concept almost immediately.
  • Compositionality: Because the steering is applied at the patch/token level, you can "compose" edits. You can steer a landscape to be "snowy" while simultaneously steering a specific house in that landscape to be "made of wood."

Key Results

  • Quantitative Superiority: The method achieved a $\Delta$VQA (a metric for edit success) of 0.32, significantly outperforming training-free baselines like SAEdit (0.19) and FluxSpace (0.17).
  • Competitive with Trained Models: Remarkably, this few-shot, training-free approach approached the performance of specialized, trained models like SliderEdit (0.35) and Kontinuous Kontext (0.36).
  • Efficiency: Extracting a new concept takes roughly 21 to 26 seconds on a single H100 GPU, making it practical for rapid experimentation.
  • Robustness: The method proved robust to imperfect masks. Even if the segmentation was slightly off, the extracted direction remained effective.

Limitations and Future Work

  • Semantic Mismatch: If you try to steer a concept into an image that doesn't support it (e.g., adding "stripes" to a car where the model only sees a reflection), the edit may appear weak or distorted.
  • Spatial Drift: At very high steering strengths, the geometry of objects can start to shift unintentionally (e.g., changing a chair's color might slightly change its shape).
  • Concept Entanglement: Some attributes are "tangled" in the model's brain. For example, steering for "lipstick" often inadvertently increases the "smiling" attribute.
  • Future Directions: The authors suggest that future work could focus on better disentangling these concepts and extending the framework to video generation.

Summary updated

Abstract

Image steering aims to provide fine-grained control over concepts in generated images, enabling targeted and continuous manipulation without broadly altering the generation process. Recent steering methods for diffusion models primarily focus on the text-conditioning space or rely on costly auxiliary training. Deviating from that, we here propose a novel steering approach that extracts visual concept directions directly from the image-side activations of Diffusion Transformers. Specifically, we showcase few-shot concept extraction in the activation space using a simple difference-of-means estimator over intermediate residuals. We observe (a) scene-level concepts such as lighting, weather, and style are distributed across image tokens, as well as (b) local concepts such as object colors, textures, materials, and facial attributes are concentrated in specific regions. Towards controlling both concepts, we introduce a new unified training-free framework that uses global difference-of-means for (a) distributed concepts and masked difference-of-means for (b) local concepts, preventing local signals from being diluted by global averaging. Notably, using only a handful of positive and negative prompt pairs, our method identifies and extracts reusable concept directions that can be injected at inference for continuous steering. Quantitatively, our method outperforms text-encoder steering and training-free activation-editing baselines, as well as is competitive with specialized editing models. Our results suggest that as concept directions can be applied at patch level, our method enables precise targeted and compositional steering, manipulating multiple concepts in the same image.
BibTeX
@misc{quignon2026few,
  author = {Nabyl Quignon and Antitza Dantcheva},
  title = {{Few-Shot Visual Concept Extraction for Steering Diffusion Transformers}},
  year = {2026},
  eprint = {2611.19440},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.19440},
}
Figure