Steering Fields: Adaptive Vector Fields for Safe Image Generation and Beyond

Simone Facchiano, Jan Eric Lenssen, Bernt Schiele, Wolfgang Stammer, Fabio Galasso, Jonas FischerarXiv 2026 · arXiv

As text-to-image models like FLUX.1 and Stable Diffusion 3.5 reach near-photorealistic quality, the research frontier has shifted from how to generate to how to control. Whether it's suppressing harmful content (safety steering) or modifying specific attributes (image editing), we need precise ways to guide the generative process without breaking the underlying image structure.

A new paper, "Steering Fields: Adaptive Vector Fields for Safe Image Generation and Beyond," introduces a powerful generalization of current control methods that adapts to the generative process in real-time.

TL;DR

Steering Fields replaces static steering vectors with adaptive vector fields that re-estimate the optimal control direction at every step of the generative trajectory. This approach achieves state-of-the-art results in safety steering and enables high-fidelity, inversion-free image editing by naturally preserving local structure.

What Problem Does This Paper Address?

The current state-of-the-art for controlling generative models is activation steering. This involves adding a fixed "steering vector"—pre-calculated to represent a concept like "clothed" vs. "nude"—to the model's internal activations during inference.

However, this paradigm has a major flaw: it is static. A single, global vector is applied uniformly throughout the entire generation process, from the initial random noise to the final detailed image. Because the geometry of the latent space changes as an image forms, a fixed vector often causes unintended global changes, distorting the composition or artistic style of the image while trying to change just one concept.

Methods

The authors propose Steering Fields, which "lifts" steering from a fixed vector to a dynamic vector field.

Instead of a constant shift, they define a steered velocity $v^*$ at each integration step $t$. This is achieved by solving a quadratic objective that balances three goals:

  1. Fidelity: Staying close to the original "source" velocity field ($v_{src}$).
  2. Attraction: Moving toward a desired "target" concept ($v_{tar}$).
  3. Repulsion: Moving away from an undesired "away" concept ($v_{away}$).
Figure 1: Steering Fields improve Safe Image Generation. We propose Steering Fields as a generalization of steering vectors that adapts steering directions in the latent space locally across space and time (left), blending between original (source) and desired (target) vector fields of a flow model. Among other applications, it provides a new state-of-the-art for safety steering in image generation (see examples on the right).

The resulting closed-form solution is a weighted average of these velocity fields. Crucially, because these velocities are re-calculated at every step based on the current state $z_t$, the steering direction adaptively "curves" with the generative trajectory. This makes the method model-agnostic; it works on the latent velocity of any flow-based model (like FLUX or SD3) without needing access to internal layers or specific architectures.

Datasets

The authors evaluated Steering Fields across several rigorous benchmarks:

  • Safety Steering: Tested on Ring-a-Bell (unsafe prompts) and P4D (adversarial prompts designed to bypass safety filters). They also used T2IRiskyPrompt and T2ISafetyViolence for broader category testing.
  • Semantic Retention: Used COCO-1k to ensure that steering doesn't degrade the quality of benign, everyday prompts.
  • Image Editing: Evaluated on PieBench++, a benchmark of 700 images across nine edit categories (e.g., changing objects, styles, or colors).

Key Insights and Ideas

  • Emergent Structure Preservation: Perhaps the most surprising finding is that Steering Fields preserve local image structure without any explicit spatial masks or object priors. The per-step adaptivity naturally keeps the changes focused on the relevant areas, a behavior usually only seen in complex image-editing pipelines.
  • Compositionality: The framework allows for simultaneous induction and inhibition. You can pull the model toward "clothed" while pushing it away from "nude" in a single unified step.
  • Inversion-Free Editing: By starting the process with a noised version of a real image rather than pure noise, Steering Fields functions as a high-performance image editor. Unlike many other methods, it doesn't require a costly "inversion" step to map the image back into the model's latent space.
  • Latent vs. Activation Space: By operating directly on the latent velocity rather than internal activations, the method avoids the "brittleness" of choosing which specific layer to steer.

Key Results

Steering Fields sets a new state-of-the-art in safety. On the Ring-a-Bell benchmark using FLUX.1, it achieved the lowest NudeNet detection rate (37.32%) compared to existing methods like UCE (72.25%) or ESD (41.93%), while maintaining higher semantic fidelity.

Figure 2: Comparison of safety steering methods on Ring-a-Bell. Left: qualitative comparison using FLUX1. Right: trade-off between NSFW suppression and semantic fidelity on FLUX1 (top) and SD3.5 (bottom).

In image editing, Steering Fields outperformed dedicated editing models in semantic adherence (measured by CLIP-txt and VQAScore), proving that its trajectory-adaptive mechanism is exceptionally good at following complex instructions while maintaining visual coherence.

Limitations and Future Work

Despite its strengths, the authors note a few hurdles:

  • Inference Cost: Because the method requires two additional forward passes per step (one for the target and one for the "away" concept), it roughly triples the inference time compared to a standard unsteered generation.
  • Hyperparameter Tuning: The strength of attraction ($\mu$) and repulsion ($\lambda$) currently requires manual tuning per application to find the sweet spot between safety and image quality.
  • Subtle Content: The method is not a perfect shield; it can still struggle with prompts that imply intimacy without explicit nudity, which are harder for the underlying embeddings to distinguish.

Future work may involve automatically learning these steering parameters or adapting them dynamically based on real-time safety classifier signals.

Summary updated

Abstract

As state-of-the-art text-to-image flow models achieve near-photorealistic quality, controlling their outputs, e.g., suppressing harmful content while promoting benign alternatives, has become a central challenge. The current steering paradigm consists of adding a global steering vector to selected activations. While functional, a fixed and example-agnostic vector applied uniformly along the entire trajectory cannot adapt to the changing state of the generation and often causes unintended global changes. We introduce Steering Fields, a generalization of steering vectors that adaptively re-estimates the steering direction at each step of the generative process. Steering Fields operate on the noisy states of flow models, expose a continuous trade-off between steering strength and content preservation, and are compositional, enabling the simultaneous induction and inhibition of concepts, setting a new state of the art on safety steering benchmarks. Despite using no explicit spatial masks or object priors, the trajectory-adaptive estimation naturally preserves local structure, in a manner reminiscent of image editing. In fact, Steering Fields can serve as a structure-preserving image-editing technique that achieves state-of-the-art semantic fidelity (CLIP, VQAScore), while remaining model-agnostic and inversion-free.
BibTeX
@misc{facchiano2026steering,
  author = {Simone Facchiano and Jan Eric Lenssen and Bernt Schiele and Wolfgang Stammer and Fabio Galasso and Jonas Fischer},
  title = {{Steering Fields: Adaptive Vector Fields for Safe Image Generation and Beyond}},
  year = {2026},
  eprint = {2609.39573},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2609.39573},
}
Figure