Content-Preserving Principal Feature Blending for Training-Free Style Personalization

Le-Hoang Bui, Hoang-Nguyen Cao, Trung-Nghia Le, Minh-Triet Tran, Dinh-Khoi VoarXiv 2026 · arXiv

TL;DR

CoPFB is a training-free method for style personalization in visual autoregressive (VAR) models that solves the problem of "content leakage" from reference images. By projecting style features away from a low-rank content basis estimated on-the-fly, it preserves the requested subject while adopting the reference's aesthetic, all while doubling inference speed compared to previous state-of-the-art methods.

What Problem Does This Paper Address?

In the world of AI image generation, "style personalization" means taking the look and feel of a reference image (like a Van Gogh painting) and applying it to a new subject (like a "Shiba Inu"). While diffusion models have dominated this space, Visual Autoregressive (VAR) models—which generate images scale-by-scale from coarse to fine—are gaining ground.

The challenge is content leakage. When you inject features from a reference image to capture its style, you often accidentally inject its objects and spatial structure too. If your style reference is a portrait of a man and your prompt is "a dragon," the model might struggle to keep the dragon's shape, instead producing a weird man-dragon hybrid. Existing methods like Principal Feature Blending (PFB) try to solve this by using Singular Value Decomposition (SVD) to isolate "dominant" features, but the authors show that these dominant features are still heavily contaminated with the reference image's content.

Methods

The authors propose Content-Preserving Principal Feature Blending (CoPFB). It is a "training-free" approach, meaning it requires no fine-tuning or learnable parameters, making it highly efficient.

Key components include:

  1. Content-Conditioned Residual Projection: The core innovation. The system runs a "clean" generation branch (using only the text prompt) to estimate a low-rank content basis. It then takes the style residual (the "edit" intended to add style) and removes the components that align with this content basis. This effectively "filters out" the parts of the style reference that would overwrite the prompt's subject.
  2. Hybrid Multi-Scale Schedule: Based on an empirical analysis of how VAR models build images (see Figure 2), the authors found that different scales have different roles. CoPFB applies standard rank-one blending at the very coarse scales ($R_1–R_2$) to set the color palette, followed by the new content-projected refinement at the intermediate scales ($R_3–R_5$).
  3. Structural Attention Correction (SAC): They combine their method with SAC, an existing technique that uses the clean generation stream as a structural guide to ensure the layout remains faithful to the prompt.
CoPFB Overview: The black arrows show the clean content path used to estimate the content basis, while purple arrows show the blending path.

Datasets

The researchers evaluated CoPFB using a benchmark of 1,900 content-style pairs:

  • Content Prompts: 190 object-centered prompts from the Parti dataset.
  • Style References: 10 diverse reference images from the FineStyle dataset.
  • Model: The experiments were conducted using the frozen Infinity-2B visual autoregressive model.

Key Insights and Ideas

  • The "Pivotal Scale": Through step-wise ablation, the authors identified $R_3$ (the third scale in a 10-scale generation) as the most critical for balancing style and content. Injecting style here yields the highest "harmonic score."
  • SVD is not a Magic Bullet: A major insight is that the "dominant" singular components used in previous methods (like PFB) are not purely "style." They are content-sensitive. If you change the object in a reference image while keeping the style the same, these components change significantly (see Figure 4).
  • Early Exit for Efficiency: By realizing that style injection is most effective in the early-to-mid scales ($R_1–R_5$), the authors stop the intervention after scale 5. This leads to a massive reduction in computation time.
Fig. 2: Coarse-to-fine dynamics of next-scale prediction.

Key Results

CoPFB outperforms the previous state-of-the-art (PFB+SAC) in several key metrics:

  • Prompt Fidelity ($S_{txt}$): Increased from 0.293 to 0.297, indicating the generated images are more faithful to the text prompt.
  • Harmonic Score: Improved from 0.391 to 0.393, showing a better overall balance between content and style.
  • Inference Speed: This is the most dramatic win. CoPFB reduces inference time from ~3.8 seconds per image to just ~2.0 seconds, a nearly 2x speedup.
  • Qualitative Improvement: As seen in Figure 1, CoPFB successfully preserves specific subjects (like a "blue box" or a "train") while accurately adopting complex artistic styles, avoiding the "muddiness" or subject-erasure seen in earlier methods.
Fig. 1: CoPFB results. Each row shows a style reference followed by outputs for different content prompts.

Limitations and Future Work

  • Style-Content Trade-off: The authors note a slight decrease in raw style similarity ($S_{img}$) compared to PFB+SAC (0.577 vs 0.590). This is the price paid for better content preservation; by removing content-aligned features, some stylistic nuances might also be filtered out.
  • Future Directions: The team plans to investigate adaptive content-basis ranks (adjusting the "strength" of the filter based on the prompt) and extending the method to other autoregressive architectures beyond Infinity. They also suggest that spatially localized projections could help preserve fine object details while allowing for more textured style transfer.

Summary updated

Abstract

Reference-based style personalization should reproduce the appearance of a style reference while preserving the subject specified by a content prompt. In visual autoregressive models, dominant reference features can also carry object and spatial information, causing content leakage. We propose Content-Preserving Principal Feature Blending (CoPFB), a training-free method that estimates a low-rank content basis from a clean generation branch and removes the content-aligned component of the style residual. CoPFB applies rank-one blending at the first two autoregressive scales, followed by weaker content-projected refinement at the next three scales. On 1,900 combinations of Parti content prompts and FineStyle reference images, CoPFB improves text alignment from 0.293 to 0.297 and the harmonic score from 0.391 to 0.393 over PFB+SAC, while style similarity decreases from 0.590 to 0.577. CoPFB requires no fine-tuning or learnable parameters. Project page: https://vdkhoi20.github.io/CoPFB/.
BibTeX
@misc{bui2026content,
  author = {Le-Hoang Bui and Hoang-Nguyen Cao and Trung-Nghia Le and Minh-Triet Tran and Dinh-Khoi Vo},
  title = {{Content-Preserving Principal Feature Blending for Training-Free Style Personalization}},
  year = {2026},
  eprint = {2611.00683},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.00683},
}
Figure