TL;DR
WaveletLoRA is a frequency-aware framework that achieves state-of-the-art content-style disentanglement from a single reference image. By leveraging the discrete wavelet transform (DWT) to assign dedicated LoRA branches to different frequency subbands, the model cleanly separates structural identity from artistic style, enabling high-fidelity personalized image generation.
What Problem Does This Paper Address?
Personalized image generation aims to capture specific concepts—like a particular pet or a unique art style—from reference images to create new content. While methods like DreamBooth or StyleDrop are effective, they typically require multiple images or focus on either content or style in isolation.
Disentangling both factors simultaneously from a single image remains a "holy grail" challenge. Existing spatial-domain methods often suffer from "style leakage" (where the style bleeds into content-only generations) or "subject residue" (where the original subject's shape haunts a style transfer). The authors argue this happens because these methods ignore a fundamental signal-processing truth: content and style naturally live in different frequency bands.
Methods
The core innovation of WaveletLoRA is using frequency decomposition as a principled inductive bias for disentanglement.
- Wavelet-based Decomposition: The researchers apply a two-level Discrete Wavelet Transform (DWT) to the reference image. This breaks the image into one low-frequency subband ($LL_2$), which captures global appearance and color (style), and six high-frequency subbands that capture edges, textures, and structural details (content).
- Frequency-Specific LoRA Branches: Instead of training a single LoRA, WaveletLoRA instantiates seven independent branches. The low-frequency branch is optimized for style, while the high-frequency branches focus on content.
- Dense Mixture-of-Experts (MoE): To integrate the high-frequency information, the authors use a dense MoE module. A gating network dynamically weights the different high-frequency experts based on the input, allowing the model to prioritize specific structural cues (like horizontal edges or fine textures) depending on the subject.
- Disentanglement Objectives: The training is guided by a composite loss function:
- Weight-level frequency separation: Penalizes high-frequency energy in style weights and low-frequency energy in content weights.
- AdaIN-based style alignment: Ensures the generated style matches the global statistics (color palette, contrast) of the reference.
- Orthogonality regularization: Encourages the content and style LoRA masks to be column-wise orthogonal to prevent overlap.
Datasets
The authors introduced WaveBench, a large-scale benchmark specifically designed for single-image content-style decomposition.
- Scale: 1,000 diverse examples spanning animals, humans, objects, and scenery.
- Diversity: Includes styles ranging from traditional Chinese painting to futuristic digital art.
- Construction: Prompts were expanded using ChatGPT-4o, manually filtered for quality, and reference images were synthesized using Gemini-3 Pro to ensure high fidelity.
Key Insights and Ideas
- Frequency as a Natural Prior: Unlike Fourier transforms, which provide global spectral data, wavelets offer localized multi-scale decomposition. This makes them uniquely suited for separating "what" an object is (high-freq structure) from "how" it looks (low-freq style).
- The MoE Advantage: The visualization of MoE gating weights (Figure 7 in the paper) reveals that the model adaptively chooses experts. For a pencil sketch, it prioritizes edge-heavy subbands; for pixel art, it triggers diagonal-heavy subbands. This avoids "representation collapse" where the model treats all structural details the same.
- VLM-based Evaluation: The authors found that standard metrics like DINOv2 can be misleading because they measure holistic similarity rather than purity of separation. They developed a VLM-based protocol (using Qwen2.5-VL) that explicitly rates "Separation" and "Similarity," which correlates much more closely with human judgment.
Key Results
WaveletLoRA consistently outperformed existing SOTA methods like B-LoRA, UnZipLoRA, and USO.
- Quantitative: On WaveBench, WaveletLoRA achieved the highest VLM-based scores for both content preservation (0.8563) and style fidelity (0.8730).
- Qualitative: As shown in the comparison figures, WaveletLoRA produces much "cleaner" results. For instance, when generating a subject in a new context, it successfully strips away the original artistic medium (like watercolor textures) that other models fail to remove.
- User Preference: In a user study, WaveletLoRA's combined score (0.7750) significantly surpassed the next best specialized method, USO (0.5825).
Limitations and Future Work
- Computational Overhead: Training seven LoRA branches and an MoE module increases the parameter count and training time. WaveletLoRA takes ~320 minutes to train for 600 steps, compared to ~230 minutes for a standard dual-LoRA setup.
- Inference Speed: Inference is slightly slower (~10s vs ~6s per image) due to the additional frequency-specific processing.
- Future Directions: The authors suggest exploring even deeper wavelet decompositions or applying this frequency-aware logic to video generation, where temporal consistency adds another layer of complexity to content-style separation.
Summary updated