KIND: Knowledge Integration and Diversion in Diffusion Models

Yucheng Xie, Fu Feng, Jing Wang, Xin Geng, Yong Rui

Abstract

Modern pre-trained models are typically monolithic and fixed-size, leading to deployment rigidity and negative transfer during domain adaptation. We present KIND (Knowledge Integration and Diversion), a novel pre-training framework that constructs "decomposable" models by separating general, class-agnostic "learngenes" from specific, class-related "tailors." By employing Singular Value Decomposition (SVD) as a structural constraint during training, KIND enables modular resizing and efficient adaptation for downstream tasks. Our evaluations on Diffusion Transformers (DiTs) demonstrate that KIND significantly outperforms traditional fine-tuning methods, particularly in scenarios involving massive domain shifts.

Introduction

The current paradigm of AI relies on massive, fixed-size pre-trained models. However, this approach faces two primary challenges: deployment rigidity and negative transfer. Deployment on resource-constrained devices often requires expensive post-hoc compression (e.g., pruning or distillation) for every target size. Furthermore, when adapting a model from a general source (like ImageNet) to a highly specialized target (like medical MRI scans), the specific knowledge learned during pre-training can hinder performance—a phenomenon known as negative transfer.

Conceptual overview comparing traditional pretraining with the modular KIND approach.

KIND addresses these issues by making the pre-training process itself modular and decomposable. Instead of a single weight matrix, KIND learns a structured representation that distinguishes between universal features and class-specific details.

Methodology

The core innovation of KIND lies in its weight representation and the "Diversion" mechanism during training.

SVD-based Structural Constraint

KIND represents each weight matrix $W$ as a product of SVD-inspired components: $W = U \times \Sigma \times V^\top$. This decomposition allows the model to be partitioned into two distinct groups:

  1. Learngenes: Shared across all classes to capture fundamental, task-agnostic features (the "DNA" of the model).
  2. Tailors: Dedicated to specific classes to capture unique, class-specific details.

Knowledge Diversion and Class Gating

To ensure strict separation, KIND introduces a "class gate" mechanism. During training, for a given input image, only the universal learngenes and the specific tailor corresponding to that image's class are updated. This forces the model to isolate general generative logic from specific class attributes.

The KIND architecture within a DiT block, illustrating the SVD-based knowledge integration and class-gated diversion.

Flexible Recombination

Post-training, the model can be dynamically reconfigured. For limited-resource deployment, engineers can select only the learngenes and relevant tailors. For new domains, old tailors can be discarded and replaced with fresh, randomly initialized ones, preventing negative transfer by avoiding irrelevant source-domain "baggage."

Experiments and Results

We evaluated KIND using Diffusion Transformers (DiTs) pre-trained on ImageNet-1K.

Performance on Novel Classes

On 461 unseen classes of ImageNet, KIND (DiT-L) achieved a Fréchet Inception Distance (FID) of 12.87, significantly outperforming LoRA (22.55) and even Full Fine-Tuning (14.51). This demonstrates superior generalization to related but novel categories.

Adaptation to Large Domain Shifts

KIND was tested on four distinct datasets: CelebA-HQ (faces), Hubble (space), MRI (medical), and Pokemon. In the Hubble dataset, KIND achieved an FID of 0.124, nearly doubling the performance of LoRA (0.232). This success is attributed to the "less is more" principle: transferring only class-agnostic learngenes avoids negative transfer from the source domain.

Parameter Efficiency

KIND allows for high-performing models with reduced parameter counts. A decomposed DiT-L model using only 245.9M parameters achieved better results than the original 457M parameter traditional model, highlighting its efficiency for deployment.

Conclusion

KIND shifts modularity from a post-processing step to a fundamental architectural feature. By separating general and specific knowledge through SVD constraints and class gating, it provides a robust framework for efficient deployment and domain adaptation. Future work will explore its applicability to Large Language Models and automate the partitioning of learngenes and tailors.

Summary updated

Abstract

Pre-trained models have become the preferred backbone due to the expansion of model parameters, with techniques like Parameter-Efficient Fine-Tuning (PEFTs) typically fixing the parameters of these models. However, pre-trained models may not always be optimal, especially when there are discrepancies between training tasks and target tasks, potentially resulting in negative transfer. To address this, we introduce \textbf{KIND}, which performs \textbf{K}nowledge \textbf{IN}tegration and \textbf{D}iversion in diffusion models. KIND first integrates knowledge by decomposing parameter matrices of models using $U$, $\Sigma$, and $V$ matrices, formally inspired by singular value decomposition (SVD). Then it explicitly partitions the components of these matrices into \textbf{learngenes} and \textbf{tailors} to condense common and class-specific knowledge, respectively, through a class gate. In this way, KIND redefines traditional pre-training methods by adjusting training objectives from maximizing model performance on current tasks to condensing transferable common knowledge, leveraging the \textit{Learngene} framework. We conduct experiments on ImageNet-1K and compare KIND with PEFT and other learngene methods. Results indicate that KIND achieves state-of-the-art performance compared to other PEFT and learngene methods. Specifically, the images generated by KIND achieves more than 6.54 and 1.07 decrease in FID and sFID on DiT-L/2, utilizing only 45.4M trainable parameters and saving at least 35.4G FLOPs in computational cost.
BibTeX
@inproceedings{xie2025kind,
  author = {Yucheng Xie and Fu Feng and Jing Wang and Xin Geng and Yong Rui},
  title = {{KIND: Knowledge Integration and Diversion in Diffusion Models}},
  booktitle = {International Conference on Machine Learning},
  year = {2025},
  url = {https://openreview.net/forum?id=SH4FFEQ3Yv},
}
Figure