Abstract
Modern pre-trained models are typically monolithic and fixed-size, leading to deployment rigidity and negative transfer during domain adaptation. We present KIND (Knowledge Integration and Diversion), a novel pre-training framework that constructs "decomposable" models by separating general, class-agnostic "learngenes" from specific, class-related "tailors." By employing Singular Value Decomposition (SVD) as a structural constraint during training, KIND enables modular resizing and efficient adaptation for downstream tasks. Our evaluations on Diffusion Transformers (DiTs) demonstrate that KIND significantly outperforms traditional fine-tuning methods, particularly in scenarios involving massive domain shifts.
Introduction
The current paradigm of AI relies on massive, fixed-size pre-trained models. However, this approach faces two primary challenges: deployment rigidity and negative transfer. Deployment on resource-constrained devices often requires expensive post-hoc compression (e.g., pruning or distillation) for every target size. Furthermore, when adapting a model from a general source (like ImageNet) to a highly specialized target (like medical MRI scans), the specific knowledge learned during pre-training can hinder performance—a phenomenon known as negative transfer.
KIND addresses these issues by making the pre-training process itself modular and decomposable. Instead of a single weight matrix, KIND learns a structured representation that distinguishes between universal features and class-specific details.
Methodology
The core innovation of KIND lies in its weight representation and the "Diversion" mechanism during training.
SVD-based Structural Constraint
KIND represents each weight matrix $W$ as a product of SVD-inspired components: $W = U \times \Sigma \times V^\top$. This decomposition allows the model to be partitioned into two distinct groups:
- Learngenes: Shared across all classes to capture fundamental, task-agnostic features (the "DNA" of the model).
- Tailors: Dedicated to specific classes to capture unique, class-specific details.
Knowledge Diversion and Class Gating
To ensure strict separation, KIND introduces a "class gate" mechanism. During training, for a given input image, only the universal learngenes and the specific tailor corresponding to that image's class are updated. This forces the model to isolate general generative logic from specific class attributes.
Flexible Recombination
Post-training, the model can be dynamically reconfigured. For limited-resource deployment, engineers can select only the learngenes and relevant tailors. For new domains, old tailors can be discarded and replaced with fresh, randomly initialized ones, preventing negative transfer by avoiding irrelevant source-domain "baggage."
Experiments and Results
We evaluated KIND using Diffusion Transformers (DiTs) pre-trained on ImageNet-1K.
Performance on Novel Classes
On 461 unseen classes of ImageNet, KIND (DiT-L) achieved a Fréchet Inception Distance (FID) of 12.87, significantly outperforming LoRA (22.55) and even Full Fine-Tuning (14.51). This demonstrates superior generalization to related but novel categories.
Adaptation to Large Domain Shifts
KIND was tested on four distinct datasets: CelebA-HQ (faces), Hubble (space), MRI (medical), and Pokemon. In the Hubble dataset, KIND achieved an FID of 0.124, nearly doubling the performance of LoRA (0.232). This success is attributed to the "less is more" principle: transferring only class-agnostic learngenes avoids negative transfer from the source domain.
Parameter Efficiency
KIND allows for high-performing models with reduced parameter counts. A decomposed DiT-L model using only 245.9M parameters achieved better results than the original 457M parameter traditional model, highlighting its efficiency for deployment.
Conclusion
KIND shifts modularity from a post-processing step to a fundamental architectural feature. By separating general and specific knowledge through SVD constraints and class gating, it provides a robust framework for efficient deployment and domain adaptation. Future work will explore its applicability to Large Language Models and automate the partitioning of learngenes and tailors.
Summary updated