When we train a neural network on a specific source domain and deploy it in a new target domain, we typically face a frustrating choice: either retrain the entire network or fine-tune its parameters to absorb the domain shift. This tightly couples the learned domain knowledge to the specific weights and architecture of the base model. If a larger, more powerful base model is released, or if we need to migrate our system to a completely different architecture (such as moving from a CNN to a Transformer), the hard-won domain adaptation is lost, and we must start the expensive training process from scratch.
To solve this problem, a powerful paradigm has emerged: architecture-agnostic modular adaptation. Instead of altering the base network's internal weights, this approach isolates domain-specific and task-specific knowledge into lightweight, modular units—such as adapters, side networks, or boundary projection layers. By decoupling the adaptation mechanism from the underlying backbone, specialized capabilities become portable assets that can be transferred, plugged in, or migrated across entirely different model families, sizes, and hardware constraints.
The Core Intuition: Isolating the Domain Manifold
Why should domain adaptation be independent of network architecture? The core insight is that a domain shift (e.g., transitioning from digital photos to sketches, or from daytime to nighttime imagery) represents a systematic geometric transformation in data distribution. The underlying semantic content remains the same; only the style or modality changes.
If we can capture this transformation externally, we can treat the massive base model as a frozen, black-box feature extractor. For instance, in Side-Tuning: A Baseline for Network Adaptation via Additive Side Networks, Jeffrey O. Zhang et al. demonstrate that interacting exclusively with the final representation layer of a pre-trained model allows for successful adaptation across heterogeneous architectures. Because the base network is never modified, the same adaptation strategy can be applied to entirely different structures, even extending to non-neural models like decision trees.
This boundary-level intervention can also happen at the input stage. In Prepended Domain Transformer: Heterogeneous Face Recognition without Bells and Whistles, Anjith George and Amir Mohammadi introduce a lightweight module that prepends a few thousand learnable parameters to a frozen face recognition model. This architecture-agnostic plug-in maps alternative modalities (like thermal or infrared images) into the exact representation space the frozen backbone expects, bypassing the need for end-to-end retraining.
Standard Adaptation (Coupled):
[Input] -> [ Tuned Base Network Layers ] -> [Output] (Knowledge locked in weights)
Architecture-Agnostic Adaptation (Decoupled):
[Input] -> [ Modular Adapter ] -> [ Frozen Base Network ] -> [Output] (Portable knowledge)
Migrating Adapters Without Original Data
A major milestone in this research direction is the ability to "transplant" parameter-efficient modules (like Low-Rank Adapters, or LoRAs) from an outdated source model to a state-of-the-art target model without requiring access to the original training data. This is crucial when privacy constraints or data retention policies prevent developers from reusing the training set.
How is this possible? The migration relies on a combination of synthetic data generation and cross-model knowledge distillation. In $\textit{Trans-LoRA}$: towards data-free Transferable Parameter Efficient Finetuning, Runqian Wang et al. show that a fine-tuned source model can act as a teacher to label synthetic data generated by a target model. This distillation process successfully extracts the task-specific knowledge from the source adapter and injects it into a new adapter format tailored for the target model, completely independent of the underlying model families.
Data-Free Adapter Migration Pipeline:
1. Target Model Space --> Generates Synthetic Seed Data
2. Source Model + Adapter --> Labels Synthetic Data (Teacher)
3. Target Model + New Adapter --> Learns from Labels (Student)
To make this process even more efficient, ChanJoo Jung and Jaehyung Kim developed TiTok: Transfer Token-level Knowledge via Contrastive Excess to Transplant LoRA. Their framework identifies "contrastive excess"—the specific token representations where an adapter significantly alters and improves upon its base model's default behavior. By isolating these critical knowledge hotspots, the framework selectively transplants only the most vital task capabilities across different model sizes or even from dense architectures to Mixture-of-Experts (MoE) structures.
Bridging Dimension and Structural Mismatches
A common bottleneck when transferring modular knowledge between different models is structural incompatibility. If Model A has a hidden dimension of 4096 and Model B has a hidden dimension of 2048, a standard adapter cannot be directly transferred.
To overcome this, researchers have designed dimension-invariant and representation-intercepting mechanisms. In Co-LoRA: Collaborative Model Personalization on Heterogeneous Multi-Modal Clients, Minhyuk Seo et al. utilize dimension-invariant adapters to enable collaborative federated learning across highly heterogeneous clients (e.g., a 1B parameter model on an edge device sharing knowledge with a 3B parameter model on a server).
Alternatively, instead of reshaping the adapter, one can reshape the model's internal state during inference. In Command-V: Training-Free Representation Finetuning Transfer, Barry Wang et al. propose intercepting a recipient model's internal hidden states, projecting them into a donor model's representation space, applying the donor's pre-trained adapter, and mapping the result back to the recipient model. This training-free transfer allows specialized behaviors (like safety filters or reasoning chains) to be instantly shared across completely different model architectures.
Why It Works: The Universality of Alignment
The ultimate reason architecture-agnostic domain adaptation succeeds is that the mathematical principles of domain alignment are universal. Whether a model uses convolutional layers or self-attention mechanisms, the goal of adaptation remains the same: minimizing the divergence between the source and target distributions.
This is elegantly demonstrated by joint distribution optimization techniques. Nicolas Courty et al. show in Joint distribution optimal transportation for domain adaptation that formulating domain adaptation as an optimal transport problem allows features and labels to be aligned simultaneously. Because this optimization operates on the joint distributions rather than internal network parameters, it functions as a model-agnostic framework that can be layered over kernel methods, CNNs, or Transformers alike.
By focusing on the boundaries, latent spaces, and statistical distributions of data rather than the specific weights of a neural network, architecture-agnostic adaptation transforms domain knowledge from a static, locked attribute into a fluid, reusable module.
Key Foundational Work
SCREEN: Learning a Flat Syntactic and Semantic Spoken Language Analysis Using Artificial Neural Networks (1997) — contributed 1 concept elements to this direction.
Face Sketch Matching via Coupled Deep Transform Learning (2017) — contributed 1 concept elements to this direction.
Joint distribution optimal transportation for domain adaptation (2017) — contributed 1 concept elements to this direction.