DA-MergeLoRA: Hypernetwork-Based LoRA Merging for Few-Shot Test-Time Domain Adaptation

Siobhan Reid, Zhixiang Chi, Li Gu, Omid Reza Heidari, Ziqiang Wang, Yang WangSubmission to ICLR 2026 · Rejected

TL;DR

DA-MergeLoRA is a novel framework for adapting machine learning models to new, unseen environments at test time using only a handful of unlabeled images. By training domain-specific LoRA modules and a meta-learned hypernetwork to merge them, the system achieves state-of-the-art performance across major benchmarks while remaining significantly faster and more memory-efficient than previous methods.

What Problem Does This Paper Address?

Machine learning models often fail when the data they encounter at test time looks different from their training data—a phenomenon known as domain shift. While standard Domain Adaptation (DA) techniques exist, they usually assume you have access to a large amount of target data during the training phase.

The authors focus on a much harder, more realistic scenario: Few-Shot Test-Time Domain Adaptation (FSTT-DA). In this setting, a model is trained on several "source" domains (e.g., photos of objects). At test time, it is suddenly dropped into a "target" domain it has never seen before (e.g., sketches of those same objects) and is given only a tiny batch of unlabeled images to figure out how to adapt.

Existing solutions often rely on shallow updates (like adjusting Batch Normalization layers), treat the model as a "black box" (prompt tuning), or use expensive ensembles that slow down inference. These methods often fail to capture the deep, internal relationships between different domains.

Methods

The authors introduce DA-MergeLoRA, which treats adaptation as a parameter-space merging problem. The architecture consists of three main components:

  1. Domain-Specific LoRAs: Instead of fine-tuning the entire model, they use Low-Rank Adaptation (LoRA). They train a separate, lightweight LoRA module for each source domain on top of a frozen CLIP vision encoder. This allows the model to learn internal features specific to each style (e.g., "real," "clipart," "sketch") without forgetting general knowledge.
  2. The Hypernetwork: This is the "brain" of the system. It takes a few unlabeled target images and the weights of the source LoRAs as input. Using a cross-attention mechanism, it determines which source domains are most relevant to the new target domain.
  3. Per-Column Merging: Unlike previous methods that merge entire models or layers, this hypernetwork generates weights for every individual column of the LoRA matrices. This fine-grained approach allows the model to synthesize a new "Target LoRA" by picking and choosing specific features from across the source domains.

To train the hypernetwork, the authors use meta-learning. During training, they hide one source domain and task the hypernetwork with "reconstructing" it by merging the remaining source domains. This simulates the challenge of encountering a novel domain at test time.

Datasets

The researchers evaluated their method on four diverse and challenging benchmarks:

  • DomainNet: A massive dataset with six distinct domains (Clipart, Infograph, Painting, Quickdraw, Real, Sketch) and 345 classes.
  • WILDS - iWildCam: Wildlife camera trap images across different locations.
  • WILDS - FMoW: Satellite imagery collected over different years and regions.
  • WILDS - Camelyon17: Medical imaging (tumor identification) across different hospitals.

Key Insights and Ideas

  • Internal vs. External Adaptation: By using LoRA, the authors adapt the model's internal representations. This is more powerful than "black-box" prompt tuning, which only modifies the input, yet it is more efficient than full fine-tuning.
  • Granularity Matters: The move to per-column merging is a significant "aha moment." The authors found that different columns in a weight matrix capture different features; merging at this level allows for a much more flexible and precise adaptation than merging at the layer or model level.
  • Semantic Correlation: The hypernetwork successfully learns intuitive relationships. For example, when faced with a "Sketch" target domain, the hypernetwork automatically assigns higher weights to the "Clipart" source LoRA and lower weights to the "Real" photo LoRA.
  • One-Time Cost: The merging process happens exactly once when the first few target images arrive. Once the LoRAs are fused, the model runs at standard speed with zero additional overhead during inference.

Key Results

DA-MergeLoRA achieved state-of-the-art (SOTA) results across the board:

  • DomainNet: Improved average accuracy by 1.24% over the previous SOTA (L2C).
  • FMoW: Saw a massive 11.40% boost in worst-case accuracy, showing the model is much more robust to difficult shifts.
  • Camelyon17: Improved accuracy by 2.70%.
  • Efficiency: Compared to the L2C-CPNet framework, DA-MergeLoRA uses 50% less memory and is 50% faster during inference.

Limitations and Future Work

While highly effective, the authors noted a few areas for improvement:

  • Disjoint Label Sets: The model struggled slightly on the iWildCam dataset. This is likely because different source domains in iWildCam contain different sets of animals, violating the assumption that all domains share the same label space.
  • Class Imbalance: The current merging strategy doesn't explicitly account for domains where some classes are much rarer than others.
  • Future Directions: The authors suggest exploring Reinforcement Learning to optimize merging policies and investigating "column-alignment" strategies to ensure that features from different source LoRAs are mathematically compatible before they are merged.

Summary updated

Abstract

Few-shot Test-Time Domain Adaptation (FSTT-DA) seeks to adapt models to novel domains using only a handful of unlabeled target samples. This setting is more realistic than typical domain adaptation setups, which assume access to target data during source training. However, prior FSTT-DA approaches fail to effectively leverage source domain-specific knowledge, relying on shallow batch normalization updates, prompt-based methods that treat the model as a black box, or ensembling strategies that do not capture cross-domain relationships. To address these limitations, we introduce a new FSTT-DA framework that integrates LoRA fine-tuning with model merging. In our approach, separate LoRA modules are fine-tuned on CLIP’s vision encoder for each source domain. Since LoRA modifies only a small fraction of the model’s parameters, it retains the base model’s generalized knowledge while internally learning domain-specific features. To adapt the learned knowledge to a specific target domain, we propose a hypernetwork trained via meta-learning that generates per-column merging factors to combine LoRA modules. Given a small batch of target images, the hypernetwork produces merging weights that fuse source LoRA modules into a single adapted representation. Our results demonstrate state-of-the-art performance across various domain adaptation datasets.
BibTeX
@misc{reid2026da,
  author = {Siobhan Reid and Zhixiang Chi and Li Gu and Omid Reza Heidari and Ziqiang Wang and Yang Wang},
  title = {{DA-MergeLoRA: Hypernetwork-Based LoRA Merging for Few-Shot Test-Time Domain Adaptation}},
  year = {2026},
  eprint = {2607.17467},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2607.17467},
}
Figure