ABRA: Teleporting Fine-Tuned Knowledge Across Domains for Open-Vocabulary Object Detection

Mattia Bernardi, Chiara Cappellino, Matteo Mosconi, Enver Sangineto, Angelo Porrello, Simone Calderara

Abstract

Although recent Open-Vocabulary Object Detection (OVD) architectures, such as Grounding DINO, demonstrate strong zero-shot capabilities, their performance degrades significantly under domain shifts. Moreover, many domains of practical interest, such as nighttime or foggy scenes, lack large annotated datasets, preventing direct fine-tuning. In this paper, we introduce Aligned Basis Relocation for Adaptation (ABRA), a method that transfers class-specific detection knowledge from a labeled source domain to a target domain where no training images containing these classes are accessible. ABRA formulates this adaptation as a geometric transport problem in the weight space of a pretrained detector, aligning source and target domain experts to transport class-specific knowledge. Extensive experiments across challenging domain shifts demonstrate that ABRA successfully teleports class-level specialization under multiple adverse conditions.

What Problem Does This Paper Address?

Open-Vocabulary Object Detection (OVD) models are designed to detect almost anything based on natural language prompts. While they boast impressive zero-shot capabilities, they have a significant weakness: domain shift. A model trained on clear, sunny street scenes often fails when faced with nighttime, heavy rain, or thick fog.

The problem setup: transferring knowledge from a labeled source domain (closed set) to a target domain with different environmental conditions and unavailable classes (open set).

The standard fix is fine-tuning on the new domain, but this hits a practical wall. Collecting and annotating thousands of bounding boxes for every possible class in every possible weather condition is prohibitively expensive. Even worse, for rare classes, you might not have any images in the target domain to train on. The authors tackle a specific, difficult scenario: how do you detect a "bus" in the fog if your training data only shows buses in the sun, and your foggy data only contains "cars" and "trees"?

Methods

The authors propose a modular framework that disentangles where an object is (domain knowledge) from what an object is (class knowledge). Their method, ABRA, consists of three main stages:

  1. Objectification: To capture the visual statistics of a new domain (e.g., the texture of fog or low-light noise), they fine-tune a model using a class-agnostic approach. By relabeling common objects simply as "object," the model learns the environment's characteristics without category-specific distractions.
  2. Specialization (via SVFT): In the source domain (clear weather), they use Singular Vectors guided Fine-Tuning (SVFT) to learn lightweight "class experts." Instead of updating all weights, they only learn small residuals ($\Delta\Sigma$) on the singular values of the weight matrices, creating a compact mathematical representation of specific categories.
  3. Teleportation (Weight Alignment): This is the core innovation. The authors treat the difference between domains as a geometric mismatch in the model's weight space. Using a closed-form solution to the Orthogonal Procrustes problem, they calculate optimal rotation matrices ($L$ and $R$) to align the source domain's basis with the target domain's basis.
The ABRA teleportation mechanism aligns source and target domain bases ($L$ and $R$) to transport class-specific singular value updates ($\Delta\Sigma$).

By "rotating" the class-specific knowledge from the source into the target domain's "spectral basin," they effectively teleport the expertise without needing target-domain labels for those classes.

Key Results and Insights

The researchers evaluated ABRA on two primary benchmarks: Cityscapes $\rightarrow$ Foggy Cityscapes and SDGOD (covering Day-Clear, Day-Foggy, Dusk-Rainy, Night-Rainy, and Night-Clear conditions). Crucially, for the "unavailable" classes in the target domain, the model was given zero images and zero labels during the adaptation phase.

  • Superior Performance: ABRA consistently outperformed baselines like Task Arithmetic. On Foggy Cityscapes, ABRA achieved 40.5 mAP, significantly higher than the 27.7 mAP of the zero-shot Grounding DINO baseline, nearly matching the performance of models fully fine-tuned on foggy data (41.3 mAP).
  • Robustness to Adverse Conditions: In the SDGOD benchmark, ABRA showed its strongest gains in the difficult "Night-Rainy" split, nearly doubling the performance of the zero-shot baseline (16.1 mAP vs 9.2 mAP).
  • Disentanglement is Key: By separating the "domain expert" from the "class expert," knowledge becomes modular. ABRA proves that you don't need to see a specific object in a specific environment to detect it there, provided you have a general understanding of that environment's "visual language."

Conclusion

ABRA demonstrates that model adaptation can be viewed as a geometric rotation in weight space. While its performance is bounded by the quality of the source-domain experts, it provides a powerful path forward for detecting rare classes in challenging environments where data is scarce. Future work may explore more complex representations beyond orthogonal rotations to handle even more extreme environmental shifts.

Summary updated

Abstract

Although recent Open-Vocabulary Object Detection architectures, such as Grounding DINO, demonstrate strong zero-shot capabilities, their performance degrades significantly under domain shifts. Moreover, many domains of practical interest, such as nighttime or foggy scenes, lack large annotated datasets, preventing direct fine-tuning. In this paper, we introduce Aligned Basis Relocation for Adaptation(ABRA), a method that transfers class-specific detection knowledge from a labeled source domain to a target domain where no training images containing these classes are accessible. ABRA formulates this adaptation as a geometric transport problem in the weight space of a pretrained detector, aligning source and target domain experts to transport class-specific knowledge. Extensive experiments across challenging domain shifts demonstrate that ABRA successfully teleports class-level specialization under multiple adverse conditions. Our code will be made public upon acceptance.
BibTeX
@misc{bernardi2026abra,
  author = {Mattia Bernardi and Chiara Cappellino and Matteo Mosconi and Enver Sangineto and Angelo Porrello and Simone Calderara},
  title = {{ABRA: Teleporting Fine-Tuned Knowledge Across Domains for Open-Vocabulary Object Detection}},
  year = {2026},
  eprint = {2603.12409},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.12409},
}
Figure