Domain Adaptation for Large-Vocabulary Object Detectors

Kai Jiang, Jiaxing Huang, Weiying Xie, Yunsong Li, Ling Shao, Shijian Lu

Abstract

Large-vocabulary object detectors (LVDs) demonstrate remarkable localization capabilities but often struggle with classification accuracy when deployed in novel domains due to distribution bias and vocabulary shifts. This paper presents Knowledge Graph Distillation (KGD), a novel framework that leverages the rich semantic and visual knowledge embedded in vision-language models like CLIP. By constructing and distilling Language Knowledge Graphs (LKG) and Vision Knowledge Graphs (VKG), KGD effectively bridges the domain gap. Experimental results across 11 diverse benchmarks, including autonomous driving and artistic domains, show that KGD significantly outperforms existing state-of-the-art unsupervised domain adaptation methods.

Introduction

Large-vocabulary object detectors (LVDs) are designed to identify thousands of object categories, far exceeding the scope of standard datasets like COCO. While these models develop a robust "super objectness"—the ability to locate objects across various environments—their classification performance often degrades significantly when moved to a new "downstream" domain. This performance drop is primarily driven by domain discrepancy: visual appearances change due to lighting or weather (distribution bias), and category definitions may shift (vocabulary shift).

Existing unsupervised domain adaptation (UDA) methods typically rely on pseudo-labels generated by the model itself. However, if the initial model is confused by the new domain, these labels become noisy, leading to suboptimal training. To address this, the researchers introduce Knowledge Graph Distillation (KGD), which extracts and structures the implicit knowledge within vision-language models to guide the adaptation process.

Methodology: Knowledge Graph Distillation (KGD)

The KGD framework operates by treating the knowledge within CLIP not merely as a lookup table, but as a structured network of relationships. The process is divided into two primary components: Language Knowledge Graph (LKG) and Vision Knowledge Graph (VKG).

The KGD framework extracts semantic knowledge from WordNet/CLIP (LKG) and visual relationships from the target domain (VKG) to generate robust pseudo-labels.

Language Knowledge Graph (LKG)

To resolve vocabulary issues, KGD utilizes WordNet to retrieve definitions and hyponyms (specific sub-types) for every category in the target domain. These rich text descriptions form the nodes of a graph. A Graph Convolutional Network (GCN) then processes these nodes, allowing the model to reason through hierarchies—for instance, identifying that a visual feature resembling a "sedan" likely belongs to the "car" category based on linguistic relationships.

Vision Knowledge Graph (VKG)

To mitigate visual shifts, KGD constructs a dynamic graph based on images from the target domain. It calculates "visual centroids"—the average feature vectors for detected categories in the new environment. Through manifold smoothing, the framework ensures that visually similar categories remain close in the feature space, enhancing robustness against environmental noise such as rain or fog.

Experiments and Results

The authors evaluated KGD across 11 distinct benchmarks, including autonomous driving (Cityscapes, BDD100K), surveillance (VisDrone, MIO-TCD), and artistic styles (Clipart1k, Comic2k). The inclusion of artistic datasets tests the model's ability to handle extreme style shifts far removed from natural photography.

Radar chart comparing KGD against state-of-the-art methods across 11 diverse benchmarks, illustrating KGD's superior performance in varied conditions.

KGD consistently outperformed the baseline (Detic) and several state-of-the-art domain adaptation methods like SHOT and IRG-SFDA. Notable results include:

  • Autonomous Driving: On Cityscapes, KGD achieved 53.6 AP50, a +7.1 point improvement over the baseline. In the BDD100K "Foggy" subset, performance improved by nearly 10 points.
  • Artistic Adaptation: On the Comic2k dataset, KGD reached 55.6 AP50, significantly higher than the baseline's 51.2.

Ablation studies confirmed that the LKG and VKG are complementary; while each improves performance individually, their combination yields the most significant gains.

Conclusion

Knowledge Graph Distillation provides a powerful mechanism for adapting large-vocabulary detectors to new domains by making the implicit knowledge of vision-language models explicit and structured. By anchoring the model with stable linguistic hierarchies and flexible visual context, KGD overcomes the limitations of noisy pseudo-labeling. Future work may focus on reducing the computational overhead of graph processing or extending the framework to real-time, "on-the-fly" domain adaptation.

Summary updated

BibTeX
@inproceedings{jiang2024domain,
  author = {Kai Jiang and Jiaxing Huang and Weiying Xie and Yunsong Li and Ling Shao and Shijian Lu},
  title = {{Domain Adaptation for Large-Vocabulary Object Detectors}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=deZpmEfmTo},
}
Figure