Introduction
Deploying autonomous systems in diverse real-world environments requires robust dense prediction capabilities, such as semantic segmentation and depth estimation. However, these models often suffer from domain shift when moving from training (e.g., simulation) to deployment (e.g., real-world streets). The FAMDA (Foundation model Assisted Multi-task unsupervised Domain Adaptation) framework addresses this by leveraging large-scale Vision Foundation Models (VFMs) to guide the training of efficient, multi-task student models.
Problem Statement
Robots and autonomous vehicles rely on identifying objects and their distances. While models perform well in controlled settings, performance degrades significantly under domain shift (e.g., weather changes or location transitions). Unsupervised Domain Adaptation (UDA) aims to bridge this gap without manual labeling. Traditional multi-task UDA methods often use adversarial learning, which can be unstable and produce low-quality results, especially for lightweight models suitable for edge robotics.
The FAMDA Framework
The proposed FAMDA framework utilizes a teacher-student self-training paradigm, where massive VFMs act as "super-teachers" to provide high-quality pseudo-labels for the target domain.
The framework consists of two primary pipelines:
- Semantic Segmentation Pipeline: A standard teacher-student setup is enhanced by the Segment Anything Model (SAM). SAM provides precise object boundaries to refine the noisy pseudo-labels generated by the teacher network through majority voting within segments.
- Depth Estimation Pipeline: High-quality pseudo-depth maps are generated using the Depth Anything Model (DAM). The student model learns to mimic these signals, calibrated via a scale-and-shift invariant loss to ensure physically meaningful distance measurements.
This approach allows lightweight backbones, such as MiT-B0, to inherit the knowledge of giant foundation models, achieving high accuracy with minimal computational overhead.
Experimental Setup
FAMDA was evaluated on several benchmarks:
- Synthetic-to-Real: Adapting from SYNTHIA and Virtual KITTI 2 (VK2) to the Cityscapes dataset.
- Day-to-Night: A new "DomiNite" dataset was introduced to test adaptation from daytime Cityscapes to challenging low-light nighttime environments.
Results and Discussion
FAMDA achieves state-of-the-art performance across standard benchmarks, significantly outperforming previous multi-task UDA methods like XTAM and VTAGML.
Efficiency and Real-Time Performance
A key advantage of FAMDA is its efficiency. The MiT-B2 variant is 10× smaller than Depth Anything and 27× smaller than SAM, yet it maintains competitive accuracy. On edge hardware like the NVIDIA Jetson Orin Nano, the models achieved a latency of ~28.9 ms (~7.7 Hz), enabling real-time robotics applications.
Adaptation vs. Zero-Shot
The study demonstrates that domain-specific adaptation via FAMDA outperforms zero-shot application of giant foundation models. Specialized student models learn local sensor nuances better than general-purpose giants, particularly in difficult conditions like nighttime imagery.
Conclusion and Future Work
FAMDA successfully distills the power of large-scale VFMs into efficient multi-task models for domain adaptation. While highly effective, future work could explore modeling explicit inter-task dependencies (e.g., cross-attention between segmentation and depth) and reducing the offline computational cost of generating VFM pseudo-labels.
Summary updated