Abstract
Aiming towards human-level generalization, there is a need to explore adaptable representation learning methods with greater transferability. Most existing approaches independently address task-transferability and cross-domain adaptation, resulting in limited generalization. In this paper, we propose UM-Adapt - a unified framework to effectively perform unsupervised domain adaptation for spatially-structured prediction tasks, simultaneously maintaining a balanced performance across individual tasks in a multi-task setting. To realize this, we propose two novel regularization strategies; a) Contour-based content regularization (CCR) and b) exploitation of inter-task coherency using a cross-task distillation module. Furthermore, avoiding a conventional ad-hoc domain discriminator, we re-utilize the cross-task distillation loss as output of an energy function to adversarially minimize the input domain discrepancy. Through extensive experiments, we demonstrate superior generalizability of the learned representations simultaneously for multiple tasks under domain-shifts from synthetic to natural environments. UM-Adapt yields state-of-the-art transfer learning results on ImageNet classification and comparable performance on PASCAL VOC 2007 detection task, even with a smaller backbone-net. Moreover, the resulting semi-supervised framework outperforms the current fully-supervised multi-task learning state-of-the-art on both NYUD and Cityscapes dataset.
Introduction
Deep learning models are often "brittle," failing when deployed in real-world environments due to domain shift—the discrepancy between training (often synthetic) and deployment (natural) data. While Unsupervised Domain Adaptation (UDA) aims to mitigate this without target-domain labels, most research focuses on single-task classification. Spatially-structured tasks, such as depth estimation and semantic segmentation, present unique challenges due to high-dimensional latent features and the difficulty of balancing multiple tasks simultaneously.
UM-Adapt addresses these challenges by asking: Can we use the inherent relationship between different tasks to help the model adapt to a new domain? By enforcing consistency between tasks and using image contours as a structural anchor, the framework learns robust representations that transfer effectively from synthetic to real-world data.
Methodology
The UM-Adapt framework relies on three core components designed to maintain spatial coherence and inter-task consistency during adaptation.
Contour-based Content Regularization (CCR)
While colors and textures vary significantly between synthetic and natural domains, underlying geometric contours remain relatively consistent. CCR utilizes a shallow 4-layer decoder to predict image contours, acting as a structural anchor that prevents the model from losing spatial coherence during the adaptation process.
Cross-Task Distillation
The framework trains "task-transfer networks" ($N \to t_i$) that attempt to predict the output of one task (e.g., depth) using the outputs of other tasks (e.g., segmentation and surface normals) as input. This exploits the natural correlations between tasks—for instance, depth discontinuities often align with semantic boundaries.
Adversarial Energy-Based Adaptation
The most novel aspect of UM-Adapt is the re-purposing of the cross-task distillation loss as an energy function for adversarial training. On the source (synthetic) domain, tasks are consistent, resulting in low energy. On the target (natural) domain, domain shift causes task inconsistency and high energy. The model is trained adversarially to minimize this energy on the target domain, effectively forcing the tasks to become coherent and, by extension, domain-aligned. This approach avoids the need for a standard ad-hoc binary discriminator, which often struggles with high-dimensional spatial features.
Experimental Results
The framework was evaluated on both indoor and outdoor datasets, including PBRS to NYUD-v2 and GTA5 to Cityscapes/KITTI.
- State-of-the-Art Transfer: UM-Adapt achieved 69.51% top-5 accuracy on ImageNet transfer learning, outperforming several dedicated self-supervised methods despite using a smaller ResNet-50 backbone.
- Superior Adaptation: On the NYUD-v2 dataset, unsupervised UM-Adapt outperformed previous UDA methods in depth estimation, achieving an RMS error of 0.673 compared to AdaDepth's 0.744.
- Semi-Supervised Performance: When provided with limited labeled target samples, the semi-supervised variant (UM-Adapt-S) outperformed current fully-supervised multi-task state-of-the-art models on both NYUD and Cityscapes.
Conclusion
UM-Adapt demonstrates that inter-task coherency is a powerful supervisory signal for unsupervised domain adaptation. By treating tasks as interdependent rather than isolated, and anchoring learning in domain-invariant contours, the framework achieves state-of-the-art generalization across diverse spatially-structured tasks. Future research will explore the use of auxiliary tasks to further enhance the transferability of learned representations.
Summary updated