Image Alignment in Unseen Domains via Domain Deep Generalization

Thanh-Dat Truong, Khoa Luu, Chi Nhan Duong, Ngan Le, Minh-Triet TranarXiv 2019 · arXiv

Abstract

This paper introduces DeGIA (Deep learning based Generalized Image Alignment), a framework designed to align images in unseen target domains (e.g., Depth or X-Ray) using models trained exclusively on a single source domain (e.g., RGB). By integrating discriminative feature extraction with deep generative flows and an adversarial training strategy, DeGIA achieves robust domain generalization. Experimental results demonstrate significant performance gains over standard CNN architectures, particularly in cross-modal medical and hand-pose estimation tasks.

Introduction

Image alignment—the task of identifying specific landmark points on an object—is a fundamental problem in computer vision. While performance is high when training and testing data share the same distribution, real-world applications often face domain shift. In medical imaging, for instance, labeled RGB data may be abundant, while labeled X-ray or Depth maps are scarce.

Traditional domain adaptation requires target-domain data during training. However, this paper addresses the more challenging task of Domain Generalization, where the model must perform on entirely new modalities unavailable during the training phase. This is essential for privacy-sensitive or resource-constrained environments.

Overview of the domain generalization task: training on RGB images and testing on unseen Depth and X-Ray domains.

Methodology: The DeGIA Framework

The proposed Deep learning based Generalized Image Alignment (DeGIA) framework is an end-to-end architecture comprising two main components:

  1. Discriminative Template Feature Extraction: A CNN backbone (e.g., ResNet, VGG) trained to identify landmarks by minimizing Mean Squared Error (MSE).
  2. Deep Generative Flows: These map images into a learned latent space where data is represented as Gaussian distributions. This space is more "domain-invariant" than raw pixel values, focusing on structural features rather than stylistic noise.

Training Strategy

DeGIA employs a two-phase optimization process:

  • Minimization Phase: The network learns to align landmarks on source data while maximizing the log-likelihood of the generative flow.
  • Maximization (Perturbation) Phase: The model generates "hard" adversarial samples to maximize loss. This forces the model to become robust against distortions and shifts likely to be encountered in unseen domains.

Importantly, the generative flow is used only for regularization during training; inference speed at test time remains comparable to a standard CNN.

Experimental Evaluation

The authors evaluated DeGIA across several benchmarks to test its generalization capabilities:

  • MNIST to MNIST-M: A proof-of-concept where the model trained on standard digits and was tested on digits with complex backgrounds.
  • NYU Hand Pose Dataset: The primary testbed where models were trained on RGB images and tested on Depth and X-Ray modalities.

Key Results and Discussion

DeGIA demonstrated a significant performance boost in unseen domains:

  • Digit Recognition: Improved accuracy on MNIST-M by 6.47% compared to a standard CNN.
  • Hand Alignment Precision: In RGB $\rightarrow$ Depth tests using a ResNet backbone, the MSE dropped from 0.2596 to 0.1065, indicating a massive improvement in precision.
  • Cross-Modal Robustness: The model maintained high accuracy on X-ray images despite their significant visual difference from the RGB training set.
Successful landmark alignment on unseen X-Ray images using a model trained only on RGB data.

The core insight is that mapping features to a probabilistic latent space allows the model to prioritize object structure over domain-specific pixel values. Adversarial training further simulates the variability of unseen domains, enhancing robustness.

Conclusion

DeGIA provides a flexible, backbone-agnostic framework for image alignment that generalizes to unseen domains without requiring target-domain training data. While the training phase is more computationally intensive due to the maximization phase, the resulting models offer high precision and fast inference. Future work may extend this approach to other medical tasks such as segmentation and lesion detection.

Summary updated

Abstract

Image alignment across domains has recently become one of the realistic and popular topics in the research community. In this problem, a deep learning-based image alignment method is usually trained on an available largescale database. During the testing steps, this trained model is deployed on unseen images collected under different camera conditions and modalities. The delivered deep network models are unable to be updated, adapted or fine-tuned in these scenarios. Thus, recent deep learning techniques, e.g. domain adaptation, feature transferring, and fine-tuning, are unable to be deployed. This paper presents a novel deep learning based approach to tackle the problem of across unseen modalities. The proposed network is then applied to image alignment as an illustration. The proposed approach is designed as an end-to-end deep convolutional neural network to optimize the deep models to improve the performance. The proposed network has been evaluated in digit recognition when the model is trained on MNIST and then tested on unseen domain MNIST-M. Finally, the proposed method is benchmarked in image alignment problem when training on RGB images and testing on Depth and X-Ray images.
BibTeX
@misc{truong2019image,
  author = {Thanh-Dat Truong and Khoa Luu and Chi Nhan Duong and Ngan Le and Minh-Triet Tran},
  title = {{Image Alignment in Unseen Domains via Domain Deep Generalization}},
  year = {2019},
  eprint = {1905.12028},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1905.12028},
}
Figure