Arbicon-Net: Arbitrary Continuous Geometric Transformation Networks for Image Registration

Jianchun Chen, Lingjing Wang, Xiang Li, Yi Fang

Image registration—the process of aligning two different images of the same scene or object—is a cornerstone of computer vision. Whether you are stitching panoramas, tracking a medical tumor across MRI scans, or matching a product photo to a real-world scene, you need to find a geometric transformation that maps one image onto the other.

In a paper presented at NeurIPS, researchers introduced Arbicon-Net (Arbitrary Continuous Geometric Transformation Networks), a novel approach that moves away from rigid, "hand-crafted" transformation models in favor of a more flexible, data-driven architecture.

Examples of Arbicon-Net aligning source images to target images via a predicted continuous transformation.

TL;DR

Arbicon-Net is an end-to-end deep learning framework for image registration that directly predicts dense displacement fields rather than parameters for fixed models like Affine or Thin-Plate Spline (TPS). By design, the network architecture guarantees that these transformations are spatially continuous and smooth, allowing it to handle complex, "arbitrary" deformations that stymie traditional methods.

The Tug-of-War: Flexibility vs. Smoothness

Historically, image registration has struggled to balance two competing needs: flexibility and smoothness.

  1. Parametric Models: Many recent deep learning methods try to predict the parameters of a pre-defined model (like an Affine transform or a Thin-Plate Spline). While these models are "safe" and won't tear the image apart, they are low-dimensional. They simply cannot represent the highly complex, non-linear deformations found in real-world semantic alignment.
  2. Flow-Based Methods: Other methods estimate "semantic flow" by predicting a movement vector for every single pixel. While highly flexible, these often lack global coherence. Without complex, manually-tuned "smoothness penalties" in the loss function, the resulting images often look distorted or physically impossible.

The authors of Arbicon-Net asked: Can we create a network that is as flexible as flow-based methods but inherently as smooth as parametric ones?

How Arbicon-Net Works

Arbicon-Net breaks the registration task into three distinct modules:

The Arbicon-Net architecture, detailing the Feature Extractor, Transformation Descriptor Encoder, and Displacement Field Predictor.
  1. Geometric Feature Extractor: A shared ResNet backbone extracts dense feature maps from the two input images (source and target).
  2. Transformation Descriptor Encoder: The network computes a 4D correlation tensor to find similarities between all pairs of points in the two images. This tensor is refined using 4D convolutions to capture neighborhood context and then compressed into a single global transformation descriptor ($d_{AB}$). This descriptor represents the "essence" of the required movement.
  3. Displacement Field Predictor: This is the core innovation. Instead of outputting a grid of vectors, the network uses a Multi-Layer Perceptron (MLP). For every coordinate $(x, y)$ in the source image, the MLP takes the coordinate plus the global descriptor $d_{AB}$ and predicts the displacement.

The "Aha!" Moment: Because the displacement is a function of coordinates processed through a continuous MLP with smooth activation functions (like SoftPlus), the resulting displacement field is mathematically guaranteed to be continuous and smooth. You don't need to add a complex "smoothness term" to your loss function; the architecture itself prevents the image from "breaking."

Key Insights

  • Architecture as Regularization: The most significant insight is that smoothness can be an architectural property rather than a constraint. By treating the displacement field as a continuous function of coordinates, the model avoids the "checkerboard" artifacts or discontinuities common in pixel-wise flow estimation.
  • Global-to-Local Conditioning: By conditioning local point movement on a global transformation descriptor, the network ensures that every point moves in a way that is "aware" of the rest of the image.
  • Overcoming the "Curse of Dimensionality": Traditional flow methods often struggle with high-dimensional feature maps that oscillate. Arbicon-Net’s use of a global descriptor helps stabilize the learning process.

Datasets and Results

The researchers evaluated Arbicon-Net using several standard benchmarks, including Pascal VOC (for synthesized pre-training), PF-Pascal, Proposal Flow, and a custom Synthesized Diffeomorphic dataset to test complex, non-linear warps.

Arbicon-Net demonstrated superior performance across both synthetic and real-world tests:

  • State-of-the-Art Accuracy: On the PF-Pascal dataset, Arbicon-Net achieved a PCK (Probability of Correct Keypoint) of 77.3%, outperforming previous leaders like WeakAlign (75.8%) and CNNGeo (71.9%).
  • Handling Complexity: In tests involving complex non-linear transformations, Arbicon-Net achieved a Mean Squared Error (MSE) of 0.0002, nearly 20 times better than the 0.0037 achieved by CNNGeo-4D.
  • Qualitative Smoothness: Visual results show that Arbicon-Net preserves the structural integrity of objects (like the legs of a chair or the frame of a bicycle) much better than flow-based methods, which often "shred" the object during alignment.

Limitations and Future Work

While Arbicon-Net is a leap forward, the paper notes some areas for growth:

  • Computational Cost: Processing 4D correlation tensors is memory-intensive, which may limit the resolution of the feature maps the network can handle.
  • 2D Focus: The current work is focused on 2D image registration. Extending this "continuous function" approach to 3D volumes (essential for medical imaging) is a promising but unexplored direction.
  • Weak Supervision: While the model works well with weak supervision, further refining how the global descriptor is encoded could lead to even better performance in "unseen" categories.

In summary, Arbicon-Net provides an elegant solution to a long-standing problem in computer vision by proving that we don't have to choose between a model's flexibility and its physical plausibility.

Summary updated

Abstract

This paper concerns the undetermined problem of estimating geometric transformation between image pairs. Recent methods introduce deep neural networks to predict the controlling parameters of hand-crafted geometric transformation models (e.g. thin-plate spline) for image registration and matching. However, the low-dimension parametric models are incapable of estimating a highly complex geometric transform with limited flexibility to model the actual geometric deformation from image pairs. To address this issue, we present an end-to-end trainable deep neural networks, named Arbitrary Continuous Geometric Transformation Networks (Arbicon-Net), to directly predict the dense displacement field for pairwise image alignment. Arbicon-Net is generalized from training data to predict the desired arbitrary continuous geometric transformation in a data-driven manner for unseen new pair of images. Particularly, without imposing penalization terms, the predicted displacement vector function is proven to be spatially continuous and smooth. To verify the performance of Arbicon-Net, we conducted semantic alignment tests over both synthetic and real image dataset with various experimental settings. The results demonstrate that Arbicon-Net outperforms the previous image alignment techniques in identifying the image correspondences.
BibTeX
@inproceedings{chen2019arbicon,
  author = {Jianchun Chen and Lingjing Wang and Xiang Li and Yi Fang},
  title = {{Arbicon-Net: Arbitrary Continuous Geometric Transformation Networks for Image Registration}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
}
Figure