Siamese Encoding and Alignment by Multiscale Learning with Self-Supervision

Eric Mitchell, Stefan Keselj, Sergiy Popovych, Davit Buniatyan, H. Sebastian SeungarXiv 2019 · OpenReview

Abstract

We propose SEAMLeSS (Siamese Encoding and Alignment by Multiscale Learning with Self-Supervision), a deep learning framework for dense image alignment. The method utilizes siamese convolutional networks to encode images into feature hierarchies, which are then processed by a sequence of aligner modules in a coarse-to-fine recursion. This approach addresses the limitations of traditional image pyramids, which often lose critical structural information at coarse scales. Trained in a self-supervised manner, SEAMLeSS incorporates a "masked" smoothness loss to effectively handle non-smooth deformations such as tissue cracks. Empirical results on electron microscopy (EM) data demonstrate that SEAMLeSS achieves superior accuracy and robustness compared to existing pyramid-based and one-shot alignment networks.

Introduction

Dense image alignment, or the estimation of a dense vector field that maps a source image to a target, is fundamental to computer vision. In connectomics, this task is essential for registering thousands of 2D EM sections to reconstruct 3D neural volumes. This domain presents unique challenges: large displacements between sections, significant information loss at low resolutions, and physical artifacts like cracks and folds that create discontinuous transforms.

Traditional coarse-to-fine methods rely on image pyramids. However, at the coarsest levels required to capture large displacements, thin neural processes often vanish, leading to irrecoverable alignment failures. Furthermore, standard self-supervised models enforce global smoothness, which prevents them from correctly "closing" cracks. SEAMLeSS is designed to overcome these hurdles through learned features and adaptive loss functions.

Methodology

Siamese Feature Hierarchies

Instead of simple downsampling (e.g., Gaussian pyramids), SEAMLeSS employs two identical, weight-sharing convolutional networks to encode the source and target images. These encoders produce a hierarchy of feature maps. Because these features are learned, the network can preserve structural cues that are lost in raw pixel downsampling. This ensures that the coarsest levels of the hierarchy still contain sufficient information to guide the initial alignment steps.

Recursive Aligner Modules

The alignment process is recursive. A module at the coarsest level estimates an initial displacement field. Each subsequent module in the hierarchy receives the field from the level above, upsamples it, and uses it to warp the source feature map. The module then computes a residual correction to improve the alignment. This architecture allows the model to resolve large-scale offsets before refining fine-grained details.

The SEAMLeSS architecture: (a) hierarchical siamese encoding and (b) the recursive aligner module for coarse-to-fine displacement estimation.

Self-Supervision and Masked Smoothness Loss

SEAMLeSS is trained without ground-truth labels by minimizing the Mean Squared Error (MSE) between the warped source and the target. To handle the "smoothness trap"—where models fail on discontinuous transforms—the authors introduce a masked smoothness penalty. By simulating cracks during training and deactivating the smoothness constraint at those locations, the network learns to allow sharp jumps in the displacement field, enabling the correct alignment of fractured tissue.

Experimental Results

The model was evaluated using the Pinky40 dataset, consisting of 1,000 EM sections from the mouse visual cortex. We compared SEAMLeSS against SPyNet (a pyramid-based approach) and FlowNet (a one-shot approach), using a human-corrected "Alembic" alignment as the gold standard.

Performance and Robustness

SEAMLeSS consistently outperformed both SPyNet and FlowNet across all metrics. Quantile-Quantile (QQ) plots demonstrate that SEAMLeSS is particularly effective in "worst-case" scenarios. While SPyNet performs reasonably well on simple alignments, its reliance on image pyramids causes it to fail significantly more often on complex pairs where structural information is sparse at coarse scales.

Quantile-Quantile plot comparing the performance of SEAMLeSS, SpyNet, and FlowNet against the Alembic gold-standard benchmark. SEAMLeSS (blue) shows the closest alignment to the ideal diagonal, indicating superior robustness in challenging cases.

Efficiency and Crack Recovery

Beyond accuracy, SEAMLeSS offers significant practical advantages. Once trained, the network is approximately two orders of magnitude faster than direct optimization methods. Furthermore, the masked loss allowed the model to successfully resolve real physical cracks in the tissue, a task that standard smooth-transform models cannot perform.

Conclusion

SEAMLeSS represents a significant advancement in dense image alignment for high-resolution microscopy. By combining learned feature hierarchies with a recursive, self-supervised training regime, it provides a robust solution to the problems of large displacements and non-smooth deformations. Its speed and accuracy make it an ideal candidate for the massive datasets generated in modern connectomics.

Summary updated

Abstract

We propose a method of aligning a source image to a target image, where the transform is specified by a dense vector field. The two images are encoded as feature hierarchies by siamese convolutional nets. Then a hierarchy of aligner modules computes the transform in a coarse-to-fine recursion. Each module receives as input the transform that was computed by the module at the level above, aligns the source and target encodings at the same level of the hierarchy, and then computes an improved approximation to the transform using a convolutional net. The entire architecture of encoder and aligner nets is trained in a self-supervised manner to minimize the squared error between source and target remaining after alignment. We show that siamese encoding enables more accurate alignment than the image pyramids of SPyNet, a previous deep learning approach to coarse-to-fine alignment. Furthermore, self-supervision applies even without target values for the transform, unlike the strongly supervised SPyNet. We also show that our approach outperforms one-shot approaches to alignment, because the fine pathways in the latter approach may fail to contribute to alignment accuracy when displacements are large. As shown by previous one-shot approaches, good results from self-supervised learning require that the loss function additionally penalize non-smooth transforms. We demonstrate that "masking out" the penalty function near discontinuities leads to correct recovery of non-smooth transforms. Our claims are supported by empirical comparisons using images from serial section electron microscopy of brain tissue.
BibTeX
@misc{mitchell2019siamese,
  author = {Eric Mitchell and Stefan Keselj and Sergiy Popovych and Davit Buniatyan and H. Sebastian Seung},
  title = {{Siamese Encoding and Alignment by Multiscale Learning with Self-Supervision}},
  year = {2019},
  eprint = {1904.02643},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1904.02643},
}
Figure