Abstract
We propose SEAMLeSS (Siamese Encoding and Alignment by Multiscale Learning with Self-Supervision), a deep learning framework for dense image alignment. The method utilizes siamese convolutional networks to encode images into feature hierarchies, which are then processed by a sequence of aligner modules in a coarse-to-fine recursion. This approach addresses the limitations of traditional image pyramids, which often lose critical structural information at coarse scales. Trained in a self-supervised manner, SEAMLeSS incorporates a "masked" smoothness loss to effectively handle non-smooth deformations such as tissue cracks. Empirical results on electron microscopy (EM) data demonstrate that SEAMLeSS achieves superior accuracy and robustness compared to existing pyramid-based and one-shot alignment networks.
Introduction
Dense image alignment, or the estimation of a dense vector field that maps a source image to a target, is fundamental to computer vision. In connectomics, this task is essential for registering thousands of 2D EM sections to reconstruct 3D neural volumes. This domain presents unique challenges: large displacements between sections, significant information loss at low resolutions, and physical artifacts like cracks and folds that create discontinuous transforms.
Traditional coarse-to-fine methods rely on image pyramids. However, at the coarsest levels required to capture large displacements, thin neural processes often vanish, leading to irrecoverable alignment failures. Furthermore, standard self-supervised models enforce global smoothness, which prevents them from correctly "closing" cracks. SEAMLeSS is designed to overcome these hurdles through learned features and adaptive loss functions.
Methodology
Siamese Feature Hierarchies
Instead of simple downsampling (e.g., Gaussian pyramids), SEAMLeSS employs two identical, weight-sharing convolutional networks to encode the source and target images. These encoders produce a hierarchy of feature maps. Because these features are learned, the network can preserve structural cues that are lost in raw pixel downsampling. This ensures that the coarsest levels of the hierarchy still contain sufficient information to guide the initial alignment steps.
Recursive Aligner Modules
The alignment process is recursive. A module at the coarsest level estimates an initial displacement field. Each subsequent module in the hierarchy receives the field from the level above, upsamples it, and uses it to warp the source feature map. The module then computes a residual correction to improve the alignment. This architecture allows the model to resolve large-scale offsets before refining fine-grained details.
Self-Supervision and Masked Smoothness Loss
SEAMLeSS is trained without ground-truth labels by minimizing the Mean Squared Error (MSE) between the warped source and the target. To handle the "smoothness trap"—where models fail on discontinuous transforms—the authors introduce a masked smoothness penalty. By simulating cracks during training and deactivating the smoothness constraint at those locations, the network learns to allow sharp jumps in the displacement field, enabling the correct alignment of fractured tissue.
Experimental Results
The model was evaluated using the Pinky40 dataset, consisting of 1,000 EM sections from the mouse visual cortex. We compared SEAMLeSS against SPyNet (a pyramid-based approach) and FlowNet (a one-shot approach), using a human-corrected "Alembic" alignment as the gold standard.
Performance and Robustness
SEAMLeSS consistently outperformed both SPyNet and FlowNet across all metrics. Quantile-Quantile (QQ) plots demonstrate that SEAMLeSS is particularly effective in "worst-case" scenarios. While SPyNet performs reasonably well on simple alignments, its reliance on image pyramids causes it to fail significantly more often on complex pairs where structural information is sparse at coarse scales.
Efficiency and Crack Recovery
Beyond accuracy, SEAMLeSS offers significant practical advantages. Once trained, the network is approximately two orders of magnitude faster than direct optimization methods. Furthermore, the masked loss allowed the model to successfully resolve real physical cracks in the tissue, a task that standard smooth-transform models cannot perform.
Conclusion
SEAMLeSS represents a significant advancement in dense image alignment for high-resolution microscopy. By combining learned feature hierarchies with a recursive, self-supervised training regime, it provides a robust solution to the problems of large displacements and non-smooth deformations. Its speed and accuracy make it an ideal candidate for the massive datasets generated in modern connectomics.
Summary updated