Abstract
Computing dense pixel-to-pixel image correspondences is a fundamental task of computer vision. Often, the objective is to align image pairs from the same semantic category for manipulation or segmentation purposes. Despite achieving superior performance, existing deep learning alignment methods cannot cluster images; consequently, clustering and pairing images needed to be a separate laborious and expensive step. Given a dataset with diverse semantic categories, we propose a multi-task model, Jim-Net, that can directly learn to cluster and align images without any pixel-level or image-level annotations. We design a pair-matching alignment unsupervised training algorithm that selectively matches and aligns image pairs from the clustering branch. Our unsupervised Jim-Net achieves comparable accuracy with state-of-the-art supervised methods on benchmark 2D image alignment dataset PF-PASCAL. Specifically, we apply Jim-Net to cryo-electron tomography, a revolutionary 3D microscopy imaging technique of native subcellular structures. After extensive evaluation on seven datasets, we demonstrate that Jim-Net enables systematic discovery and recovery of representative macromolecular structures in situ, which is essential for revealing molecular mechanisms underlying cellular functions. To our knowledge, Jim-Net is the first end-to-end model that can simultaneously align and cluster images, which significantly improves the performance as compared to performing each task alone.
Introduction
In computer vision, image alignment (finding pixel-to-pixel correspondences) and clustering (grouping images by category) are traditionally treated as separate problems. However, they are deeply intertwined: to align two images accurately, they generally need to belong to the same semantic category. Conversely, robust clustering often requires objects to be aligned so that their features can be compared fairly.
This "chicken-and-egg" problem is particularly acute in cryo-electron tomography (cryo-ET). Cryo-ET allows scientists to see 3D snapshots of molecules inside living cells, but the data is incredibly noisy and contains a "soup" of different molecular structures. Manually sorting and aligning thousands of these 3D volumes is a massive bottleneck for biological discovery. Existing tools either do one or the other, often requiring laborious human intervention or failing when the data is too diverse.
Methodology
The authors introduce Jim-Net (Joint image alignment and clustering Network), a multi-task framework designed to handle completely unlabeled datasets.
Dual-Branch Architecture
The model consists of an alignment branch and a clustering branch that share a common feature extractor. This sharing ensures that the features learned are useful for both geometric matching and semantic separation. The architecture utilizes feature extractors, spatial transformers, and a classifier to process source and target images through shared and task-specific paths.
Coarse-to-Fine Alignment
Instead of trying to align images in one giant leap, Jim-Net uses a multi-stage approach. It starts with coarse resolutions to handle large rotations and shifts and progressively moves to finer resolutions for precision.
Unsupervised Pair-Matching
The model uses a Gaussian Mixture Model (GMM) on its own learned features to predict cluster assignments. It then selectively picks pairs of images from the same predicted cluster to train the alignment branch. This "mutual reinforcement" is key: clustering provides the alignment branch with "clean" pairs, while the alignment process forces the feature extractor to learn robust, pose-invariant representations.
Results
Jim-Net was evaluated across seven datasets, including simulated cryo-ET subtomograms, the 2D PF-PASCAL benchmark, and real-world biological data.
- 3D Alignment and Clustering: On real-world protein data, Jim-Net achieved a 96.6% clustering accuracy, significantly outperforming standalone methods like DeepCluster (57.7%) or PICA (60.5%).
- 2D Computer Vision Benchmarks: On the PF-PASCAL dataset, Jim-Net’s unsupervised approach achieved a Percentage of Correct Keypoints (PCK) of 74.8, comparable to many weakly-supervised methods that use image labels during training.
- Biological Discovery: When applied to real bacteria cells, Jim-Net successfully discovered and reconstructed structures like ribosomes and phycobilisome arrays at resolutions better than 40 Å—a feat previously impossible with fully automated, unsupervised methods.
Conclusion
Jim-Net represents the first end-to-end model capable of simultaneous unsupervised image alignment and clustering. By overcoming the "trivial solution" problem common in unsupervised learning—where models might otherwise group all images into a single cluster—Jim-Net enables a new workflow for "visual proteomics." This allows for the systematic discovery of macromolecular structures directly inside their native cellular environment. Future work will focus on incorporating non-rigid deformations and optimizing computational efficiency for larger cellular atlases.
Summary updated