Abstract
Volumetric (3d) images are acquired for many scientific and biomedical purposes using imaging methods such as serial section microscopy, CT scans, and MRI. A frequent step in the analysis and reconstruction of such data is the alignment and registration of images that were acquired in succession along a spatial or temporal dimension. For example, in serial section electron microscopy, individual 2d sections are imaged via electron microscopy and then must be aligned to one another in order to produce a coherent 3d volume. State of the art approaches find image correspondences derived from patch matching and invariant feature detectors, and then solve optimization problems that rigidly or elastically deform series of images into an aligned volume. Here we show how fully convolutional neural networks trained with an adversarial loss function can be used for two tasks: (1) synthesis of missing or damaged image data from adjacent sections, and (2) fine-scale alignment of block-face electron microscopy data. Finally, we show how these two capabilities can be combined in order to produce artificial isotropic volumes from anisotropic image volumes using a super-resolution adversarial alignment and interpolation approach.
Introduction
In biological imaging, particularly Serial Block-face Electron Microscopy (SBEM), researchers aim to reconstruct high-resolution 3D volumes of tissues like the mouse neocortex. However, the acquisition process is inherently imperfect. SBEM typically yields a stack of 2D "photographs" that suffer from two primary limitations: anisotropy and alignment jitter.
Anisotropy refers to the disparity in resolution; while the in-plane (XY) resolution is often very high (e.g., 11.24 nm), the resolution along the cutting axis (Z) is significantly lower (e.g., 28 nm). Alignment jitter consists of sub-pixel or pixel-level shifts between consecutive slices caused by mechanical instabilities during imaging. Traditional alignment methods rely on manual labeling or complex patch-matching optimizations, which are difficult to scale. This work proposes an unsupervised approach using Generative Adversarial Networks (GANs) to simultaneously interpolate missing data and correct fine-scale alignment.
Methodology: The Adversarial Framework
The core of the proposed method is a GAN consisting of a Generator and a Discriminator. The Generator is a fully convolutional network designed for two specific tasks: interpolation (synthesizing a missing slice $k$ from neighboring slices $k-1$ and $k+1$) and alignment (processing a 3D stack to produce a corrected, jitter-free volume).
The "Isotropy Trick"
A key innovation in this approach is the "isotropy trick" used for alignment. In a perfectly aligned biological volume, a side-view (the $XZ$ or $YZ$ planes) should be statistically indistinguishable from the top-down imaging plane ($XY$). The Discriminator is trained to distinguish between "real" $XY$ slices and "fake" $XZ/YZ$ reslices produced by the Generator. To fool the Discriminator, the Generator must learn to shift and deform the 3D data until the side-views exhibit the same sharpness and continuity as the original imaging plane.
Results and Discussion
The model was evaluated using the SegEM dataset (mouse S1 neocortex), with training performed on a 200 x 200 x 150 voxel volume and testing on an out-of-sample volume of the same size. Results demonstrate that the choice of loss function is critical for visual fidelity.
Pixel-Loss vs. Adversarial Loss
Standard training using Mean Squared Error (pixel-loss) often results in blurry images because the model averages potential outcomes to minimize error. In contrast, adversarial loss encourages the model to produce sharp edges and realistic textures, including simulated shot noise. As shown in the interpolation experiments, slices generated with adversarial loss are nearly indistinguishable from real data to the human eye, successfully capturing complex ultrastructural boundaries.
Alignment and Super-Resolution
The combined super-resolution pipeline utilizes transposed convolutions to upsample the Z-axis. This allows the model to simultaneously fill in gaps and fix jitter, moving the data closer to true isotropy. Side-view reslices of the processed volumes showed a marked reduction in "stair-stepping" artifacts and discontinuities compared to the raw data.
Conclusion and Future Work
While the adversarial approach excels at fine-scale jitter correction and realistic interpolation, it faces limitations regarding global alignment (large displacements) and the potential for "hallucinating" biologically inaccurate features. Future research may focus on hybrid models that combine the geometric constraints of traditional affine transforms with the powerful objective functions of adversarial learning, potentially utilizing more complex architectures like U-Nets to further enhance performance.
Summary updated