Predicting ptychography probe positions using single-shot phase retrieval neural network

Ming Du, Tao Zhou, Junjing Deng, Daniel J. Ching, Steven Henke, Mathew J. Cherukara

TL;DR

Researchers have developed a two-stage framework that uses a neural network (PtychoNN) to predict probe positions in ptychography, even when errors are as large as 100+ pixels. By performing "single-shot" phase retrieval on individual diffraction patterns and then aligning the resulting images via robust registration, the method overcomes the convergence failures that plague traditional optimization-based algorithms.

What Problem Does This Paper Address?

Ptychography is a high-resolution imaging technique that reconstructs an object by scanning a coherent beam (the "probe") across it and recording the resulting diffraction patterns. To work, the algorithm needs to know exactly where the probe was for every shot.

In the real world, mechanical vibrations, thermal drift, or imprecise motors introduce "position errors." Standard "engines" try to fix these errors by refining positions and the image simultaneously. However, these methods rely on significant overlap between scan points to provide redundancy. If the initial position guess is too far off (e.g., tens or hundreds of pixels), the redundancy breaks, the math fails to converge, and the final image remains a blurry mess. This paper tackles these "large-error" scenarios where traditional ptychography breaks down.

Methods

The authors propose a fundamentally different workflow that doesn't rely on initial overlap for position correction:

  1. Single-Shot Phase Retrieval: They use PtychoNN, a deep convolutional neural network trained to predict the object's phase directly from a single diffraction pattern. Unlike iterative methods, the NN uses learned prior knowledge to "guess" what the object looks like at that specific scan point without needing to see neighboring shots.
  2. Hybrid Image Registration: Once they have a "tile" (a local image) for every scan point, they find the spatial offsets between them. They use a hybrid approach: an Error Map algorithm for high precision on small offsets and SIFT (Scale-Invariant Feature Transform) for robustness when offsets are large.
  3. Global Position Solving:
    • Serial Mode: Calculates positions by chaining offsets from one point to the next (useful for a quick first pass).
  4. Collective Mode: Formulates the problem as a linear system ($Ax=b$) where they solve for all positions simultaneously using multiple neighbors for each point. This includes regularization to ensure the predicted scan path is physically "smooth."

Datasets

The study evaluated the method using two primary datasets:

  • Random Etched Object: A custom calibration chart made of tungsten. This was used for large-scale training (34,596 patterns) and testing with synthetic errors (both "accumulating" drift and "independent" random jitters).
  • Siemens Star: A standard gold test pattern. This dataset contained "natural" errors that occurred during the experiment, providing a real-world test case for the algorithm's ability to correct long-range drift and localized jitter.

Key Insights and Ideas

  • Bypassing the Overlap Constraint: Traditional ptychography is "data-driven"—it needs overlap to find the phase. By using a neural network, the authors introduce "prior-driven" reconstruction. The NN provides a "good enough" local image to allow for registration, even when the physical overlap is virtually non-existent due to errors.
  • Synergy, Not Replacement: The authors don't suggest replacing traditional engines. Instead, their method acts as a "warm start." It brings the error down from 100 pixels to ~2 pixels, a range where standard optimization-based refinement can then take over to achieve sub-pixel accuracy.
  • Robustness to Data Scarcity: A "decimation study" showed the model remains stable even when trained on only 40% of the original dataset, suggesting it could be adapted to experiments where training data is harder to come by.

Key Results

  • Massive Error Correction: The method successfully predicted positions for datasets with accumulating errors on the order of $10^2$ pixels.
  • Quantifiable Accuracy: In the most extreme test case (A1-3), the initial Root-Mean-Square Pairwise Position Error (RMS-PPE) was 112.93 pixels. The proposed method slashed this to 2.43 pixels, which was further refined to 1.95 pixels using standard tools.
  • Image Quality: Reconstructions using the predicted positions showed sharp, recognizable features and higher high-frequency content (verified via Radial Power Spectrum Density) compared to reconstructions using the erroneous "nominal" positions, which often failed to produce any usable image.

Limitations and Future Work

  • Weak Scatterners: The authors noted that PtychoNN struggles with "weak phase objects" (like soft biological tissues) where diffraction patterns have very little visual variation.
  • Structural Complexity: If the object is significantly larger than the probe area or highly complex, the NN-predicted tiles can show slight deformations, which complicates the registration step.
  • Generalizability: Currently, the NN needs to be trained on data similar to the sample. The authors suggest moving toward physics-informed self-supervision or conditional generative models (like diffusion models) to make the network more flexible and capable of handling diverse, unknown samples without extensive pre-training.

Summary updated

Abstract

Ptychography is a powerful imaging technique that is used in a variety of fields, including materials science, biology, and nanotechnology. However, the accuracy of the reconstructed ptychography image is highly dependent on the accuracy of the recorded probe positions which often contain errors. These errors are typically corrected jointly with phase retrieval through numerical optimization approaches. When the error accumulates along the scan path or when the error magnitude is large, these approaches may not converge with satisfactory result. We propose a fundamentally new approach for ptychography probe position prediction for data with large position errors, where a neural network is used to make single-shot phase retrieval on individual diffraction patterns, yielding the object image at each scan point. The pairwise offsets among these images are then found using a robust image registration method, and the results are combined to yield the complete scan path by constructing and solving a linear equation. We show that our method can achieve good position prediction accuracy for data with large and accumulating errors on the order of $10^2$ pixels, a magnitude that often makes optimization-based algorithms fail to converge. For ptychography instruments without sophisticated position control equipment such as interferometers, our method is of significant practical potential.
BibTeX
@misc{du2024predicting,
  author = {Ming Du and Tao Zhou and Junjing Deng and Daniel J. Ching and Steven Henke and Mathew J. Cherukara},
  title = {{Predicting ptychography probe positions using single-shot phase retrieval neural network}},
  year = {2024},
  eprint = {2405.20910},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2405.20910},
}

Loading related content...

Figure