Super-compression of large electron microscopy time series by deep compressive sensing learning

Siming Zheng, Chunyang Wang, Xin Yuan, Huolin L. Xin

Electron microscopy (EM) is a powerful tool for exploring the physical sciences, but the massive datasets it generates pose significant challenges for storage, processing, and transmission. This paper introduces a novel approach that combines deep learning with compressive sensing to dramatically reduce the size of EM data while preserving high fidelity.

TL;DR

This paper presents a novel framework, TCS-DL, that uses deep learning and temporal compressive sensing to achieve super-compression of large electron microscopy time series. The method significantly outperforms conventional JPEG compression, offering substantial savings in power, memory, and bandwidth with high-fidelity reconstruction and built-in denoising capabilities.

What Problem Does This Paper Address?

The rapid advancement of electron microscopy (EM) techniques, particularly with the advent of ultrafast detectors, has led to an explosion in data generation. This creates a critical bottleneck for big data processing and storage. Existing compression methods, often developed for natural images, are not well-suited for the unique characteristics of EM data, leading to either insufficient compression or loss of crucial details. The challenge is to develop a compression strategy that can handle the large volume of EM data efficiently without sacrificing the high resolution and intricate details essential for scientific discovery.

Methods

The proposed framework, Temporal Compressive Sensing-Deep Learning (TCS-DL), consists of two main parts:

  1. Encoder (Temporal Compressive Sensing - TCS): This part compresses multiple sequential EM images (referred to as "B frames") into a single compressed measurement. It achieves this by applying a random binary mask to each frame and then summing these masked frames. This process significantly reduces the data size, effectively compressing B frames into one measurement. The encoder's complexity is noted as being very low, essentially O(1).
  2. Decoder (Deep Neural Network - DNN): This part reconstructs the original multi-frame EM images from the single-frame compressed measurement. The DNN is an end-to-end network composed of:
  3. A Convolutional Neural Network (CNN): A 12-layer CNN is used to reconstruct the first frame, leveraging the masked measurement and the masks themselves.
  4. Two Recurrent Neural Networks (RNNs): A forward RNN and a backward RNN are employed to reconstruct subsequent frames. These RNNs are designed to learn the temporal correlations between adjacent frames, which is crucial for accurate reconstruction. The forward RNN uses the CNN output as its initial input, while the backward RNN uses the forward RNN's output.

This combined approach leverages the sparsity of EM data for efficient compression and the power of deep learning for high-fidelity reconstruction, including an inherent denoising capability.

Datasets

The paper evaluates the TCS-DL framework on:

  • Sequential atomic-resolution images of Au nanocrystals: These images were captured during an in-situ sintering process, showcasing complex structures.
  • Sequential EM images of carbon fibers: These images were captured during an in-situ deformation process, representing objects with larger feature sizes.

The authors also mention that the original data used in this paper is available from their website, and the code is available on GitHub.

Key Insights and Ideas

  • Synergy of CS and Deep Learning: The paper demonstrates a powerful synergy between compressive sensing for efficient data reduction and deep learning for robust reconstruction. This combination overcomes limitations of traditional methods.
  • Built-in Denoising: A significant advantage of the TCS-DL framework is its inherent denoising capability, which improves the quality of the reconstructed images beyond what conventional compression methods can achieve, especially when dealing with noisy raw EM data.
  • Temporal Correlation is Key: The use of RNNs to capture temporal correlations between frames is critical for reconstructing coherent and detailed EM time series, especially when dealing with dynamic processes.
  • High Compression Ratios Achieved: The framework successfully reconstructs compressed videos with compression ratios up to 30 with high fidelity, a feat not easily achieved by standard methods.

Key Results

  • Superior Performance over JPEG: The TCS-DL framework consistently outperformed conventional JPEG compression in terms of Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM).
  • For atomic-resolution images of Au nanocrystals, TCS-DL achieved an average PSNR of 25.91 dB, compared to 23.98 dB for JPEG. This is a significant improvement of 1.93 dB.
  • For carbon fiber images, TCS-DL achieved an average PSNR of 35.41 dB, compared to 32.38 dB for JPEG, an improvement of approximately 3 dB.
  • High-Fidelity Reconstruction: Reconstructed EM images using TCS-DL showed clean backgrounds, fine details, and sharp edges, preserving crucial atomic-scale information that was lost or blurred in JPEG-compressed images.
  • Significant Data Reduction: The method enables compression ratios of up to 30, leading to substantial savings in storage space and transmission bandwidth.
  • Potential for Power and Memory Savings: The reduced data size is expected to lead to considerable savings in encoding power and in-situ memory usage.

Limitations and Future Work

  • Coherent Frames Assumption: The current TCS-DL method is most effective when the sequential EM frames are coherent. The authors acknowledge that if frames are obviously incoherent, the reconstruction might be vulnerable.
  • Future Work: The authors plan to further increase the compression ratio without compromising reconstruction quality. They also suggest that in future work, they could automatically detect sudden changes in the receptive field and switch to a more appropriate compression method when frames become incoherent. The paper also hints at a wide applicability for the EM community and other imaging techniques.

Summary updated

BibTeX
@article{zheng2021super,
  author = {Siming Zheng and Chunyang Wang and Xin Yuan and Huolin L. Xin},
  title = {{Super-compression of large electron microscopy time series by deep compressive sensing learning}},
  journal = {Patterns},
  year = {2021},
  doi = {10.1016/j.patter.2021.100292},
}
Figure