OpenICS: Open image compressive sensing toolbox and benchmark

Jonathan Zhao, Mark Lakatos-Toth, Matthew Westerham, Zhikang Zhang, Avi Moskoff, Fengbo Ren

TL;DR

This paper introduces OpenICS, a unified and standardized toolbox for image compressive sensing (CS) that consolidates numerous reconstruction algorithms. It also presents a comprehensive benchmark evaluating these methods across various datasets and compression ratios, aiming to accelerate research and real-world applications of CS.

What Problem Does This Paper Address?

Image compressive sensing (CS) promises to reduce sensing costs by acquiring fewer samples than traditional methods, but its real-world adoption has been hampered by a lack of standardization. Researchers have proposed many algorithms, but inconsistent implementations and evaluation metrics make it difficult to compare them fairly. This fragmentation hinders progress and makes it challenging for practitioners to select the best approach for their needs. OpenICS aims to solve this by providing a single, well-organized platform for various CS algorithms and a standardized benchmark to objectively assess their performance.

Methods

The core contribution of OpenICS is its unified interface and modular design. This means that all the included image CS reconstruction algorithms, whether model-based or data-driven, are implemented to work with a consistent set of parameters and a standardized structure. This makes it much easier for users to switch between different methods, train new models, and compare results.

The toolbox includes a range of algorithms, categorized as:

  • Model-based methods: These rely on prior knowledge about the signal's structure. Examples include L1 (a foundational method), NLR-CS (non-local low-rank regularization), TVAL-3 (total variation minimization), and D-AMP (based on denoising algorithms).
  • Data-driven methods: These use neural networks to learn image priors. Examples include ReconNet (CNN-based), LDAMP (learned unrolled iterative methods), ISTA-Net (unrolled ISTA algorithm), LAPRAN (Laplacian pyramid networks), CSGM (generative adversarial networks), and CSGAN (a variant of CSGM).

The paper also details a comprehensive benchmarking methodology. This involves evaluating each method on multiple datasets and compression ratios, then combining these results into a single score using weighted averages for datasets, compression ratios, and metrics (PSNR, SSIM, and reconstruction speed).

Datasets

OpenICS was evaluated on six widely used datasets:

  • MNIST
  • CIFAR10
  • CIFAR10 (grayscaled)
  • CELEBA
  • Bigset (a custom dataset)
  • Bigset (grayscaled)

The benchmark was conducted across five different compression ratios: 2, 4, 8, 16, and 32. The authors note that pre-trained networks for data-driven methods are available for these datasets and compression ratios, and the toolbox also supports training new networks from scratch.

Key Insights and Ideas

  • The critical need for standardization: The paper strongly emphasizes that the lack of standardized implementations and evaluation is a major bottleneck for the field of image CS. OpenICS directly addresses this by providing a unified framework.
  • Data-driven methods generally outperform model-based methods: The benchmark results consistently show that data-driven approaches achieve higher reconstruction accuracy and speed, especially when sufficient training data and computational resources are available.
  • Trade-offs exist even within data-driven methods: While ReconNet offers high speed, its accuracy is limited. Conversely, LAPRAN demonstrates a strong balance of both accuracy and speed, leading to the highest overall benchmark score.
  • Model-based methods still have a role: For scenarios with limited data, model-based methods remain a viable option, with TVAL-3 offering a good balance of speed and accuracy compared to other model-based approaches.
  • GAN-based methods show potential but require modern architectures: The paper notes that CSGM, using an older DCGAN architecture, had suboptimal performance. This suggests that newer GAN architectures could significantly improve performance in future work.

Key Results

The benchmark results, summarized in Table V and Figure 1, highlight several key findings:

  • LAPRAN achieved the highest overall benchmark score (58.30), demonstrating superior performance in both accuracy and speed.
  • ReconNet showed the fastest reconstruction speed (37.00 images/sec) but had moderate accuracy (19.15).
  • LDAMP also performed well, with a score of 47.46, indicating a good balance between its speed (30.25 images/sec) and accuracy (17.21).
  • Model-based methods generally had lower scores compared to data-driven methods, with L1 scoring the lowest (23.46).
  • The paper provides detailed tables (VI-XV) for raw benchmark results across all methods, datasets, and compression ratios, offering a granular view of performance.

Limitations and Future Work

  • Dataset limitations: While six datasets are used, the authors acknowledge that the "Bigset" dataset is manually composed. Future work could involve more diverse and challenging real-world datasets.
  • GAN architecture: The performance of CSGM was limited by the use of an older DCGAN architecture. The authors suggest that incorporating more recent GAN models (e.g., StyleGAN) could significantly improve performance.
  • Maintenance and updates: The authors state that the project is still under maintenance and will be updated, implying ongoing development and potential expansion of the toolbox.
  • Customizable weighting: The benchmarking framework allows users to specify their own weights for metrics (PSNR, SSIM, Speed), enabling personalized evaluations based on specific application needs. This flexibility is a strength but also implies that the presented "overall score" is one of many possible interpretations.

Summary updated

Abstract

We present OpenICS, an image compressive sensing toolbox that includes multiple image compressive sensing and reconstruction algorithms proposed in the past decade. Due to the lack of standardization in the implementation and evaluation of the proposed algorithms, the application of image compressive sensing in the real-world is limited. We believe this toolbox is the first framework that provides a unified and standardized implementation of multiple image compressive sensing algorithms. In addition, we also conduct a benchmarking study on the methods included in this framework from two aspects: reconstruction accuracy and reconstruction efficiency. We wish this toolbox and benchmark can serve the growing research community of compressive sensing and the industry applying image compressive sensing to new problems as well as developing new methods more efficiently. Code and models are available at https://github.com/PSCLab-ASU/OpenICS. The project is still under maintenance, and we will keep this document updated.
BibTeX
@article{zhao2021openics,
  author = {Jonathan Zhao and Mark Lakatos-Toth and Matthew Westerham and Zhikang Zhang and Avi Moskoff and Fengbo Ren},
  title = {{OpenICS: Open image compressive sensing toolbox and benchmark}},
  journal = {Softw. Impacts},
  year = {2021},
}
Figure