Co-Optimizing Sampling Patterns and Reconstruction Networks

20 concepts18 papers1 proposal

In physical imaging systems—such as Magnetic Resonance Imaging (MRI), Computed Tomography (CT), and compressive optical microscopy—the journey from physical reality to a digital image happens in two distinct stages: acquisition (sampling the physical signal) and reconstruction (turning those sparse samples into a viewable image).

Historically, these two stages were designed in isolation. Engineers optimized hardware sampling patterns using mathematical heuristics like the Nyquist theorem or random sub-sampling patterns derived from Compressed Sensing (CS) theory. Separately, computer scientists developed algorithms to reconstruct the full signal from these sparse measurements.

This decoupled approach creates a fundamental joint optimization gap. A fixed sampling pattern is entirely agnostic to the strengths, weaknesses, and structural priors of the reconstruction algorithm. If the decoder is exceptionally skilled at interpolating high-frequency details but requires a solid low-frequency anchor, a generic random sampling pattern might waste its limited measurement budget collecting redundant high-frequency points while missing the critical low-frequency anchor.

To bridge this gap, a powerful research direction has emerged: jointly optimizing data acquisition and signal reconstruction in an end-to-end, differentiable pipeline. By treating the physical sampling mechanism as a learnable encoder and the reconstruction network as a decoder, the system learns exactly how to look at the world to make the reconstruction task as easy as possible.


The Core Intuition: The Imaging Pipeline as an Autoencoder

To understand why joint optimization works, it helps to view the entire physical imaging system through the lens of a deep autoencoder.

Physical Signal (x) ──> [ Learnable Sampler / Encoder ] ──> Sparse Measurements (y)
                                                                   │
                                                                   ▼
Digital Image  (x̂) <─── [ Deep Reconstruction / Decoder ] ─────────┘

In a traditional autoencoder, both the encoder and decoder are digital neural network layers trained simultaneously via backpropagation. In co-optimized imaging, the encoder is constrained by physical hardware (e.g., the trajectories of magnetic gradients in an MRI machine or a physical mask in a camera), while the decoder is a digital neural network.

As shown by Wuzhen Shi et al. in their work on CSNet, mapping traditional compressed sensing steps directly to neural network layers allows the sampling matrix and the reconstruction network to adapt to one another. Similarly, Shuaichao Li et al. demonstrated in Joint Design of Measurement Matrix and Sparse Support Recovery Method via Deep Auto-encoder that treating sparse signal recovery as an autoencoder allows the system to exploit hidden structures in data distributions that classical, data-agnostic methods like LASSO completely miss.


Overcoming the Differentiability Bottleneck

The primary technical challenge in joint optimization is differentiability. Hardware sampling patterns are often binary choices: you either sample a specific point in data space or you do not. Because the derivative of a binary mask is zero everywhere (or undefined), standard backpropagation fails; gradients cannot flow from the reconstruction network back to the sampling parameters.

Researchers have solved this using two main strategies: probabilistic relaxation and continuous parameterization.

Probabilistic Relaxation (Learning Masks)

The LOUPE (Learning-based Optimization of the Under-sampling PattErn) framework, introduced by Adrian V Dalca et al. in Learning-based Optimization of the Under-sampling Pattern in MRI and expanded by Alan Q. Wang 0003 et al. in Deep Learning-Based Optimization of the Under-Sampling Pattern in MRI, bypasses the binary bottleneck by parameterizing the sampling mask as a continuous probability map.

During training, the system optimizes these continuous probabilities. To simulate physical sampling, binary masks are sampled from this distribution using differentiable relaxations (such as the Gumbel-Softmax or straight-through estimators). Over training epochs, the probabilities converge toward 0 or 1, yielding a highly optimized, hardware-compatible binary mask that is explicitly tailored to the reconstruction network's behavior.

This probabilistic approach is also utilized in the PUERT framework by Jian Zhang et al. (PUERT: Probabilistic Under-Sampling and Explicable Reconstruction Network for CS-MRI), which couples learned probabilistic sampling with an unrolled, physics-informed reconstruction network.

Continuous Parameterization (Learning Trajectories)

In many imaging modalities, like MRI, data is collected along continuous physical trajectories (k-space) rather than discrete grids. In B-Spline Parameterized Joint Optimization of Reconstruction and K-Space Trajectories (BJORK) for Accelerated 2D MRI, Guanhua Wang et al. parameterized these continuous sampling paths using B-splines. Because B-splines are fully differentiable with respect to their control points, the authors successfully backpropagated reconstruction errors directly into the hardware control parameters, ensuring the scanner moves along paths that capture the most informative data.


Tailoring Acquisition to Modality and Context

When the sampling mechanism is free to adapt to the decoder, it uncovers unique, highly specialized sampling strategies for different imaging modalities.


Beyond Pixels: Task-Driven Acquisition

The ultimate validation of this paradigm is that we do not always sample data just to create a pretty picture for human eyes; often, the goal is a downstream clinical decision, such as segmentation or diagnostic grading.

If the final goal is semantic, the sampling pattern should adapt to the task, not just the image reconstruction.

Raw Data ──> [ Learned Sampler ] ──> [ Reconstruction ] ──> [ Downstream Task (Segmentation/Grading) ]
   ▲                                                                      │
   └─────────────────────────── Backpropagate Task Loss ──────────────────┘

In Empirical evidence of the task-adapted reconstruction framework for joint CT reconstruction and segmentation, Emilien Valat et al. proved that when a CT reconstruction network is trained jointly with a downstream segmentation network, the pipeline achieves significantly higher segmentation accuracy in low-dose or sparse-view scenarios. The reconstruction network learns to preserve edges and semantic boundaries critical for the segmentation model, rather than wasting capacity minimizing pixel-wise mean squared error.

Similarly, Jun Cheng et al. introduced Sparse Range-constrained Learning (Sparse Range-Constrained Learning and Its Application for Medical Image Grading), which integrates image reconstruction and clinical grading into a single optimization step. This ensures that the sparse coding framework selects reference images that are both visually similar and clinically relevant, directly maximizing diagnostic accuracy.

Application to Optical Systems

This concept is not limited to medical imaging; it applies broadly to optical hardware design. In snapshot compressive imaging, Xuanyu Zhang et al. developed HerosNet, which jointly optimizes a physical, hardware-friendly binary coding mask alongside a deep unfolding reconstruction network for hyperspectral imaging.

In biological microscopy, where excessive light exposure destroys living samples (phototoxicity), Alan Q. Wang 0003 et al. demonstrated in Joint Optimization of Hadamard Sensing and Reconstruction in Compressed Sensing Fluorescence Microscopy that optimizing Hadamard sensing patterns in tandem with a deep decoder allows microscopes to capture high-fidelity biological structures at a fraction of the light dose required by traditional patterns.

Summary

Co-optimizing sampling and reconstruction transforms physical sensors from passive data collectors into active, intelligent encoders. By forcing hardware and software to speak the same differentiable language, this paradigm eliminates the joint optimization gap, allowing imaging systems to see less, yet discover more.

Key Foundational Work

Deep Networks for Compressed Image Sensing (2017) — contributed 1 concept elements to this direction.
Sparse Range-Constrained Learning and Its Application for Medical Image Grading (2018) — contributed 1 concept elements to this direction.
Learning-Based Compressive MRI (2018) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_decoder_aware_sampling_optimization_a16a2e4d,
  title = {decoder-aware sampling optimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/decoder-aware-sampling-optimization/rdr_a16a2e4d6d694956ab1a7dd3a6e99532},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_decoder_aware_sampling_optimization_a16a2e4d_v1,
  title = {decoder-aware sampling optimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/decoder-aware-sampling-optimization/rdr_a16a2e4d6d694956ab1a7dd3a6e99532/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake