Joint Optimization of Data Acquisition and Signal Reconstruction

275 concepts153 papers2 proposals

In physical imaging and signal processing systems—such as Magnetic Resonance Imaging (MRI), compressed sensing cameras, and X-ray scanners—we face a fundamental bottleneck: collecting raw data takes time, energy, or communication bandwidth. For decades, the standard engineering pipeline treated data acquisition and signal reconstruction as two entirely separate problems. First, hardware engineers designed a fixed sampling strategy (such as random or equidistant undersampling) based on generic mathematical guarantees like the Nyquist theorem or classical Compressed Sensing (CS) theory. Second, software engineers developed an independent algorithm (such as iterative optimization or a deep neural network) to reconstruct the full signal from those sparse measurements.

This modular approach, however, leaves significant performance on the table. A fixed sampling pattern is blind to the specific inductive biases, strengths, and weaknesses of the reconstruction model. If a reconstruction network is exceptionally good at interpolating smooth regions but struggles with sharp edges, a generic random sampling pattern will waste valuable measurements collecting data from the smooth areas.

The core insight of joint sampling-reconstruction optimization is to break down this wall. By treating both the physical data acquisition process and the digital recovery algorithm as a single, end-to-end differentiable pipeline, the system can co-adapt. The hardware sampler learns exactly what features are most informative to collect, while the reconstructor learns to specifically counteract the exact aliasing artifacts caused by that specific sampling pattern.


The Mechanism of Co-Adaptation

To understand why joint optimization works so well, consider how information flows during training. In a standard deep learning reconstruction setup, an undersampled signal is fed into a network, and the network is penalized based on how much its output deviates from the ground truth.

When we make the sampling process learnable, we parameterize the physical sensing mechanism—such as an MRI k-space mask or a camera's coded aperture—and place it before the reconstruction network. During backpropagation, gradients flow from the final image loss, back through the layers of the reconstruction network, and all the way into the parameters governing the hardware sampler.

As highlighted by Tomer Weiss et al. in their work on accelerated MRI, Joint learning of cartesian undersampling and reconstruction for accelerated MRI, this joint training creates a powerful synergy. The reconstruction network learns specific compensations for the gaps in the sampled data, allowing it to recover fine structural details—such as small vessels or sharp boundaries—even at extreme acceleration factors where traditional independent sampling methods would produce severe blurring or artifacts.

This co-design principle dates back to foundational concepts in dictionary learning. For instance, Jean Ponce demonstrated in Task-Driven Dictionary Learning that simultaneously optimizing a sensing matrix alongside a data dictionary yields significantly higher reconstruction quality than using fixed or random projections. In the era of deep learning, Wuzhen Shi et al. formalized this for neural networks with CSNet (Deep Networks for Compressed Image Sensing), showing that a data-driven measurement matrix specialized for a specific recovery network consistently outperforms mathematically guaranteed random matrices.


Bridging the Differentiability Gap

The primary technical challenge in joint optimization is making the sampling process differentiable. Physical sampling is often inherently binary or discrete: a data point is either collected or it is not. Because discrete selections have zero gradients almost everywhere, standard backpropagation fails.

Researchers have developed several elegant mechanisms to bypass this bottleneck:

1. Probabilistic Relaxation

One prominent solution is to model the sampling mask as a set of continuous probabilities rather than hard binary choices. In the LOUPE (Learning-based Optimization of the Under-sampling PattErn) framework introduced by Adrian V Dalca et al. (Learning-based Optimization of the Under-sampling Pattern in MRI), the network learns a continuous weight map. During the forward pass, these weights are relaxed using stochastic approximations (like the Gumbel-Softmax trick or uniform noise thresholds) to simulate binary sampling, while allowing smooth gradient flow during the backward pass. Iris A. M. Huijben et al. utilized a similar approach with their Deep Probabilistic Subsampling (DPS) layer in Learning Sampling and Model-Based Signal Recovery for Compressed Sensing MRI, proving that probabilistic masks allow the sampler to discover non-intuitive, high-information features.

2. Learnable Linear Layers

In compressed sensing applications where the hardware allows for weighted linear combinations of pixels (such as single-pixel cameras), the sampling matrix can be directly represented as a convolutional or fully connected layer without a bias term. As shown in the SCSNet architecture by Wuzhen Shi et al. (Scalable Convolutional Neural Network for Image Compressed Sensing) and LapCSNet by Wenxue Cui (An efficient deep convolutional laplacian pyramid architecture for CS reconstruction at low sampling ratios), the weights of this initial layer correspond directly to the physical sensing patterns, making the entire system natively differentiable.

3. Continuous Trajectory Parameterization

When data collection relies on continuous physical trajectories—such as the magnetic gradient waveforms in MRI—sampling cannot be optimized pixel-by-pixel due to hardware constraints like maximum gradient strength and slew rates. To solve this, Guanhua Wang et al. introduced the BJORK framework (B-Spline Parameterized Joint Optimization of Reconstruction and K-Space Trajectories (BJORK) for Accelerated 2D MRI), which parameterizes the sampling trajectories using B-splines. Similarly, Wei Peng et al. leveraged Neural Ordinary Differential Equations (Neural ODEs) in Learning Optimal K-space Acquisition and Reconstruction using Physics-Informed Neural Networks. By optimizing the continuous control parameters rather than discrete points, these frameworks ensure that the learned sampling paths are fully differentiable yet strictly adhere to physical hardware limits.


Static vs. Dynamic Acquisition Policies

Once we accept that sampling and reconstruction should be optimized together, a new question arises: should the learned sampling pattern be static (the same for every object) or dynamic (adapting on-the-fly based on the specific subject being scanned)?

Static Joint Optimization:
[Training Data] ---> [Joint Optimization] ---> Fixed Optimal Mask ---> Deploy to Hardware

Dynamic/Sequential Optimization:
[Initial Measurement] ---> [Policy Network] ---> Next Best Measurement Point ---> [Reconstructor]
                               ^                                                    |
                               |____________________________________________________|

In the dynamic paradigm, acquisition is framed as a sequential game or a Markov Decision Process. For example, Luis Pineda et al. applied Reinforcement Learning to this problem in Active MR k-space Sampling with Reinforcement Learning, using a policy network to evaluate existing measurements and sequentially select the next most informative k-space line. Tianwei Yin et al. expanded on this with an end-to-end differentiable sequential framework (End-to-End Sequential Sampling and Reconstruction for MRI), demonstrating that active, patient-specific adjustments can yield superior image fidelity at extreme acceleration rates.

However, dynamic adaptation introduces significant computational overhead and hardware complexity. Interestingly, Tim Bakker et al. discovered a surprising nuance in On learning adaptive acquisition policies for undersampled multi-coil MRI reconstruction: when trained end-to-end on large multi-coil datasets, the best-performing sequential policies often naturally converged to "non-adaptive" behaviors. They discovered that a single, globally optimized static mask—tailored to the reconstruction network's capabilities—can perform just as well as computationally expensive slice-by-slice adaptive policies, offering a highly practical route for clinical deployment.


Broad Applications Across Domains

While accelerated MRI is the most prominent testbed for this research direction, the underlying principle of joint optimization generalizes across a vast array of imaging modalities:

  • Single-Pixel Video: Catherine F. Higham et al. demonstrated in Deep learning for real-time single-pixel video that optimizing physical illumination patterns alongside a convolutional autoencoder allows a single-pixel camera to achieve real-time, high-resolution video reconstruction at a 98% compression ratio.
  • X-Ray Fluorescence (XRF) Imaging: Traditional XRF scans suffer from grueling acquisition times. Qiqin Dai et al. developed an adaptive framework (Adaptive Image Sampling Using Deep Learning and Its Application on X-Ray Fluorescence Image Reconstruction) that trains a mask generator and an inpainting network simultaneously, forcing the sampler to prioritize scanning the exact pixels that the reconstruction model finds hardest to predict.
  • Multi-Task Medical Pipelines: The benefits of joint optimization can extend beyond simple visual reconstruction. Wenjun Xia proposed SemuNet (One Network to Solve Them All: A Sequential Multi-Task Joint Learning Network Framework for MR Imaging Pipeline), an end-to-end framework that links k-space sampling, image reconstruction, and downstream tissue segmentation. By allowing the final clinical task (segmentation) to backpropagate all the way to the initial data acquisition stage, the system learns to preserve the exact raw data points critical for diagnostic accuracy.

Ultimately, joint sampling-reconstruction optimization shifts the focus of imaging science from how to perfectly recover data from a generic sub-sampled state to how to intelligently capture only the data that matters. By forcing hardware acquisition and software reconstruction to speak the same mathematical language via backpropagation, this paradigm achieves unprecedented speed and fidelity across the physical sciences.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 155 papers
  2. Version 1 · 2026-10-04 · 171 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_joint_sampling_reconstruction_optimizati_dc0ad84b,
  title = {joint sampling-reconstruction optimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/joint-sampling-reconstruction-optimization/rdr_dc0ad84b663d489dbe27bfd49c2ce9e4},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 2, frozen text):

@misc{lacuna_joint_sampling_reconstruction_optimizati_dc0ad84b_v2,
  title = {joint sampling-reconstruction optimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/joint-sampling-reconstruction-optimization/rdr_dc0ad84b663d489dbe27bfd49c2ce9e4/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake