In traditional signal processing, the way we capture data and the way we reconstruct it are treated as two separate problems. In Magnetic Resonance Imaging (MRI), physicists design sampling "masks" to decide which frequencies to collect. Separately, computer scientists develop algorithms to turn those sparse frequencies into a clean image.
This decoupling is a missed opportunity. As Tomer Weiss and Sanketh Vedula argue in Joint learning of cartesian undersampling and reconstruction for accelerated MRI, the optimal way to sample data depends entirely on the specific strengths and weaknesses of the reconstruction algorithm. By merging these two stages into a single, end-to-end differentiable pipeline, we can "co-design" the hardware's acquisition strategy and the software's recovery logic.
The Synergy of Co-Adaptation
The core intuition behind this research direction is co-adaptation. When you train a reconstruction network (like a U-Net or an unrolled iterative solver) alongside a learnable sampling mask, the two components evolve to help each other.
- The Sampler learns to identify the most informative data points for that specific restorer. Instead of using a generic "one-size-fits-all" random pattern, it discovers which frequencies are essential for the network to resolve fine structures.
- The Reconstructor learns to specifically compensate for the artifacts and gaps created by that learned mask. It doesn't just learn to "deblur"; it learns to fill in the specific "holes" it knows the sampler will leave behind.
This synergy allows for much higher acceleration rates. In Learning Fast Magnetic Resonance Imaging, Alexander Bronstein and colleagues show that this joint approach consistently outperforms methods where the sampling pattern is fixed or designed independently.
Differentiable Acquisition
To make this work, the sampling process must be differentiable. In the LOUPE (Learning-based Optimization of the Under-sampling PattErn) framework, Adrian V Dalca and Cagla Deniz Bahadir solve this by parameterizing the sampling mask as a probabilistic layer. During training, the model learns a continuous probability for each k-space coordinate; during the forward pass, these are relaxed (often using a sigmoid or Gumbel-softmax) to allow gradients to flow from the final image loss back to the acquisition parameters.
This concept extends beyond simple masks to complex physical trajectories. In PILOT: Physics-Informed Learned Optimal Trajectories for Accelerated MRI, the authors incorporate hardware constraints—such as maximum gradient strength and slew rates—directly into the loss function. This ensures that the "optimal" path the network discovers is actually physically executable by an MRI scanner.
Beyond MRI: Learned Compressive Sensing
While MRI is a primary driver, this logic applies to any system where we "sense" a signal to reconstruct it later. In the field of Compressive Sensing (CS), researchers like Amir Adler and Michael Elad demonstrated in A Deep Learning Approach to Block-based Compressed Sensing of Images that replacing fixed random Gaussian matrices with learned linear sensing layers significantly improves recovery quality.
Similarly, Catherine F. Higham applied this to single-pixel cameras, using a deep convolutional auto-encoder to jointly optimize the binary patterns projected onto a scene and the network that reconstructs the video. By treating the physical measurement as the first layer of a neural network, the system learns to extract the most "compressed" yet "descriptive" features possible.
The Shift Toward Adaptive Intelligence
Recent work has pushed this even further into adaptive or active acquisition. Instead of learning one fixed mask for all patients, researchers are exploring policies that change the sampling pattern in real-time based on the data already collected. Kyong Hwan Jin and Michael Unser proposed using Monte Carlo Tree Search to guide this active sampling in Self-Supervised Deep Active Accelerated MRI.
Whether the sampling is fixed or adaptive, the conclusion across these 109 papers is clear: the most efficient imaging systems are those that treat acquisition and reconstruction as a single, unified conversation. When the "eye" (the sampler) knows what the "brain" (the reconstructor) is looking for, the entire system becomes more than the sum of its parts.