Overview
Many real-world tasks, such as MRI reconstruction or image deblurring, are inverse problems: we have measured data $d$ and want to recover the original signal $p$ given a forward model $d = A(p, n)$. Modern deep learning approaches often use "iterative inverse models" (like Recurrent Inference Machines) that mimic optimization algorithms by repeatedly updating an estimate of the signal.
However, the primary bottleneck for these models is memory. To train these models using backpropagation through time (BPTT), a system must store the intermediate activations of every layer at every iteration. As models get deeper or data gets larger (like 3D medical volumes), memory requirements explode, forcing researchers to rely on shallow, less expressive models.
To solve this, the authors introduce the Invertible Recurrent Inference Machine (i-RIM), a framework for solving inverse problems that achieves constant memory complexity $O(1)$ relative to model depth. By utilizing invertible neural networks, they successfully train models with up to 400 layers on large 3D MRI volumes—a feat previously impossible due to the linear memory scaling of standard backpropagation.
Methods: Breaking the Memory Barrier
The paper proposes two primary innovations to overcome the memory limitations of standard iterative models:
- The i-RIM Architecture: The authors modify the standard Recurrent Inference Machine (RIM) update equations to make the entire iterative process mathematically invertible. Because the network is invertible, the activations from the forward pass do not need to be stored in memory; they can be exactly recomputed from the output during the backward pass.
- Stable Invertible Layers: Standard invertible layers (like those in Glow or RealNVP) can become numerically unstable when stacked hundreds of layers deep. The authors introduce a new layer using Orthogonal $1 \times 1$ Convolutions constructed via Householder reflections. This ensures that the eigenvalues of the transformation remain at 1, preventing gradients from vanishing or exploding and allowing for the stable training of extremely deep networks.
Key Insights and Results
The researchers evaluated their model on the fastMRI challenge dataset (single-coil task), utilizing 973 volumes (over 34,000 slices) for training. The task involved reconstructing images from sub-sampled K-space data at 4x and 8x acceleration factors across both 2D slices ($480 \times 320$) and full 3D volumes ($32 \times 480 \times 320$).
Depth is Now "Free"
By moving to an $O(1)$ memory scheme, the depth of the model is limited only by computation time, not by GPU RAM. This allows for vastly more expressive models. The authors successfully trained a 400-layer model, demonstrating that their Householder-based layers effectively solve the instability issues common in deep invertible architectures.
The 3D Breakthrough
Standard iterative models are often restricted to 2D slices because the "machine state" (the memory the model carries across iterations) for a 3D volume is too large to store alongside activations. The i-RIM is the first iterative model capable of handling these large 3D observations effectively.
State-of-the-Art Performance & Efficiency
- Accuracy: The i-RIM 2D consistently outperformed both the standard U-Net baseline and the original RIM across all metrics (NMSE, PSNR, and SSIM). For 8x acceleration, the i-RIM 2D achieved a PSNR of 30.41 compared to 29.45 for the U-Net.
- Memory Efficiency: During training, a standard RIM with 8 steps required ~10.5 GB of memory. The i-RIM 2D required only ~2.5 GB, despite being significantly deeper.
Limitations and Future Work
While the i-RIM presents a massive leap in memory efficiency, it comes with a computational trade-off: training time increases because the backward pass requires recomputing the forward activations.
Additionally, while the 3D i-RIM was feasible to train, its performance was slightly below the 2D version. The authors suggest this is due to a lack of hyperparameter tuning and the need for more specialized 3D convolutional architectures. Finally, because the model is an Invertible Neural Network (INN), it inherently defines a probability density. The authors note that this opens the door for future work in unsupervised or semi-supervised learning for inverse problems using the change-of-variables formula.
Summary updated