How Much Pruning Can a Metalens Afford?

A metalens is a flat sheet of nanoscale pillars that focuses light the way a stack of curved glass lenses does, but in a layer about a micron thick. Cameras built this way can be an order of magnitude shorter than glass ones. The cost is blur that changes across the image and across colors, and that blur is removed afterwards by a large neural network. Edge devices like endoscopes and wearables, the targets that motivate flat optics in the first place, cannot run those networks at full size. They have to be pruned, meaning weights are removed so the model runs faster. That raises a choice nobody has measured for this setting. Removing scattered individual weights preserves quality but rarely speeds up real hardware. Removing whole channels, or using the hardware-friendly "2 of every 4 weights" pattern, speeds things up but damages quality by an unknown amount.

We propose to measure that damage as a function of the optics themselves. Our central hypothesis is that the quality cost of hardware-friendly pruning is not a fixed property of the network. It is set by how strongly the lens blur varies across the field of view. Shorter, more aggressive flat optics produce more field-dependent blur, so they should make coarse pruning disproportionately expensive. If this holds, it yields a practical result: an exchange rate between millimetres of optical track length and milliseconds of pruned inference. We also propose a pruning procedure that allocates sparsity using the lens's own blur profile, and we test it against the strongest generic methods.

Core Claim

Recent meta-imaging systems shorten the camera by moving aberration correction into software. Neural array meta-imaging reaches a 4.3 mm track length, 13 times shorter than a matched compound lens, but needs about 30 ms per frame in its Wiener-deconvolution fusion network (Neural array meta-imaging). A 1 cm, 1 µm-thick broadband meta-optic relies on a diffusion-model deblurrer, and its authors name mobile deployment as future work (Beating spectral bandwidth limits for large aperture broadband nano-optics). MetaFormer needs 13 Wiener filters plus two kinds of self-attention to reach 34.29 dB on non-uniform metalens blur (MetaFormer). Even the explicitly lightweight burst pipeline on a Jetson Orin was never pruned systematically (Enabling High-Quality In-the-Wild Imaging from Severely Aberrated Metalens Bursts). These papers motivate the work: each shows the digital cost and none compresses it.

Pruning research supplies the methods but evaluates them on classification and language models whose inputs carry no optical structure. Those methods include one-stage 2:4 training with SR-STE (Learning N: M Fine-grained Structured Sparse Neural Networks From Scratch), small rotation matrices wrapped around a 2:4 core (ARMOR), mixed dense/2:4/empty regions per layer (FlexHiNM-GP), and pruning in a decorrelated eigenbasis (EigenDamage).

Our claim has two parts. The first is that the granularity penalty grows with the spatial variability of the point spread function (PSF), the blur pattern a single point of light makes on the sensor. The penalty is the restoration quality lost by a hardware-friendly sparsity pattern relative to unstructured pruning at equal measured latency. The second is that the penalty is concentrated in the off-axis field. A pruning criterion weighted by where the optic is hardest should recover most of it.

We define spatial variability as

$$V = \frac{1}{K}\sum_{k=1}^{K} \frac{\lVert h_k - h_0 \rVert_2}{\lVert h_0 \rVert_2},$$

where $h_k$ is the PSF at field position $k$ of $K$ sampled positions, and $h_0$ is the on-axis PSF, with the norm pooled over color channels. The prediction is that the penalty $\Delta_g$ for pattern $g$ increases monotonically in $V$ at fixed latency. The reasoning is that a channel-sparse or 2:4 network has fewer distinct filters available to represent many different local blurs. A shift-invariant blur, with $V$ near 0, needs only one deconvolution and should tolerate coarse pruning.

Research Program

We first build a controlled family of optics. Using differentiable wave-optics simulation of radially symmetric phase profiles, the same reduced parameterization Fröch et al. used to make a 10⁹-scatterer lens tractable, we generate about eight metalens designs. They sweep track length and field of view at fixed aperture, so $V$ ranges from nearly shift-invariant to severely field-dependent. Each design produces a bank of spatially varying RGB PSFs. We blur Div2K and Flickr2K images with these banks, add sensor noise, and train one restoration network per design. We use two architectures: a MetaFormer-style Wiener-guided transformer and the burst pipeline's conditioned U-Net. The burst pipeline gives us an edge-sized model with a published deployment target.

We then prune every trained network at three latency targets: 1.5×, 2× and 3× speedup measured on an Ampere-class Jetson Orin, which has 2:4 sparse acceleration. We use each granularity family, so every design yields a curve of penalty against latency. We report PSNR overall and separately for the central, mid and peripheral field zones, because the hypothesis predicts where the losses occur.

Our method, aberration-weighted allocation, changes two things in an existing pipeline. First, the calibration loss that scores weight importance is weighted per pixel by the dense model's own residual error map, averaged over training. This concentrates importance on regions the optic makes hard, which are mostly peripheral and chromatic. Second, the dense/2:4/empty region assignment from FlexHiNM-GP is chosen per layer by sensitivity measured on peripheral patches rather than on whole images. Everything else, including the permutation search and the ARMOR-style wrappers where used, is reused unchanged. We are not proposing a new sparsity format; we are proposing a new signal for deciding where sparsity goes.

Finally, we compute the system-level trade-off. For each latency budget we ask which design, after pruning, gives the best quality. That traces a Pareto front of track length against latency and quality, and states the exchange rate in millimetres per millisecond. If the hypothesis is right, the front bends: below some track length, extra pruning-friendly latency buys little, and the right move is a slightly longer optic.

Baselines

The naive reference is the dense, unpruned restoration network for each optic. It sets the quality ceiling and the latency floor. For real data, the published anchor is MetaFormer at 34.29 dB PSNR on its non-uniform metalens benchmark, against 27.84 dB for Restormer (MetaFormer). For the burst setting the anchor is BRISQUE 23.75, NIQE 4.31 and PIQE 28.77 on the real in-the-wild dataset (Enabling High-Quality In-the-Wild Imaging from Severely Aberrated Metalens Bursts). We found no reported result on Lacuna for any pruned metalens restoration network, so every pruned number here will be new. Our method must stay within 0.5 dB of the dense model at 2× measured speedup to count as a success.

Unstructured magnitude pruning at matched sparsity is the accuracy reference. The survey by Cheng et al. describes it as the upper bound for accuracy at a given sparsity. We report its measured latency honestly, including any kernel from MACKO for moderate sparsity, rather than assuming FLOP savings translate into speed.

Uniform 2:4 trained with SR-STE is the simplest hardware-native competitor. On ImageNet it nearly matches dense accuracy, 77.0% versus 77.3% top-1 for ResNet-50 (Learning N: M Fine-grained Structured Sparse Neural Networks From Scratch). That result is exactly why it might also suffice here.

The strongest existing method is FlexHiNM-GP with ARMOR-style correction wrappers. FlexHiNM-GP reports 79.46% top-1 on DeiT-Base at 80% sparsity against 73.52% for the vector-wise structured baseline (FlexHiNM-GP). ARMOR roughly halves the 2:4 perplexity gap on Llama-2-13B (ARMOR). Channel pruning (and, for the convolutional U-Net, EigenDamage) is the coarse end of the range.

All methods share the same pretrained dense network and data, and the same number of fine-tuning steps after pruning. Every method also receives the same hyperparameter search budget, about 20 trials. Comparisons use measured latency on the same device, not nominal sparsity.

The key ablation is identical to our method except that the calibration loss and the layer-sensitivity measurement are unweighted. That is plain FlexHiNM-GP with ARMOR, run under our schedule. Beating it in peripheral-zone PSNR at equal latency, with no loss in the center, is the evidence that the optical signal matters.

Why not just…?

Why not just train a smaller dense network to the same latency? This is the most serious threat. A narrower dense network runs fast on any hardware and avoids sparsity formats entirely. We include it as a baseline: a width- and depth-scaled dense network, trained with distillation from the full model, matched to each latency target. If it equals the best pruned model across all optics, then the granularity question is moot for this domain, and we would report that as the finding. The proposal would then reduce to the track-length-versus-latency map, which remains useful. We expect it to lose at high $V$, because shrinking width removes the filter diversity that field-dependent blur needs. This is the same mechanism we expect to hurt channel pruning, and the experiment tests both together.

Why not just use unstructured pruning, which preserves quality best? Unstructured zeros generally do not reduce latency on SIMD hardware at moderate sparsity, as MUSTAFAR and MACKO document. A real-time 25 Hz pipeline like Dun et al.'s needs actual speedup. We nonetheless measure its latency with the best available kernel rather than assume. If it does achieve real speedup on the Orin, it becomes the winner and our penalty curves become moot.

Why not just apply uniform 2:4 and accept the loss? It may be enough at low $V$, and our hypothesis predicts exactly that. The question is whether the loss stays small as optics get shorter. FlexHiNM-GP and ARMOR show that uniform 2:4 loses substantially on language and vision transformers at higher compression. The sweep over $V$ will show directly where uniform 2:4 stops being adequate.

Why not just use better optics, such as a compound refractive lens, so no heavy network is needed? Glass can be the right answer when length is not constrained. However, the matched Edmund compound lens is 13 times longer than the neural array system (Neural array meta-imaging), and meta-optics cut endoscope tip length by a third (Fröch et al.). Our Pareto analysis includes longer, low-$V$ designs. It will tell practitioners when adding optical length beats adding compute.

Proposed Experiments

The decisive test is the slope of the penalty against $V$. For each granularity we regress the penalty at 2× measured speedup on $V$ across the eight simulated optics. The hypothesis fails if channel pruning and 2:4 show flat penalties, or if the penalty does not concentrate in the peripheral field. In that case aberration-weighted allocation has nothing to recover, and we would report that hardware-friendly pruning is optics-agnostic.

The second test transfers to real captures. We prune models trained on MetaFormer's fabricated-lens data and on the burst pipeline's OLED-projected and in-the-wild data, then check that the ranking of methods from simulation holds. For downstream use we follow Dun et al.: we feed pruned restorations to an unchanged YOLOv5s and compare against their reported detection precision above 80%, to see whether pruning artifacts that PSNR averages away still break detection.

The third test checks whether the field weighting is just a way to oversample hard patches. We train the unweighted ablation with peripheral patches oversampled to the same effective weight. If that matches our method, the contribution reduces to data sampling, and we will say so.

All latencies will be reported at batch size one on the Orin, with end-to-end frame time, so the final track-length/latency map reflects what a device builder would actually see.