Spend Capacity Where Pixels Clipped, Precision Where They Didn't

When a camera pixel saturates, its value hits the sensor's ceiling (pure white) or floor (black), and the scene detail under it is gone. Restoring such a region is no longer an inverse problem. It is guesswork guided by a learned prior. That guess needs a large model, but it does not need an exact one. The true answer is already uncertain by far more than any arithmetic rounding error. Pixels in the camera's linear range are the reverse case: they have a single right answer that a small model can reach, so every computational error shows up in the output.

We propose to use this split to decide how a power-limited chip spends its hardware. Large, low-voltage, low-precision accelerator tiles would handle clipped regions. Small, exact cores would handle everything else. The rest of the chip stays switched off, which is the "dark silicon" that post-Dennard processors must leave unpowered anyway to stay within their heat budget.

The core claim is a reversal of the obvious pairing. The natural move at this intersection is to send saturated regions to a heavy generative accelerator running at full precision, since those regions are "hard." We argue that hardness of this kind calls for capacity, not precision. We also argue that the standard way approximate-computing systems decide what is safe to compute sloppily gets clipped regions exactly backwards. That method measures how much the output changes when an intermediate value changes.

Core Claim

Write a restored pixel as $\hat x_i$, the unknown true radiance as $x_i$, and the observed clipped image as $y$. The expected squared error splits into two parts:

$$\mathbb{E}\big[(x_i-\hat x_i-\delta_i)^2 \mid y\big] = \big(\mathbb{E}[x_i\mid y]-\hat x_i-\delta_i\big)^2 + \mathrm{Var}(x_i\mid y),$$

Here $\delta_i$ is the error added by approximate hardware, such as reduced bit-width, voltage over-scaling or injected soft faults. $\mathrm{Var}(x_i\mid y)$ is the irreducible spread of plausible radiances given what the sensor recorded.

In the linear range, that variance is roughly sensor noise, so $\delta_i$ dominates the error budget. Under a clipped plateau, the variance is large because many radiances map to the same $V_{\max}$. The same $\delta_i$ then changes measured fidelity by a small fraction.

This suggests a significance score proportional to $1/\mathrm{Var}(x_i\mid y)$, not to the output's gradient. It predicts that clipped regions tolerate much coarser arithmetic than linear ones, at the same quality loss.

This directly contradicts how Energy Efficiency through Significance-Based Computing assigns work. That work classifies tasks by the derivatives of outputs with respect to intermediates. It runs low-sensitivity tasks on near-threshold "unreliable" cores, saving up to 67% energy on a Jacobi solver. On a saturated plateau the image gradient is zero, so a derivative-based score would call clipped pixels insignificant. In a restoration network, the internal sensitivities near clipped regions may in fact be large, because the network must synthesize content there. We do not know which way gradient sensitivity actually falls, and measuring it is part of the program.

The paper constrains our design in two ways. It supplies the core-assignment method we reuse. It also supplies a warning we must respect: errors in floating-point exponents are catastrophic, while mantissa errors are absorbed.

Positioning

Three lines of prior work motivate the proposal without making it.

Spatially adaptive restoration is one. Spatially-Adaptive Image Restoration using Distortion-Guided Networks (SPAIR) predicts a degradation mask and concentrates sparse convolutions on degraded pixels. It reports roughly 22% lower MSE and +2.16 dB on deraining. ECMamba selects high-information tokens inside a 1.75M-parameter exposure-correction model. Both change where computation happens. Neither changes how exactly it happens, and neither treats sensor clipping as a distinct regime.

Hardware specialization under dark silicon is the second. Conservation cores report up to 16× energy reduction from code-specific circuits. QsCores extend this to shared patterns across related code. The survey Approximate Computing and the Efficient Machine Learning Expedition covers quantization, in-memory computing and voltage over-scaling for neural networks. That survey treats inputs as generic tensors with no spatial structure in their reliability.

The third line establishes the physics. RAW Image Reconstruction Using a Self-Contained sRGB-JPEG Image with Only 64 KB Overhead argues that clipped radiance cannot be analytically inverted. Model-Based Image Signal Processors via Learnable Dictionaries calls the resulting failure a "ceiling effect." Flare7K shows that bright flares clip color channels and force networks to synthesize texture.

What is new here is a per-pixel rule that sets precision from how informative the sensor reading is. That rule is then mapped onto heterogeneous, partly dark hardware.

Research Program

The work has three parts.

Part one: measure tolerance by region. We take trained exposure-correction and flare-removal networks, then perturb them in a region-controlled way. The perturbations are quantization to 8, 6, 4 and 3 bits, and random mantissa-versus-exponent bit flips that emulate near-threshold faults. Each perturbation is applied only to computations whose receptive field is centered in one of three pixel classes. Fully clipped pixels have all channels at 0 or $V_{\max}$. Partially clipped pixels have one or two channels clipped, which is the regime where Color Shift Estimation-and-Correction for Image Enhancement and Local Color Distributions Prior for Image Enhancement find color distributions break down. Linear pixels are the remainder.

We report the error increase per class and compare it with the network's own uncertainty. That uncertainty is estimated from an ensemble, or from samples of a conditional diffusion model trained on the same splits. This tests the equation above directly.

Part two: build a two-path system. A small exact path, starting from ECMamba's architecture, runs on linear pixels. A higher-capacity path runs on clipped regions plus a dilated boundary band. We test two versions of the higher-capacity path: a larger convolutional/attention network of the Uformer family, and a few-step diffusion model. The routing signal comes from one of two sources: the trivial clip mask (thresholding $y$), or the learned variance map. The high-capacity path runs at reduced precision with exponent bits protected, following the bit-level finding in Energy Efficiency through Significance-Based Computing.

Part three: model the hardware honestly. We cannot fabricate silicon. That gap is the same one the file notes for most dark-silicon work, so we will not claim chip-level savings. Instead we report bit-operation counts, and compute energy with an explicit, published-parameter energy-per-operation model in which tile activity is power-gated.

We will also map the two paths onto an FPGA. Reconfigurable fabric, as discussed in Reconfigurable computing in the era of post-silicon scaling, is the closest substrate available to real heterogeneous gated tiles. On it we can measure board power with the high-capacity tile clock-gated when no clipped patches are present.

Baselines

The naive reference is a uniform full-precision model applied to every pixel. ECMamba is the strongest such model in the file: it reports 23.76 dB PSNR on the ME multi-exposure dataset and 27.69 dB on LOLv1 ECMamba. For flare, the reference is Uformer trained on Flare7K, at 26.98 dB PSNR and 0.890 SSIM on the 100-pair real test set Flare7K. The Exposure Normalization and Compensation method, at 22.35 dB on ME Exposure Normalization and Compensation for Multiple-Exposure Correction, is a secondary reference. All of these are retrained by us on identical splits with the same tuning budget. That way reported and reproduced numbers are both visible.

The strongest competitor on this specific question is uniform quantization of the whole model, at the same average bit-width our region split achieves. If clipped and linear regions really differ in tolerance, the region split should beat uniform quantization at equal bit-operations. A tie would falsify the core claim.

The second strong competitor is SPAIR-style mask-guided sparsity Spatially-Adaptive Image Restoration using Distortion-Guided Networks. It uses our clip mask as its localization input and runs at full precision. This isolates whether gains come from spatial routing alone or from precision allocation.

The third is derivative-based significance Energy Efficiency through Significance-Based Computing. It computes output sensitivity per region and assigns low precision to the least sensitive regions, again at matched average bit-width.

Three ablations isolate the key ingredient. Inverse assignment puts low precision on linear pixels and high precision on clipped ones, and should be clearly worse. Random region assignment matches the clipped area. The trivial threshold mask is compared against the learned variance map.

Beating the baselines means two things. First, at equal estimated energy, PSNR within the linear-pixel mask must be no worse than full precision. Second, overall PSNR must exceed uniform quantization by a margin larger than the seed-to-seed spread across three training runs.

For energy, there is no reported result found on Lacuna for image restoration under near-threshold or power-gated execution. The only reported savings for significance-based assignment is 67% on the Weighted Jacobi solver, which is not comparable to imaging.

Why not just…?

Why not just quantize the whole network uniformly? This is the real threat, because uniform quantization is simple and well supported by tooling. Our argument predicts it wastes precision on clipped regions and starves linear ones. The matched-bit-width comparison settles this directly. If uniform 6-bit matches our split, we drop the claim.

Why not just run SPAIR-style sparsity on a GPU? Sparsity reduces how many pixels are processed, but it says nothing about how coarse their arithmetic can be. The two ideas are complementary, and we run both separately and combined. SPAIR itself never tests clipped plateaus, and Gap 1 in the file notes that its savings are counted in operations, not gated power. Our FPGA measurement checks whether either idea survives contact with real power draw.

Why not just run the light model everywhere and accept the loss in clipped areas? That is a fair option, and it may win if capacity buys little where information is absent. We measure PSNR and a perceptual metric restricted to the clipped mask, for the light path versus the high-capacity path. If the difference is within noise, the heavy tile is unjustified. The proposal then narrows to precision allocation alone, which is still testable and still useful.

Why not just use the clip threshold instead of a learned uncertainty map? If the threshold works as well, we will recommend it, since it is essentially free to compute. We expect it to fail on partially clipped pixels and flare halos, where variance is high but values sit below $V_{\max}$. The per-class breakdown shows whether that expectation holds.

Why not just use HDR sensors? Flare from lens dirt clips before photons reach the sensor well Flare7K, and bracketing introduces ghosting in motion. We treat sensor hardware as out of scope rather than as solved.

Proposed Experiments

Evaluation uses the ME and SICE multi-exposure datasets, for which no reported SICE number was found on Lacuna. For flare we use Flare7K's synthetic pairs (Flickr24K backgrounds) and its real-world test set. LOLv1 serves as a control with little highlight clipping, where our split should give little benefit.

The ground truth of over-exposed ME inputs supplies the radiance under clipped plateaus. That lets us measure both terms of the error decomposition.

The decisive check is the tolerance curve from part one. It is error increase versus bit-width, plotted separately for each pixel class. If clipped pixels do not degrade substantially more slowly than linear ones, the proposal fails at its foundation, whatever the hardware results.

Two further checks bound the hallucination risk. First, we report a perceptual metric alongside PSNR, to catch artifacts that squared error hides. Second, we run a downstream stereo-disparity test on the restored images: if low-precision synthesis corrupts disparity near highlights, the "precision does not matter there" argument fails for metric-sensitive uses.

A last diagnostic measures where derivative-based significance actually ranks clipped regions. This tells us whether the reversal we predict is real, or whether gradient sensitivity already happens to track uncertainty.