Spending Heavy Enhancement Compute Only Where the Light Fails

Photos and video of real scenes are often lit unevenly. A single frame can hold a floodlit sign next to a nearly black doorway. Low-light enhancement networks that handle such frames well, such as transformer or diffusion models, are expensive. They run that full expense on every pixel, including the large parts of the frame that need almost nothing. This proposal tests a simple idea. First, a cheap pixel-level exposure estimator marks which regions of each frame are truly degraded. Only those regions are sent as separate jobs to a pool of heavy restoration workers. A tiny network handles the rest immediately. When heavy results finish in time, they are blended back in. A job queue sits between the estimator and the workers so the delivered frame rate does not depend on how many dark regions the worst frame happens to contain.

The second half of the claim concerns measurement. The field mostly scores enhancement with whole-image PSNR on small, low-resolution paired sets such as LOL-v1. That protocol cannot show whether a method fixed the shadows or the highlights, and it cannot reward saving compute. We therefore score every method separately inside each exposure zone, on high-resolution and video data. We also test directly whether this changes which methods look best.

Core Claim

On high-resolution, unevenly lit inputs, sending heavy restoration only to regions flagged by a pixel-level exposure map should come close to the shadow-region quality of running the heavy model on the whole frame. It should do so at a fraction of the compute and with a bounded tail latency. Neither of the two obvious reference points gets there. Running the light model everywhere is too weak in dark regions, and running the heavy model on a downscaled frame loses fine detail.

We state the queue's role narrowly. It is a mechanism for meeting deadlines on streams where the number of dark regions varies from frame to frame. It is not a source of image quality. If a synchronous version with the same routing and the same models matches it, we will say so.

Relation to Prior Work

Several papers motivate the proposal by showing that different regions need different processing. SNR-Aware Low-light Image Enhancement builds a per-pixel signal-to-noise map. It sends high-SNR regions through a short-range CNN branch and low-SNR regions through a long-range transformer branch, but both branches run over the whole frame in one graph. DALE predicts a dark-region attention map so its enhancement network targets underexposed areas. Attention Guided Low-Light Image Enhancement uses separate underexposure and noise maps. All of these apply their attention inside one monolithic forward pass. Our proposal takes their observation literally and turns it into a compute budget: regions that do not need the heavy model never run it.

Fast Enhancement for Non-Uniform Illumination Images using Light-weight CNNs supplies two things.

  • Our light path. Its 5,000-parameter network estimates illumination on a coarse bilateral grid and runs at about 50 fps on 0.5-megapixel images.
  • The gap we fill. The paper itself notes that such small networks cannot recover detail in regions where exposure has failed badly. That is the gap the heavy workers address.

Rethinking No-reference Image Exposure Assessment plays two roles.

  • The router. Its P-IEANet is a 2.7M-parameter model that outputs a pixel-level exposure residual map. We use it as the signal for which regions get heavy processing.
  • The evaluation data and a warning. Its IEA40K benchmark (40,000 images annotated with the Zone System, a photographer's scale of exposure levels from black to white) anchors our per-zone evaluation. The paper also warns that resizing large images discards the fine detail pixel-level assessment needs. Tiling avoids that, and a downscaling baseline makes the cost of resizing visible.

The heavy path is borrowed as-is: the SNR-aware network, and optionally the diffusion model from BSCNet, which constrains brightness to vary smoothly with a Bézier surface. BSCNet's "decoupling" refers to separating brightness from color inside one network, not to scheduling work, so it is complementary to our proposal rather than a competitor.

On the systems side, Parsl turns Python functions into asynchronous tasks that return futures. Merlin distributes tasks through a message broker (a server that holds queued jobs until a worker takes them) and generated a 100-million-sample dataset that way. Both papers constrain the design more than they supply it:

  • Merlin reports roughly 30 ms of overhead per task.
  • Parsl reports throughput of about 1,200 tasks per second.

A 4K stream at 30 fps cut into 64 tiles per frame would produce about 1,920 tasks per second. That exceeds Parsl's reported rate, and at Merlin's overhead it would blow any real-time deadline. The design therefore groups flagged tiles into one heavy job per frame per worker. Parsl-style futures run within a node, and the broker is used only when work spans several machines. Merlin's point that brokers can conflict with cluster security policies is another reason to keep the default single-node.

Research Program

Each frame goes through five steps.

  1. Global light pass. The light network produces a full-frame enhanced image at once. Its coarse illumination grid is kept.
  2. Routing. P-IEANet runs on a downsampled copy of the frame, and its exposure residual map is upsampled. Tiles whose predicted residual falls in the deepest-shadow or clipped-highlight zones are marked for heavy processing.
  3. Heavy jobs. Marked tiles are packed into batched jobs with overlapping borders. Each tile is conditioned on the global illumination grid and the SNR map. This conditioning is what keeps tiles consistent in brightness despite losing the long-range context SNR-Net's transformer normally has.
  4. Deadline assembly. At the frame deadline, finished heavy tiles are blended into the light output with feathered seams. Late tiles are either dropped or applied to the following frame, following optical flow from the SDSD video.
  5. Offline mode. For still images the deadline is removed, so the question becomes pure quality per unit of compute.

We will study two things analytically.

  • Coverage versus quality. We sweep the router threshold to see how the fraction of the frame sent to heavy workers trades against quality in each zone, and whether a small coverage recovers most of the heavy model's shadow gains.
  • Seam artifacts. We measure them separately using gradient error along tile borders.

Baselines

The do-nothing reference is the light network alone on every pixel, following Lv et al.. It sets the floor on compute and on quality.

The strongest quality competitor is the SNR-aware network run synchronously on the full-resolution frame. It is the most widely evaluated region-aware model in the file. The file reports a LOL-v1 PSNR of 24.61 dB for it, against a previous best of 24.14 dB (SNR-Aware Low-light Image Enhancement). On SDSD I found no reported number on Lacuna for its per-subset results, so we will measure it ourselves. BSCNet is reported only as gains of up to 5.38 dB PSNR on LOLv2-real and up to 3.34 dB on SDSD-out over baselines, with no absolute values in the file. It is included as a second heavy model where compute allows.

A resize baseline runs the heavy model on a copy downscaled to match our compute budget, then restores full resolution with the light model's bilateral-grid guidance. This is the cheap way to spend less compute, and it isolates whether tiling at native resolution matters.

The ablation that isolates the key ingredient keeps everything fixed and changes only the routing signal. It compares four routers: the P-IEANet exposure map, the SNR map alone, random tiles at the same coverage, and sending all tiles.

A synchronous tile router uses identical routing and models but runs the heavy tiles as one blocking batch per frame. It separates the benefit of routing from the benefit of the queue.

All methods share the same pretrained weights, fine-tuned for the same number of steps on the same training split, and run on the same GPUs. Compute is reported as GPU-seconds per megapixel.

Two separate bars count as beating the SNR-aware full-frame baseline:

  • Quality at lower cost. The proposal must stay within 0.3 dB of that baseline's PSNR in the darkest zone while using at most half its compute.
  • Deadline performance. On streams, it must hit a higher share of frame deadlines at matched quality.

Why not just…?

Why not just use a synchronous gated network or spatial attention, as DALE and SNR-Net do? Attention inside one graph still computes every pixel through the heavy branch. It re-weights outputs but does not skip work, so it cannot cut compute by itself. A synchronous tile router that actually skips unmarked tiles is the real threat. On a single GPU with still images, we expect it to match our pipeline, because the queue adds nothing there. The synchronous-router baseline settles this. The queue earns its place only if it gives a lower 99th-percentile latency or a higher deadline hit rate on SDSD streams, where the dark fraction changes over time. If it does not, the contribution is region-conditional compute plus the evaluation protocol, and the paper will say so.

Why not just use CLAHE or classical Retinex on the dark tiles? CLAHE is contrast-limited adaptive histogram equalization. Yin et al. show that histogram mappings are skewed by dominant bright pixels. Lv et al. report halos and color casts from Retinex near high-contrast edges. Neither method denoises photon-starved regions, which is exactly where the heavy model's value lies. CLAHE applied to routed tiles is included as a cheap heavy-path substitute, so its shortfall is measured rather than assumed.

Why not just run the light network everywhere and accept the quality? It is the right choice if dark regions are rare or mild. The coverage-versus-quality curve answers this. If light-only quality in the darkest zone sits within noise of the heavy model on IEA40K-stratified inputs, the proposal is unnecessary. Lv et al.'s own admission that small networks fail under extreme exposure suggests it will not be.

Why not just downscale the frame and run the heavy model once? He et al. note that resizing destroys the fine detail pixel-level exposure work depends on. The resize baseline at matched compute tests whether that loss shows up in texture recovery in the shadows.

Proposed Experiments

Training uses the LOL-v1 and LOL-v2 training splits plus Lv et al.'s 82,000 retouched COCO pairs. We also add DALE-style superpixel re-darkening, which simulates local shadows by assigning random brightness to irregular superpixel regions rather than dimming the whole image. The router is validated against IEA40K pixel annotations before use.

Evaluation covers:

  • LOL-v1 and LOL-v2 test sets, for comparability with prior work.
  • Full-resolution SID Sony and MIT-Adobe FiveK (Expert C), for the high-resolution regime.
  • SDSD indoor and outdoor video, for streaming.
  • DICM, LIME, MEF, NPE and VV, with no-reference scores.
  • DARK FACE face-detection AP and ExDark detection, as downstream tasks.

Every paired metric is reported both over the full frame and within each Zone System bin, where bins are assigned from the input's exposure map. System metrics are GPU-seconds per megapixel, median and 99th-percentile frame latency, and deadline hit rate at 24 and 30 fps.

There are three decisive checks.

  1. Compute savings. On full-resolution SID and FiveK, does exposure-routed heavy processing at 50% or lower coverage hold darkest-zone PSNR within 0.3 dB of the full-frame SNR-aware model?
  2. Queue benefit. On SDSD streams with a varying dark fraction, does the asynchronous pipeline beat the synchronous tile router on deadline hit rate at equal quality?
  3. Evaluation lens. Does ranking the heavy, light, resize and routed methods by full-frame LOL-v1 PSNR disagree with ranking them by zone-stratified PSNR on the high-resolution sets? We measure disagreement with Kendall's rank correlation. A low correlation would show directly that the legacy protocol hides the failure mode this whole line of work claims to fix.