Spectrum-Keyed Out-of-Core Training for Demosaicing Networks

A camera sensor records only one color per pixel, through a mosaic of red, green and blue filters called a color filter array. Rebuilding the two missing colors at every pixel ("demosaicing") is easy on smooth regions. It fails on a small minority of patches: fine fabric, fences and grids whose detail is close to the mosaic's own repetition rate. There the network cannot tell real color from aliased brightness, and the output shows rainbow moiré. Deep demosaicers are trained on random crops drawn from large raw-image corpora. As those corpora grow past what fits in memory, the loader has to read data from disk in large contiguous chunks to stay fast. Reading in chunks makes it even harder to see enough of the rare hard patches.

We propose a fix that only works because of the physics of the mosaic. A Bayer mosaic places color information at known spatial frequencies, called carrier frequencies. The amount of image energy that lands near those carriers in a small Fourier transform of the raw mosaic patch measures how ambiguous that patch is. The score needs no demosaicer and no ground truth. It is cheap enough to compute during a single sequential pass over a terabyte-scale corpus. We use it as the routing key of a bucketed out-of-core loader. Patches are written to disk grouped by difficulty, and training reads contiguous blocks from those groups at controlled rates. The aim is a demosaicer that, under a fixed RAM and GPU budget, reaches the hard-texture quality of hard-patch mining without running a reference network over the whole corpus.

Core Claim

The obvious bridge between these two fields is out-of-core demosaicing inference: tiling a gigapixel mosaic so a heavy network can process it. We argue this is not a research problem for local networks. A convolutional or windowed network has a bounded receptive field, so overlapping tiles with a halo of that width give exactly the same output as processing the whole image. The tiling seams studied in Tiling artifacts and trade-offs of feature normalization in the segmentation of large biological images come from normalization statistics that change from tile to tile. Demosaicers can avoid that design choice. The harder problem is on the training side, at the point where out-of-core loading and demosaicing difficulty interact.

Our claim has two parts. First, a carrier-energy score computed on the raw mosaic predicts per-patch demosaicing error at least as well as generic edge-energy scores, and much better than random. Second, using this score as the bucket key in a quasi-random block loader improves accuracy on hard textures at equal compute and memory, while preserving the I/O efficiency of sequential block reads.

The score builds directly on the analysis in Color Demosaicing by Estimating Luminance and Opponent Chromatic Signals in the Fourier Domain. That paper shows a Bayer mosaic keeps brightness at the center of the spectrum and modulates the two color-difference signals onto carriers at the corner and edge frequencies. A patch is ambiguous when brightness energy spreads out to those carrier locations. For a raw patch $P$ with 2D Fourier magnitude $|\hat P(\omega)|$, we define

$$s(P) = \frac{\sum_{\omega \in C_\epsilon} |\hat P(\omega)|^2}{\sum_{\omega} |\hat P(\omega)|^2},$$

where $C_\epsilon$ is the set of frequencies within radius $\epsilon$ of the mosaic's carrier frequencies. For a general periodic mosaic, $C_\epsilon$ is computed from the mosaic's period, following the arbitrary-mosaic treatment in A generic variational approach for demosaicking from an arbitrary color filter array. One normalized ratio per 64×64 patch costs one small FFT.

Research Program

The pipeline borrows its two-phase structure from Woody in Training Big Random Forests with Little Resources. Woody streams a billion instances once through a shallow routing tree into disk buckets sized to fit in RAM, then trains on each bucket. Our routing tree is replaced by the physics-derived score.

In the distribution phase, each raw file is read sequentially exactly once and decoded. Clean training targets are produced by downsampling and re-mosaicking, following the protocol of Deep Residual Network for Joint Demosaicing and Super-Resolution (16MP to a 4MP target to a mosaicked input). The image is then cut into patches, $s$ is computed for each, and patches are appended to one of $K$ difficulty buckets, each stored as contiguous compressed HDF5 shards.

In the training phase, the loader follows scDataset. It reads contiguous blocks of $b$ patches, buffers $f$ blocks, and shuffles inside the buffer. That paper shows that $b \le f/2$ recovers the gradient variance of fully random sampling on single-cell data. Our loader adds two things. Each block is drawn from a bucket chosen with mixing weights $w_k$. And because every bucket mixes patches from thousands of images, a contiguous block is not homogeneous in scene content the way a contiguous slab of one plate is in omics data. This weakens the batch-homogeneity failure that motivated scDataset.

We will study fixed temperature-scaled weights and a simple adaptive rule that raises $w_k$ in proportion to recent per-bucket validation loss. Three analyses come with the system. The first is how well $s$ ranks patches by their true error. The second is how the gain depends on $K$, $b$ and $f$. The third is whether the score transfers to Quad-Bayer and RGBW layouts by recomputing $C_\epsilon$, using the re-mosaic setting of the MIPI 2022 RGBW challenge.

Baselines

The "do nothing" reference is a standard PyTorch loader that opens raw files at random and takes random crops, run under the same host-RAM cap (16 GB) on a corpus that exceeds it by roughly an order of magnitude. On a subset small enough to fit in memory, we also run an in-memory random-crop loader. That run serves as the accuracy reference any out-of-core scheme should match. For demosaicing raw-file loaders, no reported result was found on Lacuna for throughput, so we measure it ourselves.

The strongest systems competitor is uniform quasi-random block loading as in scDataset, which reports a 204x throughput speedup over AnnLoader on Tahoe-100M with accuracy parity to true random sampling. We implement it on the same HDF5 patch shards with no difficulty key. We give its $b$ and $f$ the same tuning grid as ours. This is the competitor that isolates whether difficulty-aware routing adds anything beyond efficient I/O. We also run Redox from Brand, which reports up to 4.57x end-to-end speedup over PyTorch DataLoader. Redox works on whole files, so it tests whether file-level redirection is already enough.

On the accuracy side, the strongest reported number on the hard benchmark is SegDem(R) at 29.45 dB PSNR on MIT-Hard, against 27.61 dB for ms-demosaic, from SegDem: Segmentation helps Demosaicing. On RAISE, the 24-block joint network reports 31.41 dB PSNR and 0.9476 SSIM in Deep Residual Network for Joint Demosaicing and Super-Resolution. We will not claim to beat SegDem's architecture or pretraining. Instead, we train one fixed architecture (a Restormer-sized model) under every loader. Within that architecture, "beating" means at least 0.3 dB higher MIT-Hard PSNR than uniform quasi-random loading at equal GPU-hours, with no more than 0.1 dB loss on Kodak.

Two ablations isolate the key ingredient. The first keeps the identical bucketing machinery but routes by a random key. The second routes by a CFA-agnostic edge score, mean gradient magnitude on the green sub-lattice. If the edge score matches the carrier score, the physics-specific part of the claim fails, and we will report it that way.

Why not just…?

Why not just mine hard patches offline once and train in memory? This is the most serious alternative. A reference demosaicer is run over the corpus, the worst patches are kept, and the mined set fits in RAM. Its costs are a full forward pass of a reference network over every patch and a set that stays fixed while the model being trained improves. Our score costs one small FFT per patch, computed in the pass that already reads the data. That does not prove offline mining is worse. We will run it with the same reference-network budget counted as compute and compare MIT-Hard PSNR at equal total GPU-hours. If mining wins at equal cost, the contribution shrinks to a cheaper approximation, and we will say so.

Why not just use online loss-based hard example mining? That approach still has to load and evaluate patches to discover they are hard, so it inherits the full I/O cost. In practice it can only re-weight patches that are already in the buffer. Routing at write time lets hard patches be read with sequential I/O. We will compare against loss-based selection inside the scDataset buffer.

Why not just buy a faster SSD and use random crops? The out-of-core component only matters if I/O actually stalls the GPU. Brand reports that I/O took over 90% of epoch time for SqueezeNet under standard loaders, but demosaicers do far more compute per byte. So we will measure the GPU stall fraction of the naive loader first. If stalls are negligible, the systems contribution is moot, and the proposal reduces to difficulty-keyed sampling, which can still be evaluated on its own terms.

Proposed Experiments

Training corpora are RAISE raw files processed with the downsampling protocol above, plus synthetic mosaics generated from LSDIR. Evaluation uses Kodak (24 images), MIT-Hard, and the LSDIR test split, with color PSNR, SSIM and LPIPS as metrics. Systems metrics are samples per second, GPU stall fraction and peak host RAM.

The first decisive check is purely about the score. On held-out patches, we compute Spearman correlation between $s$ and the per-patch error of a converged demosaicer. The carrier score must beat the green-gradient score; otherwise we drop the physics framing. The second check is the equal-budget training comparison against all baselines, with the 0.3 dB MIT-Hard and 0.1 dB Kodak thresholds above. The third is a $b$/$f$/$K$ sweep to see whether scDataset's $b \le f/2$ rule still holds when buckets are deliberately non-uniform. The fourth recomputes $C_\epsilon$ for Quad-Bayer and RGBW mosaics and repeats the correlation check, to test whether the score transfers beyond Bayer.