Real-Valued Fountain Codes for Raw Camera Streams

Camera sensors record light linearly. Before any processing, a pixel's value is proportional to the light that hit it, and its noise follows a simple, well-understood mix of photon randomness and electronic read noise. Fountain codes are a way to send data over lossy wireless links without asking for retransmissions. The sender emits an endless stream of mixtures of the source packets, and the receiver can rebuild everything from any sufficiently large subset of them. These codes do their mixing in finite-field arithmetic, the bitwise algebra used in error-correcting codes. That choice has a cost. A finite-field mixture of two pixel values is not a physically meaningful number, so the receiver learns nothing useful from the parity packets until it holds enough of them to invert the whole system. This is the decodability cliff.

We propose to do the mixing in ordinary real arithmetic instead. The sender transmits the raw pixels themselves, the systematic packets, followed by parity packets that are real-valued weighted sums of whole raw packets. Because the raw signal is linear, every parity sample is a legitimate measurement of scene light. Its noise variance can be computed exactly from the sensor's noise model. With enough packets the frame is recovered exactly, up to a quantization step we place below the sensor's own read noise. With fewer packets, every packet that did arrive, parity included, feeds a learned restoration network, so quality degrades gradually rather than collapsing. In two sentences: raw sensor data is the one kind of image data where real-valued linear coding is physically natural. Exploiting that gives a fountain code with no cliff, whose partial receptions are directly usable by raw-domain vision models.

Positioning

The closest bridging work all operates on processed images. Zhao et al. give unequal protection to JPEG 2000 wavelet layers using systematic Raptor codes, a fountain code with a sparse outer pre-code. Wang et al. apply random linear network coding over GF(256) to H.264 video and give I-frames higher priority than P-frames. The content-based secure transmission work On Learning-Assisted Content-Based Secure Image Transmission fountain-codes regions of interest separately from background. All three motivate tiered protection. All three mix bits over finite fields, so their gradual quality comes only from discrete layers, never from partial parity.

Bursalioglu et al. are nearest in spirit. They pair Raptor codes with multiresolution quantization of Gaussian sources and use the source statistics inside belief-propagation decoding. The source there is abstract and memoryless, and the coding is still binary. Our proposal replaces the bit-plane construction with real-valued combinations of a physical signal whose noise is known.

On the sensor side, ABO shows that offloading packed Bayer data to an edge server beats a JPEG pipeline in both latency and detection accuracy. It assumes a reliable transport, so it constrains our latency budget and supplies our application setting. Learning to See in the Dark and Dark-ISP supply the restoration and detection backbones that our decoder extends. Both assume a complete raw array.

Network Coding Meets TCP supplies the transport abstraction we reuse: acknowledging degrees of freedom rather than specific packets. It also warns that Gaussian elimination can become a bottleneck. That warning shapes our decoder, as described next.

Core Claim

Let $x_1,\dots,x_K$ be the $K$ raw packets of a frame, each a vector of $N$ Bayer samples after black-level subtraction. Interleaving spreads every packet across the image and across all four color sites. A parity packet is

$$y_j = \sum_{i \in S_j} a_{ji}\, x_i,$$

where $S_j$ is a small random set of packets drawn from a sparse degree distribution, as in fountain codes, and the $a_{ji}$ are real Gaussian weights generated from a seed carried in the header. The weights are shared by all $N$ samples of a packet. Decoding therefore needs only one small $M \times K$ least-squares factorization per frame, where $M$ is the number of received packets, followed by an $O(KN)$ application to the samples. There is no dense inversion over the full image.

Under the Poisson-Gaussian model, a raw sample with signal $s$ has variance $\alpha s + \sigma^2$, where $\alpha$ is the sensor gain and $\sigma$ the read-noise standard deviation. Consequently each parity sample has variance $\sum_i a_{ji}^2(\alpha s_i + \sigma^2)$, plus quantization noise. We quantize parity values with a step comparable to $\sigma$, which is affordable because the low-order raw bits are already dominated by read noise.

When fewer than $K$ packets arrive, the decoder computes a noise-whitened minimum-norm estimate on the received span. It passes this estimate, together with a confidence map, to a raw restoration network. The network is a U-Net in the style of See-in-the-Dark, or the Dark-ISP detector front end.

The claim we will test is that, at equal transmitted bits, received parity packets improve both reconstruction quality and detection accuracy below full rank. Finite-field parity cannot do this.

Research Program

The work has three parts.

Code design. We will choose the degree distribution, weight distribution, and parity quantizer to trade three things against each other: overhead at full rank, conditioning of the real-valued system, and value of the partial estimates below rank. We will also test whether giving the green sites and the high-order bits denser parity coverage helps, which is the Bayer-specific unequal protection idea, in place of uniform coverage.

Decoder. We will train a restoration network on simulated erasure patterns. Its inputs are the least-squares estimate, a per-sample residual-variance map, and an optional raw prior. It is trained end to end for image fidelity and, separately, for detection loss.

Systems measurement. We will time encoding on a Jetson-class device and decoding on an edge GPU, inside ABO's 40 ms real-time budget.

Baselines

Naive reference. Raw packets are sent uncoded over a reliable transport with retransmission. This measures the latency and frame drops that the problem creates in the first place. A second naive point is ABO-style raw offloading with lost packets simply dropped and the frame rejected when incomplete.

Strongest existing method. We will use systematic Raptor coding over GF(256) on the same interleaved raw packets, following the systematic Raptor constructions surveyed in Primer and Recent Developments on Fountain Codes, with unequal protection in the style of Zhao et al. Below rank, the receiver feeds whatever systematic packets arrived into the same restoration network, retrained with missing-sample masks. This is the critical competitor because it already degrades somewhat gracefully, through its systematic packets plus inpainting. It differs from our method only in whether parity packets are usable below rank.

Fairness. Both systems get the same transmitted-bit budget, with our parity quantization counted honestly. They also share the same erasure traces (independent and bursty two-state channels), the same restoration architecture and training budget, and the same tuning sweep over overhead.

Isolating ablation. We will replace our real-valued parity with additional systematic packets at the same bit cost. This shows whether the parity mixtures, rather than just more raw samples, carry the benefit. A second ablation turns off the noise-aware whitening.

What counts as beating the strongest baseline. - Below full rank: higher PSNR and mAP at each received fraction. - At full rank: no more than a small overhead penalty. - Robustness: lower frame-failure rate under bursty loss.

Reported numbers. No reported result was found on Lacuna for any fountain or network code applied to raw sensor data. The clean-input ceilings we will cite are 64.9 and 70.4 mAP on LOD for Dark-ISP with ResNet-18 and ResNet-50 (Dark-ISP). The latency reference is ABO's reported 37 ms end-to-end at 27 FPS (ABO). For the cooked-domain analogue, Zhao et al. report frame failure falling from 81.23% under equal protection to 16.72% under unequal protection for their shielded receiver.

Why not just…?

Why not just interleave raw pixels and inpaint what is lost? This is the most serious threat. Interleaving turns packet loss into scattered missing samples, and a strong raw network may fill these in well. Our design includes this baseline as its systematic phase. The question is only whether real parity beats spending the same bits on more raw samples, and the isolating ablation answers it directly.

Inpainting cannot recover exact values. Parity can, once rank is reached, and it helps whenever a burst removes one specific packet. If the ablation shows no gain below rank, we will narrow the contribution to the full-rank and burst-loss regime and report that honestly.

Why not just use standard Raptor or network coding over GF(256)? Over finite fields, received parity is unusable until the system can be fully inverted. Below rank only the systematic packets help, which is exactly the strongest baseline above.

Real arithmetic has two known costs: worse numerical conditioning and extra bits per parity value. We will measure both rather than assume them away. Finite fields remain preferable when payloads are compressed bitstreams, which is why we restrict the claim to linear raw data.

Why not just send ISP-processed images over the existing cooked-domain codes? Wang et al. and Zhao et al. already do that well. However, processing clips highlights and crushes shadows irreversibly. Beyond RGB reports sRGB detection dropping by up to 13.6% in snow against 2.5–3.5% for raw input, and ABO reports up to 15% higher mAP from raw offloading. We will include a Raptor-over-JPEG arm to show whether the raw advantage survives transmission loss.

Why not just retransmit? Network Coding Meets TCP reports standard TCP throughput falling to 0.007 Mbps at 5% per-link loss, against 0.39 Mbps with a coding layer. In addition, every retransmission round trip eats into a 40 ms budget. The naive baseline will quantify this on our traces.

Proposed Experiments

Datasets and channels. We will use the Sony Bayer subset of SID, measured by PSNR and SSIM against long-exposure references. We will use LOD and PASCALRAW for detection mAP with the Dark-ISP backbone, and the ABO RAW dataset for latency and frames per second on the Jetson Nano offloading setup. Erasure traces will be independent loss at 1–30% and bursty two-state loss at matched average rates.

Decisive checks. - Quality versus received fraction. We will plot quality against the fraction of $K$ packets received, from 0.5 to 1.2. The method must lie above the systematic-Raptor-plus-inpainting curve below rank and match it within a few percent of overhead at rank. - Isolating ablation. The parity-versus-extra-raw-samples ablation must favor parity, especially under bursty loss. - Quantization. Raising the parity quantization step from below $\sigma$ to several times $\sigma$ should leave quality flat up to roughly $\sigma$ and then degrade. This would confirm that read noise sets the bit budget. - Latency. End-to-end latency, including the per-frame $M \times K$ solve, must stay under 40 ms at ABO's frame sizes.

If the first two checks fail, the idea reduces to raw inpainting, and we will say so.