Cameras on vehicles, drones and factory floors send pictures that are damaged twice. Rain, haze, snow and water on the lens damage the scene before it is captured. The network then damages the compressed picture on the way, because parts of it arrive late or not at all. Research treats these two problems separately. "All-in-one" restoration networks, meaning single models trained to remove many kinds of corruption, assume the receiver has an intact image. Loss-resilient learned image compression, meaning neural codecs designed to decode sensibly when some packets are missing, assumes the scene itself is clean.
This note proposes one receiver-side model that handles both, built on an asymmetry between them. The receiver never knows how much it is raining, but it always knows exactly which packets it failed to get by the frame deadline. We therefore feed that erasure map to the restorer as an explicit input, so the network spends its capacity inferring only the corruption it cannot observe directly. We also train the codec end to end from weather-degraded input to clean target, so the encoder learns not to spend bits describing rain streaks. The link to network systems is the operating regime. The deployments that most need this are confined to narrow transport silos: browser-only JavaScript APIs, TCP-only overlays, and UDP-only hardware stacks without fragmentation. Each silo produces a different, predictable pattern of missing data, and the design and evaluation are organized around those patterns.
Core Claim
Under any protocol restriction, the receiver can reduce the transport's behavior to a known erasure mask at the frame deadline. Delay in a TCP-only silo becomes erasure once a packet misses the deadline, and loss in a UDP silo is erasure directly. Conditioning a joint decoder-restorer on this known mask should beat three things on compound degradation, measured by both pixel fidelity and downstream detection. The first is a cascade of the best loss-resilient codec followed by the best all-in-one restorer. The second is the same joint model trained without the mask, which must guess the erasures blindly. The third is transport-level protection at matched bitrate and latency.
The specific novelty is twofold: an observed channel-state input to an all-in-one restorer, and codec training whose reconstruction target is the clean scene rather than the captured frame. Neither appears in the closest work.
Where This Sits
The closest work is Every Packet Counts, which supplies our method for the network half. It makes a learned codec survive packet loss by three means. It spreads information evenly across latent channels through inter-channel redistribution. It assigns those channels to packets in an interleaved pattern, so each packet samples the whole image. It limits how far one missing packet can corrupt later decoding by keeping the decoder's sequential dependencies to two layers. It reports a 1.84 dB PSNR gain over LossResilientLIC at 20% packet loss on Kodak and CLIC2020. Our proposal differs in three ways:
- Its targets are clean images, so it never removes weather.
- It assumes the side-information stream (the hyperprior, a small header carrying the codec's probability model, under 8% of the data) arrives losslessly through out-of-band error correction.
- Its packetization lives in tensor space, with no connection to real datagram sizes or transport behavior.
The restoration half draws on three all-in-one designs that motivate the approach and also constrain it:
- AirNet learns a degradation "fingerprint" by contrasting patches. It reports 34.90 dB on Rain100L.
- M2Restore sends image regions to specialist sub-networks using degradation probabilities from a frozen DA-CLIP vision-language model. It reports 31.65 dB PSNR and 0.936 SSIM averaged over the All-weather benchmark.
- The architecture-search model of Li, Tan and Cheong uses weather-specific encoders and physics-inspired operations. It reports 24.71 dB on Outdoor-Rain.
All three estimate degradation purely from pixels. The file flags a specific risk here: AirNet's patch-contrast encoder assumes a degradation is consistent across patches of one image. Blocky packet erasures break that assumption, so the encoder may classify an erasure pattern as a weather type. Our known-mask input removes that ambiguity rather than asking the vision model to resolve it.
The physical interaction of rain streaks with mist veiling, documented by Yang et al., is why we do not model weather as additive. RESIDE constrains how we evaluate. It found that dehazers winning on synthetic PSNR often helped object detection on real hazy images (its RTTS set) less than classical priors did. PSNR alone cannot be our success criterion.
On the network side, four papers define the silos we evaluate:
- Callejo et al. show that code running in the browser gets no raw sockets.
- MOR inherits a TCP-only limit from Tor.
- Tran et al. report that public mobile testbeds deny the kernel access needed to change transport behavior.
- Beehive runs UDP at line rate on an FPGA but omits IP fragmentation, TCP connection setup and congestion control.
These papers explain why "just fix the transport" is often unavailable. They supply no restoration method.
Research Program
Encoder. The model is a learned codec. Its encoder takes the weather-degraded frame. Its decoder-restorer outputs the clean frame, and the loss is computed against the clean ground truth. The encoder is kept small so that it could run on an embedded camera.
Packetization under a silo constraint. Latents are dispersed and interleaved as in Every Packet Counts. Each packet's payload is then sized to fit the silo's constraint. For a Beehive-like UDP stack, a packet must fit one datagram below the path MTU (maximum transmission unit), because nothing will fragment it. For a TCP-only or WebSocket path, packets are application-level chunks.
Protecting the hyperprior. Instead of assuming an error-protected side channel, we repeat a compressed copy of the hyperprior across several packets. We then measure what that repetition costs in bitrate.
Decoder-restorer. At the frame deadline, the receiver builds a binary mask over latent groups marking which arrived. The decoder-restorer takes three inputs: the partial latents, the mask, and a small embedding of recent loss statistics (loss rate and burst length over the last second). The restorer body is an AirNet-style degradation encoder paired with a mask-conditioned modulation layer, trained with both a pixel loss and a detection-feature loss. The detection-feature loss matches features from a frozen detector backbone, and it addresses the RESIDE concern about oversmoothing.
Training data and channel simulation. We train on All-weather pairs (Outdoor-Rain, Snow100K, Raindrop) and on RESIDE's outdoor set. Losses are simulated with a mix of uniform and Gilbert-Elliott channels, where Gilbert-Elliott is a two-state Markov model of bursty loss.
Three transport profiles. The models are then exposed to three profiles. The "UDP, no fragmentation" profile has direct losses. The "TCP-only" profile emulates head-of-line blocking over a lossy link, so late segments count as erasures at a 100 ms deadline. The "Tor-like" profile is TCP-only with added multi-hop delay variance.
Analysis questions. Two questions guide the analysis. First, does the mask let the degradation encoder's representation separate weather from erasure? We test this with a linear probe predicting weather type, run with and without the mask. Second, does clean-target training reduce bitrate at equal quality? Bits spent on rain streaks are wasted, and fewer packets per frame means less exposure to loss.
Baselines
Naive reference point. A standard learned codec receives a clean channel, and its output passes through no restoration. This fixes the floor on quality. It is also run with losses, to show how badly a codec without loss protection breaks.
Strongest existing combination. This is the cascade: the Every Packet Counts decoder, trained on degraded frames as its targets, followed by M2Restore trained on All-weather. Each component is the strongest published method for its half of the problem. The fairness rules for this comparison are as follows:
- We retrain both components on the same All-weather and RESIDE training split, with the same loss simulator.
- We match total parameters and GPU-hours to the joint model's budget, and give each an equal hyperparameter search budget.
- We compare at equal bitrate.
Published anchors are the 1.84 dB gain at 20% loss reported by Every Packet Counts, and M2Restore's 31.65 dB / 0.936 All-weather average. The latter was measured on intact inputs, so it is an upper reference, not a like-for-like number. Beating the cascade means two things at equal bitrate and 20% loss: higher PSNR on the All-weather test sets, and higher detection mAP on RTTS images passed through the channel. For RTTS we found no reported mAP result on Lacuna, so all mAP numbers will be our own runs, with a fixed detector.
The key ablation. The same joint model and training are used, but the erasure mask and loss-statistics inputs are replaced by zeros. The model still sees erasures during training. It just has to infer where they are. This isolates the central claim. If the masked and unmasked models are within noise, the claim fails.
Transport-protection baseline. A clean-target codec without loss resilience is sent with Reed-Solomon forward error correction. Its redundancy is tuned so that the total bitrate equals ours. It is then restored by M2Restore.
Why not just…?
Why not just chain the best codec and the best restorer? This is the most serious threat. The cascade's weakness is specific. The codec spends bits faithfully encoding rain. The restorer then sees erasure artifacts it was never trained on, and the file flags that AirNet-style patch-contrast encoders may misread them. Retraining the restorer on decoded, lossy outputs removes the second weakness. We therefore include that stronger cascade variant. If the retrained cascade matches the joint model at equal bitrate, the contribution shrinks to the mask input and clean-target coding, and we will report exactly that.
Why not just add forward error correction or use TCP? In the silos we target, the transport cannot be reconfigured. Browsers expose no sockets, mobile testbeds deny kernel changes, Tor carries only TCP, and Beehive's hardware stack lacks congestion control. Where TCP is available, retransmission over bursty loss converts loss into unbounded delay. That is why our TCP-only profile scores late packets as erasures. Error correction is available everywhere, but it costs bitrate. At fixed bitrate it must overcome burst losses longer than its redundancy covers. The matched-bitrate comparison under Gilbert-Elliott bursts settles this. Every Packet Counts' 77% FLOP reduction over ResiComp suggests learned dispersion is cheap, but that number was not measured against error correction.
Why not just restore on the camera before encoding? Then only channel loss remains, and existing codecs suffice. This is a real alternative whenever the sender has spare compute. M2Restore's 0.17 s per 224×224 image was measured on an RTX 3090, which is well beyond an embedded camera. Sender-side restoration also cannot recover from network loss. We run this variant with M2Restore at the sender, followed by the Every Packet Counts codec, and report quality against sender compute. If it wins at comparable sender compute, the proposal's claim narrows to devices that cannot restore locally.
Why not just train one blind network on mixed weather and loss? This is the mask ablation above. The prediction is that blind inference wastes capacity localizing erasures that the transport already reports.
Proposed Experiments
Main comparison. We report PSNR, SSIM and bitrate on Outdoor-Rain, Snow100K-L and Raindrop after transmission. RTTS detection mAP is measured with a fixed pretrained detector. Settings cover uniform loss rates of 0–30%, Gilbert-Elliott bursts, and the three protocol profiles. Kodak and CLIC2020 with no weather added check that the joint model does not lose to Every Packet Counts on its own clean-scene task.
Decisive checks. Five results would decide the claim:
- The mask ablation gap at 20% loss must exceed seed variance across three seeds.
- The joint model must beat the retrained cascade on RTTS mAP, not only on PSNR.
- The linear-probe accuracy for weather type must rise when the mask is supplied.
- In-band hyperprior repetition must keep decoding alive when hyperprior packets are dropped, where the baseline fails outright. We will report the bitrate overhead of repetition.
- Performance must hold when a model trained on uniform loss is tested on the TCP-only deadline profile. This checks whether "delay becomes erasure" truly unifies the silos, or whether each silo needs its own model.
References
- Every Packet Counts: Dispersing Information for Loss-Resilient Learned Image Compression
- All-In-One Image Restoration for Unknown Corruption
- M2Restore: Mixture-of-Experts-Based Mamba-CNN Fusion Framework for All-in-One Image Restoration
- All in One Bad Weather Removal Using Architectural Search
- Deep Joint Rain Detection and Removal from a Single Image
- RESIDE: A Benchmark for Single Image Dehazing
- Opportunities and Challenges of Ad-based Measurements from the Edge of the Network
- MOR: Monitoring and Measurements through the Onion Router
- Voice-Activated Applications and Multipath TCP: A Good Match?
- Beehive: A Flexible Network Stack for Direct-Attached Accelerators