Most image and video formats store color as one brightness channel (luma, Y) plus two color-difference channels (chroma, Cb and Cr). The chroma channels are usually stored at half the width and half the height, a format called 4:2:0, and codecs quantize them coarsely. Restoration networks mostly ignore this. Compression-artifact removers such as AR-CNN and DMCNN train and report results on Y alone. Video enhancers such as QE-CNN, STResNet and DCNGAN each list chroma as future work. The color is left to bicubic interpolation, so it stays blocky and bleeds across object edges even after the luma has been sharpened.
The obvious fix is to reuse the standard tool for keeping fine detail: an encoder-decoder with skip connections, which are shortcuts that copy high-resolution early features straight to the decoder. For chroma, though, the standard tool copies the wrong thing. The early chroma features are exactly where the quantization damage (8×8 blocking, ringing, banding) lives. A skip connection carries that damage around the network's denoising bottleneck and into the output. We propose that the chroma decoder's fine-scale skip connections should come from the luma encoder instead. Luma is stored at twice the resolution, is quantized more gently, and marks almost all object boundaries. The chroma stream's own early features would be admitted only through a learned gate that opens where luma carries no edge. The claim is testable in a narrow and falsifiable way. At matched parameter count, luma-routed skips should reduce color bleeding and chroma blocking near edges more than either ordinary same-channel skips or a plain three-channel network, and the gain should grow as chroma quantization gets harsher.
Core Claim and How It Differs From Prior Work
The work that motivates this proposal documents the neglect itself. Dong et al. (AR-CNN) and Hou et al. (Quality Enhancement of Compressed Video via CNNs) both note that luma-only models leave color-specific quantization noise in place, and that this noise is most visible in vivid color transitions. The work that supplies the method is RED-Net, which showed that symmetric skip connections let very deep restoration networks recover fine detail and train stably. RED-Net passes every early feature through unfiltered and assumes all channels share one resolution. We keep its mirrored encoder-decoder shape but change where each skip originates.
The closest existing work is Hewei et al. (Luminance-Guided Chrominance Image Enhancement for HEVC Intra Coding). They restore 4:2:0 chroma with a recursive gated chroma branch and add in luma features that a strided convolution has reduced to chroma resolution. They report average BD-rate savings of 28.96% for U and 16.74% for V over HEVC intra coding. BD-rate is the bitrate saved at equal quality. Their design differs from ours in three ways that matter here. It injects luma at a single scale. It reaches chroma resolution with a strided convolution, which throws away the sub-pixel position of luma edges. Its output stays at 4:2:0, so it cannot put a color edge between two chroma samples. In our design, luma enters at every decoder scale through pixel-unshuffle, a lossless rearrangement that folds each 2×2 luma block into four channels at chroma resolution. The finest decoder stage then predicts full-resolution 4:4:4 chroma, so color edges can land exactly on luma edges.
Two other papers constrain the design rather than supply it. ICLR runs luma and chroma as two interacting U-Net streams in the HVI color space for low-light images, reaching 28.603 dB PSNR on LOLv1. It finds that pixel losses give conflicting gradients in flat color regions, which produce blotches. That tells us gating must not let luma texture leak into flat chroma. ICLR assumes full-resolution, uncompressed chroma input, so it does not face our resolution mismatch or quantization damage. SFIGF links cross-attention to the classical guided filter for transferring structure from a guidance image to a target. We borrow its insight as a baseline: a guided-filter branch keeps the output locally linear in the guide, which limits hallucination. SFIGF was never applied to luma guiding chroma inside a compressed image.
For decoder scale $s$, let $d_s$ be the chroma decoder feature, $e^Y_s$ the luma encoder feature at twice the chroma resolution, $e^C_s$ the chroma encoder feature, $P$ pixel-unshuffle followed by a 1×1 projection, and $g_s \in [0,1]$ a per-pixel gate predicted from local luma and chroma gradient magnitudes. The update is
$$d_s = D_s(d_{s+1}) + P(e^Y_s) + g_s \odot e^C_s .$$
Here $D_s$ is the ordinary upsampling decoder block, and $\odot$ is element-wise multiplication. The luma path is always open. The chroma self-path is open only where the gate chooses. The gate exists because of isoluminant edges, meaning color boundaries with no brightness change, such as red text on a green background of equal brightness. There luma carries no information, and only chroma's own features can locate the edge.
Research Program
We will build two matched networks: a small one of about 1M parameters and a medium one of about 4M, the latter comparable to ICLR's 4.27M. Each has a luma encoder at full resolution and a chroma encoder-decoder at 4:2:0 resolution with a final 2× stage to 4:4:4. We will train them on Flickr2K, converted to 4:2:0 and compressed with HEVC intra coding at QP 22/27/32/37 following Hewei et al.'s protocol, and separately with JPEG at quality factors 10 to 40 using 4:2:0 subsampling. The luma branch is either restored jointly or frozen after pretraining. Both settings are reported, because a frozen luma branch isolates the routing question from any benefit of joint training.
The analysis has two parts. The first is a skip-contamination probe. We measure how much energy each skip feature map carries at the 8-pixel and 16-pixel block-grid frequencies, where blocking artifacts appear in the spectrum. We compare same-channel skips, luma-routed skips, and gated skips. If our mechanism is right, same-channel chroma skips should carry measurable grid energy, and luma-routed skips should carry little. The second part is a failure-mode study on a synthetic isoluminant set: rendered text and shapes whose colors are matched in luma, then compressed. It tests whether the gate actually recovers edges that luma cannot see.
Baselines
The do-nothing reference is the decoded image with bicubic chroma upsampling to 4:4:4. The chroma-neglect reference applies the same luma restoration our network uses and still upsamples chroma bicubically, which mirrors the AR-CNN and DMCNN convention. Their reported LIVE1 quality-10 results, 28.98 dB for AR-CNN and 29.73 dB for DMCNN, are luma-only. They show what sharpened luma alone achieves, not a chroma target.
The classical guided reference upsamples and filters chroma with a guided filter driven by the restored luma. Its window size and regularization are tuned on a held-out split. This baseline tests whether any learning is needed.
The strongest dedicated competitor is Hewei et al.'s luma-guided chroma network, reimplemented at matched parameter count and trained on the identical Flickr2K/HEVC data with the same number of iterations and the same hyperparameter-search budget. Its 4:2:0 output is upsampled bicubically for the 4:4:4 comparison. We will also compare in its native 4:2:0 setting on Classical, McMaster and Kodak, against its reported 28.96% (U) and 16.74% (V) BD-rate savings. Beating it means lower BD-rate on both chroma channels in that native setting, plus lower edge-band chroma error at 4:4:4.
The general-purpose competitor is a RED-Net-style three-channel U-Net. It receives bicubic-upsampled 4:4:4 YCbCr input, has ordinary skips, and matches our parameter count and training budget. SFIGF is run with luma as guide and chroma as target at matched size. No reported result was found on Lacuna for SFIGF, RED-Net or ICLR on compressed 4:2:0 chroma.
The key ablation swaps only the skip source in our network. It compares chroma-only skips, luma-only skips with no gate, and the full gated design, with everything else held fixed.
Why not just…?
Why not just train a three-channel U-Net on upsampled chroma? It sees luma and could learn to suppress chroma noise itself. This is the real threat. Nothing in the file shows that it fails; RED-Net and aRGB report only RGB or luma results. The matched-parameter comparison settles the question directly, and so does the skip-contamination probe. If the three-channel net matches us on edge-band chroma error and bleeding, and its skips carry no grid energy, the routing claim is false. We would then report that the network learned the separation implicitly.
Why not just use Hewei et al.'s network and upsample its output? A fixed strided convolution fixes where luma information can enter and discards sub-pixel edge positions. Bicubic upsampling afterward then reintroduces blur exactly at edges. The prediction is that Hewei's network closes most of the gap in flat regions but not in the one- to two-pixel band around luma edges. The edge-band metric is designed to show this.
Why not just use a guided filter on the chroma? The guided filter is cheap and edge-aware. It assumes chroma is locally linear in luma, so it cannot remove quantization noise that has no luma counterpart, and it copies luma texture into flat color regions. SFIGF's gap of 5.60 against 10.9 RMSE over a plain guided filter on depth super-resolution suggests learned fusion helps a lot in guided tasks. We test it here rather than assume it.
Why not just change the loss, for example with aRGB? aRGB replaces the pixel loss with one computed in a learned 128-dimensional color-structure space and improves texture, lowering Urban100 FID from 20.61 to 17.73 with ESRGAN. A loss changes what the output is pushed toward, not what the decoder is allowed to see. The two are complementary, so we will also train our network and the three-channel U-Net with aRGB to check that our advantage survives a better loss.
Proposed Experiments and Decisive Checks
We evaluate on Kodak, McMaster and Classical under HEVC intra coding, on LIVE1 and BSDS500 under 4:2:0 JPEG, and on DIV2K validation. The metrics are PSNR on Cb and Cr separately, PSNR-B on chroma, chroma error in a narrow band around luma edges versus flat regions, and color-bleeding confinement measured with the Cluster Discrepancy Ratio from Deep Edge-Aware Interactive Colorization. That paper used this super-pixel metric only for interactive colorization; we use it for automatic restoration. For the codec setting we also report BD-rate on U and V.
Three results would support the proposal. First, edge-band chroma error and bleeding confinement improve over both the three-channel U-Net and Hewei's network at matched size. Second, the margin widens from QP 22 to QP 37 and from quality factor 40 to 10, as expected if the gain comes from avoiding corrupted chroma skips. Third, the gated model beats the luma-only-skip ablation on the isoluminant set while matching it elsewhere. If luma-routed skips give no gain over the three-channel U-Net at high QP, the hypothesis is rejected.
As an extension, the same routing could be applied to the chroma branch of a multi-frame enhancer on JCT-VC sequences with LPIPS and DISTS, addressing the chroma gap DCNGAN names. It is a secondary test, run only if the image results hold.
References
- Compression Artifacts Reduction by a Deep Convolutional Network
- Dmcnn: Dual-Domain Multi-Scale Convolutional Neural Network for Compression Artifacts Removal
- Enhancing Quality for HEVC Compressed Videos
- Spatial-temporal residue network based in-loop filter for video coding
- DCNGAN: A Deformable Convolution-Based GAN with QP Adaptation for Perceptual Quality Enhancement of Compressed Video
- Quality Enhancement of Compressed Video via CNNs
- Image Restoration Using Very Deep Convolutional Encoder-Decoder Networks with Symmetric Skip Connections
- Luminance-Guided Chrominance Image Enhancement for HEVC Intra Coding
- ICLR: Inter-Chrominance and Luminance Interaction for Natural Color Restoration in Low-Light Image Enhancement
- Guided Image Restoration via Simultaneous Feature and Image Guided Fusion
- Rethinking RGB Color Representation for Image Restoration Models
- Deep Edge-Aware Interactive Colorization against Color-Bleeding Effects