RB-LDC: Redundancy-Balanced Latent Coding for Noise-Robust Diffusion Latents

Hyunseok Jeong, Jaeho Jeon, Young-Sik KimarXiv 2026 · arXiv

TL;DR

RB-LDC (Redundancy-Balanced Latent Coding) introduces an invertible linear layer between the encoder and the diffusion process to protect the latent directions that the decoder is most sensitive to. By redistributing signal-to-noise ratio (SNR) according to a theoretically derived power law, it significantly improves image reconstruction from noisy latents and enhances the robustness of latent diffusion models (LDMs) without altering clean reconstruction performance.

What Problem Does This Paper Address?

Latent Diffusion Models (LDMs) operate by adding and removing noise in a compressed latent space. However, not all parts of this latent space are created equal. After standard "whitening" (normalizing the data), the forward diffusion process treats every latent direction as equally important, corrupting them at the same SNR.

In reality, the decoder—the component that turns latents back into pixels—is much more sensitive to some directions than others. A small error in a "high-sensitivity" direction might cause a massive artifact in the final image, while a larger error in a "low-sensitivity" direction might be invisible. This paper addresses this "isotropic noise vs. anisotropic sensitivity" mismatch by asking: How should we distribute latent energy across directions to minimize the final image distortion?

Methods

The authors propose inserting an invertible linear layer, $G$, between the tokenizer's encoder and the diffusion process. This layer acts as a "precoder" that shifts the log-SNR of each latent direction.

Key technical components include:

  • Sensitivity Estimation: They use the "expected pullback metric" of the decoder. This is essentially a Jacobian-based measure of how much the output image changes relative to changes in the latent input.
  • LMMSE Risk Optimization: They minimize a decoder-sensitivity-weighted Linear Minimum Mean-Square Error (LMMSE) risk, integrated over the entire diffusion schedule.
  • Deterministic Spectral Solver: Instead of heavy training, they provide an algorithm that finds the optimal layer $G$ by aligning its eigenvectors with the decoder's sensitivity matrix and solving for the optimal eigenvalues (energy allocation).
  • The Power Law: A major theoretical contribution is proving that in the large-budget limit, the optimal energy allocation follows a power law relative to sensitivity.

Datasets

The researchers evaluated their method on:

  • ImageNet 256 × 256: Used for primary reconstruction and generation experiments.
  • Tokenizers: They tested with the standard Stable Diffusion VAE (SD-VAE) and the more recent VA-VAE.
  • Pre-trained Models: Evaluation included the LightningDiT-XL/1 model.

Key Insights and Ideas

  • The Schedule Matters: The optimal way to protect latents isn't just a property of the decoder; it depends heavily on the diffusion loss weighting (e.g., whether the model predicts noise or the original data).
  • Fundamental Exponents: Under a standard cosine schedule, the optimal energy allocation follows a power law with an exponent of 1/2 for noise prediction and 2/3 for data prediction. This is illustrated in Figure 1, which shows how the slope of the allocation changes as the budget increases.
Figure 1: Slope of energy allocation vs. sensitivity. SNR-based weighting (noise prediction) approaches the 1/2 square-root law, while uniform weighting (data prediction) stays near 2/3.
  • Redundancy Balancing: The name RB-LDC comes from the idea of "redundancy," referring to the extra protection energy assigned to critical latent directions to balance the marginal risk reduction across the entire space.
  • Oracle Validation: In an "oracle" experiment (Table 1), they showed that correcting errors in high-sensitivity directions reduces image MSE by nearly 19%, whereas correcting low-sensitivity directions with the same energy budget reduces MSE by only 0.33%.
Table 1: Oracle correction of DiT predictions on 256 images (seed 42). High and Low correct the top and bottom 16 eigendirections of the primary identity-Φ sensitivity estimate. Removed energy is the mean percentage of total whitened error energy removed per image, matched between corrections to numerical precision. Image MSE is averaged over images against their clean reconstructions; lower is better.

Key Results

RB-LDC consistently outperformed standard isotropic coding (treating all directions equally) across multiple metrics:

  • Reconstruction Gains: On VA-VAE with block-local coding, it reduced the average perturbed FID (pFID) by 17.07 points.
  • Metric Improvements: It showed gains in PSNR (+0.855 dB), SSIM (+0.0337), and LPIPS (-0.0346) on the primary configuration. Detailed comparisons across different tokenizers and coding schemes are shown in Table 2.
Table 2: Mean ± standard deviation across seeds 42–44 of the 15-SNR-average metric differences (log10 ρ ∈[−4, 3], Φ equal to the identity). Differences are RB-LDC minus whitened isotropic coding. Lower is better for pFID and LPIPS; higher is better for PSNR (dB) and SSIM.
  • Frozen Model Robustness: Even with a frozen diffusion model, RB-LDC yielded lower FID point estimates in all 12 tested combinations of sampling progress and channel noise.
  • Zero Cost to Clean Data: Because the layer $G$ is invertible, the reconstruction of "clean" (non-noisy) latents remains mathematically unchanged.

Limitations and Future Work

  • Linearity: The current method uses a linear layer. While efficient, non-linear coding layers might offer even better protection.
  • Block-Local Assumption: The block-diagonal coding assumes spatial independence between blocks, which might ignore some cross-block sensitivities.
  • Surrogate Risk: The theory relies on LMMSE as a surrogate for perceptual quality. While it correlates well, it doesn't perfectly capture the complex "distortion-perception" trade-off at very high noise levels.
  • Future Directions: The authors suggest that joint training of the tokenizer and the coding layer, as well as input-dependent (dynamic) allocations, are natural next steps.

Summary updated

Abstract

Latent diffusion models decode approximately correct latents, making decoder sensitivity to perturbations relevant to image quality. After whitening, forward diffusion corrupts every direction at the same signal-to-noise ratio (SNR), despite unequal decoder sensitivities. We study an invertible linear layer between the encoder and diffusion process that shifts each direction's log-SNR. Its linear minimum mean-square-error (LMMSE) recovery risk has an exact spectral expression for any whitened latent distribution. Under an energy budget and a floor ensuring invertibility, minimizing decoder-sensitivity-weighted LMMSE risk integrated over the diffusion schedule yields a unique optimal allocation. The layer aligns with the decoder's expected pullback metric and equalizes marginal risk reduction across directions, computed by a deterministic spectral solver. At large budgets, the allocation follows a power law in sensitivity whose exponent is determined by the low-SNR tail of the loss weighting: one half for noise prediction and two thirds for data prediction under the cosine schedule, with a crossover on truncated schedules. We instantiate this principle as RB-LDC using Jacobian-based sensitivity estimates. On ImageNet at 256 x 256 resolution with SD-VAE and VA-VAE, trained RB-LDC improves reconstruction from noisy latents over budget-matched isotropic coding in all four metrics and five coding configurations, averaged over 15 SNRs and three pipeline seeds. It reduces average perturbed FID by 17.1 for VA-VAE block-local coding while preserving clean reconstruction. We also report intermediate SNRs where perturbed FID worsens. With a frozen diffusion model, RB-LDC yields lower FID point estimates in all 12 tested combinations of sampling progress and channel SNR under corruption during sampling.
BibTeX
@misc{jeong2026rb,
  author = {Hyunseok Jeong and Jaeho Jeon and Young-Sik Kim},
  title = {{RB-LDC: Redundancy-Balanced Latent Coding for Noise-Robust Diffusion Latents}},
  year = {2026},
  eprint = {2611.13955},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.13955},
}
Figure