Universal Representations for the Rate-Distortion-Perception Tradeoff

16 concepts5 papers1 proposal

In traditional lossy compression, we typically optimize for a single objective: minimizing the mathematical difference between the original and the reconstruction, usually measured by Mean Squared Error (MSE). However, as neural compression has matured, a tension has emerged. Models that achieve the lowest MSE often produce "blurry" images, while models optimized for perceptual realism (looking "real" to a human) often introduce higher mathematical error.

This tension is formalized as the Rate-Distortion-Perception (RDP) tradeoff. For a long time, the assumption was that navigating this tradeoff required training separate, specialized models for every desired point on the spectrum. Recent research, however, reveals a surprising "universality": a single, fixed encoder can capture all the information necessary to satisfy any point on the RDP curve.

The Perception Tax and the Three-Way Tradeoff

To understand why universality matters, we must first understand the "perception tax." As Yan et al. (2021) quantify, there is a fundamental cost to making an image look real. If you force a decoder to produce an output that is statistically indistinguishable from real images, you inevitably increase the mathematical distortion (MSE).

This creates a three-way tug-of-war:

  1. Rate: How many bits are used?
  2. Distortion: How close are the pixels to the original?
  3. Perception: How "authentic" does the reconstruction look?

The Case for a Universal Encoder

The core insight across this body of work is that the encoder’s job is simpler than we thought. Instead of needing to know whether the final user wants a "perfectly accurate" or a "perfectly realistic" image, the encoder only needs to be a good feature extractor.

Zhang et al. (2021) and Yan et al. (2021) demonstrate that an encoder optimized strictly for minimizing MSE is theoretically optimal for all subsequent perceptual tasks. The intuition is that an MSE-optimized bitstream contains the maximum possible information about the source. Once that information is captured, the specific balance between accuracy and realism can be decided entirely by the decoder.

This means we don't need specialized "perceptual encoders." As Salehkalaibar et al. (2023) show in the context of video, an MSE-trained encoder remains agnostic to the final perception metric. Whether the receiver cares about visual sharpness or temporal consistency, the same bitstream provides the necessary raw material.

Decoupling the Pipeline

If the encoder is fixed, how do we actually navigate the tradeoff? The answer lies in the decoder.

Yan et al. (2022) propose a remarkably simple architecture: pair one universal MSE encoder with two specialized decoders—one for maximum accuracy and one for maximum realism. By interpolating between the outputs of these two decoders, a user can traverse the entire optimal distortion-perception curve in real-time without changing the bitstream or retraining the model.

Is There a Penalty for Universality?

One might worry that a "jack-of-all-trades" encoder would be less efficient than one custom-tailored for a specific perceptual goal. However, information-theoretic analysis suggests otherwise.

Zhang et al. (2025) prove that using a universal representation incurs a negligible, and often zero, rate penalty compared to using specialized encoders. For Gaussian sources, the authors derive closed-form expressions showing that a fixed encoder can achieve the absolute theoretical limits of the RDP region. Further, Qu et al. (2025) establish that even with limited common randomness, the optimal encoder's structure remains independent of the specific perception constraint.

Why It Works: Intuition

The reason universality holds is that "distortion" and "perception" are not looking for different kinds of information; they are just different ways of interpreting the same information.

  • Distortion-optimized decoders use the bitstream to place pixels as close to their original values as possible, resulting in a "mean" reconstruction that looks blurry where information is missing.
  • Perception-optimized decoders use the same bitstream to ensure the output stays on the manifold of natural images, "hallucinating" plausible details where information is missing.

Because the MSE-optimized encoder captures the most information about where the original image sits relative to that manifold, it serves as the perfect foundation for both strategies. This shift from "specialized codecs" to "universal encoders with flexible decoders" simplifies the deployment of neural compression, allowing a single compressed file to satisfy diverse viewing requirements.

Key Foundational Work

Optimally Controllable Perceptual Lossy Compression (2022) — contributed 4 concept elements to this direction.
Universal Rate-Distortion-Perception Representations for Lossy Compression (2021) — contributed 2 concept elements to this direction.
On Perceptual Lossy Compression: The Cost of Perceptual Reconstruction and An Optimal Training Framework (2021) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_encoder_universality_e3edc09f,
  title = {encoder universality},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/encoder-universality/rdr_e3edc09f4db74c608b1a6df4fdea18de},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_encoder_universality_e3edc09f_v1,
  title = {encoder universality},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/encoder-universality/rdr_e3edc09f4db74c608b1a6df4fdea18de/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake