The Unreasonable Effectiveness of Linear Prediction as a Perceptual Metric

Daniel Severo, Lucas Theis, Jona Ballé

In computer vision, measuring how "similar" two images appear to a human observer is a notoriously difficult problem. Standard mathematical metrics like Mean Squared Error (MSE) or Peak Signal-to-Noise Ratio (PSNR) are computationally cheap but fail to capture human perception, as they focus on absolute pixel-level differences rather than structural integrity.

To bridge this gap, the field has increasingly relied on "Perceptual Metrics." The current gold standard, LPIPS, utilizes features extracted from massive, pre-trained deep neural networks (such as VGG or AlexNet) and often requires fine-tuning on expensive, human-annotated datasets. This creates a heavy dependency on large-scale data and computationally expensive models.

This paper asks a provocative question: Do we actually need deep features to align with human preference, or can simple linear math do the job? The authors introduce the Linear Autoregressive Similarity Index (LASI), a new perceptual image metric that requires zero training data and no deep neural networks, yet achieves performance competitive with state-of-the-art deep learning models.

The LASI Approach: Perception as Local Compression

Instead of relying on a neural network to extract high-level features, LASI treats perceptual embedding as a local compression problem. The method is built on psychological theories suggesting the human visual system performs "lossy compression" to manage sensory input. By forcing the metric to find the most efficient linear prediction for a pixel, LASI captures the structural regularities the human eye naturally prioritizes.

Comparison of LASI and LPIPS architectures

The algorithm operates through a few straightforward steps at inference time:

  1. Causal Neighborhoods: For every pixel in an image, the algorithm examines a small "causal neighborhood"—a set of nearby pixels that would have been visited earlier in a standard raster scan.
  2. Weighted Least Squares (WLS): The algorithm solves a WLS problem for each pixel to find a vector of weights that best predicts the current pixel's value based on its neighbors. Closer neighbors are given more weight using a Manhattan distance-based decay.
  3. Embeddings as Solutions: The resulting weight vectors (the solutions to the WLS problem) serve as the "perceptual embedding" for that pixel, representing the local structural logic of the image.
  4. Distance Calculation: To compare two images, LASI simply calculates the $L_2$ distance between their respective embedding matrices.

Crucially, this entire process is fully differentiable, meaning LASI can be used as a loss function to train other models, such as image compressors or generative networks.

Evaluation and Key Insights

The researchers evaluated LASI using the Berkeley Adobe Perceptual Patch Similarity (BAPPS) dataset, the industry standard for perceptual metrics. BAPPS includes Two-Alternative Forced Choice (2-AFC) tests—where humans choose which of two distorted images looks more like a reference—and Just-Noticeable Difference (JND) tests. The dataset covers both "traditional" distortions (noise, blurs) and "CNN-based" distortions (artifacts from neural network upsampling).

Without ever seeing a single training image or human label, LASI demonstrated remarkable capabilities:

  • Competitive Performance: On the BAPPS 2-AFC benchmark, LASI scored within 2-5% of the best unsupervised deep learning methods.
  • Superiority in Traditional Distortions: In categories involving standard image processing distortions like blurs and noise, LASI actually outperformed the "features-only" version of LPIPS.
  • The "More is Better" Property: Unlike deep learning—where better classification doesn't always yield better perceptual metrics—increasing the dimensionality of LASI's embedding (the neighborhood size $N$) consistently improves both mathematical prediction and alignment with human judgment.
  • Efficiency: LASI is faster than other unsupervised deep metrics like PIM, which require multiple forward passes through a neural network.

Complementary Failure Modes and Future Work

Interestingly, LASI and LPIPS do not fail in the same ways. Using a "Maximum Differentiation" (MAD) competition, the authors showed that LASI is more sensitive to neighborhood-grid artifacts, while LPIPS is sensitive to smoother, high-level distortions. Because these metrics have different "blind spots," the authors suggest that a combination of the two could yield an even more robust perceptual metric.

While highly effective, LASI does have limitations. Solving the WLS problem involves a Singular Value Decomposition (SVD) for every pixel. Although highly parallelizable on GPUs, the computational complexity scales cubically ($O(N^3)$) with the neighborhood size. Furthermore, experiments were primarily conducted on $64 \times 64$ image patches; future work will need to explore how LASI handles high-resolution, full-frame images where global structure may take precedence over local autoregressive patterns.

Summary updated

Abstract

We show how perceptual embeddings of the visual system can be constructed at inference-time with no training data or deep neural network features. Our perceptual embeddings are solutions to a weighted least squares (WLS) problem, defined at the pixel-level, and solved at inference-time, that can capture global and local image characteristics. The distance in embedding space is used to define a perceptual similary metric which we call \emph{LASI: Linear Autoregressive Similarity Index}. Experiments on full-reference image quality assessment datasets show LASI performs competitively with learned deep feature based methods like LPIPS \citep{zhang2018unreasonable} and PIM \citep{bhardwaj2020unsupervised}, at a similar computational cost to hand-crafted methods such as MS-SSIM \citep{wang2003multiscale}. We found that increasing the dimensionality of the embedding space consistently reduces the WLS loss while increasing performance on perceptual tasks, at the cost of increasing the computational complexity. LASI is fully differentiable, scales cubically with the number of embedding dimensions, and can be parallelized at the pixel-level. A Maximum Differentiation (MAD) competition \citep{wang2008maximum} between LASI and LPIPS shows that both methods are capable of finding failure points for the other, suggesting these metrics can be combined.
BibTeX
@inproceedings{severo2024unreasonable,
  author = {Daniel Severo and Lucas Theis and Jona Ballé},
  title = {{The Unreasonable Effectiveness of Linear Prediction as a Perceptual Metric}},
  booktitle = {International Conference on Learning Representations},
  year = {2024},
  url = {https://openreview.net/forum?id=e4FG5PJ9uC},
}
Figure