In computer vision, measuring how "similar" two images appear to a human observer is a notoriously difficult problem. Standard mathematical metrics like Mean Squared Error (MSE) or Peak Signal-to-Noise Ratio (PSNR) are computationally cheap but fail to capture human perception, as they focus on absolute pixel-level differences rather than structural integrity.
To bridge this gap, the field has increasingly relied on "Perceptual Metrics." The current gold standard, LPIPS, utilizes features extracted from massive, pre-trained deep neural networks (such as VGG or AlexNet) and often requires fine-tuning on expensive, human-annotated datasets. This creates a heavy dependency on large-scale data and computationally expensive models.
This paper asks a provocative question: Do we actually need deep features to align with human preference, or can simple linear math do the job? The authors introduce the Linear Autoregressive Similarity Index (LASI), a new perceptual image metric that requires zero training data and no deep neural networks, yet achieves performance competitive with state-of-the-art deep learning models.
The LASI Approach: Perception as Local Compression
Instead of relying on a neural network to extract high-level features, LASI treats perceptual embedding as a local compression problem. The method is built on psychological theories suggesting the human visual system performs "lossy compression" to manage sensory input. By forcing the metric to find the most efficient linear prediction for a pixel, LASI captures the structural regularities the human eye naturally prioritizes.
The algorithm operates through a few straightforward steps at inference time:
- Causal Neighborhoods: For every pixel in an image, the algorithm examines a small "causal neighborhood"—a set of nearby pixels that would have been visited earlier in a standard raster scan.
- Weighted Least Squares (WLS): The algorithm solves a WLS problem for each pixel to find a vector of weights that best predicts the current pixel's value based on its neighbors. Closer neighbors are given more weight using a Manhattan distance-based decay.
- Embeddings as Solutions: The resulting weight vectors (the solutions to the WLS problem) serve as the "perceptual embedding" for that pixel, representing the local structural logic of the image.
- Distance Calculation: To compare two images, LASI simply calculates the $L_2$ distance between their respective embedding matrices.
Crucially, this entire process is fully differentiable, meaning LASI can be used as a loss function to train other models, such as image compressors or generative networks.
Evaluation and Key Insights
The researchers evaluated LASI using the Berkeley Adobe Perceptual Patch Similarity (BAPPS) dataset, the industry standard for perceptual metrics. BAPPS includes Two-Alternative Forced Choice (2-AFC) tests—where humans choose which of two distorted images looks more like a reference—and Just-Noticeable Difference (JND) tests. The dataset covers both "traditional" distortions (noise, blurs) and "CNN-based" distortions (artifacts from neural network upsampling).
Without ever seeing a single training image or human label, LASI demonstrated remarkable capabilities:
- Competitive Performance: On the BAPPS 2-AFC benchmark, LASI scored within 2-5% of the best unsupervised deep learning methods.
- Superiority in Traditional Distortions: In categories involving standard image processing distortions like blurs and noise, LASI actually outperformed the "features-only" version of LPIPS.
- The "More is Better" Property: Unlike deep learning—where better classification doesn't always yield better perceptual metrics—increasing the dimensionality of LASI's embedding (the neighborhood size $N$) consistently improves both mathematical prediction and alignment with human judgment.
- Efficiency: LASI is faster than other unsupervised deep metrics like PIM, which require multiple forward passes through a neural network.
Complementary Failure Modes and Future Work
Interestingly, LASI and LPIPS do not fail in the same ways. Using a "Maximum Differentiation" (MAD) competition, the authors showed that LASI is more sensitive to neighborhood-grid artifacts, while LPIPS is sensitive to smoother, high-level distortions. Because these metrics have different "blind spots," the authors suggest that a combination of the two could yield an even more robust perceptual metric.
While highly effective, LASI does have limitations. Solving the WLS problem involves a Singular Value Decomposition (SVD) for every pixel. Although highly parallelizable on GPUs, the computational complexity scales cubically ($O(N^3)$) with the neighborhood size. Furthermore, experiments were primarily conducted on $64 \times 64$ image patches; future work will need to explore how LASI handles high-resolution, full-frame images where global structure may take precedence over local autoregressive patterns.
Summary updated