Deep learning models have become the standard judges of digital media quality. In applications ranging from streaming video optimization to content moderation, No-Reference Image Quality Assessment (NR-IQA) and Video Quality Assessment (NR-VQA) models are deployed to mimic human perception without requiring a pristine "ground truth" reference image. On static benchmarks, these models achieve remarkably high correlations with human quality scores.
However, as Weixia Zhang et al. demonstrate in Perceptual Attacks of No-Reference Image Quality Models with Human-in-the-Loop, high dataset correlation does not guarantee robustness. These automated judges suffer from a critical vulnerability: they can be easily manipulated by imperceptible adversarial perturbations. This vulnerability decouples automated quality scores from actual human visual perception, as shown by Ao-Xiang Zhang et al. in Vulnerabilities in Video Quality Assessment Models: The Challenge of Adversarial Attacks. By injecting tiny, pixel-level changes, an attacker can trick a model into rating a heavily degraded video as high-quality, creating a massive security gap in automated media pipelines.
Why Quality Metrics Fail
To understand why quality metrics are so fragile, we must look at what they actually learn. Instead of understanding semantic quality or true visual fidelity, deep neural networks often rely on low-level statistical shortcuts, such as specific textures and high-frequency patterns, as proxies for sharpness and clarity.
In Adversarial Photo Frame: Concealing Sensitive Scene Information of Social Images in a User-Acceptable Manner, Zhengyu Zhao et al. observed that automated aesthetic models inherently prefer certain textures, mistakenly assigning higher scores to images containing adversarial noise. Attackers exploit this preference by hiding perturbations in high-frequency and highly textured regions where human eyes are naturally less sensitive to distortion. For instance, Ekaterina Shumitskaya et al. developed the IOI: Invisible One-Iteration Adversarial Attack on No-Reference Image- and Video-Quality Metrics, which injects fast, single-iteration noise into these exact texture zones to artificially inflate quality scores without human detection.
Crucially, the lack of a baseline reference exacerbates this issue. As Anna Chistyakova et al. explain in Robustness as Architecture: Designing IQA Models to Withstand Adversarial Perturbations, NR-IQA models evaluate an image in isolation. Without a clean reference image to verify the structural integrity of the input, the model cannot distinguish between genuine high-quality details (like sharp edges) and adversarial noise designed to mimic those details.
The Mechanics of Score-Boosting
Unlike traditional adversarial attacks that aim to change a discrete classification label (e.g., turning a "stop sign" into a "speed limit sign"), attacks on quality metrics target a continuous regression value. The primary objective is typically "score-boosting"—forcing the model to output a higher numerical score for an objectively worse image or video.
In a comprehensive benchmark, Khaled Abud evaluated 15 modern metrics in Comparing the Robustness of Modern No-Reference Image- and Video-Quality Metrics to Adversarial Attacks, finding that deep-learning-based metrics are highly susceptible to these score-boosting strategies. These attacks can manipulate model outputs by up to 50% of their total range while remaining completely invisible to humans, as shown by Yu Ran et al. in Black-box Adversarial Attacks Against Image Quality Assessment Models.
This vulnerability creates a dangerous feedback loop when quality metrics are used to optimize other AI models. For example, in diffusion-based generative models, directly optimizing the network using gradients from an IQA model backfires. As explored in IQA-Adapter: Exploring Knowledge Transfer from Image Quality Assessment to Diffusion-based Generative Models, this gradient-based optimization causes the generator to exploit the IQA model's blind spots, producing visually distorted outputs filled with adversarial artifacts that nevertheless receive perfect quality scores.
Furthermore, this vulnerability extends across modalities and tasks:
- Cross-Modal Transferability: Ekaterina Shumitskaya et al. demonstrated in Cross-Modal Transferable Image-to-Video Attack on Video Quality Metrics that adversarial noise can transfer from image models and multi-modal proxies (like CLIP) to fool video quality metrics.
- Perceptual Similarity Metrics: Metrics like LPIPS, often used to verify that an adversarial attack is invisible, are themselves vulnerable. Abhijay Ghildyal et al. warn in Attacking Perceptual Similarity Metrics that using a non-robust metric to validate an attack's imperceptibility creates a circular security risk.
Designing Robust Defenses
Defending quality metrics is uniquely challenging because the defense must restore an accurate numerical score without altering the image's true visual quality, a distinct requirement from classification defenses (Aleksandr Gushchin et al. in Guardians of Image Quality: Benchmarking Defenses Against Adversarial Attacks on Image Quality Metrics). Traditional adversarial training is computationally expensive and hard to scale for regression tasks, leading researchers to explore architectural and mathematical safeguards.
Gradient Regularization
One approach is to smooth the model's loss landscape. Yujia Liu et al. introduced a defense in Defense Against Adversarial Attacks on No-Reference Image Quality Models with Gradient Norm Regularization that regularizes the $\ell_1$ norm of the model's gradient during training. This stabilizes the predicted scores against subtle pixel shifts, preventing small input changes from causing massive output swings.
Architectural Immunity
Instead of teaching the model to ignore noise via data augmentation, Anna Chistyakova et al. proposed embedding robustness directly into the model's structure. By introducing orthogonal convolutions and frequency-domain processing, their architecture naturally suppresses high-frequency adversarial noise before it can propagate through the network.
Temporal Consistency
For video quality assessment, defenses can exploit the temporal dimension. Because video frames are sequentially linked, adversarial noise injected into frames creates temporal discrepancies that violate natural motion. Chaowei Xiao et al. leveraged this in AdvIT: Adversarial Frames Identifier Based on Temporal Consistency in Videos by using optical flow to warp previous frames into "pseudo frames" and checking for prediction consistency.
Similarly, Ao-Xiang Zhang et al. developed the SecureVQA framework in Secure Video Quality Assessment Resisting Adversarial Attacks, combining temporal consistency with spatial grid sampling and randomized "guardian maps" to break the deterministic pixel alignments that white-box and black-box attacks rely on.
Key Foundational Work
Black-box Adversarial Attacks on Video Recognition Models (2019) — contributed 2 concept elements to this direction.
AdvIT: Adversarial Frames Identifier Based on Temporal Consistency in Videos (2019) — contributed 1 concept elements to this direction.
Attacking Automatic Video Analysis Algorithms: A Case Study of Google Cloud Video Intelligence API (2017) — contributed 1 concept elements to this direction.