Abstract
Vision-Language Models (VLMs) have demonstrated great potential in real-world applications. While existing research primarily focuses on improving their accuracy, the efficiency remains underexplored. Given the real-time demands of many applications and the high inference overhead of VLMs, efficiency robustness is a critical issue. However, previous studies evaluate efficiency robustness under unrealistic assumptions, requiring access to the model architecture and parameters—an impractical scenario in ML-as-a-service settings, where VLMs are deployed via inference APIs. To address this gap, we propose VLMInferSlow, a novel approach for evaluating VLM efficiency robustness in a realistic black-box setting. VLMInferSlow incorporates fine-grained efficiency modeling tailored to VLM inference and leverages zero-order optimization to search for adversarial examples. Experimental results show that VLMInferSlow generates adversarial images with imperceptible perturbations, increasing the computational cost by up to 128.47%. We hope this research raises the community’s awareness about the efficiency robustness of VLMs.
Introduction
As Vision-Language Models (VLMs) like GPT-4V and Gemini are increasingly deployed via Machine Learning as a Service (MLaaS) APIs, the focus of adversarial research is shifting. While traditional attacks target integrity (forcing incorrect outputs), a new threat emerges: efficiency attacks.
VLMs are computationally intensive, and their operational costs are often tied to token generation or GPU time. An efficiency attack aims to sabotage the model's performance by forcing it to generate excessively long or complex responses from seemingly normal inputs. This can lead to increased cloud costs for providers, battery drain on mobile devices, and potential Denial-of-Service (DoS) conditions. Unlike previous "white-box" methods that require internal model access, we introduce VLMInferSlow, a framework designed for the realistic black-box setting of commercial APIs.
The VLMInferSlow Framework
VLMInferSlow utilizes Zero-Order Optimization to estimate gradients without internal model access. It iteratively perturbs an input image to maximize a three-part objective function designed to degrade inference efficiency:
- Sequence Length Maximization ($L_{len}$): Directly incentivizes the model to produce more tokens.
- Slower Termination ($L_{eos}$): Targets the "End-of-Sequence" (EOS) token. By applying a dynamic weight decay strategy, the attack discourages the model from terminating the sequence, effectively making it "forget" how to stop.
- Variable Output Production ($L_{var}$): Encourages the model to produce a diverse set of tokens by aligning the output distribution with a uniform distribution, forcing the decoder to work harder on complex text generation.
To optimize these objectives in a black-box environment, the framework employs Natural Evolution Strategies (NES). By sampling random noise and observing the API's response, the system moves image pixels in the direction that most effectively increases the computational "slowness."
Experimental Evaluation
We evaluated VLMInferSlow on standard image captioning benchmarks, including MS-COCO and ImageNet-1k, testing against prominent models such as Flamingo, BLIP, GIT, and Florence.
Key Results
- Significant Latency Increase: On the Flamingo model, the attack increased sequence length by 128.47% and response latency by 105.56%.
- Resource Exhaustion: Energy consumption rose by 115.19%, demonstrating the potential for battery-drain attacks on mobile assistive technologies.
- Stealthiness: The perturbations are virtually invisible to the human eye, maintaining an $L_2$ distance of approximately 10.63, yet they consistently trigger the efficiency drain.
- The "Ramble" Effect: Qualitatively, the attack forces the model to describe irrelevant details or hallucinate long-winded descriptions. For example, a simple prompt for "a man walking a dog" might result in a multi-sentence paragraph about dappled sunlight and forest serenity.
- Performance Degradation: As a side effect of the verbosity, accuracy (measured by BLEU score) dropped by up to 38.46%.
Conclusion
VLMInferSlow demonstrates that VLMs are highly vulnerable to efficiency-based adversarial attacks, even in black-box scenarios. By shifting the focus from accuracy to efficiency robustness, this work highlights a critical security gap in MLaaS deployments. Future research must focus on developing VLMs that are inherently resistant to these "slow-down" inputs to ensure the reliability and cost-effectiveness of vision-language services.
Summary updated