VLMInferSlow: Evaluating the Efficiency Robustness of Large Vision-Language Models as a Service

Xiasi Wang, Tianliang Yao, Simin Chen, Runqi Wang, Lei Ye, Kuofeng Gao, Yi Huang, Yuan Yao

Abstract

Vision-Language Models (VLMs) have demonstrated great potential in real-world applications. While existing research primarily focuses on improving their accuracy, the efficiency remains underexplored. Given the real-time demands of many applications and the high inference overhead of VLMs, efficiency robustness is a critical issue. However, previous studies evaluate efficiency robustness under unrealistic assumptions, requiring access to the model architecture and parameters—an impractical scenario in ML-as-a-service (MLaaS) settings, where VLMs are deployed via inference APIs. To address this gap, we propose VLMInferSlow, a novel approach for evaluating VLM efficiency robustness in a realistic black-box setting. VLMInferSlow incorporates fine-grained efficiency modeling tailored to VLM inference and leverages zero-order optimization to search for adversarial examples. Experimental results show that VLMInferSlow generates adversarial images with imperceptible perturbations, increasing the computational cost by up to 128.47%.

Introduction

The rapid advancement of Vision-Language Models (VLMs) has led to their widespread deployment in diverse fields, from autonomous driving to assistive technologies. These models are increasingly offered as Machine Learning as a Service (MLaaS), where users interact with models like GPT-4o or Gemini through restricted APIs. While the security community has extensively studied adversarial attacks targeting model accuracy, the efficiency robustness of these models remains a significant blind spot.

Efficiency robustness refers to a model's ability to maintain consistent computational performance across various inputs. In an MLaaS context, an efficiency-based adversarial attack—where an input is designed to maximize resource consumption—can lead to Denial-of-Service (DoS) conditions, excessive cloud billing, or rapid battery depletion on edge devices. VLMInferSlow addresses this vulnerability by demonstrating how black-box attacks can significantly degrade VLM efficiency without requiring internal model access.

The VLMInferSlow Framework

VLMInferSlow is designed to operate in a black-box environment where only the model's outputs (logits and generated text) are accessible. The attack utilizes a zero-order optimization framework, specifically employing Natural Evolution Strategies (NES) to estimate gradients by sampling noise around the input image.

The optimization process targets a composite objective function designed to maximize inference latency through three primary mechanisms:

  1. Sequence Length Maximization ($L_{len}$): This component directly rewards the generation of a higher number of tokens, pushing the model toward verbosity.
  2. End-of-Sequence (EOS) Suppression ($L_{eos}$): By applying a dynamic weight decay strategy, the attack specifically lowers the probability of the EOS token. This prevents the model from concluding its response, especially as the sequence grows.
  3. Distributional Variance ($L_{var}$): This objective encourages the model to produce a more uniform distribution among its top-k token candidates. By increasing uncertainty, the model is forced to explore a more diverse and often redundant vocabulary, leading to longer and more complex sentence structures.

Experimental Evaluation

The researchers evaluated VLMInferSlow across two benchmark datasets: MS-COCO (complex scene captioning) and ImageNet-1k (diverse object categories). The attack was tested against four prominent VLM architectures: Flamingo, BLIP, GIT, and Florence.

Key Findings

The experimental results demonstrate a significant vulnerability across all tested models:

  • Computational Overhead: VLMInferSlow increased the relative sequence length by an average of 82.08% on MS-COCO and 76.98% on ImageNet-1k.
  • Peak Impact on Flamingo: The Flamingo model proved particularly susceptible, experiencing a 128.47% increase in sequence length and a corresponding 115.19% increase in energy consumption.
  • Visual Imperceptibility: The adversarial perturbations remain largely invisible to the human eye, ensuring the attack can be delivered without raising immediate suspicion.
  • Ineffectiveness of Standard Optimizations: Common techniques like quantization, often used to improve efficiency, failed to mitigate the attack, suggesting the vulnerability is inherent to the model's learned logic.

Analysis of the "Chatty" Vulnerability

The success of VLMInferSlow stems from the autoregressive nature of VLMs. Because these models generate tokens sequentially, any delay in reaching the termination token has a multiplicative effect on total resource usage.

Grad-CAM visualizations revealed a fascinating qualitative shift: while benign images lead the model to focus on salient objects, adversarial images cause the model's attention to "scatter" across the entire frame. This dispersed attention forces the model to describe irrelevant background details or repeat information, resulting in "chatty" and inefficient outputs. Furthermore, while the primary goal is slowing the model, the attack also results in a degradation of captioning accuracy (e.g., lower BLEU scores), indicating a dual impact on performance.

Conclusion and Future Directions

VLMInferSlow highlights a critical security gap in the deployment of VLMs as services. By demonstrating that significant efficiency degradation is possible in a black-box setting, this research underscores the need for "efficiency-aware" adversarial training and more robust defense mechanisms.

Future work should explore the query efficiency of these attacks to determine if they can be executed with fewer API calls, potentially bypassing rate-limiting defenses. Additionally, extending this evaluation to multi-modal tasks like Visual Question Answering (VQA) and long-form video analysis will be essential as VLMs continue to evolve.

Summary updated

Abstract

Vision-Language Models (VLMs) have demonstrated great potential in real-world applications. While existing research primarily focuses on improving their accuracy, the efficiency remains underexplored. Given the real-time demands of many applications and the high inference overhead of VLMs, efficiency robustness is a critical issue. However, previous studies evaluate efficiency robustness under unrealistic assumptions, requiring access to the model architecture and parameters -- an impractical scenario in ML-as-a-service settings, where VLMs are deployed via inference APIs. To address this gap, we propose VLMInferSlow, a novel approach for evaluating VLM efficiency robustness in a realistic black-box setting. VLMInferSlow incorporates fine-grained efficiency modeling tailored to VLM inference and leverages zero-order optimization to search for adversarial examples. Experimental results show that VLMInferSlow generates adversarial images with imperceptible perturbations, increasing the computational cost by up to 128.47%. We hope this research raises the community's awareness about the efficiency robustness of VLMs.
BibTeX
@inproceedings{wang2025vlminferslow,
  author = {Xiasi Wang and Tianliang Yao and Simin Chen and Runqi Wang and Lei Ye and Kuofeng Gao and Yi Huang and Yuan Yao},
  title = {{VLMInferSlow: Evaluating the Efficiency Robustness of Large Vision-Language Models as a Service}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
  url = {https://openreview.net/forum?id=ydaJVCo0qJ},
}
Figure