ChronoPatch: Black-Box Energy-Latency Attacks on Vision-Language Models

Jiawei Xing, Li Wang, Xiangtao Meng, Chuanchao Zang, Rui Wen, Zheng LiarXiv 2026 · arXiv

TL;DR

ChronoPatch is a novel black-box attack that exploits the autoregressive nature of Vision-Language Models (VLMs) to significantly amplify their inference latency and energy consumption. By observing only the final response length, it crafts localized visual perturbations that trick models into generating excessively long and repetitive outputs, posing a serious availability risk for commercial AI services.

What Problem Does This Paper Address?

As Vision-Language Models (VLMs) like GPT-4o or LLaVA become central to automated services, their computational efficiency becomes a target. These models generate text "autoregressively"—one token at a time. Because each token requires a full forward pass of the model, the total time (latency) and energy cost of a request are directly tied to the length of the response.

This creates a vulnerability known as an energy-latency attack. An attacker can craft "adversarial" images designed to prevent the model from stopping, forcing it to generate long, rambling, or repetitive text. While previous attacks required "white-box" access (knowing the model's internal gradients or seeing intermediate probability scores), ChronoPatch addresses the more realistic final-response-only setting. It asks: Can an attacker crash your service's budget if they can only see how long the final answer was?

Figure 1: Motivating example of ChronoPatch. Localized visual perturbations induce prolonged and repetitive VLM generation under final-response-only black-box access.

Methods

The authors propose a two-stage framework to overcome the lack of internal model feedback:

  1. Phase I: Delay-Anchor Discovery: The attack uses a population-based evolutionary search to find "visual delay anchors." It places small, high-frequency noise patches on the image and evolves their appearance and location. The "fitness" of a patch is simply the length of the resulting VLM response. This phase identifies the specific visual patterns that trigger the model's "over-generation" behavior.
  2. Phase II: Perturbation Refinement: Once a delay anchor is found, it's usually visually jarring. This phase uses subspace-calibrated decision-based optimization to shrink the perturbation's magnitude while keeping the response length above a target threshold. Because the attack only modifies a small patch (e.g., 5% of the image), standard optimization techniques fail due to the mismatch in dimensionality. ChronoPatch specifically calibrates its search radius to the "effective dimensionality" of the patch area to remain efficient.
Figure 2: Design overview of ChronoPatch. Phase I discovers visual delay anchors using response-length feedback. Phase II refines the anchor within a fixed spatial mask through subspace-calibrated decision-based refinement, producing a localized low-distortion adversarial image that induces longer, more repetitive responses and increases latency and energy usage.

Datasets

The attack was evaluated using 1,000 images randomly sampled from three standard computer vision benchmarks:

  • MSCOCO 2017: A dataset for object detection and captioning.
  • ImageNet: The classic large-scale image classification dataset.
  • Flickr30K: Used for evaluating image-to-sentence descriptions.

Key Insights and Ideas

  • Length is a Sufficient Signal: The most surprising finding is that the mere count of tokens in a final response provides enough signal to guide a complex adversarial optimization. You don't need gradients or logits to break these models.
  • Repetitive Degeneration: The attack often works by trapping the model in "n-gram repetition" loops. Instead of generating meaningful long text, the model gets stuck repeating phrases like "the dog is in the grassy field with the blue ball..." indefinitely.
  • Localized Vulnerability: You don't need to change the whole image. A tiny, localized patch (often looking like a QR code or colorful noise) is enough to hijack the model's attention and decoding logic.
  • Transferability to Commercial APIs: The attack successfully amplified costs on closed-source models like ChatGPT-5 and Claude, proving that the underlying vulnerability is a fundamental property of current autoregressive VLM architectures, not just a quirk of smaller open-source models.

Key Results

The quantitative impact of ChronoPatch is substantial across various models:

  • Open-Source Models: On models like InternVL3-8B and Qwen3-VL, inference latency increased by up to 293% and GPU energy consumption by up to 249% within a 5% spatial budget.
  • Commercial APIs: Under a strict budget of 500 queries, the attack increased response lengths by 37.6%–81.5% and client-observed latency by 1.82×–2.29× on services like ChatGPT-5 and Gemini.
  • Repetition: On InternVL3, the "rep-4" ratio (a measure of 4-gram repetition) skyrocketed from ~8% to over 65%, indicating a total collapse of coherent generation.
Figure 3: Visual comparison of clean images (top) and their corresponding adversarial images (bottom).

Limitations and Future Work

  • Optimization Cost: While effective, the attack requires hundreds to thousands of queries per image. This might be expensive for an attacker unless they can reuse optimized images across many requests.
  • Perceptibility: Although the perturbations are small and have low global distortion scores (LPIPS), the "patches" themselves are often visible as localized blocks of noise. Future work could focus on making these patches semantically meaningful (e.g., looking like a sticker or a piece of jewelry) to improve stealth.
  • Defense Trade-offs: The authors tested defenses like repetition penalties and early stopping. While these mitigate the attack, they also interfere with the model's ability to generate legitimate long-form content, highlighting a difficult trade-off for service providers.

Summary updated

Abstract

Vision-language models (VLMs) are increasingly deployed as generative inference services, where autoregressive decoding makes response length a key determinant of latency and computational cost. This creates an availability risk: adversarial visual inputs can prolong generation and increase serving overhead. Existing energy-latency attacks typically rely on gradients, logits, or token-level feedback, leaving open whether such risks persist when an attacker observes only the final response. We present ChronoPatch, a black-box attack that induces prolonged VLM generation requiring only the final response. It discovers visual delay anchors through response-length-guided evolutionary search, then minimizes perturbation magnitude through subspace-calibrated decision-based optimization. Across four open-source VLMs, ChronoPatch increases inference latency by 47.86%--293.34% and GPU energy consumption by 41.44%--249.70% under a 5% spatial budget. It also induces more repetitive outputs, as reflected by increased n-gram repetition ratios. On three commercial VLM APIs, ChronoPatch increases response length by 37.6%--81.5% and client-observed latency by 1.82x--2.29x under query budgets of 500 per API. These results reveal a generation-based availability risk that persists under final-response-only access and motivate generation-aware safeguards for deployed generative AI services.
BibTeX
@misc{xing2026chronopatch,
  author = {Jiawei Xing and Li Wang and Xiangtao Meng and Chuanchao Zang and Rui Wen and Zheng Li},
  title = {{ChronoPatch: Black-Box Energy-Latency Attacks on Vision-Language Models}},
  year = {2026},
  eprint = {2611.26117},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.26117},
}
Figure