TL;DR
ChronoPatch is a novel black-box attack that exploits the autoregressive nature of Vision-Language Models (VLMs) to significantly amplify their inference latency and energy consumption. By observing only the final response length, it crafts localized visual perturbations that trick models into generating excessively long and repetitive outputs, posing a serious availability risk for commercial AI services.
What Problem Does This Paper Address?
As Vision-Language Models (VLMs) like GPT-4o or LLaVA become central to automated services, their computational efficiency becomes a target. These models generate text "autoregressively"—one token at a time. Because each token requires a full forward pass of the model, the total time (latency) and energy cost of a request are directly tied to the length of the response.
This creates a vulnerability known as an energy-latency attack. An attacker can craft "adversarial" images designed to prevent the model from stopping, forcing it to generate long, rambling, or repetitive text. While previous attacks required "white-box" access (knowing the model's internal gradients or seeing intermediate probability scores), ChronoPatch addresses the more realistic final-response-only setting. It asks: Can an attacker crash your service's budget if they can only see how long the final answer was?
Methods
The authors propose a two-stage framework to overcome the lack of internal model feedback:
- Phase I: Delay-Anchor Discovery: The attack uses a population-based evolutionary search to find "visual delay anchors." It places small, high-frequency noise patches on the image and evolves their appearance and location. The "fitness" of a patch is simply the length of the resulting VLM response. This phase identifies the specific visual patterns that trigger the model's "over-generation" behavior.
- Phase II: Perturbation Refinement: Once a delay anchor is found, it's usually visually jarring. This phase uses subspace-calibrated decision-based optimization to shrink the perturbation's magnitude while keeping the response length above a target threshold. Because the attack only modifies a small patch (e.g., 5% of the image), standard optimization techniques fail due to the mismatch in dimensionality. ChronoPatch specifically calibrates its search radius to the "effective dimensionality" of the patch area to remain efficient.
Datasets
The attack was evaluated using 1,000 images randomly sampled from three standard computer vision benchmarks:
- MSCOCO 2017: A dataset for object detection and captioning.
- ImageNet: The classic large-scale image classification dataset.
- Flickr30K: Used for evaluating image-to-sentence descriptions.
Key Insights and Ideas
- Length is a Sufficient Signal: The most surprising finding is that the mere count of tokens in a final response provides enough signal to guide a complex adversarial optimization. You don't need gradients or logits to break these models.
- Repetitive Degeneration: The attack often works by trapping the model in "n-gram repetition" loops. Instead of generating meaningful long text, the model gets stuck repeating phrases like "the dog is in the grassy field with the blue ball..." indefinitely.
- Localized Vulnerability: You don't need to change the whole image. A tiny, localized patch (often looking like a QR code or colorful noise) is enough to hijack the model's attention and decoding logic.
- Transferability to Commercial APIs: The attack successfully amplified costs on closed-source models like ChatGPT-5 and Claude, proving that the underlying vulnerability is a fundamental property of current autoregressive VLM architectures, not just a quirk of smaller open-source models.
Key Results
The quantitative impact of ChronoPatch is substantial across various models:
- Open-Source Models: On models like InternVL3-8B and Qwen3-VL, inference latency increased by up to 293% and GPU energy consumption by up to 249% within a 5% spatial budget.
- Commercial APIs: Under a strict budget of 500 queries, the attack increased response lengths by 37.6%–81.5% and client-observed latency by 1.82×–2.29× on services like ChatGPT-5 and Gemini.
- Repetition: On InternVL3, the "rep-4" ratio (a measure of 4-gram repetition) skyrocketed from ~8% to over 65%, indicating a total collapse of coherent generation.
Limitations and Future Work
- Optimization Cost: While effective, the attack requires hundreds to thousands of queries per image. This might be expensive for an attacker unless they can reuse optimized images across many requests.
- Perceptibility: Although the perturbations are small and have low global distortion scores (LPIPS), the "patches" themselves are often visible as localized blocks of noise. Future work could focus on making these patches semantically meaningful (e.g., looking like a sticker or a piece of jewelry) to improve stealth.
- Defense Trade-offs: The authors tested defenses like repetition penalties and early stopping. While these mitigate the attack, they also interfere with the model's ability to generate legitimate long-form content, highlighting a difficult trade-off for service providers.
Summary updated