SPARTA: Evaluating Reasoning Segmentation Robustness through Black-Box Adversarial Paraphrasing in Text Autoencoder Latent Space

Viktoriia Zinkovich, Anton Antonov, Andrei Spiridonov, Denis Shepelev, Andrey Moskalenko, Daria Pugacheva, Elena Tutubalina, Andrey Kuznetsov, Vlad Shakhuro

Abstract

Multimodal large language models (MLLMs) have shown impressive capabilities in vision-language tasks such as reasoning segmentation, where models generate segmentation masks based on textual queries. While prior work has primarily focused on perturbing image inputs, semantically equivalent textual paraphrases—crucial in real-world applications where users express the same intent in varied ways—remain underexplored. To address this gap, we introduce a novel adversarial paraphrasing task: generating grammatically correct paraphrases that preserve the original query meaning while degrading segmentation performance. We introduce SPARTA—a black-box, sentence-level optimization method that operates in the low-dimensional semantic latent space of a text autoencoder, guided by reinforcement learning. SPARTA achieves significantly higher success rates, outperforming prior methods by up to 2x on both the ReasonSeg and LLMSeg-40k datasets.

Introduction

Reasoning segmentation involves generating precise masks for objects based on complex, implicit queries (e.g., "the thing the girl is using to eat her dessert"). While researchers have extensively studied robustness to image noise, the impact of textual variation remains a critical blind spot. In real-world applications, users express the same intent in countless ways. A robust model should produce the same segmentation mask regardless of whether a user asks "What is the man throwing?" or "What's the item being tossed by the guy in the red shirt?". This paper addresses the lack of systematic evaluation for this "textual robustness" and introduces a method to identify linguistic weak points that cause models to fail.

Adversarial paraphrasing examples where minor linguistic changes (e.g., 'something' to 'a thing') lead to complete segmentation failure.

Methodology: The SPARTA Framework

The authors introduce SPARTA, a black-box optimization method that avoids the pitfalls of discrete token-level attacks and heuristic LLM-based paraphrasing.

  1. Latent Space Search: SPARTA uses a pretrained text autoencoder (SONAR) to map sentences into a continuous, high-dimensional latent space.
  2. Reinforcement Learning (PPO): It employs Proximal Policy Optimization to explore this latent space. The goal is to find a nearby vector that, when decoded back into text, preserves the original meaning but minimizes the model's segmentation performance (measured by Intersection over Union, or IoU).
  3. Reward Function: The RL agent is rewarded for maximizing the drop in IoU while being penalized for straying too far from the original semantic meaning.
  4. Evaluation Protocol: To ensure validity, the authors developed an automated protocol using LLMs combined with cosine similarity filters, validated against human judgment.

Key Insights and Results

The study evaluates robustness across two primary benchmarks: ReasonSeg and LLMSeg-40k.

  • Continuous vs. Discrete Optimization: Searching in a continuous latent space proved far more effective than discrete token-level substitutions, allowing for natural, sentence-level structural changes.
  • Universal Fragility: No single model emerged as truly robust. While larger models like LISA [13B] showed higher baseline resistance, they remained vulnerable to well-crafted adversarial paraphrases.
  • Performance Gains: SPARTA consistently outperformed all baselines. On the LLMSeg-40k dataset, it achieved an average success rate improvement of ~84% over the strongest baselines.
Success rate comparison of SPARTA against baseline methods across different IoU thresholds on ReasonSeg and LLMSeg-40k.

Conclusion

SPARTA exposes the fragility of reasoning segmentation models by generating semantically equivalent but performance-degrading paraphrases. These findings highlight the limitations of current multimodal models in handling linguistic diversity. Future work will focus on developing defenses, such as adversarial training using SPARTA-generated prompts, to build more linguistically resilient systems.

Summary updated

Abstract

Multimodal large language models (MLLMs) have shown impressive capabilities in vision-language tasks such as reasoning segmentation, where models generate segmentation masks based on textual queries. While prior work has primarily focused on perturbing image inputs, semantically equivalent textual paraphrases-crucial in real-world applications where users express the same intent in varied ways-remain underexplored. To address this gap, we introduce a novel adversarial paraphrasing task: generating grammatically correct paraphrases that preserve the original query meaning while degrading segmentation performance. To evaluate the quality of adversarial paraphrases, we develop a comprehensive automatic evaluation protocol validated with human studies. Furthermore, we introduce SPARTA-a black-box, sentence-level optimization method that operates in the low-dimensional semantic latent space of a text autoencoder, guided by reinforcement learning. SPARTA achieves significantly higher success rates, outperforming prior methods by up to 2x on both the ReasonSeg and LLMSeg-40k datasets. We use SPARTA and competitive baselines to assess the robustness of advanced reasoning segmentation models. We reveal that they remain vulnerable to adversarial paraphrasing-even under strict semantic and grammatical constraints. All code and data will be released publicly upon acceptance.
BibTeX
@inproceedings{zinkovich2026sparta,
  author = {Viktoriia Zinkovich and Anton Antonov and Andrei Spiridonov and Denis Shepelev and Andrey Moskalenko and Daria Pugacheva and Elena Tutubalina and Andrey Kuznetsov and Vlad Shakhuro},
  title = {{SPARTA: Evaluating Reasoning Segmentation Robustness through Black-Box Adversarial Paraphrasing in Text Autoencoder Latent Space}},
  booktitle = {EACL (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-long.126},
}
Figure