TL;DR
This paper tackles the vulnerability of Ukrainian multimodal models to synonym substitution attacks, a problem exacerbated by the language's rich morphology and low-resource status. They introduce a novel hybrid synonym generation method and a synonym-augmented fine-tuning strategy that significantly improves robustness without sacrificing original performance, offering a practical solution for low-resource languages.
What Problem Does This Paper Address?
Multimodal models, like those that understand both images and text, are increasingly powerful. However, they can be surprisingly brittle. Even small, human-imperceptible changes to the input text, such as replacing a word with a synonym, can drastically degrade their performance. This is particularly true for languages like Ukrainian, which have complex grammar (rich morphology) and less available training data compared to English. Existing methods for testing and improving robustness against synonym substitution are often designed for English and don't translate well to Ukrainian. This paper aims to bridge this gap by evaluating and enhancing the robustness of Ukrainian multimodal models against these "synonym substitution attacks" (SSA).
Methods
The researchers developed three key methods for generating synonyms to attack multimodal models:
- Dictionary-based: This method uses an existing Ukrainian synonym dictionary. However, it requires careful inflection of synonyms to match the original sentence's grammar.
- LLM-based (GPT-40): This approach uses a large language model (GPT-40) to generate context-aware synonyms, aiming for semantic and grammatical correctness.
- Hybrid: This novel method combines the broad coverage of the dictionary with the contextual understanding of the LLM. The LLM refines dictionary candidates, selecting the best synonyms that preserve both meaning and grammar.
To improve model robustness, they proposed synonym-augmented fine-tuning. This involves creating new training data by replacing nouns in existing captions with LLM-generated synonyms. The model is then fine-tuned on this augmented dataset, alongside the original data.
Datasets
- Multi30K: The primary dataset used for training and evaluation. This is a multilingual dataset containing image-text pairs.
- Ukrainian extension of Multi30K: A version of Multi30K with human-translated Ukrainian captions.
- Czech version of Multi30K: Used for cross-lingual replication of the study.
- Ukrainian Visual-WSD benchmark: Used to establish a baseline for Ukrainian multimodal models.
Key Insights and Ideas
- Ukrainian is more vulnerable: Ukrainian multimodal models are significantly more affected by synonym substitution attacks than English models, highlighting the impact of rich morphology and lower data availability.
- Hybrid approach is superior: Combining dictionary coverage with LLM's contextual understanding (the Hybrid method) yields the most accurate and grammatically sound synonyms for attacking models.
- Synonym-augmented fine-tuning works: Exposing models to synonym-substituted data during fine-tuning significantly boosts their robustness without harming performance on original, unperturbed data.
- Cross-lingual generalization: The proposed methods (Hybrid substitution and synonym-augmented fine-tuning) show promising results not only for Ukrainian but also for Czech, suggesting their applicability to other morphologically rich, low-resource languages.
Key Results
- Performance drop: Unperturbed Ukrainian models achieved HIT@1 of 32.1%, which dropped to 7.6% with dictionary-based SSA and 10.9% with GPT-40 SSA. English models showed smaller drops.
- Hybrid method effectiveness: The Hybrid SSA method resulted in the smallest performance degradation, with Ukrainian models maintaining HIT@1 at 16.8%.
- Manual audit: A manual evaluation showed that Hybrid replacements were the most grammatically and semantically valid (90.4% correct).
- Fine-tuning gains: Synonym-augmented fine-tuning significantly improved robustness. For example, HIT@1 for the Hybrid method increased from 16.8% to 28.1% after fine-tuning.
- Czech replication: Similar patterns were observed in Czech, with Hybrid substitution and synonym-augmented fine-tuning showing consistent improvements.
Limitations and Future Work
- Reliance on external resources: The synonym generation methods depend on existing dictionaries and LLMs, which can introduce biases from their training data.
- Cross-lingual scope: While tested on Czech, further research is needed to assess the generalizability to non-Slavic languages with different morphological structures.
- Tokenizer limitations: The study kept the original XLM-R tokenizer unchanged. Exploring selective vocabulary expansion or adapter-level token embeddings could further improve robustness against unseen sub-words.
- Future work: The authors suggest extending the study to other generation tasks like Visual Question Answering (VQA) and to a wider range of languages. They also propose exploring adapter-level token embeddings for better handling of unseen sub-words.
Summary updated