TL;DR
This paper introduces AnyTrans, a novel framework for translating text within images (Translate AnyText in the Image - TATI). It leverages large language models (LLMs) and diffusion models to achieve accurate translations that seamlessly blend into the original image, preserving its style and realism, all without requiring any training.
What Problem Does This Paper Address?
The task of translating text embedded within images (Text Image Translation or TIT) is crucial for global communication and accessibility. However, existing methods often treat text translation and image integration as separate steps. Traditional approaches might use OCR to extract text, translate it with standard machine translation, and then simply paste it back. This can lead to translations that don't fit the visual context, break the image's aesthetic, or even misinterpret the text's meaning due to a lack of contextual understanding. The authors argue that a truly effective solution needs to consider both the textual content and its visual integration, aiming for a seamless fusion that maintains the original image's style and realism.
Methods
AnyTrans tackles this challenge with a three-step pipeline:
- Text Detection and Recognition: It starts by using a robust, pre-trained OCR model (specifically PP-OCR) to accurately locate and identify text within the source image. This step is crucial for knowing what text to translate and where it is.
- Text Image Translation: Here, the core innovation lies in leveraging Large Language Models (LLMs). The recognized text segments are concatenated with special tags to preserve their positional information. Then, using a few-shot prompt learning strategy, LLMs translate these segments. This approach allows the LLM to understand the context of fragmented text, leading to more accurate translations. The authors also explore using Vision-Language Models (VLMs) to further refine translations by incorporating visual cues from the image.
- Text Fusion: Finally, the translated text is seamlessly integrated back into the original image. This is achieved using a modified version of the AnyText diffusion model. To ensure a natural look, AnyTrans incorporates techniques like stroke-level text erasure and an "Anticipated Box Resize" strategy. This resizing step adjusts the target text box based on the length difference between the original and translated text, preventing awkward empty spaces or text overflow and preserving the original image's style and realism.
A key advantage of AnyTrans is that it's training-free and can be built entirely from open-source models, making it highly accessible.
Datasets
The paper introduces a new benchmark dataset called MTIT6. This dataset consists of multilingual text image translation data across six language pairs: English-to-Chinese, Japanese-to-Chinese, Korean-to-Chinese, Chinese-to-English, Chinese-to-Japanese, and Chinese-to-Korean. Each language pair contains approximately 200 images. The dataset includes text position information, translation details, and crucially, manually corrected translation order, which is vital for accurate contextual translation.
Key Insights and Ideas
- Context is King (Even for Text in Images): Simply translating text boxes in isolation is insufficient. LLMs' ability to process concatenated text with positional information allows for a much deeper understanding of the intended meaning, especially for fragmented or context-dependent text.
- Seamless Fusion is Non-Negotiable: The visual integration of translated text is as important as the translation itself. Traditional methods fail here, leading to jarring results. Diffusion models, with their advanced editing capabilities, are key to achieving realistic text fusion.
- Open-Source and Training-Free is Achievable: The paper demonstrates that a high-performing system for this complex task can be built using readily available open-source models without the need for extensive, task-specific training. This significantly lowers the barrier to entry for research and application.
- Dataset Curation Matters: The creation of MTIT6 highlights the need for high-quality, annotated datasets that capture the nuances of text image translation, including correct text ordering, to drive progress in the field.
Key Results
- Quantitative Performance: AnyTrans, particularly when using larger LLMs like
qwen1.5-110B-chatandqwen-vl-max, achieves competitive or superior results compared to commercial products on metrics like BLEU and COMET for both image-to-text (I2T) and image-to-image (I2I) translation. For instance,qwen-vl-maxachieved an average BLEU score of 48.7 for zh→en I2T. - Qualitative Superiority: The paper presents compelling qualitative comparisons (Figure 8) showing AnyTrans's ability to preserve font style, color, and seamlessly integrate text, outperforming Google, Microsoft, and Apple's image translation products in terms of authenticity and style consistency.
- Ablation Studies: An ablation study confirmed that translating the entire recognized text as a whole (using context) significantly improves performance over translating individual boxes, underscoring the power of LLMs' contextual understanding. The "Anticipated Box Resize" strategy was also shown to be effective, improving I2I translation results by 2.9 points in the zh→en task.
Limitations and Future Work
- Text Length Constraint: Due to the underlying AnyText model's limitation, AnyTrans cannot effectively translate texts exceeding 20 letters or characters at a time.
- Language Support: The text editing proficiency of the underlying diffusion model is limited to Chinese, English, Korean, and Japanese. This restricts AnyTrans's ability to translate into other languages, such as Arabic.
- Future Directions:
- Unified OCR and Translation: Integrating OCR and translation into a single step using advanced VLMs could further enhance efficiency and accuracy.
- Dynamic Text Editing: Developing text editing models that can dynamically adjust font sizes would eliminate the need for manual resizing and better preserve image aesthetics.
Summary updated