Reading the Screen to Translate the Button

Software is translated one short string at a time. A translator, human or machine, receives "Volume", "Run", or "Home" with no sentence around it. Whether "Volume" means loudness (German Lautstärke) or a disk partition (Datenträger), and whether "Run" is a verb on a button or a noun in a log viewer, is settled entirely by where the string sits on screen. These words name abstract functions, not physical objects. A picture of the object does not exist, and an object detector has no category for "Run". The only visual evidence of their meaning is relational: the widget the text lives in, the icons and labels beside it, and the typographic conventions it follows.

We propose to translate interface strings from rendered screenshots instead of from text metadata. The model would read the target string in place with a character-aware visual encoder, meaning an image encoder pretrained to align individual characters with image regions, so it sees the glyphs, their casing and truncation, the widget around them, and the surrounding layout. Our bet is that interface layout conventions are shared across applications in a way that project-specific metadata is not. If so, grounding in rendered layout should get past the cross-application "generalization wall" that text-only context methods hit.

Core Claim

The meaning of an abstract interface string is grounded in the geometry and typography of its rendered context, not in any depicted object. A translation model that reads the rendered string and its neighbors at the character level should therefore generalize to applications it never saw in training, where metadata-based context gives little or no gain. The work is new because it changes the source of context. Analyzing Challenges in NMT for Software Localization attached text proxies to each string: domain tags, neighbor strings, and code file paths. Those proxies nearly doubled targeted disambiguation accuracy within an application, but the gains were only marginal across applications, because models memorized project tokens. The authors called for real visual UI input, and no paper in our file supplies it. Visually grounded translation work does use images, but it grounds concrete nouns through an object detector, as in Tackling Ambiguity with Images (VGAMT, which uses MDETR boxes to guide attention). That design cannot represent an abstract action, and the abstract-grounding literature argues that such concepts are not localized in boxes at all (Hu et al.).

The character-level component is the reading mechanism, not decoration. Several decisive cues exist only at the glyph level. Title Case often marks menu headers while sentence case marks buttons and actions. Truncation ("Vol…") and accelerator markers carry information. The rendered width of the source box limits how long a German translation can be. Sub-word tokens discard all of these. Language Matters (oCLIP) showed that a character-aware text encoder trained with masked character prediction teaches a visual backbone to localize individual text instances from transcriptions alone. We reuse that encoder to read many co-occurring strings on one screen, which is the setting where ordinary one-caption-per-image pretraining fails.

Research Program

The first deliverable is a dataset, since no screenshot-paired localization corpus appears in our file. Open-source Android and Qt applications ship declarative layout files together with per-language string resources. Rendering these layouts with the toolkit's own preview renderer gives, for each English string, a screenshot, the bounding box of the string's widget, and the reference German translation. We will split by application, not by string, so the test set measures transfer to unseen software. Following the contrastive design of CoMMuTE, we will also build a cross-application contrastive suite of roughly a thousand items. Each item pairs one English string with two screenshots that imply different senses and two candidate translations. This addresses the low statistical power of the 95-entry Targeted Disambiguation set from Koneru et al. For that original set, we will recover or reconstruct screenshots of the source applications wherever possible, so results stay comparable to the published numbers.

The model keeps a frozen multilingual translation backbone (mBART, as in VGAMT) and adds trainable adapters that cross-attend to visual features. The visual features come from an oCLIP-pretrained backbone applied to the full screenshot. The string to translate is marked with a highlighted box drawn on the image, so the model knows which text is the source and which is context. Training mixes the standard translation loss with a consistency term borrowed from Soul-Mix. Soul-Mix applies the term between clean and degraded inputs to force reliance on the image. We apply it between a full screenshot and one where the neighboring context is masked, so that context use is actively encouraged rather than optional. A secondary decoder signal predicts whether the output fits the source box width. We treat this as a check on whether glyph geometry is actually being read, not as a headline result.

The analysis asks which part of the screen carries the disambiguating signal. We will compare three conditions: the target glyphs alone, the layout with target glyphs blanked, and the full screen. We will also probe the adapters' attention on the contrastive suite to see whether attention goes to the paired icon or slider, or spreads across the screen. That distinction speaks directly to the abstract-grounding claim that such meanings are relational and diffuse.

Baselines

The naive reference point is text-only translation of the isolated string with the same frozen backbone and adapters, given no context. Koneru et al. report 21.05% accuracy for this setting on their Targeted Disambiguation set. We will rerun it on our splits under the same fine-tuning data and steps.

The strongest published method on this exact question is Koneru et al.'s metadata conditioning. They prepend domain tags, neighbor strings, or source-path references to the input. The domain tag raised targeted accuracy to 41.03%, and source references raised intra-application BLEU from 49.03 to 51.09. For the cross-application setting, which is our main target, they report only that improvements were "marginal", and no cross-application number is reported on Lacuna. We will retrain all three metadata variants on our corpus with the same backbone, data, and tuning budget, and evaluate them on the identical application-held-out split. Beating this baseline means a statistically significant accuracy gain on held-out applications on the contrastive suite, with matched or better chrF and BLEU. Matching it only within applications would not count.

The strongest general multimodal competitor is a vision-language model given the screenshot and string. We will use InternLM-XComposer2-VL, the model Bao et al. showed can extract semantics from rendered glyphs. We will run it zero-shot, and then fine-tuned with the same data and a matched GPU-hour budget. No reported result was found on Lacuna for any VLM on UI string translation. We also include a VGAMT-style detector-guided variant, with MDETR regions replacing the character-aware features, because it is the best published image-grounded translation design. Its reported 67.1% on CoMMuTE (Futeral et al.) concerns concrete nouns, and we expect a closed vocabulary to hurt it on interface actions.

Two ablations isolate the key ingredient. The first swaps the character-aware oCLIP backbone for a vanilla CLIP image encoder with identical adapters. This tests whether character-level reading matters beyond generic visual features. The second is incongruent decoding: each string is paired with a screenshot from a different context, following Soul-Mix's test. A model that truly grounds should lose accuracy here, and a model that ignores the image will not.

Why not just…?

Why not just prepend neighbor strings and file paths as text? This is exactly what Koneru et al. tried. Their models memorized project-specific tokens, broke under false context, and gained little across applications. Our application-held-out split will measure whether that failure persists on our data. If text neighbors matched the screenshot model across applications, we would conclude that the visual channel adds nothing, and we would say so.

Why not just serialize the layout file as text and give it to the translator? This is the most serious threat. A layout tree already encodes widget type and nesting, and may carry most of the signal. We will train this variant under the same budget. If it matches our model, the honest conclusion is that layout structure matters and pixels do not, at least wherever layout files exist. The rendered-pixel route would then be justified only where layout files are unavailable, such as web pages, games, and closed-source binaries. We would demonstrate that case separately on screenshots without layout sources instead of claiming a general win. The glyph-level cues of casing, truncation, and box width are partly absent from serialized trees, and the target-glyph-only condition will show how much they contribute.

Why not just ask a large vision-language model? It may well read screens competently, and the fine-tuned VLM baseline is there precisely to test that. Our model's argument is efficiency and targeted reading: a frozen translator with small adapters and a character-aware encoder, against a general model. If the fine-tuned VLM wins on the held-out applications at matched compute, the contribution narrows to the dataset and the contrastive suite. We regard that as a useful negative result, not a reason to drop the comparison.

Proposed Experiments and Decisive Checks

The primary metric is accuracy on the application-held-out contrastive suite, with chrF and BLEU on the full held-out split. Secondary metrics are accuracy on the reconstructed 95-entry set for comparison with Koneru et al.'s 21.05% and 41.03%, and the rate at which outputs fit the source box. Three outcomes would decide the question. First, the full model must beat the metadata baselines across applications, not just within them. Second, accuracy must fall under incongruent screenshots, which shows the model actually uses the image. Third, the oCLIP-versus-vanilla-CLIP ablation and the glyph-blanked condition must show whether character-level reading carries weight. If the vanilla CLIP encoder performs as well, the character-structural part of the thesis fails, and the project reduces to layout grounding. We will report that outcome as such.