Abstract: An emerging family of language models (LMs), capable of processing both text and images within a single visual view, has the promise to unlock complex tasks such as chart understanding and UI navigation. We refer to these models as screenshot language models. Despite their appeal, existing screenshot LMs substantially lag behind text-only models on language understanding tasks. To close this gap, we adopt a simplified setting where the model inputs are plain-text-rendered screenshots, and we focus on improving the text ability of screenshot LMs. We propose a novel Patch-and-Text Prediction (PTP) objective, which masks and recovers both image patches of screenshots and text within screenshots. We also conduct extensive ablation studies on masking rates and patch sizes, as well as designs for improving training stability. Our pre-trained model, while solely taking visual inputs, achieves comparable performance with BERT on 6 out of 8 GLUE tasks (within 2%) and improves up to 8% over prior work. Additionally, we extend PTP to train autoregressive screenshot LMs and demonstrate its effectiveness--our models can significantly reduce perplexity by utilizing the screenshot context. Together, we hope our findings can inspire future research on developing powerful screenshot LMs and extending their reach to broader applications.
TL;DR
Screenshot Language Models (LMs) process text and images through a single visual lens, but they have historically lagged behind text-only models on standard language tasks. This paper introduces Patch-and-Text Prediction (PTP), a novel training objective that forces models to reconstruct both visual pixels and semantic text, bringing screenshot LMs to near-parity with BERT on the GLUE benchmark.
The Challenge of Screenshot Language Models
We are seeing a shift toward "Screenshot LMs"—models like Pix2Struct or PIXEL that "see" a webpage or document as a single image rather than a string of characters. This approach is elegant because it bypasses messy OCR pipelines and naturally handles visually situated text (like text in charts or UI buttons).
However, there is a catch: these models are surprisingly bad at pure language understanding. While a text-only model like BERT excels at logic and semantics, screenshot LMs often struggle with the "modality mismatch." They spend so much effort learning to recognize shapes that they miss the deeper linguistic meaning. The authors aim to close this performance gap, making screenshot LMs as linguistically capable as their text-only cousins.
Methods: Patch-and-Text Prediction (PTP)
The authors propose the Patch-and-Text Prediction (PTP) objective. Instead of just predicting missing pixels (like the PIXEL model) or just predicting missing text (like Pix2Struct), PTP does both simultaneously.
- Dual Masking: The model takes a screenshot of text, then masks out certain image patches (visual) and certain words (semantic).
- Dual Decoding: The architecture uses a shared Transformer encoder but employs two separate decoders. One decoder uses Mean Squared Error (MSE) to reconstruct the raw pixels of the masked patches, while the other uses Cross-Entropy loss to recover the underlying text tokens.
- Stabilizing Training: Training on screenshots is notoriously unstable because the images are "highly polarized" (mostly high-contrast black and white). To fix this, the authors introduced several "tricks":
- Text Prefixes: Adding a standard string like "Beginning of the sequence:" to every screenshot to act as a training anchor.
- Pixel Normalization: Standardizing pixel values to prevent loss spikes.
- Embedding Layernorm: Adding normalization layers immediately after the input to handle high masking rates.
The researchers also extended this to Autoregressive Screenshot LMs, creating a decoder-only version (similar to the LLaMA architecture) that predicts the next patch or next token in a sequence.
Pre-training and Evaluation Datasets
- Pre-training: The models were pre-trained on the English Wikipedia and BookCorpus (the same data used for the original BERT) to ensure a fair comparison.
- Evaluation: The primary benchmark was GLUE (General Language Understanding Evaluation), which includes tasks like sentiment analysis (SST-2), linguistic acceptability (CoLA), and natural language inference (MNLI).
- Autoregressive Evaluation: For the decoder-only models, the authors measured Perplexity (PPL) on Wikipedia and BookCorpus to see how well the visual context helped predict subsequent text.
Key Insights and Results
- The "Aha" of the Dual Objective: The authors found that pixel prediction helps the model learn local visual features (the "font" and "shape" of letters), while text prediction is what actually drives language understanding. You need both to excel.
- Low Masking is Better for Text: In computer vision, it’s common to mask 75% of an image (MAE). However, for text-rendered screenshots, the authors found that a much lower masking rate (10% for patches, 25% for text) is optimal. Text is dense with information; if you hide too much, the model has nothing to learn from.
- Closing the Gap: The PTP model (86M parameters) achieved an average score of 80.2 on GLUE, significantly outperforming the previous state-of-the-art screenshot model, PIXEL (76.0).
- BERT Parity: On 6 out of 8 GLUE tasks, PTP performed within 2% of BERT, proving that a model taking only visual inputs can match the reasoning of a text-only model.
- Autoregressive Success: Their autoregressive screenshot LM successfully used visual context to reduce text perplexity (dropping from 10.28 to 9.66), showing that the model can effectively "read" the screenshot to predict what comes next.
Limitations and Future Work
- Efficiency: Screenshot LMs are computationally expensive. Because they process image patches, they require roughly twice the number of tokens as a text-only LM to represent the same amount of information.
- Simplified Setting: This paper focused on "text-only screenshots" (cleanly rendered text on a white background). Real-world screenshots contain noise, complex layouts, and non-textual UI elements, which remains a challenge for future research.
- Scaling: While the results at 86M and 1.3B parameters are promising, it remains to be seen how these objectives scale to "frontier" model sizes (70B+ parameters).
Summary updated