Not All Visitors are Bilingual: A Measurement Study of the Multilingual Web from an Accessibility Perspective

Masudul Hasan Masud Bhuiyan, Matteo Varvello, Yasir Zaki, Cristian-Alexandru Staicu

TL;DR

This paper introduces LangCrUX, the first large-scale dataset of 120,000 popular websites across 12 non-Latin script languages, to evaluate web accessibility for non-English speakers. The study uncovers a severe "language mismatch" where websites with visible content in a native language provide accessibility metadata (like image alt text) almost entirely in English, rendering screen readers ineffective for local users. To address this, the authors develop Kizuki, an open-source, language-aware extension for automated accessibility auditing.

What Problem Does This Paper Address?

For visually impaired users, screen readers are a lifeline to the digital world. These tools rely heavily on hidden metadata—such as alt attributes for images and aria-label tags for buttons—to describe web content. However, while the web is increasingly multilingual, screen readers frequently stumble when encountering non-Latin scripts (like Devanagari, Arabic, or Thai) or mixed-language pages.

Prior web accessibility research has largely ignored this intersection of multilingualism and assistive technology, primarily due to a lack of data. Standard web datasets like Tranco rank sites by popularity but offer no insight into their linguistic makeup. This paper tackles this gap by systematically measuring how well the modern web supports accessibility across diverse linguistic contexts.

Methods

The authors designed a multi-step pipeline to collect and analyze localized web accessibility data:

  1. Localized Web Crawling: Using Puppeteer, the authors built a custom crawler to visit websites programmatically. Crucially, they routed all traffic through country-specific VPNs. This step is vital because many websites dynamically alter their language and layout based on the user's IP address; crawling from generic cloud IPs would mistakenly pull global, English-dominant versions of the sites.
  2. Language-Sensitive Auditing: They isolated 12 key HTML elements defined by the Axe-core engine (which powers Google Lighthouse) where natural language is critical for user comprehension (e.g., image-alt, button-name, link-name).
  3. Heuristic Text Filtering: To ensure they were analyzing meaningful data, they built a rule-based filtering pipeline to discard uninformative accessibility text, such as generic single words ("button"), placeholders ("icon"), raw URLs, file names ("banner.jpg"), or text strings that were too short to convey meaning.
  4. Language-Aware Testing (Kizuki): They developed Kizuki, a Google Lighthouse extension that goes beyond checking for the mere presence of accessibility tags. Kizuki uses language detection to verify whether the language of the accessibility metadata actually matches the language of the visible page content.
Figure 1: Overview of our methodology for construct- ing and analyzing the LangCrUX dataset.

Datasets

The primary contribution of this work is LangCrUX, a newly introduced, open-source dataset containing 120,000 popular websites spanning 12 distinct non-Latin script languages (10,000 sites per language-country pair).

The dataset covers:

  • East Asian Scripts: Mandarin Chinese (China), Cantonese (Hong Kong), Japanese (Japan), Korean (South Korea).
  • South Asian Scripts: Hindi (India), Bangla (Bangladesh).
  • Arabic Scripts: Modern Standard Arabic (Algeria), Egyptian Arabic (Egypt).
  • Other Scripts: Russian (Cyrillic), Thai (Thai), Greek (Greek), Hebrew (Israel).

Websites were selected from the Chrome User Experience Report (CrUX) and filtered using a Unicode-based heuristic to ensure that at least 50% of the visible text on each site was in the target native language.

Key Insights and Ideas

  • The Compliance Illusion: Standard accessibility tools suffer from a major blind spot. They mark an image as "accessible" as long as an alt attribute exists—even if it is completely empty or written in a language the user doesn't speak. This allows developers to pass automated compliance checks without actually helping the user.
  • The English Default Bias: Even on websites where the visible text is almost entirely in a native language (such as a Bangladeshi government portal written 98% in Bangla), developers overwhelmingly default to writing accessibility metadata in English.
  • Mixed-Language Chaos: Many sites mix English and native scripts within a single accessibility label. Because screen readers generally cannot handle mid-label language switching, this results in severe mispronunciations and unintelligible audio output.

Key Results

  • Widespread Neglect: Accessibility tags are frequently missing entirely. For instance, label tags are missing on average 98.5% of the time, and link-name attributes are missing 95.9% of the time. While image-alt tags are missing less often (17.12%), they exhibit the highest rate of being left completely empty (25.39%).
  • Severe Language Mismatch: In India and Bangladesh, over 40% of websites feature less than 10% of their accessibility text in the native language. In Bangladesh, a staggering 79% of informative accessibility text is written in English despite the visible content being in Bangla.
CDFs of native language usage in visible vs. accessibility text
  • The Kizuki Reality Check: When evaluating 10,000 websites from Bangladesh and Thailand using standard Lighthouse, 43% of the sites received a "good" accessibility score above 90. However, when Kizuki factored in language consistency, that number plummeted to just 15.8%, exposing a massive gap between automated compliance and real-world usability.
Accessibility score distribution before and after applying Kizuki

Limitations and Future Work

  • Traffic Bias: Because LangCrUX relies on the Google CrUX database, the study is biased toward high-traffic websites, potentially overlooking the accessibility state of smaller, highly localized community portals.
  • VPN Evasion: Some websites detect commercial VPNs and serve generic or restricted pages, which required the authors to manually filter and replace affected sites during crawling.
  • Exclusion of Video Captions: Due to the technical complexity of programmatically playing and evaluating video streams at scale, the authors omitted video-caption checks from their automated pipeline.
  • Future Directions: The authors suggest expanding language-aware auditing to dynamic web components and exploring how AI-driven, localized alt-text generation can help developers automatically bridge this linguistic divide.

Summary updated

Abstract

English is the predominant language on the web, powering nearly half of the world's top ten million websites. Support for multilingual content is nevertheless growing, with many websites increasingly combining English with regional or native languages in both visible content and hidden metadata. This multilingualism introduces significant barriers for users with visual impairments, as assistive technologies like screen readers frequently lack robust support for non-Latin scripts and misrender or mispronounce non-English text, compounding accessibility challenges across diverse linguistic contexts. Yet, large-scale studies of this issue have been limited by the lack of comprehensive datasets on multilingual web content. To address this gap, we introduce LangCrUX, the first large-scale dataset of 120,000 popular websites across 12 languages that primarily use non-Latin scripts. Leveraging this dataset, we conduct a systematic analysis of multilingual web accessibility and uncover widespread neglect of accessibility hints. We find that these hints often fail to reflect the language diversity of visible content, reducing the effectiveness of screen readers and limiting web accessibility. We finally propose Kizuki, a language-aware automated accessibility testing extension to account for the limited utility of language-inconsistent accessibility hints.
BibTeX
@inproceedings{bhuiyan2025not,
  author = {Masudul Hasan Masud Bhuiyan and Matteo Varvello and Yasir Zaki and Cristian-Alexandru Staicu},
  title = {{Not All Visitors are Bilingual: A Measurement Study of the Multilingual Web from an Accessibility Perspective}},
  booktitle = {IMC},
  year = {2025},
  doi = {10.1145/3730567.3764505},
}
Figure