TL;DR
This paper introduces LangCrUX, the first large-scale dataset of 120,000 popular websites across 12 non-Latin script languages, to evaluate web accessibility for non-English speakers. The study uncovers a severe "language mismatch" where websites with visible content in a native language provide accessibility metadata (like image alt text) almost entirely in English, rendering screen readers ineffective for local users. To address this, the authors develop Kizuki, an open-source, language-aware extension for automated accessibility auditing.
What Problem Does This Paper Address?
For visually impaired users, screen readers are a lifeline to the digital world. These tools rely heavily on hidden metadata—such as alt attributes for images and aria-label tags for buttons—to describe web content. However, while the web is increasingly multilingual, screen readers frequently stumble when encountering non-Latin scripts (like Devanagari, Arabic, or Thai) or mixed-language pages.
Prior web accessibility research has largely ignored this intersection of multilingualism and assistive technology, primarily due to a lack of data. Standard web datasets like Tranco rank sites by popularity but offer no insight into their linguistic makeup. This paper tackles this gap by systematically measuring how well the modern web supports accessibility across diverse linguistic contexts.
Methods
The authors designed a multi-step pipeline to collect and analyze localized web accessibility data:
- Localized Web Crawling: Using Puppeteer, the authors built a custom crawler to visit websites programmatically. Crucially, they routed all traffic through country-specific VPNs. This step is vital because many websites dynamically alter their language and layout based on the user's IP address; crawling from generic cloud IPs would mistakenly pull global, English-dominant versions of the sites.
- Language-Sensitive Auditing: They isolated 12 key HTML elements defined by the Axe-core engine (which powers Google Lighthouse) where natural language is critical for user comprehension (e.g.,
image-alt,button-name,link-name). - Heuristic Text Filtering: To ensure they were analyzing meaningful data, they built a rule-based filtering pipeline to discard uninformative accessibility text, such as generic single words ("button"), placeholders ("icon"), raw URLs, file names ("banner.jpg"), or text strings that were too short to convey meaning.
- Language-Aware Testing (Kizuki): They developed Kizuki, a Google Lighthouse extension that goes beyond checking for the mere presence of accessibility tags. Kizuki uses language detection to verify whether the language of the accessibility metadata actually matches the language of the visible page content.
Datasets
The primary contribution of this work is LangCrUX, a newly introduced, open-source dataset containing 120,000 popular websites spanning 12 distinct non-Latin script languages (10,000 sites per language-country pair).
The dataset covers:
- East Asian Scripts: Mandarin Chinese (China), Cantonese (Hong Kong), Japanese (Japan), Korean (South Korea).
- South Asian Scripts: Hindi (India), Bangla (Bangladesh).
- Arabic Scripts: Modern Standard Arabic (Algeria), Egyptian Arabic (Egypt).
- Other Scripts: Russian (Cyrillic), Thai (Thai), Greek (Greek), Hebrew (Israel).
Websites were selected from the Chrome User Experience Report (CrUX) and filtered using a Unicode-based heuristic to ensure that at least 50% of the visible text on each site was in the target native language.
Key Insights and Ideas
- The Compliance Illusion: Standard accessibility tools suffer from a major blind spot. They mark an image as "accessible" as long as an
altattribute exists—even if it is completely empty or written in a language the user doesn't speak. This allows developers to pass automated compliance checks without actually helping the user. - The English Default Bias: Even on websites where the visible text is almost entirely in a native language (such as a Bangladeshi government portal written 98% in Bangla), developers overwhelmingly default to writing accessibility metadata in English.
- Mixed-Language Chaos: Many sites mix English and native scripts within a single accessibility label. Because screen readers generally cannot handle mid-label language switching, this results in severe mispronunciations and unintelligible audio output.
Key Results
- Widespread Neglect: Accessibility tags are frequently missing entirely. For instance,
labeltags are missing on average 98.5% of the time, andlink-nameattributes are missing 95.9% of the time. Whileimage-alttags are missing less often (17.12%), they exhibit the highest rate of being left completely empty (25.39%). - Severe Language Mismatch: In India and Bangladesh, over 40% of websites feature less than 10% of their accessibility text in the native language. In Bangladesh, a staggering 79% of informative accessibility text is written in English despite the visible content being in Bangla.
- The Kizuki Reality Check: When evaluating 10,000 websites from Bangladesh and Thailand using standard Lighthouse, 43% of the sites received a "good" accessibility score above 90. However, when Kizuki factored in language consistency, that number plummeted to just 15.8%, exposing a massive gap between automated compliance and real-world usability.
Limitations and Future Work
- Traffic Bias: Because LangCrUX relies on the Google CrUX database, the study is biased toward high-traffic websites, potentially overlooking the accessibility state of smaller, highly localized community portals.
- VPN Evasion: Some websites detect commercial VPNs and serve generic or restricted pages, which required the authors to manually filter and replace affected sites during crawling.
- Exclusion of Video Captions: Due to the technical complexity of programmatically playing and evaluating video streams at scale, the authors omitted
video-captionchecks from their automated pipeline. - Future Directions: The authors suggest expanding language-aware auditing to dynamic web components and exploring how AI-driven, localized alt-text generation can help developers automatically bridge this linguistic divide.
Summary updated