Translating Data Visualizations into Accessible Semantic Structures

45 concepts34 papers2 proposals

Data visualizations are powerful because they exploit human pre-attentive visual processing. A sighted reader can glance at a line chart and instantly grasp a complex trend, notice an outlier, or compare two distributions. However, this spatial compression of information creates a severe accessibility barrier. When a chart is rendered as a flat raster image (like a PNG) or inside a high-performance graphics environment like WebGL, the underlying data structure is lost. As Venkatesh Sivaraman et al. and Frank Elavsky et al. point out, these formats do not automatically expose their content to the browser's accessibility tree, rendering them completely opaque to assistive technologies like screen readers.

To bridge this gap, researchers are developing methods to translate visual charts back into accessible semantic structures. This problem is not just about generating text; it requires reconstructing the underlying data, extracting high-level insights, and designing interactive, non-linear ways to explore spatial information without sight.

The Semantic Gap in Chart Descriptions

The traditional solution to image inaccessibility is alternative text (alt-text). However, traditional alt-text for data visualizations suffers from a profound "descriptive insight gap."

In an analysis of thousands of academic publications, Sanjana Shivani Chintalapati et al. discovered that even when authors do provide alt-text, it almost exclusively focuses on low-level construction details—such as axis labels or chart types—while completely omitting the high-level trends and statistical takeaways. A description that merely states "a line chart showing GDP over time" provides a structural skeleton but leaves the actual "story" of the data a black box.

[Visual Chart] ──(Traditional Alt-Text)──> "A line chart of GDP over time" (Low-Level)
               ──(Semantic Summary)─────> "GDP grew steadily until a sharp drop in 2020" (High-Level)

To automate the creation of high-quality descriptions, researchers have turned to vision-language models (VLMs). However, standard image-captioning models that excel at describing everyday scenes (e.g., "a cat in a park") struggle with the precise, abstract nature of scientific figures. To train models to extract actual analytical insights, researchers have introduced large-scale, specialized datasets:

  • SCICAP: Ting-Yao Hsu et al. curated over 2 million real-world scientific figures from arXiv to highlight the steep challenge of automated figure captioning.
  • AutoChart: Jiawen Zhu et al. used a linguistic framework called "Rhetorical Moves" to generate thousands of charts paired with sophisticated, human-like analytical descriptions.
  • LineCap: Anita Mahinpei et al. focused specifically on multi-line scientific plots, training models to describe high-level trends rather than just reading off isolated labels.

A major hurdle in automated chart-to-text generation is "hallucination"—the tendency of AI models to invent statistical facts. To combat this, Enamul Hoque et al. introduced a data variable substitution technique that forces the model to anchor its descriptions strictly to the verified data points within the chart. Similarly, Omar Moured et al. developed AltChart, which improves VLM accuracy by pretraining models on geometric "pretext tasks," such as solving chart jigsaws and predicting rotations, ensuring the model truly understands the visual structure before attempting to summarize it.

Reconstructing Data from Pixels

Relying on authors to write alt-text or expecting VLMs to perfectly summarize a chart from a distance is not always sufficient. True equity in data access requires giving the user the ability to inspect the data directly. This has led to the development of automated chart parsing tools that reverse-engineer visualizations back into structured data.

For instance, ChartParser, developed by Anukriti Kumar et al., uses computer vision pipelines to locate axes, decode legends, isolate bar colors, and extract raw numerical values from static bar charts, converting them into screen-reader-friendly data tables.

[Static Raster Chart (PNG)] ──> [ChartParser / ChartFormer] ──> [Accessible Table / Tactile SVG]

Going beyond text and tables, Omar Moured et al. introduced ChartFormer, a model that converts standard raster chart images into tactile-accessible Scalable Vector Graphics (SVGs). By transforming flat pixels into structured vector paths that adhere to accessibility standards, this approach allows blind and visually impaired users to interact with the chart's geometry directly using embossed paper or refreshable tactile displays.

Beyond Text: Interactive and Multimodal Exploration

While a well-written text summary or a raw data table is a massive improvement over an empty image, linear text is fundamentally limited. Sighted users do not read a chart linearly from top to bottom; they jump back and forth, compare specific bars, and explore relationships dynamically. As Gerhard Weber et al. observe, translating non-linear graphical notations into non-visual formats without losing semantic meaning remains one of the core challenges of accessibility.

To replicate the exploratory nature of visual data analysis, researchers are moving toward interactive, multimodal interfaces.

Non-Visual Graph Navigation

Instead of forcing a chart into a paragraph of text, Data Navigator, a toolkit designed by Frank Elavsky et al., decouples the navigation logic of a visualization from its visual rendering. It models the data points, axes, and trends as an interconnected graph structure. Users can traverse this graph using keyboard commands, screen readers, or gestures, allowing them to independently choose whether to listen to a high-level summary of a trend or dive deep into individual data points.

Sonification and Audio Cues

Another powerful alternative to visual perception is data sonification—mapping data dimensions to sound properties like pitch, volume, or panning. Boris Schauerte et al. developed a web-based platform that leverages the Web Audio API to turn touchscreen surfaces into interactive audio maps and graphs. By sliding a finger across a tablet, users receive real-time auditory feedback, allowing them to map out the shape of a curve or the boundaries of a region purely through sound.

Combining Touch and Speech

The most effective non-visual interfaces mimic the multi-sensory way sighted people interact with data. Samuel Reinders et al. investigated a multimodal approach that pairs Refreshable Tactile Displays (RTDs) with Conversational Agents (CAs).

                       ┌──> Conversational Agent ──> "What is the peak value?" (Quick Facts)
[Multimodal Interface] ┤
                       └──> Tactile Display ───────> [Physical Shape of Curve] (Spatial Context)

The intuition behind this combination is a division of cognitive labor:

  1. Speech (The Conversational Agent) is excellent for extracting quick, discrete facts (e.g., "What is the exact value of the highest bar?").
  2. Touch (The Tactile Display) is uniquely suited for spatial sense-making, allowing the user to feel the physical slope of a line or the distribution of a scatter plot.

Their study demonstrated that while language provides immediate answers, the physical sensation of touch allows users to form an independent mental model of the data's structure. By shifting the focus from passive consumption (listening to a static caption) to active exploration (interrogating a semantic graph through touch, sound, and speech), these frameworks aim to fulfill the broader goal outlined by Silvia Miksch et al.: democratizing data insights by ensuring that complex information is never locked behind a purely visual wall.

Key Foundational Work

Voice search for development (2010) — contributed 1 concept elements to this direction.
Haptic Granular Synthesis: Targeting, Visualisation and Texturing (2004) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_visualization_accessibility_gap_90747698,
  title = {visualization accessibility gap},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/visualization-accessibility-gap/rdr_90747698ad38407b8c0fef8ba49d535c},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_visualization_accessibility_gap_90747698_v1,
  title = {visualization accessibility gap},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/visualization-accessibility-gap/rdr_90747698ad38407b8c0fef8ba49d535c/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake