Much of the world’s most valuable data does not live in clean, plaintext strings. Instead, it is locked inside visual formats: scientific plots, chemical structures, infographics, maps, and user interface designs. When an expert looks at a chart, they instantly extract trends, data-to-pixel mappings, and semantic relationships. Yet, traditional information retrieval systems treat these graphics as opaque blocks of pixels or rely entirely on the text surrounding them.
This disconnect creates a visual searchability gap. To bridge it, researchers are developing techniques that parse the internal structural logic of graphics and allow users to query data using visual, structural, or multimodal inputs.
The Visual Searchability Gap
Traditional search engines index documents by parsing their text. If a research paper contains a crucial data plot or a patent contains a novel molecular structure, a text-based crawler only indexes the caption or the surrounding prose. As Li et al. point out in their work on infographic retrieval, this approach neglects the unique, implicit insights contained within the graphics themselves, leading to a massive gap in knowledge discovery.
This problem spans multiple domains:
- Scientific Literature: Critical experimental evidence is routinely trapped in figures. Yang et al. highlight that biomedical figures contain essential data labels that text mining workflows completely miss, while Ghosh et al. demonstrate that text-only pipelines lose a substantial portion of materials science evidence reported in plots.
- Intellectual Property: In chemistry patents, vital molecular structures are stored as visual depictions rather than searchable text strings, preventing researchers from effectively indexing or querying them for drug discovery, as noted by Morin.
- The Web and Digital Archives: Antonacopoulos et al. observe that web designers frequently embed semantically important text directly into images for styling purposes, rendering it invisible to standard web crawlers. As the web becomes increasingly visual, keyword-based exploration of digital and historical archives becomes highly inadequate, a challenge explored by Adewoye et al..
When search engines rely solely on document-level text, they suffer from high false-positive rates. For example, Mitra et al. show that a document might mention a geographic location or a "map" in its text without actually containing the relevant visual map data a user needs.
Why Traditional Parsers and OCR Fail
Why can't we just use standard Optical Character Recognition (OCR) to extract text from these images and feed it into a standard search engine?
Standard OCR is designed for linear, structured documents—like a scanned page of a book. It assumes text flows left-to-right and top-to-bottom. Graphical data breaks these assumptions completely. Infographics and charts feature multi-oriented text, labels wrapped around axes, and text embedded in complex, low-resolution color backgrounds. To solve the background styling issue, Antonacopoulos et al. designed a bottom-up approach mimicking human color perception, using fuzzy inference to merge components based on connectivity and color distance rather than standard RGB processing. Similarly, to capture text oriented at arbitrary angles within charts, Scherp et al. developed a pipeline combining hierarchical binarization and density-based clustering.
Standard OCR Pipeline:
[Pixels] ──> [Linear Text Scan] ──> Fails on non-linear/multi-oriented text
Graphical Parsing Pipeline:
[Pixels] ──> [Spatial/Structural Decomposition] ──> [Context-Aware Extraction]
Furthermore, extracting text labels is only half the battle. Graphics convey meaning through spatial and structural relationships. As Bhatt et al. note, standard OCR fails to interpret and localize graphical elements like tables, figures, and formulas. A chart's true meaning lies in its internal "DNA"—the data-to-pixel mappings, mark types (bars, lines, scatter points), and stylistic choices. Hoque et al. demonstrate that standard engines cannot index these properties, meaning a developer cannot search a repository for a specific technical attribute, such as a "scatterplot with negative values," without a system that explicitly parses the visualization's underlying code and visual attributes.
The Query Formulation Bottleneck
The visual searchability gap isn't just an extraction problem; it is also a user interface problem. Even if a system perfectly indexes every graphic, users face a query formulation difficulty.
Humans are remarkably good at remembering visual layouts, colors, and spatial arrangements, but we struggle to translate those visual memories into precise text strings. As Furusawa et al. observe, a user might easily recall the spatial layout and color scheme of a web page long after they have forgotten its specific textual content or URL.
When forced to use a traditional text search bar for visual concepts, users hit a wall. Wang et al. and Ogawa et al. point out that users frequently lack the technical vocabulary required to describe complex visual observations, mechanical failures, or abstract artistic styles in text. For instance, in software engineering, developers looking to reuse existing web application models find it incredibly difficult to express complex structural and graph-based requirements using keywords alone, as shown by Bozzon et al..
How Structural and Multimodal Retrieval Works
To make graphical data truly accessible, modern systems attack the problem from both sides: unlocking the data via multimodal extraction and allowing flexible exploration via visual queries.
1. Multimodal and Deep Structural Extraction
Instead of choosing between text or vision, state-of-the-art systems fuse them. Early foundations by Rafkind et al. and You et al. proved that combining image processing features with semantic information from captions and inline text pointers drastically improves biomedical figure classification and retrieval.
Recently, Multimodal Large Language Models (VLMs) have fundamentally changed how systems "read" graphics. Rather than relying on rigid, hand-crafted heuristics to find axes and labels, systems can leverage the open-ended reasoning of models like GPT-4V. For example, Zheng et al. built an automated pipeline powered by GPT-4V to categorize thousands of scientific pages and extract complex graphical data from characterization plots in chemistry literature with over 93% accuracy, successfully bridging the gap between locked image data and digital databases.
2. Visual Queries and Query-by-Example
To bypass the limitations of text descriptions, researchers are building interfaces that accept visual inputs directly:
- Query-by-Example: In model-driven software engineering, Bozzon et al. implemented graph-matching algorithms that allow developers to use a visual fragment or a structural sketch as the query itself, matching it against a repository of complex models.
- Sketch-Guided Generation: For consumer applications where users want to find a product with a specific aesthetic or spatial arrangement, Sun introduced a framework that pairs rough freehand sketches with natural language. The system uses a language agent to refine the intent and a diffusion model to generate an "ideal" query image, turning an imprecise human sketch into a high-accuracy product search.
- Multimodal Query Suggestions: When a user provides an image but doesn't know how to proceed, Wang et al. leverage multi-agent reinforcement learning to automatically generate text-based query suggestions based on the input image, helping the user find the right vocabulary to navigate their search.
By treating graphics as structured, semantic data rather than flat arrays of pixels, this research direction transforms how we interact with visual knowledge—turning unsearchable images into rich, navigable databases.
Key Foundational Work
A Texture Thesaurus for Browsing Large Aerial Photographs (1998) — contributed 1 concept elements to this direction.
Text Extraction from Web Images Based on Human Perception and Fuzzy Inference (2001) — contributed 1 concept elements to this direction.