Traditional data integration relies on Extract, Transform, Load (ETL) pipelines that physically consolidate heterogeneous data into a centralized warehouse. While effective for static, predictable workloads, this batch-oriented approach breaks down under modern pressures: high-velocity streaming data, rapidly changing external APIs, and the prohibitive storage and synchronization costs of maintaining duplicate data silos.
Virtualized semantic data integration shifts the integration boundary from offline preprocessing to real-time query execution or ingestion. Instead of physically moving and transforming data upfront, systems maintain data at its source and resolve structural, semantic, and entity-level differences on the fly. This approach ensures data freshness, eliminates the latency of data materialization pipelines, and allows applications to interact with a unified, up-to-date view of disparate data sources.
The Shift from Centralized Warehousing to Virtualization
The core intuition behind data virtualization is that rewriting a query is often far cheaper than rewriting the underlying data. Early foundations for this paradigm were established by Gingras in nD-SQL: A Multi-Dimensional Language for Interoperability and OLAP, which treated metadata—such as table and column names—as first-class data. This allowed users to perform complex restructurings and multi-dimensional aggregations directly across federated databases without the overhead of a pre-integrated repository.
In modern contexts, this has evolved into Ontology-Based Data Access (OBDA), where an abstract semantic layer (an ontology) masks technical database complexities. As detailed by Giese et al. in The Optique Project: Towards OBDA Systems for Industry (Short Paper) and Optique: OBDA Solution for Big Data, virtualizing data access allows organizations to maintain data freshness and avoid massive data consolidation costs.
When users query the ontology, the system translates those queries into optimized native queries (like SQL) on the fly. This eliminates what Skjæveland et al. describe in Publishing the Norwegian Petroleum Directorate's FactPages as Semantic Web Data as the "synchronization gap"—the window where a static semantic snapshot becomes outdated the moment the underlying database is modified.
Traditional ETL:
[Raw Sources] ───(Batch ETL Pipeline)───> [Centralized Warehouse] ───> [User Query]
(Stale Data)
Virtualized Integration:
[User Query] ───> [Semantic Layer] ───(On-the-fly Translation)───> [Live Sources]
(Fresh Data)
On-the-Fly Query Translation
To make virtualization work, a system must map user queries to external endpoints in real-time. For example, rather than downloading and converting massive geographic datasets into static Resource Description Framework (RDF) triples, Lukasiewicz et al. developed a virtual RDF graph approach in LOSM: a SPARQL Endpoint to Query Open Street Map. The system accepts standard SPARQL queries and translates them into native Overpass API calls in real-time, treating a live external database as a queryable semantic endpoint.
However, building these semantic mappings manually can be a bottleneck. To scale this process, Goel et al. introduced an interactive system called Karma in Interactively Mapping Data Sources into the Semantic Web. Karma uses machine learning and graph algorithms to semi-automatically map raw data sources to existing ontologies, enabling virtualized semantic access instead of generating fixed, static output files.
Query-Time Entity Resolution and Enrichment
Data integration requires more than just schema mapping; it also requires resolving entity conflicts (e.g., determining if two records refer to the same real-world object) and enriching records with external context. Performing these steps on the fly requires shifting from global batch processing to targeted, context-aware execution.
Adaptive Entity Resolution
Traditional entity resolution scales poorly because it compares every record against every other record upfront. In Online Collective Entity Resolution, Bhattacharya et al. propose an "expand and resolve" strategy that operates at query time. By only fetching and resolving records that are highly informative to the user's active request, the system delivers clean, deduplicated results interactively without needing to clean the entire database beforehand.
Dynamic Enrichment
Instead of storing copies of external datasets locally, systems can enrich data dynamically during the consumption phase.
- Quercini et al. demonstrate this in Facetted Browsing on Extracted Fusion Tables Data for Digital Cities by resolving local entities to global identifiers during browsing, pulling real-time information from external knowledge graphs and multimedia APIs on demand.
- Srinivas et al. formalize this vision in Self-Curating Databases, where the database engine unifies data and metadata into a holistic model. The system automatically discovers, links, and enriches information in real-time during query execution based on the specific context of the user's request.
Integrating Ephemeral and Streaming Data
On-the-fly integration is uniquely suited for ephemeral data—such as live sensor feeds, GPS logs, or stock prices—where updates occur too rapidly for traditional database materialization to be efficient.
API-Driven Query Operators
To incorporate volatile web data directly into graph queries, Junemann et al. introduced a specialized query operator (BIND_API) in Incorporating API Data into SPARQL Query Answers. This operator dynamically generates request URIs from existing variable bindings during query execution, fetching and processing external JSON API data as a native part of the query flow without permanently storing it.
SPARQL Execution Flow with BIND_API:
1. Evaluate local graph patterns ──> ?city = "Dublin"
2. BIND_API(?city) ───────────────> Calls api.weather.com/Dublin
3. Parse JSON response ───────────> ?temp = "14C"
4. Stream unified result ─────────> { "Dublin", "14C" }
Near-to-the-Source Stream Fusion
For high-velocity streaming data, integration can happen at the moment of ingestion. In FAIMUSS: Flexible Data Transformation to RDF from Multiple Streaming Sources, Santipantakis et al. present a "near-to-the-source" integration tool. As streaming records (like live ship coordinates) arrive, the system instantly performs archival lookups and resolves external references, creating enriched linked data fragments on the fly and avoiding post-processing latencies.
This real-time fusion allows engines to handle complex urban and personal environments:
- Urban Monitoring: Tallevi-Diotallevi et al. demonstrate in Real-Time Urban Monitoring in Dublin Using Semantic and Stream Technologies that directly integrating raw, unstructured data formats with structured linked data within a single query engine eliminates expensive pre-conversion steps, allowing high-velocity sensor feeds to be queried alongside static knowledge bases.
- Personal Analytics: Giunchiglia et al. apply this to personal devices in A Context Model for Personal Data Streams, transforming flat, messy numerical sensor logs into structured, human-understandable knowledge graphs to enable real-time, context-aware reasoning.
Downstream Impact
By shifting the computational burden of data transformation and consolidation to the query engine, downstream applications can remain lightweight and responsive. For example, Graux et al. leverage this in Timelining Knowledge Graphs in the Browser (short paper) to build browser-based visualization tools. Because the application queries public SPARQL endpoints on the fly, it requires zero local data storage or preprocessing, ensuring that the rendered interface consistently reflects the most recent updates in the underlying knowledge graph.
Ultimately, this paradigm fulfills the broader vision outlined by Pan in The Ubiquitous Semantic Web: The Story So Far: transitioning from a document-centric web to a data-centric web requires making structured information actionable in ubiquitous, mobile environments, where data must be constantly generated, enriched, and integrated on the fly.
Key Foundational Work
Automatic Data Fusion with HumMer (2005) — contributed 2 concept elements to this direction.
nD-SQL: A Multi-Dimensional Language for Interoperability and OLAP (1998) — contributed 1 concept elements to this direction.