Abstract
Reasoning over semantically annotated data is an emerging trend in stream processing aiming to produce sound and complete answers to a set of continuous queries. It usually comes at the cost of finding a trade-off between data throughput and the cost of expressive inferences. Strider-lsa proposes such a trade-off and combines a scalable RDF stream processing engine with an efficient reasoning system. The main reasoning tasks are based on a query rewriting approach for SPARQL that benefits from an intelligent encoding of RDFS+ (RDFS + owl:sameAs) ontology elements. Strider-lsa runs in production at a major international water management company to detect anomalies from sensor streams. The system is evaluated along different dimensions and over multiple datasets to emphasize its performance.
TL;DR
Strider-lsa is a scalable RDF stream processing engine that enables real-time reasoning over massive data streams by combining Spark/Kafka with an intelligent query-rewriting approach. It achieves million-level throughput for complex continuous queries by transforming semantic hierarchies into efficient integer range checks and managing entity equivalence through "clique" representatives.
Introduction: The Challenge of Real-Time Semantic Reasoning
In the era of the Internet of Things (IoT), data is frequently "semantically annotated," meaning it is linked to ontologies that provide context. To derive meaningful insights, systems must perform reasoning. For instance, a query for "Sensors" should automatically include results for "PressureSensors" based on ontological relationships.
However, two major hurdles exist for real-time applications:
- Scalability: Reasoning is computationally intensive, often causing traditional engines to lag behind high-velocity Big Data streams.
- The Identity Problem (
owl:sameAs): Different identifiers often refer to the same real-world entity. Reconciling these aliases in real-time without massive data duplication (materialization) or slow query rewriting is a significant bottleneck.
System Architecture and Methodology
Strider-lsa addresses these challenges by building upon the Strider RSP engine and integrating the LiteMat reasoning approach. The architecture leverages Apache Spark and Kafka for distributed processing, as illustrated below.
The system relies on three core pillars:
- Intelligent Encoding (LiteMat): Ontology terms (classes and properties) are encoded into numerical identifiers using a prefix-based binary representation. This transforms complex semantic checks into fast integer range comparisons (e.g.,
ID >= LowerBound AND ID < UpperBound) during execution. - Clique-Based Equivalence: To handle
owl:sameAs, the system identifies "cliques" of equivalent individuals. It assigns a unique ID to each clique and replaces all aliases in the incoming stream with a single representative ID at ingestion, preventing data explosion. - Query Rewriting: SPARQL queries are parsed and rewritten into optimized Spark SQL/Scala code that operates directly on the encoded numerical data.
Evaluation and Datasets
The system was rigorously evaluated using several benchmarks:
- LUBM (Lehigh University Benchmark): Extended with
owl:sameAstriples to simulate complex identity scenarios. - LOD (Linked Open Data) Datasets: Real-world data from DBpedia, Yago, and Drugbank were used to model realistic entity equivalence.
- Waves Project: A practical use case involving anomaly detection in water distribution networks for a major international utility company.
Core Innovations
- Semantic Reasoning as Range Filtering: By encoding hierarchies correctly, category membership checks become simple range filters, avoiding expensive runtime graph traversals.
- Ingestion-Time Normalization: Resolving
owl:sameAsaliases during Abox encoding keeps the internal stream compact and avoids "join-bombs" during query processing. - Hybrid Metadata Management: Static metadata (TBox) is encoded offline, while dynamic stream data (Abox) is encoded on the fly, balancing preparation and execution efficiency.
Performance Analysis
Strider-lsa demonstrates significant performance gains over traditional materialization-based approaches (SAM).
Key findings include:
- Million-Level Throughput: The system achieves up to 2 million triples per second with second-level latency.
- Efficiency: LiteMat (LM) was consistently faster than SAM, showing a 75% improvement on complex queries involving multiple inferences.
- Robustness: While materialization-based systems often crashed on complex queries (Q5, Q7, Q8) due to memory exhaustion, Strider-lsa remained stable and efficient.
Conclusion and Future Directions
While highly effective, the authors identify several areas for future optimization:
- Memory Optimization: Reducing the footprint of large encoding dictionaries.
- Distributed Dictionaries: Implementing efficient dictionary partitioning across clusters to improve scalability.
- Expanded Expressiveness: Extending support to more complex logic, such as transitive properties.
- Spark Filter Optimization: Further tuning how Spark handles the specific numeric range filters generated by LiteMat.
Summary updated