QUASL: A Framework for Question Answering and its Application to Business Intelligence

Nicolas Kuchmann-Beauger, Falk Brauer, Albert BifetSPIRE 2013

TL;DR

QUASL is a declarative framework designed to bridge the gap between natural language questions and complex Business Intelligence (BI) databases. By leveraging RDF and SPARQL to map extracted semantics to structured query models, it allows non-technical users to perform sophisticated data exploration—like range filters and personalized "top-k" queries—without knowing the underlying database schema.

What Problem Does This Paper Address?

In the world of Business Intelligence, data is stored in massive, complex "data warehouses." Currently, if a business user wants a specific insight (e.g., "Who are my top 5 middle-aged customers in Palo Alto?"), they usually have two choices: wait for the IT department to build a custom report or try to navigate a daunting technical interface.

Existing solutions like keyword search are often too simplistic; they struggle with "fuzzy" concepts like "middle-aged" (which implies a numerical range) or contextual phrases like "my city." The challenge is not just recognizing words, but understanding how those words relate to a specific database schema and ensuring the resulting query is mathematically and structurally valid.

Methods

The authors introduce QUASL, which follows a multi-stage pipeline to transform a question into a ranked list of executable queries:

  1. Entity Recognition: Using a combination of dictionary matching and linguistic patterns, the system identifies "artifacts" (measures, dimensions, or values). For example, "top 5" is recognized as a truncation rule, while "customers" is mapped to a dimension in the schema.
  2. The Parse Graph: All identified entities are stored in an RDF-based "parse graph." This graph captures the relationship between the user's words and the database metadata.
  3. Declarative Mapping (The Core Innovation): Instead of hard-coding rules, the authors use SPARQL graph patterns to define constraints. These patterns check if a query is valid (e.g., "Do these two dimensions actually connect to the same fact table?") and inject missing information (e.g., if a user asks for "revenue," the system might suggest "per year" to make the data meaningful).
  4. Query Generation: A SPARQL CONSTRUCT clause transforms the parse graph into an intermediate query model, which is then serialized into SQL or MDX.
  5. Scoring & Ranking: Since one question can have multiple interpretations, QUASL ranks them using three metrics:
  6. Confidence: How well did the text match the metadata?
  7. Complexity: More complex queries (using more recognized entities) are often more relevant.
  8. Selectivity: Queries generated by very specific, rare graph patterns are prioritized.

Datasets

The framework was evaluated using a corpus derived from ManyEyes, a collaborative visualization platform. Specifically, the authors used:

  • The American Census Bureau dataset: Integrated into a test data warehouse.
  • 50 "Gold Standard" Questions: These were the actual titles of the 50 most popular visualizations on ManyEyes related to the census data (e.g., "Home ownership by State").

Key Insights and Ideas

  • Declarative vs. Imperative: By using SPARQL to define mapping rules, the system is highly configurable. Administrators can add custom vocabulary (like defining "rich" as a specific income range) without rewriting the core engine.
  • Handling Ambiguity through Suggestion: Users often ask incomplete questions. QUASL’s ability to "propose" measures or dimensions based on schema relationships allows it to generate valid visualizations even from vague input.
  • Contextualization: The framework can leverage a "User Profile" (e.g., knowing the user is in Palo Alto) to automatically inject filters into the query, making the results immediately relevant.

Key Results

The authors compared QUASL against WolframAlpha, a leading computational knowledge engine.

  • Raw Performance: On the original 50 questions, QUASL significantly outperformed WolframAlpha. This is largely because QUASL was tailored to the specific census schema, whereas WolframAlpha is a general-purpose tool.
  • Success@k: In terms of "Success@1" (the correct query being the top result), QUASL achieved high accuracy. When questions were slightly "updated" to use terms more familiar to the systems (e.g., changing "rich people" to "highest household income"), both systems performed comparably well, with QUASL maintaining a slight edge in the top 4 results.

Limitations and Future Work

While powerful, the authors note several areas for improvement:

  • Linguistic Limitations: The current system relies heavily on exact or near-exact matches. It lacks a deep linguistic component (like WordNet) to handle synonyms (e.g., knowing "dwellings" refers to "home ownership").
  • Manual Setup: Currently, the mapping between natural language patterns and the schema requires manual configuration by an administrator.
  • Future Directions: The authors suggest using machine learning to improve the scoring of results based on user feedback and automating the setup process by generating patterns from existing query logs.

Summary updated

Abstract

Question Answering (Q&A) from structured data is a technique that may revolutionize enterprise search. A very promising use-case for such technology is Business Intelligence (BI). In order to make BI more accessible to end-users, some efforts have been made in the field of search for existing reports. However, the problem of converting an end-user’s natural language input to a valid structured query in an ad-hoc fashion hasn’t been sufficiently solved yet. In this paper we present a framework for Q&A systems that operate on structured data. The main innovation is that the framework allows defining a mapping between recognized semantics of a user’s questions to a structured query model that can be executed on arbitrary data sources. It bases on popular standards like RDF and SparQL and is therefore very easy to adapt to other domains or usecases. We will describe the application of this framework at hand of a BI question answering use-case, which also includes the personalization of generated queries, demonstrating the realworld applicability of our approach. In our experiments, we demonstrate that with our approach one can easily achieve a similar answering quality as one of the most popular Q&A systems on the Web. I.
BibTeX
@inproceedings{kuchmannbeauger2013quasl,
  author = {Nicolas Kuchmann-Beauger and Falk Brauer and Albert Bifet},
  title = {{QUASL: A Framework for Question Answering and its Application to Business Intelligence}},
  booktitle = {SPIRE},
  year = {2013},
}
Figure