LiCQA : A Lightweight Complex Question Answering System

Sourav Saha, Dwaipayan Roy, Mandar MitraarXiv 2026 · arXiv

TL;DR

LiCQA is a lightweight, unsupervised question-answering (QA) system designed to tackle complex queries by synthesizing evidence across multiple documents. By discarding expensive end-to-end deep learning architectures in favor of a clean, modular pipeline, LiCQA achieves an approximate 8x speedup over state-of-the-art baselines while significantly improving answer accuracy.

What Problem Does This Paper Address?

Most modern open-domain QA systems excel at "simple" factoid questions where the answer is explicitly stated within a single document (e.g., "Who won the Turing Award in 1970?"). However, real-world queries are often "complex," involving multiple entities and relationships that require aggregating information from various sources (e.g., "Which Nolan films won an Oscar, but missed a Golden Globe?").

Current approaches to complex QA generally fall into two camps:

  1. Knowledge Graph (KG) systems, which suffer from the inherent incompleteness of structured databases.
  2. Supervised Deep Learning models (like dense transformers), which require massive volumes of human-annotated training data and heavy GPU infrastructure for deployment.

The authors introduce LiCQA to bridge this gap: an unsupervised, corpus-evidence-based system that handles complex questions efficiently without requiring end-to-end neural training or massive computational budgets.

Methods

LiCQA processes questions using a streamlined, four-stage pipeline that runs locally and efficiently:

Figure 1: Architecture of the LiCQA system.
  1. Question Type Classification: The system determines the expected answer type (e.g., PERSON, LOCATION, NUMERIC). The authors experimented with both a traditional linear Support Vector Machine (SVM) using syntactic features and a neural classifier based on the Universal Sentence Encoder (USE).
  2. Answer Extraction and Filtering: Concurrently, a standard Information Retrieval (IR) engine fetches relevant documents. LiCQA extracts Named Entities (NEs) from these texts using the Flair NLP framework and filters out any entities that do not match the predicted question type using a handcrafted mapping table to OntoNotes 5 tags.
  3. Answer Scoring: To evaluate candidate entities, LiCQA computes the semantic similarity between the user's question and the sentences where the candidate entity appears. Sentence embeddings are generated using either InferSent or Sentence-BERT. The authors found that using the single best-matching sentence (max-score) yielded the best results.
  4. Answer Ranking: Finally, the semantic similarity score is multiplied by the entity's normalized document frequency (comb-score*) to generate a final ranked list of the top 5 answers.

Datasets

The system was evaluated using two complex question datasets, each containing 150 multi-entity queries:

  • WikiAnswers (CQ-W)
  • Google Trends (CQ-T)

To ensure robust evaluation and simulate variations in search engine performance, the authors tested the pipeline across six distinct document collections per query: Top10 (the actual top 10 results returned by Google) and Strata-1 through Strata-5 (constructed using stratified sampling from the top 50 Google results).

Key Insights and Ideas

  • Shallow ML Can Beat Deep Learning: Surprisingly, the traditional SVM-based question classifier slightly outperformed the neural USE-based classifier, proving that heavy deep-learning architectures aren't always necessary for specialized sub-tasks.
  • The Power of the "Best Match": When scoring an entity's relevance, looking only at the single most semantically relevant sentence (max-score) performed significantly better than averaging across all sentences containing that entity. This prevents highly relevant context from being diluted by mundane mentions elsewhere in the text.
  • Exposing Baseline Flaws via Tie-Aware Metrics: Traditional evaluation metrics like Mean Reciprocal Rank (MRR) and Precision@1 (P@1) can artificially inflate the performance of systems that output many tied answers. By implementing tie-aware metrics (tMRR, tP@1), the authors revealed that prior graph-based systems like QUEST frequently output large blocks of tied, incorrect answers, whereas LiCQA maintains high precision.

Key Results

  • Accuracy Improvements: On the CQ-W dataset using the Top10 collection, LiCQA achieved a tie-aware MRR (tMRR) of 0.412, vastly outperforming the state-of-the-art unsupervised baseline, QUEST (0.067), and the strong supervised baseline, DrQA (0.226).
  • Consistency: LiCQA consistently retrieved at least one correct answer within its top 5 choices (tHit@5) for over 63% of the complex questions across all datasets.
  • 8x Latency Reduction: Because LiCQA avoids building complex, on-the-fly knowledge graphs, it achieves a massive computational speedup. It processes queries in roughly 13 to 18 seconds, compared to the 120 to 195 seconds required by DrQA and QUEST.

Limitations and Future Work

While LiCQA is highly efficient, its modular nature means it is susceptible to upstream errors. For instance, if the initial question classifier misidentifies the target answer type, or if the Named Entity Recognition (NER) module fails to extract the correct entity from the text, the pipeline cannot recover the correct answer.

In the future, the authors plan to explore low-dimensional vector representations for complex concepts to make the answer extraction module even more robust against variations in natural language phrasing.

Summary updated

Abstract

Over the last twenty years, significant progress has been made in designing and implementing Question Answering (QA) systems. However, addressing complex questions, the answers to which are spread across multiple documents, remains a challenging problem. Recent QA systems that are designed to handle complex questions work either on the basis of knowledge graphs, or utilise contem- porary neural models that are expensive to train, in terms of both computational resources and the volume of training data required. In this paper, we present LiCQA, an unsupervised question answer- ing model that works primarily on the basis of corpus evidence. We empirically compare the effectiveness and efficiency of LiCQA with two recently presented QA systems, which are based on different underlying principles. The results of our experiments show that LiCQA significantly outperforms these two state-of-the-art systems on benchmark data with noteworthy reduction in latency.
BibTeX
@misc{saha2026licqa,
  author = {Sourav Saha and Dwaipayan Roy and Mandar Mitra},
  title = {{LiCQA : A Lightweight Complex Question Answering System}},
  year = {2026},
  eprint = {2602.22182},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2602.22182},
}
Figure