Unsupervised Question Duplicate and Related Questions Detection in e-learning platforms

Maksimjeet Chowdhary, Sanyam Goyal, Venktesh V, Mukesh K. Mohania, Vikram Goyal

TL;DR

Researchers from IIIT Delhi have developed QDup, an unsupervised tool designed to automatically detect duplicate and related questions in massive e-learning repositories. By combining traditional statistical methods with modern neural NLP techniques in a multi-stage pipeline, QDup identifies near-duplicates with 81.5% accuracy without requiring any labeled training data—a major win for platforms struggling to manage rapidly growing question banks.

What Problem Does This Paper Address?

Online learning platforms rely on vast databases of questions to assess students. As these platforms grow, they constantly "onboard" new questions from external sources or manual entries. This creates a massive deduplication headache: academicians cannot manually check thousands of new entries against a million existing ones.

The challenge is that "similarity" in questions is nuanced. Two questions might share 90% of their words but ask completely different things (e.g., "What is GDP?" vs. "What is the significance of GDP?"). Conversely, they might use different terminology for the same concept. Existing supervised models require expensive labeled datasets, while simple "bag-of-words" approaches fail to capture these semantic subtleties.

Methods

The authors propose an unsupervised hybrid pipeline that uses an "iterative elimination" strategy. Instead of one complex model, QDup passes questions through several specialized filters:

  1. Preprocessing & Taxonomy Tagging: Questions are normalized (e.g., converting "Cl" to "chlorine") and tagged using TagRec, a hierarchical model that categorizes questions by subject, chapter, and topic. This immediately narrows the search space to relevant subjects.
  2. Token-Level Comparison: A Jaccard similarity filter removes questions that have very low word overlap (threshold < 0.4).
  3. Named Entity Recognition (NER): Using the spaCy library, the tool ensures that questions involve the same entities. If one question asks about "Google" and another about "Apple," they are discarded as duplicates even if the rest of the sentence structure is identical.
  4. Keyphrase Extraction: The EmbedRank algorithm extracts core concepts. The tool calculates the overlap of these keyphrases to ensure the "heart" of the question is the same.
  5. Negation Detection: This is a critical step to avoid "false positives." It checks for words like "not" to distinguish between questions like "Which of these is a metal?" and "Which of these is not a metal?"
  6. Related Question Retrieval: For questions that aren't exact duplicates but are topically similar, the tool uses Sentence Transformers and ScANN (Scalable Nearest Neighbors) to provide recommendations for "related practice."

Datasets

The study utilized a dataset of 114,804 secondary high school questions curated from the CBSE (India) curriculum. The dataset spans subjects like Physics, Chemistry, Biology, and Social Science. While the paper doesn't introduce a new public benchmark, it demonstrates the tool's efficacy on a large-scale, real-world educational repository.

Key Insights and Ideas

  • The Power of "No Labels": The most significant insight is that a well-constructed pipeline of unsupervised tools (NER + Keyphrase + Negation) can rival or outperform complex supervised models in niche domains like education.
  • Negation is the "Silent Killer" of Accuracy: Most similarity models struggle with negation. By explicitly adding a negation detection stage, the authors significantly reduced false positives where opposite questions were flagged as duplicates.
  • Efficiency via Taxonomy: By first indexing questions into a hierarchical taxonomy (Subject -> Chapter -> Topic), the system avoids the "all-vs-all" comparison problem, making it fast enough for real-time bulk onboarding.

Key Results

  • Accuracy: QDup achieved 81.5% accuracy in detecting duplicates. This significantly outperformed a keyphrase-only approach (76.5%) and a standard vector-based "closest neighbors" approach (51.5%).
  • User Satisfaction: A study with 14 academicians rated the tool on a scale of 1–3. It scored high across the board: 2.46 for intuitiveness, 2.78 for responsiveness, and 2.68 for relevance.
  • Speed: By leveraging ScANN for vector searches, the tool maintains high performance even when searching through hundreds of thousands of embeddings.

Limitations and Future Work

The authors acknowledge that while the current pipeline is highly effective, it still relies on multiple independent stages. They suggest that future work could involve using the data collected by QDup to train a single-stage bi-encoder transformer model using contrastive learning. This would potentially simplify the architecture while maintaining the "unsupervised" benefit by using the pipeline's current outputs as "weakly supervised" labels. They also plan to explore knowledge distillation to make the model even lighter for deployment on resource-constrained environments.

Summary updated

Abstract

Online learning platforms provide diverse questions to gauge the learners' understanding of different concepts. The repository of questions has to be constantly updated to ensure a diverse pool of questions to conduct assessments for learners. However, it is impossible for the academician to manually skim through the large repository of questions to check for duplicates when onboarding new questions from external sources. Hence, we propose a tool QDup in this paper that can surface near-duplicate and semantically related questions without any supervised data. The proposed tool follows an unsupervised hybrid pipeline of statistical and neural approaches for incorporating different nuances in similarity for the task of question duplicate detection. We demonstrate that QDup can detect near-duplicate questions and also suggest related questions for practice with remarkable accuracy and speed from a large repository of questions. The demo video of the tool can be found at https://www.youtube.com/watch?v=loh0_-7XLW4.
BibTeX
@inproceedings{chowdhary2023unsupervised,
  author = {Maksimjeet Chowdhary and Sanyam Goyal and Venktesh V and Mukesh K. Mohania and Vikram Goyal},
  title = {{Unsupervised Question Duplicate and Related Questions Detection in e-learning platforms}},
  booktitle = {WSDM},
  year = {2023},
  url = {https://openreview.net/forum?id=j3T2xYm06l},
}

Loading related content...

Figure