TL;DR
Researchers from IIIT Delhi have developed QDup, an unsupervised tool designed to automatically detect duplicate and related questions in massive e-learning repositories. By combining traditional statistical methods with modern neural NLP techniques in a multi-stage pipeline, QDup identifies near-duplicates with 81.5% accuracy without requiring any labeled training data—a major win for platforms struggling to manage rapidly growing question banks.
What Problem Does This Paper Address?
Online learning platforms rely on vast databases of questions to assess students. As these platforms grow, they constantly "onboard" new questions from external sources or manual entries. This creates a massive deduplication headache: academicians cannot manually check thousands of new entries against a million existing ones.
The challenge is that "similarity" in questions is nuanced. Two questions might share 90% of their words but ask completely different things (e.g., "What is GDP?" vs. "What is the significance of GDP?"). Conversely, they might use different terminology for the same concept. Existing supervised models require expensive labeled datasets, while simple "bag-of-words" approaches fail to capture these semantic subtleties.
Methods
The authors propose an unsupervised hybrid pipeline that uses an "iterative elimination" strategy. Instead of one complex model, QDup passes questions through several specialized filters:
- Preprocessing & Taxonomy Tagging: Questions are normalized (e.g., converting "Cl" to "chlorine") and tagged using TagRec, a hierarchical model that categorizes questions by subject, chapter, and topic. This immediately narrows the search space to relevant subjects.
- Token-Level Comparison: A Jaccard similarity filter removes questions that have very low word overlap (threshold < 0.4).
- Named Entity Recognition (NER): Using the spaCy library, the tool ensures that questions involve the same entities. If one question asks about "Google" and another about "Apple," they are discarded as duplicates even if the rest of the sentence structure is identical.
- Keyphrase Extraction: The EmbedRank algorithm extracts core concepts. The tool calculates the overlap of these keyphrases to ensure the "heart" of the question is the same.
- Negation Detection: This is a critical step to avoid "false positives." It checks for words like "not" to distinguish between questions like "Which of these is a metal?" and "Which of these is not a metal?"
- Related Question Retrieval: For questions that aren't exact duplicates but are topically similar, the tool uses Sentence Transformers and ScANN (Scalable Nearest Neighbors) to provide recommendations for "related practice."
Datasets
The study utilized a dataset of 114,804 secondary high school questions curated from the CBSE (India) curriculum. The dataset spans subjects like Physics, Chemistry, Biology, and Social Science. While the paper doesn't introduce a new public benchmark, it demonstrates the tool's efficacy on a large-scale, real-world educational repository.
Key Insights and Ideas
- The Power of "No Labels": The most significant insight is that a well-constructed pipeline of unsupervised tools (NER + Keyphrase + Negation) can rival or outperform complex supervised models in niche domains like education.
- Negation is the "Silent Killer" of Accuracy: Most similarity models struggle with negation. By explicitly adding a negation detection stage, the authors significantly reduced false positives where opposite questions were flagged as duplicates.
- Efficiency via Taxonomy: By first indexing questions into a hierarchical taxonomy (Subject -> Chapter -> Topic), the system avoids the "all-vs-all" comparison problem, making it fast enough for real-time bulk onboarding.
Key Results
- Accuracy: QDup achieved 81.5% accuracy in detecting duplicates. This significantly outperformed a keyphrase-only approach (76.5%) and a standard vector-based "closest neighbors" approach (51.5%).
- User Satisfaction: A study with 14 academicians rated the tool on a scale of 1–3. It scored high across the board: 2.46 for intuitiveness, 2.78 for responsiveness, and 2.68 for relevance.
- Speed: By leveraging ScANN for vector searches, the tool maintains high performance even when searching through hundreds of thousands of embeddings.
Limitations and Future Work
The authors acknowledge that while the current pipeline is highly effective, it still relies on multiple independent stages. They suggest that future work could involve using the data collected by QDup to train a single-stage bi-encoder transformer model using contrastive learning. This would potentially simplify the architecture while maintaining the "unsupervised" benefit by using the pipeline's current outputs as "weakly supervised" labels. They also plan to explore knowledge distillation to make the model even lighter for deployment on resource-constrained environments.
Summary updated