Abstract
Online learning platforms provide diverse questions to gauge the learners' understanding of different concepts. The repository of questions has to be constantly updated to ensure a diverse pool of questions to conduct assessments for learners. However, it is impossible for the academician to manually skim through the large repository of questions to check for duplicates when onboarding new questions from external sources. Hence, we propose a tool QDup in this paper that can surface near-duplicate and semantically related questions without any supervised data. The proposed tool follows an unsupervised hybrid pipeline of statistical and neural approaches for incorporating different nuances in similarity for the task of question duplicate detection. We demonstrate that QDup can detect near-duplicate questions and also suggest related questions for practice with remarkable accuracy and speed from a large repository of questions.
Introduction
As online learning platforms scale, the volume of educational content—specifically assessment questions—grows exponentially. To maintain high-quality assessments, these platforms frequently "onboard" new questions from external sources or manual entries. This growth introduces a significant administrative challenge: deduplication. Manually checking thousands of new entries against a million-item repository is labor-intensive and error-prone for academicians.
The core difficulty lies in the nuance of educational content. Two questions may share high lexical overlap but ask fundamentally different things (e.g., "What is GDP?" vs. "What is the significance of GDP?"). Conversely, questions might be semantically identical while using different terminology. While supervised deep learning models can address these issues, they require massive labeled datasets which are rarely available for niche educational domains. To solve this, we introduce QDup, an unsupervised tool designed to detect duplicates and related questions with high precision and efficiency.
The QDup Framework
QDup utilizes an "iterative elimination" strategy through an unsupervised hybrid pipeline. Rather than relying on a single complex model, the system passes candidate questions through several specialized filters to refine results.
1. Preprocessing and Taxonomy Tagging
Questions are first normalized (e.g., expanding chemical symbols like "Cl" to "chlorine"). To avoid the computational cost of an "all-vs-all" comparison, QDup uses TagRec, a hierarchical model that categorizes questions by subject, chapter, and topic. This narrows the search space significantly, ensuring that a biology question is only compared against other biology questions.
2. Token-Level and Entity Filtering
After taxonomy tagging, the system applies a Jaccard similarity filter to remove pairs with very low word overlap (threshold < 0.4). To further refine the search, Named Entity Recognition (NER) via the spaCy library is employed. This ensures that questions involve the same entities; for instance, if one question mentions "Google" and another mentions "Apple," they are discarded as duplicates even if their sentence structures are identical.
3. Keyphrase Extraction and Negation Detection
To capture the "heart" of a question, QDup uses the EmbedRank algorithm for keyphrase extraction. The tool calculates the overlap of these keyphrases to ensure semantic alignment. Crucially, the pipeline includes a Negation Detection stage. This distinguishes between questions like "Which of these is a metal?" and "Which of these is not a metal?"—a common pitfall for standard similarity models that often leads to false positives.
4. Related Question Retrieval
For questions that are not exact duplicates but are topically relevant, QDup provides recommendations for "related practice." This is achieved using Sentence Transformers and ScANN (Scalable Nearest Neighbors), allowing for high-speed vector searches across large repositories.
Experimental Results
Dataset
The system was evaluated using a dataset of 114,804 secondary high school questions curated from the CBSE (India) curriculum, covering subjects such as Physics, Chemistry, Biology, and Social Science.
Performance Metrics
QDup achieved an accuracy of 81.5% in detecting duplicates, significantly outperforming baseline approaches:
- Keyphrase-only approach: 76.5% accuracy
- Standard vector-based (closest neighbors) approach: 51.5% accuracy
By explicitly addressing negation and entity matching, the hybrid pipeline successfully mitigated the high false-positive rates seen in purely vector-based models.
User Evaluation
A study involving 14 academicians assessed the tool on a scale of 1 to 3. The results indicated high practical utility:
- Intuitiveness: 2.46 / 3.0
- Responsiveness: 2.78 / 3.0
- Relevance: 2.68 / 3.0
Conclusion and Future Work
QDup demonstrates that a well-constructed pipeline of unsupervised tools can rival supervised models in specialized domains. The integration of negation detection and hierarchical taxonomy tagging provides the necessary precision and speed for real-time bulk onboarding in e-learning environments.
Future research will explore using the outputs of the QDup pipeline as "weakly supervised" labels to train a single-stage bi-encoder transformer model via contrastive learning. Additionally, the team plans to investigate knowledge distillation techniques to create lighter models suitable for resource-constrained deployment environments.
Summary updated