EdTec-QBuilder: A Semantic Retrieval Tool for Assembling Vocational Training Exams in German Language

Alonso Palomino, Andreas Fischer, Jakub Kuzilek, Jarek Nitsch, Niels Pinkwart, Benjamin PaassenNAACL 2024 · DOI

TL;DR

Assembling high-quality vocational training exams is a time-consuming challenge for educators. This paper introduces EdTec-QBuilder, a semantic search tool that leverages advanced language models to help vocational trainers efficiently find and select relevant test items from a large database. The system demonstrates that using cross-encoders for re-ranking initial search results significantly improves the relevance of retrieved items, making exam assembly more manageable.

What Problem Does This Paper Address?

Creating comprehensive and psychometrically valid exams is crucial for assessing student competence. While generating individual test items has seen considerable research, the process of assembling these items into a complete exam form from a database of existing, validated questions is less explored. This task is complex because exams need to cover a broad range of skills, vary in difficulty, and align with curriculum standards. Vocational educators, in particular, face this challenge, often spending significant time manually sifting through item banks. EdTec-QBuilder aims to alleviate this burden by providing a semantic search tool that understands the intent behind a query and retrieves the most relevant items, while also serving as a platform to collect valuable data for future research into assisted and automated exam assembly.

Methods

The core of EdTec-QBuilder's functionality lies in its semantic search capabilities. The authors explored eight different search and retrieval strategies, evaluating them in combination with 25 pre-trained sentence similarity models. The goal was to find the optimal approach for matching user queries (e.g., a topic description) to relevant test items.

Key strategies included:

  • BM25: A traditional probabilistic baseline.
  • LM+ANN: Using pre-trained language models (LMs) to generate embeddings and Approximate Nearest Neighbor (ANN) search for efficiency.
  • TF-IDF Weighted Average: Combining TF-IDF scores with LM embeddings.
  • Query-term expansion: Using synonyms to broaden search results.
  • Vertical Search: Limiting the search to the most semantically similar topics.
  • MILP and LambdaMART: Re-ranking initial candidate lists using optimization techniques.
  • Cross-encoder: A powerful approach where the query and candidate item are processed together by a transformer model to directly score their relevance.

The authors found that a combination of initial retrieval using LM+ANN and subsequent re-ranking with a cross-encoder yielded the best results.

Datasets

The primary dataset used in this work is a collection of 2,812 test items from the bfz group, a German vocational training provider, covering 34 vocational topics. To augment this dataset and create a more robust evaluation set, an additional 2,812 items were generated using ChatGPT 3.5, followed by manual inspection and cleaning.

For evaluation, a synthetic TREC-style dataset was constructed. This involved generating 15 syntactically diverse queries for each of the 34 topics. Relevance judgments were then assigned based on the average similarity scores from a set of language models, with scores above 0.75 considered "highly relevant" and scores between 0.60 and 0.75 considered "moderately relevant."

Key Insights and Ideas

  • Exam assembly as Information Retrieval: Framing the complex task of exam assembly as an information retrieval problem simplifies its conceptualization and allows for the application of established search techniques.
  • Cross-encoders reign supreme for re-ranking: The most significant finding is that cross-encoder models, which process query-item pairs jointly, dramatically outperform other strategies for re-ranking an initial set of candidate items. This suggests that a two-stage approach (initial broad retrieval followed by deep contextual re-ranking) is highly effective.
  • Semantic understanding is key: Relying solely on keyword matching (like BM25) is insufficient. Semantic similarity, captured by modern language models, is crucial for understanding the nuances of vocational topics and user queries.
  • Crowdsourcing for future research: EdTec-QBuilder is designed not just as a tool but also as a data collection platform. By logging user interactions and feedback, it aims to build a valuable dataset for training future assisted and automated exam assembly models, addressing the lack of such data in prior research.

Key Results

The evaluation revealed that the cross-encoder re-ranking strategy achieved the highest performance, with an nDCG@100 score of 0.516 and an MRR@100 score of 0.713. This indicates that, on average, about half of the top 100 retrieved items were relevant, and the highest-ranked relevant item appeared very early in the list (rank 1 or 2). This was significantly better than baseline methods like BM25. The system is now in use at bfz, demonstrating its practical utility.

Limitations and Future Work

The authors acknowledge several limitations:

  • Synthetic Relevance Judgments: The reliance on a synthetic dataset for relevance judgments, derived from language models themselves, could introduce bias. Future work should incorporate human judgments to create a more robust ground truth.
  • Cold Start Problem: To facilitate rapid deployment with their industry partner, the initial evaluation was based on synthetic data. Future work should focus on collecting real user data to overcome the "cold start" issue.
  • Popularity Bias: There's a risk of popularity bias, where frequently selected items become even more popular, potentially overshadowing other relevant items. The current synthetic data generation helps mitigate this, but it's an area for future investigation.

Promising future directions include:

  • Leveraging Crowdsourced Data: Further developing models using the data collected from user interactions on EdTec-QBuilder.
  • Estimating Item Difficulty: Developing models to predict test item difficulty without relying solely on respondent data.
  • Incorporating Large Language Models (LLMs): Exploring how LLMs can further enhance the exam assembly process.

Summary updated

BibTeX
@inproceedings{palomino2024edtec,
  author = {Alonso Palomino and Andreas Fischer and Jakub Kuzilek and Jarek Nitsch and Niels Pinkwart and Benjamin Paassen},
  title = {{EdTec-QBuilder: A Semantic Retrieval Tool for Assembling Vocational Training Exams in German Language}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.naacl-demo.3},
}
Figure