Automatic Question Generation in Multimedia-Based Learning

Yvonne Skalban, Le An Ha, Lucia Specia, Ruslan MitkovCOLING 2012

TL;DR

This paper explores how automatically generated questions can enhance multimedia learning. They found that pre-questions, especially those paired with images, significantly improve test-taker performance on comprehension tests, and their own question generation system performs comparably to a state-of-the-art system.

What Problem Does This Paper Address?

Creating effective learning materials, particularly assessment questions, is a time-consuming and expertise-intensive process for educators. This paper addresses the challenge of automating question generation to assist in creating these materials, specifically for multimedia-based learning. The authors investigate whether automatically generated questions can be used as "pre-questions" – questions presented to learners before they engage with the learning material – to improve comprehension. They also aim to evaluate the quality of these automatically generated questions by comparing their psychometric properties to manually created ones.

Methods

The core of this research involves developing and evaluating an automatic Question Generation (QG) system. The authors' system, "System A," utilizes existing Natural Language Processing (NLP) tools (like GATE) for pre-processing documentary video subtitles. It then employs a rule-based approach to generate factual "wh-questions" (who, what, where, when, how much) from declarative sentences. A key feature of their system is its ability to link generated questions to specific time-stamps in the video, allowing for the extraction of relevant screenshots to accompany the questions.

They compare their system to a state-of-the-art QG system ("System B," developed by Heilman) and manually generated questions. The experiment involved dividing participants into three groups, each receiving different types of pre-questions (text-only, text with screenshots, or no pre-questions) before watching a documentary. After each section of the documentary, participants answered comprehension questions (post-questions). The study then analyzed:

  • Accuracy: How accurately participants answered post-questions based on the type of pre-questions they received.
  • Time: The time taken by participants to answer post-questions.
  • Psychometric Parameters: Specifically, the item discriminating power (DP) of the generated questions.

Datasets

The study used a documentary on "nuclear fusion" as the multimedia learning material. The subtitles of this documentary served as the primary input for the QG systems. No new datasets were introduced; rather, existing QG systems were applied to a specific domain (documentary videos) for evaluation.

Key Insights and Ideas

  • Pre-questions are beneficial: The presence of pre-questions, regardless of whether they are text-only or include images, significantly improves test-takers' performance on comprehension tests compared to having no pre-questions.
  • Images enhance pre-questions: Pre-questions accompanied by screenshots from the video led to a more significant improvement in accuracy than text-only pre-questions. This highlights the value of visual context in pre-questioning.
  • Automatic QG is competitive: The authors' QG system (System A) generated questions whose psychometric parameters (specifically, discriminating power) were comparable to, and in some aspects better than, those of a state-of-the-art system (System B).
  • Manual questions still lead: While automatic systems are competitive, manually generated questions still exhibited the highest discriminating power, suggesting that human expertise still holds an edge in creating questions that effectively differentiate between high and low performers.
  • Time is not significantly affected: The presence of pre-questions did not significantly impact the time test-takers took to answer the post-questions.

Key Results

  • Participants receiving pre-questions with screenshots answered 58.33% correctly, compared to 50.29% for text-based pre-questions and 39.83% for no pre-questions.
  • The difference in accuracy between pre-questions with screenshots and no pre-questions was statistically significant (p=0.00085).
  • The average time to answer post-questions was similar across all conditions, with no statistically significant differences.
  • Manually generated questions had the highest average discriminating power (0.37), followed by System A (0.16), and then System B (0.07).
  • There was a statistically significant difference in DP between manual questions and System A (p=0.0434), but no significant difference between System A and System B (p=0.356988).

Limitations and Future Work

The authors acknowledge that while their system's questions are comparable to the state-of-the-art, manually generated questions still perform better in terms of discriminating power. They suggest several avenues for future research:

  • Automating pre-question selection: Developing methods to automatically or semi-automatically select suitable pre-questions from a larger pool.
  • Exploring other image sources: Investigating the use of images from sources other than direct video screenshots, such as Google Image Search, for pre-questions.
  • Large-scale productivity study: Planning a larger experiment to compare the time taken to post-edit automatically generated questions versus generating questions from scratch.

Summary updated

BibTeX
@inproceedings{skalban2012automatic,
  author = {Yvonne Skalban and Le An Ha and Lucia Specia and Ruslan Mitkov},
  title = {{Automatic Question Generation in Multimedia-Based Learning}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2012},
}
Figure