MineObserver 2.0: A Deep Learning & In-Game Framework for Assessing Natural Language Descriptions of Minecraft Imagery

Jay Mahajan, Samuel Hum, Jack Henhapl, Diya Yunus, Matthew Gadbury, Emi Brown, Jeffrey Ginger, H. Chad LaneAAAI 2024 · DOI

Abstract

MineObserver 2.0 is an AI-driven framework that uses computer vision and natural language processing to assess and provide real-time feedback on students' scientific observations within Minecraft. By comparing student descriptions to AI-generated "expert" captions using RoBERTa-based semantic similarity, the system significantly improves the quality of learner observations and provides teachers with a real-time monitoring dashboard.

Introduction

In science education, the ability to observe and articulate natural phenomena is a foundational skill. The WHIMC (What-If Hypothetical Implementations in Minecraft) project uses the popular "sandbox" game to let students explore scientifically grounded "what-if" scenarios, such as "What if Earth had no moon?".

However, in a classroom or summer camp setting, a single instructor cannot manually review every student's written observations in real-time. Previous attempts at automated grading (MineObserver 1.0) acted as "outside graders" rather than interactive participants and lacked the depth needed to actually improve a student's observational skills. MineObserver 2.0 addresses this by integrating an AI feedback loop directly into the game.

Methodology

The framework consists of three main components: the Photographer, the AI Architecture, and the Visualizer. This integrated approach allows for a seamless transition between student exploration and automated assessment.

System Architecture Overview

The Photographer

To capture exactly what a student is looking at, the system uses an invisible Minecraft client (a "spectator" bot). When a student makes an observation, the bot teleports to their coordinates and takes a screenshot of their Point of View (POV).

AI Architecture and Feedback Loop

The core of the MineObserver 2.0 system is its dual-model evaluation pipeline, which translates visual data into semantic comparisons.

AI Feedback Loop and Semantic Analysis
  • Image Captioning Model: A hybrid CNN-LSTM model (using ResNet and visual attention) acts as an "expert observer," generating a description of the screenshot.
  • RoBERTa Model: The system uses a RoBERTa model fine-tuned on Semantic Textual Similarity (STSb) and Natural Language Inference (NLI) datasets. It calculates the cosine similarity between the student’s observation and the AI-generated "expert" caption.
  • Feedback System: If the similarity score meets a threshold ($\gamma = 0.5$), the student receives positive reinforcement highlighting specific keywords. If it falls below, the system prompts the student to look for those specific keywords and try again.

The Visualizer

A web-based dashboard for teachers displays student observations, AI captions, and feedback in real-time. This "teacher-in-the-loop" component allows for immediate pedagogical intervention, helping instructors identify students who are off-task or struggling with complex scientific concepts.

Datasets

Because no standard dataset exists for Minecraft image-caption pairs, the authors created a custom dataset of over 2,000 unique datapoints. Screenshots from the WHIMC server were paired with descriptions sourced via Amazon Mechanical Turk, manual labeling, and contributions from children (ages 8-14) to ensure the model understood the linguistic style of the target demographic.

Key Results

The authors conducted a double-blind experiment with 73 participants (ages 8-14) across several summer camps, comparing MineObserver 2.0 to its predecessor.

  • Perceived Accuracy: Students rated the AI's generated observations as significantly more accurate than the previous version (Welch t-test, $p = 0.03196$).
  • Feedback Usefulness: The keyword-based feedback was rated significantly more useful than the generic feedback of version 1.0 ($p = 0.001719$).
  • Standard Metrics: The new model outperformed the old one across standard NLP metrics, including BLEU-1 (0.2 vs 0.197) and METEOR (0.146 vs 0.121).
  • Latency: The median rating for system speed was 3/5, with feedback arriving roughly 20 seconds after an observation was made.

Limitations and Future Work

While MineObserver 2.0 represents a significant step forward, several areas for improvement remain. The 20-second delay is adequate but could be improved by scaling the number of "Photographer" clients. Future iterations aim to use Large Language Models (LLMs) to provide deeper, probing feedback. Additionally, the authors plan to implement continuous learning, allowing the model to retrain itself on novel observations made by learners, effectively growing alongside the students.

Summary updated

BibTeX
@inproceedings{mahajan2024mineobserver,
  author = {Jay Mahajan and Samuel Hum and Jack Henhapl and Diya Yunus and Matthew Gadbury and Emi Brown and Jeffrey Ginger and H. Chad Lane},
  title = {{MineObserver 2.0: A Deep Learning \& In-Game Framework for Assessing Natural Language Descriptions of Minecraft Imagery}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i21.30367},
}
Figure