Rebo Junior: Analysis of Dialogue Structure Quality for a Reflection Guidance Chatbot

Irmtraud Wolfbauer, Viktoria Pammer-Schindler, Carolyn P. RoséEc-tel (impact Papers) 2020

Introduction

Reflection—the systematic review of past experiences to improve future performance—is a cornerstone of professional learning. However, one-on-one reflection with a human instructor is expensive and difficult to scale, especially in vocational training or remote learning environments. Researchers developed Rebo Junior, a chatbot designed to guide apprentices through structured reflection on their daily learning tasks. The study found that even a non-adaptive, static dialogue structure can successfully lead students to deep levels of reflection and maintain high user engagement over a three-month period.

Methodology: Designing for Depth

The researchers designed Rebo Junior based on established pedagogical models, specifically those by Boud et al. and Fleck & Fitzpatrick. The chatbot follows a hierarchical dialogue structure that moves the user through six distinct levels of reflection, ranging from simple description to future planning.

Table 1: Rebo Junior's reflective questions mapped to pedagogical levels of reflection.

The study was conducted in two phases with 17 apprentices in electrical engineering and mechatronics:

  • Phase 1 (4 weeks): Tightly spaced, static interactions where the questions remained identical across sessions.
  • Phase 2 (8 weeks): More widely spaced interactions using "question pools" to randomly vary the phrasing of the prompts, intended to prevent boredom.

Three metrics were manually coded by researchers to evaluate the 153 valid reflective conversations: Coherence (logical flow), Reflectivity (evidence of learning/change), and Engagement (user cooperation).

Key Results

The analysis revealed that Rebo Junior was remarkably effective at facilitating high-quality reflection:

  • High Coherence: 97% of the conversations were rated as coherent, indicating that the sequence of questions and answers remained logical throughout the interaction.
  • Deep Reflection: 87% of all interactions reached the highest level of reflection ("Learning or Change"), where apprentices articulated a specific planned behavior change for future tasks.
  • Sustained Engagement: In Phase 1, 95% of conversations showed high engagement. While this dropped to 72% in Phase 2, the overall results exceeded expectations for a long-term deployment.
  • User Acceptance: 94% of apprentices reported a positive experience, with some noting it felt like a "real talk" and was preferable to standard text boxes or paper forms.

Discussion: Structure vs. Adaptivity

A primary insight from the study is that structure trumps adaptivity. Surprisingly, the chatbot was highly effective despite having zero adaptivity—it did not change its questions based on the content of the user's responses. This suggests that a well-designed pedagogical structure is more critical than "smart" AI for reflection guidance.

Figure 1: The visual design of Rebo Junior was intentionally robotic to avoid the "uncanny valley."

Furthermore, the "familiarity" of the static structure in Phase 1 appeared to be an advantage rather than a drawback. Engagement and reflectivity actually dropped in Phase 2 when randomized questions were introduced. The researchers speculate that the alternative questions might have been less precisely worded, or that the unpredictability disrupted the reflective flow. The bot’s visual design also played a role; by choosing a "cool," robotic appearance, the designers avoided the "uncanny valley" effect, making the tool more appealing for technical apprentices.

Conclusion and Future Work

Rebo Junior demonstrates that a relatively simple, structured chatbot can effectively facilitate deep professional reflection. While the current study focused on structural quality, future work will aim to develop automated "Reflection Analytics" to assess the depth of conversations in real-time and investigate the technical correctness of the insights gained by apprentices.

Summary updated

BibTeX
@inproceedings{wolfbauer2020rebo,
  author = {Irmtraud Wolfbauer and Viktoria Pammer-Schindler and Carolyn P. Rosé},
  title = {{Rebo Junior: Analysis of Dialogue Structure Quality for a Reflection Guidance Chatbot}},
  booktitle = {Ec-tel (impact Papers)},
  year = {2020},
}
Figure