TL;DR
This paper explores how Generative AI (GenAI) can be effectively integrated into secondary classrooms, focusing on understanding teacher needs and student learning outcomes. It proposes a novel approach by first gathering teacher design insights for a GenAI chatbot and then evaluating its impact on student learning, aiming to bridge the gap between cutting-edge AI and practical educational applications.
What Problem Does This Paper Address?
While Generative AI (GenAI) and Large Language Models (LLMs) are rapidly transforming various fields, their application in secondary education remains underexplored. Much of the existing research focuses on higher education, leaving a gap in understanding how these powerful tools can best benefit younger students and their teachers. Secondary classrooms face unique challenges, including increasing student-teacher ratios, making efficient and effective teaching tools crucial. This paper aims to address this by investigating how to design and implement GenAI tools tailored to the specific needs of secondary education, ensuring they are both pedagogically sound and practically useful for instructors and students alike.
Methods
The research is structured in two proposed studies.
Proposed Study 1 (P1): This study focuses on understanding secondary teachers' perspectives on GenAI chatbots. It involves:
- Workshops: A series of workshops were conducted for secondary teachers on GenAI (specifically ChatGPT), building upon previous work with updated materials (GPT-4).
- Design Sessions: Participants engaged in planning and design sessions where they physically "crafted" their ideal chatbots using low-fidelity methods (paper, pencil, note cards).
- Interviews: Focus group interviews were conducted before, after, and several months after the workshops to capture teachers' attitudes and the longer-term impacts.
- Thematic Analysis: Transcripts from planning, design sessions, and interviews are being thematically analyzed using inductive coding and affinity diagramming to identify common themes and features of teacher-designed chatbots.
Proposed Study 2 (P2): This study aims to create and evaluate a GenAI chatbot for students based on the insights from P1. It involves:
- Chatbot Development: A Retrieval-Augmented Generation (RAG)-based chatbot will be developed, informed by teacher feedback from P1. This approach allows for tailoring the chatbot to instructor needs and ensuring age/discipline-appropriate content.
- Quasi-Experimental Evaluation: The chatbot will be evaluated with secondary students in a quasi-experimental setup. One group of students will use the chatbot for their lessons, while a control group will not.
- Data Collection: Learning outcomes will be measured using pre- and post-tests. Students using the chatbot will complete surveys with Likert-scale questions and open-ended responses. Teachers will also be surveyed on their perceptions of tool deployment and ease of use.
- Analysis: Survey data will be analyzed using descriptive statistics and thematic analysis.
Datasets
Not discussed in the paper. The paper focuses on the design and evaluation of a novel GenAI chatbot, with data collected from teacher workshops and student evaluations rather than pre-existing datasets.
Key Insights and Ideas
- Teacher-Centric Design is Crucial: The research emphasizes that effective GenAI tools for secondary education must be designed with teachers, not just for them. Understanding their specific needs, desired functionalities, and pedagogical approaches is paramount.
- Tailored GenAI Over Generic Tools: Preliminary results suggest secondary teachers prefer a "tailored GenAI agent" over simply using general-purpose tools like ChatGPT. This highlights the need for specialized, context-aware AI assistants in education.
- RAG for Educational Customization: The choice of a Retrieval-Augmented Generation (RAG) approach for chatbot development is a key methodological insight. RAG allows for grounding AI responses in specific, relevant information, which is critical for ensuring accuracy and appropriateness in an educational context.
- Bridging the Tech-Ed Gap: The overarching goal is to bridge the significant gap between the rapid advancements in AI technology and the practical implementation and understanding within secondary education.
Key Results
From Proposed Study 1 (P1): * Data collection is complete, and data analysis is in progress. * Thematic analysis using inductive coding and affinity diagramming is being used to understand teacher design processes and opinions. * The study aims to answer: (1) perceived roles of LLM-based chatbots in classrooms, (2) desired features and functionalities, and (3) disciplinary similarities/differences in chatbot design. * Completion of P1 is anticipated by February 2025.
From Proposed Study 2 (P2): * This study is planned to use the results of P1 to inform the creation of a RAG-based chatbot. * The evaluation will involve a quasi-experimental setup to measure student learning outcomes. * The study aims to answer: (1) how well secondary teacher design considerations map to a RAG-based chatbot, and (2) the chatbot's effectiveness in supporting or improving student learning. * This work is expected to be in-progress by February 2025.
Limitations and Future Work
Limitations: * The paper is a proposal and outlines anticipated progress. The full results of the studies are not yet available. * The evaluation in P2 is a quasi-experimental setup, which may have limitations in establishing causality compared to a randomized controlled trial. * The focus is on secondary education, and the findings may not directly translate to other educational levels.
Future Work: * The immediate future work involves completing the data analysis for P1 and proceeding with the development and evaluation of the RAG-based chatbot in P2. * Further research could explore the long-term impacts of such chatbots on student engagement, critical thinking, and overall academic performance. * Investigating the ethical implications and potential biases of GenAI tools in secondary classrooms will be crucial. * Expanding the scope to include a wider range of disciplines and student demographics would enhance the generalizability of the findings.
Summary updated