TL;DR
This paper introduces "Prompty," a novel AI literacy tool designed for high school students to foster critical engagement with Large Language Models (LLMs). By providing a structured yet flexible interface, Prompty aims to transform students from passive consumers of AI-generated text into informed users who can effectively prompt, evaluate, and understand LLMs.
What Problem Does This Paper Address?
The rapid proliferation of powerful LLMs like ChatGPT has created a significant gap: while these tools can effortlessly generate text, many users, particularly students, lack the skills to critically evaluate this output or use LLMs effectively and responsibly. This poses a challenge for educators seeking to integrate AI into curricula without compromising critical thinking and digital literacy. The paper addresses the need for accessible, classroom-ready tools that can teach students about LLMs, prompt engineering, and the ethical considerations surrounding AI, all within a safe and supportive learning environment.
Methods
Prompty is a web-based learning tool built using React and OpenAI's text-davinci-003 LLM. Its core design is centered around pedagogical principles and co-designed with high school teachers. Key methodological features include:
- Contrasting Cases: The interface presents users with three distinct LLM-generated responses to a single prompt, encouraging comparison, evaluation, and deeper understanding of variations in output.
- Prompt Guides: A structured prompting system breaks down prompt creation into three components: Role, Context, and Task. This scaffolding helps students systematically construct effective prompts.
- Limited Tries: A capped number of generation attempts per session is implemented to promote conscious and intentional usage, encouraging students to reflect on their prompts and outputs.
- Guardrails: Robust safety measures are in place, including OpenAI's Moderation endpoint to filter inappropriate content and server-side modifications to ensure responses are student-friendly.
Datasets
Not discussed in the paper. The tool utilizes OpenAI's LLM API for generation and does not appear to be trained on a specific dataset for this project.
Key Insights and Ideas
- Shifting from Consumption to Creation: The "Contrasting Cases" approach is crucial. By seeing multiple outputs, students are naturally prompted to ask why one response is better than another, moving beyond simply accepting the first result.
- Scaffolding for Prompt Engineering: The Role-Context-Task structure is an effective way to demystify prompt engineering, making it accessible to students without prior technical knowledge. It breaks down a complex skill into manageable components.
- Intentionality through Limitation: Limiting generation tries, while initially met with student requests for more, was perceived by the teacher as a valuable mechanism for fostering mindfulness and deliberate prompting. It forces students to think critically about each attempt.
- AI Literacy as a Foundational Skill: The paper highlights that AI literacy isn't just about understanding AI's technical aspects but also about developing critical evaluation skills and an understanding of its societal impact, aligning with broader educational goals.
- Teacher Co-Design is Key: The success of Prompty is heavily attributed to the collaborative design process with a high school teacher, ensuring the tool is relevant, practical, and adaptable for classroom use.
Key Results
The paper presents preliminary findings from a pilot study conducted in a high school English class. While quantitative results are not detailed, qualitative feedback from the participating teacher, Ms. L, indicates:
- Students quickly understood and adopted the Prompty interface.
- The tool effectively facilitated learning objectives related to AI literacy and ELA standards, particularly in comparing and evaluating texts.
- Students were able to compose creative writing pieces using AI-generated responses.
- The Prompt Guides were helpful in breaking down prompt components.
- Ms. L observed students developing a deeper understanding that AI requires human attention and evaluation, aligning with higher levels of cognitive demand (Depth of Knowledge levels 3-4).
Limitations and Future Work
- Teacher Adaptation: While Prompty is designed to be adaptable, the paper acknowledges that teachers with less prior AI literacy experience might require more support and resources to integrate it effectively.
- Prompt Revision Guidance: While students could generate responses, some struggled with identifying how to revise prompts based on output weaknesses, suggesting a need for more explicit guidance on iteration.
- Limited LLM Choice: Currently, Prompty only uses
text-davinci-003. Future work could involve allowing students to experiment with different LLMs to understand their varying capabilities. - Advanced Prompting: The paper suggests adding an "advanced prompting mode" to explore parameter settings like temperature and context window, offering a deeper dive into LLM mechanics.
- Broader Applications: The team aims to expand Prompty's use beyond ELA, exploring its potential in other disciplines and for different AI literacy topics, such as understanding chatbot interactions and the ethical implications of AI.
Summary updated