TL;DR
This paper proposes "experiment-inspired design," a paradigm shift where educational experiments are used not just to validate hypotheses, but to actively explore and improve instructional content. By treating experiments as iterative design tools, researchers can uncover richer insights and create more effective learning experiences.
What Problem Does This Paper Address?
Traditional educational research often frames experiments as a means to accept or reject a specific hypothesis about teaching or learning. This approach can be limiting, as it may overlook the potential of experiments to serve as a powerful engine for designing and improving instructional content. The authors argue that we can gain more value by viewing experimentation as a way to explore the complex design space of educational interventions, rather than solely as a validation tool. They aim to operationalize this "experiment-inspired design" approach by examining its implications in a real-world educational setting.
Methods
The authors introduce "experiment-inspired design" as a framework that reframes the purpose of educational experiments. Instead of focusing on hypothesis testing, this approach emphasizes using experiments to explore the design space of instructional content. They operationalize this by running a series of A-B interventions over three years with thousands of first-year computer science students, focusing on teaching "meta-skills" like stress management and planning.
The paper highlights two key themes within experiment-inspired design:
- Iterative Design & Parallel Prototyping: This involves viewing interventions as opportunities for continuous improvement through a sequence of experiments. It includes generating multiple alternative designs simultaneously (parallel prototyping) and iterating on these designs over time by re-deploying interventions and incorporating new variations. Crucially, it emphasizes actively soliciting user feedback to inform these iterations.
- Divergent Thinking: This theme encourages rethinking and sometimes breaking traditional constraints associated with experimentation. This includes widening constraints to accommodate diverse stakeholder needs (e.g., instructors' concerns about fairness) and normalizing experimentation even if it's not perfectly executed. It also involves imposing constraints to drive creativity, such as shortening intervention durations to force more concise and impactful design solutions.
Datasets
Not discussed in the paper. The paper focuses on the design and implementation of interventions rather than specific datasets for training or evaluation.
Key Insights and Ideas
- Experiments as Design Tools: The most significant insight is the reframing of experiments from hypothesis validators to active design tools. This opens up a broader spectrum of opportunities for innovation in educational content.
- Embracing Imperfection for Learning: It's valuable to run experiments even if they aren't perfectly designed or immediately analyzed. The act of regular experimentation can lead to systematic improvement and a deeper understanding of how resources impact learning over time.
- User Feedback is Crucial for Iteration: Actively seeking and incorporating qualitative user feedback is essential for iterative design. Understanding how students perceive and interact with interventions provides invaluable insights for refinement.
- Constraints Can Drive Creativity: Rather than always relaxing constraints, sometimes imposing them (e.g., time limits for interventions) can force more creative and effective solutions.
- Transparency and Autonomy Enhance Engagement: When dealing with multiple intervention versions, being transparent with students and offering them autonomy in choosing which versions to engage with can address concerns about fairness and improve the overall experience.
Key Results
The paper doesn't focus on reporting specific quantitative results of the interventions themselves, as the primary goal was to explore the experiment-inspired design process. However, they mention that the Stress Reappraisal intervention had a "significant positive effect on exam scores (p = 0.003, d = 0.252)" and a larger impact on first-year students. The authors emphasize that checking for this statistical significance was not the primary goal; the aim was to explore the design space and produce multiple ways to communicate the idea.
Limitations and Future Work
The authors acknowledge that their work is based on preliminary lessons derived from their experience. They suggest that "experiment-inspired design" is an active area for researchers to explore more broadly in the future. They encourage a deeper consideration of alternative ways to think about experiments and, in doing so, to "experiment with experimentation."
Summary updated