TL;DR
This paper introduces NexP, an open-source toolkit designed to simplify the process of designing and conducting controlled experiments for beginners in Human-Computer Interaction (HCI). By providing a guided, question-driven approach, NexP helps novice researchers bridge the gap between informal ideas and formal experimental design, making the process more intuitive and educational.
What Problem Does This Paper Address?
Designing controlled experiments is a cornerstone of HCI research for evaluating user interfaces and interactions. However, this process is often complex and error-prone, even for experienced researchers. Existing toolkits, while helpful, can be challenging for beginners who may lack formal statistical training or experience in experimental design. These novices often struggle to translate research questions into testable hypotheses, identify appropriate variables, and structure experimental conditions. This paper addresses this gap by creating a tool specifically tailored to the needs of these beginner researchers, aiming to improve both their productivity and their understanding of experimental design principles.
Methods
NexP employs a novel 5-stage, question-answer-driven approach to guide users through experiment design. This "Question-Answer Structure" starts with broad, informal questions about the research goal and progressively refines them into formal hypotheses, variables, and experimental conditions. Key features include:
- Hypothesis Development: Scaffolds users in bridging informal language to formal statistical hypotheses.
- Variable Identification: Helps users distinguish between independent and dependent variables.
- Condition Structuring: Guides users in organizing experimental conditions and test orders.
- Streamlined Process: A step-by-step flow that breaks down complex decisions into manageable parts.
- Web-Based Platform: An open-source toolkit with both design and execution platforms.
- Simulation and Overview: Provides a preliminary simulation of the experiment design for iterative refinement.
This approach is informed by scaffolding problem-solving techniques and aims to reduce cognitive load for novice users.
Datasets
Not discussed in the paper. The paper focuses on the design and evaluation of the NexP toolkit itself, rather than on specific datasets used for training or evaluation in a machine learning sense. The "datasets" involved in the evaluation were the experimental designs created by participants using NexP and Touchstone.
Key Insights and Ideas
- Bridging the Formal-Informal Gap: The most significant insight is the effectiveness of the Question-Answer Structure in helping beginners translate everyday language into the precise terminology of experimental design. This is crucial for making complex research methods accessible.
- Scaffolding for Learning: NexP's guided, step-by-step process acts as a powerful scaffold, not just for completing a task, but for learning the underlying principles of experimental design. This is particularly valuable for educational purposes.
- Beginner-Centric Design: The tool's success hinges on its focus on the specific challenges faced by novice researchers, such as understanding statistical terms and the purpose of different design elements.
- End-to-End Solution: NexP aims to be a comprehensive solution, offering not only design but also execution and data support, which is a notable advancement.
Key Results
A workshop study involving 24 participants with limited prior knowledge of controlled experiments was conducted to compare NexP with Touchstone, a more established toolkit.
- Replication Task Success: 75% of groups successfully replicated existing experiments using NexP, compared to 33.3% using Touchstone.
- Design Task Preference: 91.7% of participants preferred NexP for a design task.
- Post-Questionnaire Feedback: Participants rated NexP significantly higher than Touchstone across most usability and satisfaction metrics, including productivity, ease of use, identifying hypotheses, and identifying independent variables.
- Qualitative Feedback: Participants found NexP easier to understand, more intuitive, and better at supporting their thinking process for experiment design. They specifically highlighted the hypothesis feature and the tree structure simulation as beneficial.
Limitations and Future Work
Limitations:
- The paper acknowledges that while NexP aims to simplify the process, some terms in later stages (e.g., "between subjects," "counter-balancing") still require higher background knowledge, though NexP provides more elaborate help text for these.
- The study was conducted with novice users, and its effectiveness with more experienced researchers is not the primary focus.
Future Work:
- Expand NexP as a learning support tool by conducting further studies to understand the needs of researchers aiming to improve their experimental design skills.
- Improve and publish the open-source NexP design and running platform for wider community use.
- Explore how NexP can be used as a pedagogical tool to help instructors teach controlled experiments.
Summary updated