Introduction
Digital learning environments generate "clickstreams"—massive logs of every action, click, and pause a student makes. While this data is a goldmine for understanding student behavior, it is notoriously difficult to analyze. Currently, researchers rely on slow manual annotation or brittle feature engineering. ClickSight is a new LLM-based pipeline that translates these messy, high-dimensional student clickstream data into readable interpretations of their learning strategies, providing a scalable way to generate theory-driven educational insights.
The ClickSight Pipeline
The authors developed ClickSight as a three-stage pipeline designed to bridge the gap between raw log data and high-level educational theory.
- Context Gathering: Raw clickstreams are converted into a structured "function call" format (e.g.,
explore(variable=width, value=1.2cm)). This is paired with target learning strategies and environment descriptions. - LLM Interpretation: The data is processed by GPT-4o using four prompting strategies: Zero-shot, Chain-of-Thought (CoT), Meta-Prompting, and Chain-of-Prompts.
- Self-Refinement: An optional loop where the LLM critiques its own output against a quality rubric and revises it.
- Expert Evaluation: Human experts rated the outputs based on Completeness, Correctness, Justifiedness, and Comprehensibility.
Experimental Setup
The study evaluated ClickSight across two distinct open-ended learning environments:
- PharmaSim: A pharmacy assistant simulation involving 9 diagnostic and research strategies.
- Beer’s Law Lab: A virtual chemistry lab focusing on 3 core inquiry strategies, such as the Control of Variables Strategy.
The researchers sampled clickstreams from 50 students across both environments, generating 400 total interpretations for evaluation.
Key Insights
- Simplicity Wins: Contrary to many LLM benchmarks, Zero-shot prompting performed best. Chain-of-Thought (CoT) often became bogged down in intermediate steps, leading to incomplete final interpretations.
- Refinement Risks: While self-refinement helped complex prompts like Chain-of-Prompts, it occasionally caused the LLM to "hallucinate" errors in previously correct Zero-shot outputs in the PharmaSim environment.
- Accessibility: LLMs excel at turning dry logs into natural language. Comprehensibility scores were nearly perfect, making the data accessible to non-technical stakeholders like teachers.
- Domain Sensitivity: The "geometry" of the task matters; chemistry lab strategies were harder to interpret than those in the pharmacy simulation, likely due to the complexity of simultaneous variable changes.
Results
Zero-shot achieved the highest overall quality scores in both environments. Interestingly, while self-refinement improved multi-step methods, its impact on Zero-shot was mixed, sometimes leading to a decrease in performance due to over-correction.
- Top Performance: Zero-shot scored 0.79 for PharmaSim and 0.76 for Beer’s Law Lab on a 0–1 scale.
- Completeness: Zero-shot achieved a perfect 1.0, meaning it never missed a strategy it was asked to identify.
- CoT Failure: In the chemistry lab, CoT scored a dismal 0.15, largely because it failed to link its conclusions back to specific evidence in the clickstream.
Conclusion
ClickSight demonstrates that LLMs can effectively scale the interpretation of student behavior. While complex reasoning chains are popular in AI research, simple Zero-shot prompting provides a robust and accurate baseline for extracting educational insights from raw data. Future work will explore Few-shot prompting and the development of real-time "strategy reports" to assist teachers during active classroom sessions.
Summary updated