ClickSight: Interpreting Student Clickstreams to Reveal Insights on Learning Strategies via LLMs

Bahar Radmehr, Ekaterina Shved, Fatma Betül Güres, Adish Singla, Tanja KäserarXiv 2025 · DOI · arXiv

Introduction

Digital learning environments generate "clickstreams"—massive logs of every action, click, and pause a student makes. While this data is a goldmine for understanding student behavior, it is notoriously difficult to analyze. Currently, researchers rely on slow manual annotation or brittle feature engineering. ClickSight is a new LLM-based pipeline that translates these messy, high-dimensional student clickstream data into readable interpretations of their learning strategies, providing a scalable way to generate theory-driven educational insights.

The ClickSight Pipeline

The authors developed ClickSight as a three-stage pipeline designed to bridge the gap between raw log data and high-level educational theory.

The ClickSight pipeline: (1) Gathering Context from environments like PharmaSim and Beer's Law Lab, (2) Interpreting Clickstreams using various prompting strategies, and (3) Evaluating Interpretations with domain experts.
  1. Context Gathering: Raw clickstreams are converted into a structured "function call" format (e.g., explore(variable=width, value=1.2cm)). This is paired with target learning strategies and environment descriptions.
  2. LLM Interpretation: The data is processed by GPT-4o using four prompting strategies: Zero-shot, Chain-of-Thought (CoT), Meta-Prompting, and Chain-of-Prompts.
  3. Self-Refinement: An optional loop where the LLM critiques its own output against a quality rubric and revises it.
  4. Expert Evaluation: Human experts rated the outputs based on Completeness, Correctness, Justifiedness, and Comprehensibility.

Experimental Setup

The study evaluated ClickSight across two distinct open-ended learning environments:

  • PharmaSim: A pharmacy assistant simulation involving 9 diagnostic and research strategies.
  • Beer’s Law Lab: A virtual chemistry lab focusing on 3 core inquiry strategies, such as the Control of Variables Strategy.

The researchers sampled clickstreams from 50 students across both environments, generating 400 total interpretations for evaluation.

Key Insights

  • Simplicity Wins: Contrary to many LLM benchmarks, Zero-shot prompting performed best. Chain-of-Thought (CoT) often became bogged down in intermediate steps, leading to incomplete final interpretations.
  • Refinement Risks: While self-refinement helped complex prompts like Chain-of-Prompts, it occasionally caused the LLM to "hallucinate" errors in previously correct Zero-shot outputs in the PharmaSim environment.
  • Accessibility: LLMs excel at turning dry logs into natural language. Comprehensibility scores were nearly perfect, making the data accessible to non-technical stakeholders like teachers.
  • Domain Sensitivity: The "geometry" of the task matters; chemistry lab strategies were harder to interpret than those in the pharmacy simulation, likely due to the complexity of simultaneous variable changes.

Results

Zero-shot achieved the highest overall quality scores in both environments. Interestingly, while self-refinement improved multi-step methods, its impact on Zero-shot was mixed, sometimes leading to a decrease in performance due to over-correction.

Overall quality scores for PharmaSim and Beer's Law Lab. Zero-shot consistently outperforms Chain of prompts, though self-refinement shows benefits for the latter.
  • Top Performance: Zero-shot scored 0.79 for PharmaSim and 0.76 for Beer’s Law Lab on a 0–1 scale.
  • Completeness: Zero-shot achieved a perfect 1.0, meaning it never missed a strategy it was asked to identify.
  • CoT Failure: In the chemistry lab, CoT scored a dismal 0.15, largely because it failed to link its conclusions back to specific evidence in the clickstream.

Conclusion

ClickSight demonstrates that LLMs can effectively scale the interpretation of student behavior. While complex reasoning chains are popular in AI research, simple Zero-shot prompting provides a robust and accurate baseline for extracting educational insights from raw data. Future work will explore Few-shot prompting and the development of real-time "strategy reports" to assist teachers during active classroom sessions.

Summary updated

Abstract

Clickstream data from digital learning environments offer valuable insights into students' learning behaviors, but are challenging to interpret due to their high dimensionality and granularity. Prior approaches have relied mainly on handcrafted features, expert labeling, clustering, or supervised models, therefore often lacking generalizability and scalability. In this work, we introduce ClickSight, an in-context Large Language Model (LLM)-based pipeline that interprets student clickstreams to reveal their learning strategies. ClickSight takes raw clickstreams and a list of learning strategies as input and generates textual interpretations of students' behaviors during interaction. We evaluate four different prompting strategies and investigate the impact of self-refinement on interpretation quality. Our evaluation spans two open-ended learning environments and uses a rubric-based domain-expert evaluation. Results show that while LLMs can reasonably interpret learning strategies from clickstreams, interpretation quality varies by prompting strategy, and self-refinement offers limited improvement. ClickSight demonstrates the potential of LLMs to generate theory-driven insights from educational interaction data.
BibTeX
@misc{radmehr2025clicksight,
  author = {Bahar Radmehr and Ekaterina Shved and Fatma Betül Güres and Adish Singla and Tanja Käser},
  title = {{ClickSight: Interpreting Student Clickstreams to Reveal Insights on Learning Strategies via LLMs}},
  year = {2025},
  eprint = {2505.15410},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.15410},
}
Figure