Collaboration-Type Identification in Educational Datasets

Andrew E. Waters, Christoph Studer, Richard G. BaraniukZenodo 2014 · DOI

TL;DR

This paper introduces a novel Bayesian methodology for automatically identifying and classifying collaboration between learners in educational datasets. By combining advanced learning analytics with new statistical models of collaborative behavior, the approach can detect cheating and provide a more accurate assessment of individual student competence, even in large online courses.

What Problem Does This Paper Address?

Educators face a significant challenge in identifying when students collaborate on assignments or exams. This is crucial for two main reasons: first, to maintain academic integrity and prevent cheating, and second, to accurately assess individual student understanding. In traditional settings with small classes, instructors might spot suspicious patterns. However, this becomes nearly impossible in large online courses (MOOCs) with thousands of students and limited instructor interaction. Existing automated methods often focus on multiple-choice tests and lack the sophistication to model diverse collaboration types or handle general right/wrong response data.

Methods

The proposed methodology integrates three key components:

  1. Learning Analytics (LA): The system uses LA techniques to estimate the probability of each learner answering each question correctly. The paper highlights two established LA methods: the Bayesian Rasch model and Sparse Factor Analysis (SPARFA). These methods go beyond simple answer comparisons by considering individual learner abilities and question difficulties.
  2. Collaboration Models: The authors introduce four novel statistical models to represent different types of collaborative behavior between pairs of learners:
  3. Independence Model (M1): Learners work entirely on their own.
  4. Parasitic Model (M2): One learner copies another's answers.
  5. Dominance Model (M3): Learners discuss answers, with one dominating the other's choice.
  6. OR Model (M4): Learners collaborate such that the pair gets the answer right if at least one of them does. These models are probabilistic and account for the possibility that collaboration might only occur on a subset of questions.

  7. Collaboration Detection Algorithms: Two algorithms are proposed to jointly identify collaborators and classify their collaboration type:

  8. Sequential Hypothesis Testing (SHT): This method uses a Bayesian hypothesis test to compare the likelihood of observed answer patterns arising from independent work versus a specific collaboration model.
  9. Collaborative Model Selection (CMS): This approach uses Bayesian model selection to jointly infer which collaboration model is most likely for each pair of learners, along with the associated model parameters. CMS is presented as a more comprehensive approach that avoids the need for multiple pairwise comparisons.

Datasets

The paper evaluates its methodology on:

  • Synthetic Data: Generated using the SPARFA model to create controlled scenarios with known ground truth collaboration. This allowed for systematic testing of performance under varying parameters (collaboration probability, number of questions).
  • Real-World Educational Datasets:
    • Undergraduate Signal Processing Course: Data from homework assignments and a take-home final exam from an electrical and computer engineering course. This dataset provided a "rough ground truth" for homework assignments due to group structures and instructor knowledge.
  • Undergraduate Computer Engineering Course: Data from a take-home final exam.

Key Insights and Ideas

  • Beyond Simple Copying: The paper moves beyond detecting simple answer copying to model more nuanced collaborative behaviors like symbiotic work (OR model) and dominant influence. This is crucial for understanding the true nature of student interaction.
  • Bayesian Framework for Robustness: The use of a fully Bayesian approach allows for the estimation of posterior distributions, providing a measure of uncertainty and leading to more robust collaboration detection.
  • Generalizability to Right/Wrong Data: The methodology is designed to work with simple right/wrong response data, making it applicable to a much wider range of educational datasets than methods limited to multiple-choice formats.
  • Detecting Cheating in Real-World Scenarios: The application to non-collaborative take-home exams successfully identified instances of collaboration, demonstrating the practical utility of the method for flagging potential academic misconduct.

Key Results

  • Synthetic Data Performance: The proposed Collaborative Model Selection (CMS) method consistently outperformed existing methods (Wesolowsky's method and AHT) in identifying collaborating learner pairs, achieving higher normalized percentile rankings. CMS showed excellent performance even with collaboration probabilities as low as 0.5 (i.e., learners collaborating on every other question).
  • Real-World Data Findings:
    • In the signal processing course homework, CMS identified several intra-group collaborations, with the parasitic model being most frequent for one group, supported by analysis of midterm performance.
  • For the signal processing final exam, CMS detected a symbiotic collaboration between two learners whose open-form responses showed remarkable similarity, strengthening the evidence.
  • In the computer engineering final exam, CMS identified three groups of collaborating learners, with two groups exhibiting identical answer patterns and one group missing only a single question.

Limitations and Future Work

  • Decoupled Collaboration Rate: The current models treat the collaboration rate parameter (ε₁) independently of learner success probabilities (pᵢ). The authors suggest future work on modeling how learners might strategically choose to collaborate more on difficult questions.
  • Pairwise Focus: The current methodology primarily focuses on pairwise collaboration. Future work aims to extend this to model larger groups with more complex social dynamics.
  • Integrating Side Information: The paper acknowledges that incorporating additional information beyond answer patterns (e.g., open-form responses, forum discussions) could significantly improve collaboration detection and reduce false positives.
  • "Denoising" LA: The authors propose using collaboration detection to "denoise" learning analytics, estimating true learner ability by accounting for collaborative influences.
  • Multi-Modal Data Fusion: Fusing data from different modalities (e.g., text responses, interaction logs) is identified as a promising avenue for more comprehensive insights.

Summary updated

Abstract

Identifying collaboration between learners in a course is an important challenge in education for two reasons: First, depending on the courses' rules, collaboration can be considered a form of cheating. Second, it helps one to more accurately evaluate each learner's competence. While such collaboration identification is already challenging in traditional classroom settings consisting of a small number of learners, the problem is greatly exacerbated in the context of both online courses or massively open online courses (MOOCs) where potentially thousands of learners have little or no contact with the course instructor. In this work, we propose a novel methodology for collaboration-type identification, which both identifies learners who are likely collaborating and also classifies the type of collaboration employed. Under a fully Bayesian setting, we infer the probability of learners' succeeding on a series of test items solely based on graded response data. We then use this information to jointly compute the likelihood that two learners were collaborating and what collaboration model (or type) was used. We demonstrate the efficacy of the proposed methods on both synthetic and real-world educational data; for the latter, the proposed methods find strong evidence of collaboration among learners in two non-collaborative takehome exams.
BibTeX
@inproceedings{waters2014collaboration,
  author = {Andrew E. Waters and Christoph Studer and Richard G. Baraniuk},
  title = {{Collaboration-Type Identification in Educational Datasets}},
  booktitle = {Zenodo},
  year = {2014},
  doi = {10.5281/zenodo.3554681},
}

Loading related content...

Figure