Abstract: Qualitative analysis of verbal data is of central importance in the learning sciences. It is labor-intensive and time-consuming, however, which limits the amount of data researchers can include in studies. This work is a step towards building a statistical machine learning (ML) method for achieving an automated support for qualitative analyses of students' writing, here specifically in score laboratory reports in introductory biology for sophistication of argumentation and reasoning. We start with a set of lab reports from an undergraduate biology course, scored by a four-level scheme that considers the complexity of argument structure, the scope of evidence, and the care and nuance of conclusions. Using this set of labeled data, we show that a popular natural language modeling processing pipeline, namely vector representation of words, a.k.a word embeddings, followed by Long Short Term Memory (LSTM) model for capturing language generation as a state-space model, is able to quantitatively capture the scoring, with a high Quadratic Weighted Kappa (QWK) prediction score, when trained in via a novel contrastive learning set-up. We show that the ML algorithm approached the inter-rater reliability of human analysis. Ultimately, we conclude, that machine learning (ML) for natural language processing (NLP) holds promise for assisting learning sciences researchers in conducting qualitative studies at much larger scales than is currently possible.
Introduction
In the learning sciences, "qualitative coding"—the manual process of identifying complex intellectual practices in student work—is a vital but resource-heavy task. Researchers often spend hundreds of hours reading essays to determine how students construct arguments or utilize evidence. While existing Natural Language Processing (NLP) tools can effectively identify "conceptual correctness" (e.g., the presence of specific keywords), they frequently struggle to assess the quality and sophistication of reasoning. Distinguishing between a "tautological" argument and a "relational" one requires a model that understands context and structure rather than just vocabulary.
Methodology: The CoReL-W Framework
To address these challenges, researchers at Tufts University developed CoReL-W (Contrastive Representation Learning in the Wasserstein Space). This framework is specifically designed to handle the nuances of scientific argumentation even when training data is scarce.
The CoReL-W pipeline consists of several key stages:
- Word Representation: Words are converted into vectors using embeddings. To de-emphasize common filler words, these are often weighted by their frequency (e.g., SIF weighting).
- Feature Encoding: The document is processed through an encoder, such as a Convolutional Neural Network (CNN) or a Long Short-Term Memory (LSTM) network, to extract local and sequential features.
- Point Cloud Representation: Instead of compressing an entire essay into a single fixed vector, the model represents each lab report as a "point cloud" (an empirical distribution) of its constituent feature vectors.
- Contrastive Learning & Wasserstein Distance: The model is trained using a triplet loss function. It compares an "anchor" report to a "positive" example (same score) and a "negative" example (different score). The distance between these point clouds is measured using the Wasserstein distance (also known as Earth Mover’s Distance), which calculates the minimum cost to transform one distribution into another.
Dataset and Evaluation
The study utilized a dataset of 146 discussion sections from undergraduate biology lab reports. These reports were scored on a 1–4 scale based on an adapted SOLO taxonomy, evaluating three dimensions: Argument Structure, Scope of Evidence, and Consistency/Closure.
The primary evaluation metric was the Quadratic Weighted Kappa (QWK), which measures the agreement between raters while accounting for the degree of disagreement (e.g., a model predicting a '1' for a '4' is penalized more than predicting a '3' for a '4').
Key Results
The experimental results demonstrate that the CoReL-W framework significantly outperforms traditional NLP baselines.
As shown in the table above, the baseline c-rater-ML system failed to generalize, achieving a QWK of 0. This confirms that simple feature-based models cannot capture the complexity of scientific reasoning. However, the CoReL-W framework with a CNN encoder achieved a QWK of 0.645, which is considered "substantial agreement." While this does not yet match the human-to-human reliability of 0.839, it represents a major step forward in automating qualitative analysis for small-scale, high-complexity datasets.
Discussion and Conclusion
Beyond quantitative performance, the study revealed a "second opinion" effect. In cases where the model's score disagreed with the original human score, re-evaluation by human experts often showed that the model was more consistent, leading the experts to revise their own codes. This suggests that CoReL-W can serve as a valuable calibration tool for human researchers.
Despite these successes, limitations remain. The model struggled with "Level 1" reports due to their scarcity in the training set (only 12 examples). Future work will focus on addressing this data imbalance, testing the framework's transferability to other scientific disciplines, and improving the interpretability of the model's outputs to provide more transparent insights for educators.
Summary updated