Abstract
In this paper we implement and compare 7 different data augmentation strategies for the task of automatic scoring of children's ability to understand others' thoughts, feelings, and desires (or "mindreading"). We recruit in-domain experts to re-annotate augmented samples and determine to what extent each strategy preserves the original rating. We also carry out multiple experiments to measure how much each augmentation strategy improves the performance of automatic scoring systems. To determine the capabilities of automatic systems to generalize to unseen data, we create UK-MIND-20 - a new corpus of children's performance on tests of mindreading, consisting of 10,320 question-answer pairs. We obtain a new state-of-the-art performance on the MIND-CA corpus, improving macro-F1-score by 6 points. Results indicate that both the number of training examples and the quality of the augmentation strategies affect the performance of the systems. The task-specific augmentations generally outperform task-agnostic augmentations. Automatic augmentations based on vectors (GloVe, FastText) perform the worst. We find that systems trained on MIND-CA generalize well to UK-MIND-20. We demonstrate that data augmentation strategies also improve the performance on unseen data.
Introduction: Automating the Measurement of Theory of Mind
In developmental psychology, "mindreading" (Theory of Mind) refers to a child's ability to understand that other people have distinct thoughts, feelings, and desires. Measuring this is crucial for tracking social development and identifying potential neurodevelopmental conditions. However, scoring these tests is a significant bottleneck; it requires highly trained experts to manually rate open-ended responses, making large-scale studies expensive and slow.
While Natural Language Processing (NLP) can automate this scoring, these models are data-hungry. Gathering high-quality, human-annotated data in this niche field is difficult. Data augmentation (DA)—the process of artificially creating new training examples—is a common solution, but it is risky in psychological contexts where subtle wording changes can completely alter a child's score. This paper evaluates which DA strategies preserve "psychological truth" and which ones introduce counterproductive noise.
Augmentation Strategies: Task-Specific vs. Task-Agnostic
The researchers implemented and compared seven different data augmentation strategies, categorized by their reliance on domain expertise:
Task-Specific Strategies: * Dictionary: Replacing words with synonyms curated by experts specifically for this context (e.g., replacing "men" with "burglars" in a specific test scenario). * Phrase: Adding introductory "mental state" phrases like "I think that..." to the start of a response.
Task-Agnostic (Generic) Strategies: * Order: Swapping the position of two neighboring words. * WordNet/PPDB: Replacing words using generic synonym databases. * GloVe/FastText: Replacing words with their nearest neighbors in a high-dimensional vector space.
Datasets and Benchmarks
The study utilized two primary datasets to train and evaluate the models, including a newly introduced benchmark for testing generalization.
- MIND-CA: The primary training corpus containing 11,311 question-answer pairs from children aged 7–14.
- UK-MIND-20: A new benchmark containing 10,320 pairs from a different group of children, collected at a different time and location. This was used specifically to test how well the models generalize to "unseen" real-world data.
Key Insights: Vectors vs. Experts
- Vectors aren't experts: Vector-based methods (GloVe, FastText) performed the worst. They often replaced words with "similar" ones that broke the logic of the sentence or changed the psychological score, sometimes producing semantically incoherent text.
- The "Order" Surprise: Simply swapping the order of two words was surprisingly effective. It preserved the original score 94.5% of the time and helped the model become more robust, likely because it mimics the slightly ungrammatical or varied sentence structures children actually use.
- Quantity vs. Quality: While the volume of data added was the primary driver of raw performance, the quality of the augmentation determined the model's consistency. High-quality, expert-led augmentation led to much lower standard deviation in scores across different questions.
Experimental Results and State-of-the-Art Performance
By combining high-quality augmentation strategies, the authors achieved a new state-of-the-art result, improving the macro-F1 score on the MIND-CA corpus by 6 points and the F1-per-Question by 10.3 points.
Intrinsic evaluation showed that "high-quality" strategies (Phrase, Order, Dictionary) preserved the original human label in over 94% of cases. In contrast, the GloVe-based strategy only preserved the label 68% of the time and produced "invalid" (incoherent) text in 17% of samples. Furthermore, the best-augmented model achieved an F1 score of .898 on the new UK-MIND-20 dataset, proving the system is robust to variations in test administration.
Conclusion and Future Directions
While task-specific strategies are superior, they require an initial investment from human experts to build contextual dictionaries and phrase lists. Future work could explore whether these strategies hold up across different languages and cultures, and how these NLP tools can be integrated into the daily workflows of developmental psychologists to facilitate large-scale longitudinal studies.
Summary updated