Abstract
Theory of Mind (ToM), or "mindreading," is a critical developmental milestone. Automating its assessment via the "Strange Story Task" is challenging due to the scarcity of expert-labeled data. This study investigates seven data augmentation (DA) strategies to address this "small data" bottleneck. We compare task-specific, expert-informed methods against generic, task-agnostic vector-based techniques. Our findings reveal a significant "quality gap," where expert-informed methods preserve psychological signals far better than generic ones. Combining high-quality augmentations leads to a new state-of-the-art Macro-F1 of 0.985 on the MIND-CA corpus and shows excellent generalization to the new UK-MIND-20 benchmark.
Introduction
"Mindreading"—the ability to understand that others have thoughts, feelings, and desires different from one's own—is a cornerstone of child development. Assessing this ability typically involves standardized tests like the "Strange Story Task," where children provide open-ended responses that must be manually scored by highly trained experts. This process is expensive, time-consuming, and difficult to scale.
While automated scoring systems have been proposed, they are hampered by the "small data" problem: psychological datasets are often small and imbalanced. Data augmentation (DA) is a common solution, but in the nuanced world of psychology, a single word change can completely alter the "correctness" of a child's answer. This paper evaluates which DA strategies preserve the psychological signal and which ones introduce detrimental noise.
Methods
The researchers implemented and compared seven DA strategies, categorized into two groups:
- Task-Specific (Expert-Informed):
- Dictionary: Replacing words with synonyms curated by experts specifically for the context of the test questions.
- Phrase: Appending mental state introductory phrases (e.g., "I think that...") to the start of responses.
- Task-Agnostic (Generic):
- Order: Swapping the position of neighboring words.
- WordNet/PPDB: Standard synonym substitution using public databases.
- GloVe/FastText: Replacing words with their nearest neighbors in a vector space.
Intrinsic Quality Assessment
A critical finding of the study is the "quality gap" between different DA strategies. The authors conducted an intrinsic evaluation where experts re-rated augmented samples to determine if the original psychological label remained valid.
As shown in Figure 3, task-specific methods like "Phrase" and "Dictionary" preserved the original label over 94% of the time. In contrast, vector-based methods like GloVe performed poorly, preserving the label only 68% of the time and producing semantically incoherent text in 17% of cases. Generic word embeddings often group words by association rather than strict synonymy, leading to inappropriate replacements that break the logic required for a "passing" score.
Performance and Generalization
The researchers evaluated these strategies extrinsically by training DistilBERT and BiLSTM classifiers on the augmented data. They tested these models on two datasets: the primary MIND-CA corpus (11,311 pairs) and a new benchmark, UK-MIND-20 (10,320 pairs), which serves as a "gold standard" for testing generalization to unseen data.
The results, visualized in Figure 1, demonstrate that combining high-quality, expert-informed augmentations ("All-HQ") yields the best performance. This strategy achieved a new state-of-the-art Macro-F1 score of 0.985 on MIND-CA. Crucially, DA improved performance not just on the original dataset's test split, but also on the entirely new UK-MIND-20 corpus, suggesting the models are learning underlying psychological constructs rather than dataset-specific artifacts.
Conclusion
The study concludes that while generic vector-based augmentation is often insufficient for nuanced psychological tasks, task-specific strategies guided by human domain knowledge can significantly enhance automated scoring systems. By leveraging expert insights to inform data augmentation, researchers can overcome the "small data" bottleneck and build more reliable, generalizable tools for developmental psychology.
Summary updated