Abstract
Early identification of Language Impairment (LI) is critical for effective intervention, yet traditional automated screening often relies on simplistic "shallow" metrics. This study evaluates the impact of "deep" Natural Language Processing (NLP) features—including syntactic complexity, situational coherence, and narrative structure—on the automatic classification of LI in adolescents. Using a dataset of 118 transcripts across storytelling and spontaneous speech tasks, we demonstrate that deep NLP features significantly improve prediction accuracy in structured contexts. Our best model achieved an F-1 score of 0.914 in storytelling tasks, highlighting the importance of task-specific feature engineering in clinical NLP applications.
Introduction
Language Impairment (LI) affects a substantial portion of the pediatric population, impacting academic success and social development. Current diagnostic workflows often involve labor-intensive manual assessments by speech-language pathologists. While automated tools exist, they typically utilize shallow metrics like Mean Length of Utterance (MLUm) or type-token ratios. These metrics, while useful for measuring general productivity, often fail to capture the nuanced deficits in narrative logic and global coherence characteristic of LI.
This research explores whether deeper NLP features can provide a more robust "first-pass" screening tool. By moving beyond word counts to analyze how children construct mental models and maintain referential clarity, we aim to improve the precision and recall of automated LI detection.
Methodology
The study framed LI detection as a binary classification problem (LI vs. Typically Developing). Researchers utilized a dataset of transcripts from 118 adolescents (99 Typically Developing, 19 with Language Impairment) who participated in two distinct verbal tasks:
- Storytelling: Retelling the wordless picture book "Frog, Where Are You?"
- Spontaneous Narrative: Describing a personal annoyance.
Feature Engineering
The researchers compared a baseline of shallow features (vocabulary knowledge, productivity) against several advanced NLP feature sets:
- Coh-Metrix Features: Readability scores and "Situational Model" features that track causal, intentional, and temporal dimensions of a narrative.
- Syntactic Features: Measures of sentence complexity, including noun phrase modifiers and logical operator usage.
- Referential and Semantic Features: Latent Semantic Analysis (LSA) and anaphora resolution to measure how ideas connect across sentences.
- Entity Grid Models: Tracking the transitions of specific entities (characters/objects) through grammatical roles to assess local coherence.
- Narrative Quality: Human-annotated features such as the presence of a resolution and the use of "social engagement devices" (e.g., character voices).
A Bayesian Network classifier was found to be the most effective model for integrating these diverse feature sets.
Results and Analysis
The evaluation focused on the F-1 measure to balance precision and recall. A key finding was the stark difference in performance between the two tasks.
In the Storytelling task, the integration of deep NLP features—specifically those related to readability and situational models—led to a substantial performance boost. As shown in the results, the combination of baseline features with advanced NLP metrics reached a peak F-1 score of 0.914. This suggests that when a child is constrained by a specific narrative arc, their linguistic and cognitive deficits become more apparent to automated systems.
Conversely, performance in the Spontaneous Narrative task was significantly lower, with the best models achieving an F-1 of only 0.667. In this unconstrained context, the "deep" features provided little additional predictive value over the baseline. This disparity underscores that the effectiveness of NLP features is highly dependent on the communicative context.
Discussion: The Role of Context and Narrative Quality
The success in the storytelling domain can be attributed to the existence of a "ground truth." In a story retell, there is a predefined sequence of events and logical connections that the speaker must navigate. Deep NLP features, particularly those tracking the "Situational Model," are adept at identifying when a speaker fails to maintain this logical thread.
Furthermore, the inclusion of narrative quality features (like cognitive inferences and social engagement) proved highly beneficial. Adding these human-annotated features to the baseline for story retells resulted in an 8.7% performance increase (F-1 of 0.865). This indicates that the "flavor" and social-cognitive aspects of language are as important as syntactic correctness in identifying impairment.
Conclusion and Future Work
This study demonstrates that deep NLP features significantly enhance the automated identification of Language Impairment, particularly in structured storytelling tasks. While shallow metrics provide a baseline, features that capture situational coherence and narrative structure offer the nuance required for high-accuracy screening.
The primary limitation remains the small sample size of LI participants and the current reliance on human annotation for narrative quality features. Future research will focus on automating the extraction of these narrative features and developing more robust models for the inherently "messy" data found in spontaneous personal narratives.
Summary updated