Fluency and structural complexity as predictors of L2 oral proficiency

Jared Bernstein, Jian Cheng, Masanori SuzukiInterspeech 2010

TL;DR

This paper investigates whether the speed and structural complexity of spoken language are better predictors of second language (L2) oral proficiency than previously thought. The findings suggest that real-time aspects of speaking, like how quickly someone produces language, are as important, if not more so, than the complexity of their sentence structures in determining perceived proficiency.

What Problem Does This Paper Address?

Assessing second language oral proficiency is a complex task. Traditional methods often rely on holistic human judgments that combine various linguistic cues. While linguistic accuracy and complexity are known factors, the role of "fluency" – specifically, the automaticity and real-time aspects of speech production – in these judgments is less clearly understood and operationalized. This paper aims to disentangle the contributions of fluency (measured by speech rate and timing) and structural complexity to L2 oral proficiency, seeking to understand which aspects are most salient to expert raters and how they might inform better assessment tools.

Methods

The researchers analyzed spontaneous speech samples from L2 English and Spanish speakers. They employed a combination of automated speech processing and manual linguistic analysis.

  • Automated Speech Processing: Used to extract phonological fluency measures such as total pause time, mean pause time, phones per second, and words per minute.
  • Manual Linguistic Analysis: Transcribed speech samples and manually counted linguistic units like words, cohesive devices, clauses, and T-units (a main clause plus any subordinate clauses or non-clausal structures attached to it). These counts were used to derive structural fluency measures (e.g., words/minute, clauses/minute) and structural complexity measures (e.g., words/clause, clauses/T-unit).

The core methodology involved correlating these automatically and manually derived measures with independent, holistic human ratings of L2 oral proficiency.

Datasets

The study utilized two datasets:

  • English (ENG) Subjects: Data collected from 58 non-native English speakers between 1998-1999. Proficiency was assessed using the Test of Spoken English (TSE). Speech samples were derived from the last section of the Versant English test, specifically from "Open Questions," providing a total of 80 seconds of recorded response time per subject.
  • Spanish (SPN) Subjects: Data collected from 38 non-native Spanish speakers between 2002-2003. Proficiency was assessed using a Spoken Proficiency Test (SPT) following Interagency Language Roundtable (ILR) descriptors. Speech samples were derived from the Versant Spanish test, including "Open Questions" and "Story Retellings," providing a total of 180 seconds of recorded response time per subject.

These datasets were collected as part of concurrent validation studies for Pearson's Versant English and Versant Spanish tests.

Key Insights and Ideas

  • Fluency is a Strong Predictor: The study found that measures of spoken fluency, particularly those related to the rate of production (e.g., words per minute, phones per second), were highly correlated with perceived oral proficiency. This suggests that how quickly and smoothly someone speaks is a significant factor in how proficient they are judged to be.
  • Automaticity Matters: The findings support the idea that automaticity, or the ability to perform language tasks without conscious attention, is a key component of oral proficiency. This automaticity is reflected in real-time speech production aspects like speed and minimal hesitation.
  • Complexity is Important, but Perhaps Less So: While structural complexity measures (like the number of words per clause) also correlated with proficiency, they did not consistently outperform fluency measures, especially in the Spanish dataset. This implies that while using complex structures is part of proficiency, the ability to produce them fluently and automatically might be more immediately apparent to listeners.
  • Language Differences: The Spanish data showed stronger correlations between fluency/complexity measures and proficiency compared to the English data. The authors hypothesize this might be due to longer speaking times in the Spanish experiment, a more constrained task (story retelling), or a more reliable independent proficiency variable.
  • Rethinking the Proficiency Construct: The paper suggests that the construct of oral proficiency might need to be enriched to explicitly include timing and automaticity, moving beyond a purely functional or structural definition.

Key Results

  • Strong Correlations: In the Spanish data, fluency measures like "Words/minute" (available time) correlated as high as 0.90 with proficiency, and "Phones/second" correlated at 0.81. Structural complexity measures like "Words/T-Unit" correlated at 0.70.
  • English Data: Correlations in the English data were generally lower but still meaningful, with "Words/minute" (available time) at 0.62 and "Words/T-Unit" at 0.67.
  • Combined Variables: When combining multiple fluency and complexity variables into regression models, the predictive power increased, but the paper notes that with limited data points, these combined models didn't always outperform the best single variables.
  • Spanish vs. English: The Spanish data consistently showed higher correlations across most measures, suggesting that fluency and complexity might be more clearly delineated predictors in that context, potentially due to experimental design differences.

Limitations and Future Work

  • Dataset Size: The authors acknowledge that the dataset sizes were relatively small, which might have affected the stability of the regression analyses and the ability of combined variable models to outperform single best predictors.
  • Operationalization of Fluency: While the study used several measures, the authors note that "automaticity" itself is difficult to pin down to a single operational definition.
  • Task Constraints: The English data was derived solely from "Open Questions," which are less constrained than "Story Retellings." The authors suggest that more constrained tasks might yield clearer relationships between linguistic features and proficiency.
  • Future Work: The paper suggests that combining linguistic analysis with timing measures can lead to more efficient and accurate spoken language tests. It also calls for a revision of the oral proficiency construct to better incorporate automaticity and real-time aspects of speaking. Future research could explore a wider range of tasks and languages, as well as more sophisticated computational models for analyzing speech.

Summary updated

BibTeX
@inproceedings{bernstein2010fluency,
  author = {Jared Bernstein and Jian Cheng and Masanori Suzuki},
  title = {{Fluency and structural complexity as predictors of L2 oral proficiency}},
  booktitle = {Interspeech},
  year = {2010},
}

Loading related content...

Figure