Effects of Semantic Context and Fundamental Frequency Contours on Mandarin Speech Recognition by Second Language Learners

Linjun Zhang, Yu Li, Han Wu, Xin Li, Hua Shu, Yang Zhang, Ping LiFrontiers in Psychology 2016 · DOI

Abstract

Speech recognition by second language (L2) learners in optimal and suboptimal conditions has been examined extensively with English as the target language in most previous studies. This study extended existing experimental protocols to investigate Mandarin speech recognition by Japanese learners of Mandarin at two different levels (elementary vs. intermediate) of proficiency. The overall results showed that in addition to L2 proficiency, semantic context, F0 contours, and listening condition all affected the recognition performance on the Mandarin sentences. However, the effects of semantic context and F0 contours on L2 speech recognition diverged to some extent. Specifically, there was significant modulation effect of listening condition on semantic context, indicating that L2 learners made use of semantic context less efficiently in the interfering background than in quiet. In contrast, no significant modulation effect of listening condition on F0 contours was found. Furthermore, there was significant interaction between semantic context and F0 contours, indicating that semantic context becomes more important for L2 speech recognition when F0 information is degraded. None of these effects were found to be modulated by L2 proficiency. The discrepancy in the effects of semantic context and F0 contours on L2 speech recognition in the interfering background might be related to differences in processing capacities required by the two types of information in adverse listening conditions.

Introduction

Speech recognition is significantly more challenging for second-language (L2) learners than for native speakers, particularly in "suboptimal" conditions such as noisy environments. While this phenomenon has been studied extensively in English, Mandarin Chinese presents a unique linguistic challenge. As a tonal language, Mandarin utilizes pitch (fundamental frequency or F0) not just for prosodic emphasis, but to distinguish the lexical meaning of words. This study explores how Japanese learners of Mandarin navigate the "cocktail party problem," specifically examining how they utilize sentence meaning (semantic context) and pitch (F0 contours) to decode speech in the presence of interference.

Methods and Materials

The study recruited 59 native Japanese speakers learning Mandarin, categorized into elementary (freshmen) and intermediate (juniors) proficiency groups. Researchers employed a $2 \times 2 \times 2 \times 2$ mixed-design experiment to isolate the following variables:

  • Semantic Context: Normal sentences (semantically coherent) vs. pseudo-random "word lists" (syntactically and semantically nonsensical).
  • F0 Contours: Natural pitch vs. "flat" pitch (monotonous speech resynthesized using the PSOLA method).
  • Listening Condition: Quiet vs. an interfering background (+5 dB SNR). The interference consisted of "consonant-misplaced" sentences—speech that sounds like Mandarin but is gibberish, providing informational masking.
  • Proficiency: Elementary vs. Intermediate learners.
Figure 5: Examples of experimental stimuli showing natural vs. flat F0 contours for normal sentences (A) and pseudo-random word lists (B).

Participants listened to these stimuli and transcribed what they heard using Pinyin. Accuracy was quantified based on keyword-correct counts.

Key Insights and Results

The Cognitive Bottleneck

A primary finding of the study is that L2 learners' ability to utilize semantic context decreased in the presence of an interfering background. This stands in contrast to native speakers, who typically lean more heavily on context to fill gaps caused by noise. This suggests a cognitive bottleneck: the effortful nature of L2 sound decoding in noise leaves fewer cognitive resources available for higher-level semantic synthesis.

Pitch as a Robust Cue

Natural pitch contours significantly improved recognition accuracy regardless of whether the listening environment was quiet or noisy. This indicates that even at elementary proficiency levels, Japanese learners of Mandarin have developed a sensitivity to lexical tones and prosody that remains functional despite environmental noise.

Redundancy and Compensation

The data revealed a significant interaction between context and pitch ($F(1,55) = 4.813, p = 0.032$). When pitch information was degraded (the "flat" condition), learners relied more heavily on semantic context to identify words. This demonstrates the brain's flexibility in switching between acoustic and linguistic cues when one source of information becomes unreliable.

Proficiency Plateau

While intermediate learners performed significantly better overall than elementary learners ($F(1,58) = 41.707, p < 0.001$), proficiency level did not change the fundamental way these factors interacted. Both groups showed the same patterns of cue usage, suggesting that the strategies for integrating pitch and context may stabilize relatively early in the learning process.

Conclusion and Future Directions

The study highlights that for L2 learners, semantic context is a fragile resource in noisy environments, whereas F0 contours provide a more robust anchor for speech recognition. However, several factors warrant further investigation. Given that Japanese is a pitch-accent language, the participants may have an inherent sensitivity to F0 that learners from non-tonal backgrounds (like English) might lack. Additionally, the specific type of noise—in this case, a single female talker—may produce different results than multi-talker "babble" or white noise. Future research should compare learners across diverse native language backgrounds to determine the universality of these processing strategies.

Summary updated

Abstract

Speech recognition by second language (L2) learners in optimal and suboptimal conditions has been examined extensively with English as the target language in most previous studies. This study extended existing experimental protocols (Wang et al., 2013) to investigate Mandarin speech recognition by Japanese learners of Mandarin at two different levels (elementary vs. intermediate) of proficiency. The overall results showed that in addition to L2 proficiency, semantic context, F0 contours, and listening condition all affected the recognition performance on the Mandarin sentences. However, the effects of semantic context and F0 contours on L2 speech recognition diverged to some extent. Specifically, there was significant modulation effect of listening condition on semantic context, indicating that L2 learners made use of semantic context less efficiently in the interfering background than in quiet. In contrast, no significant modulation effect of listening condition on F0 contours was found. Furthermore, there was significant interaction between semantic context and F0 contours, indicating that semantic context becomes more important for L2 speech recognition when F0 information is degraded. None of these effects were found to be modulated by L2 proficiency. The discrepancy in the effects of semantic context and F0 contours on L2 speech recognition in the interfering background might be related to differences in processing capacities required by the two types of information in adverse listening conditions.
BibTeX
@inproceedings{zhang2016effects,
  author = {Linjun Zhang and Yu Li and Han Wu and Xin Li and Hua Shu and Yang Zhang and Ping Li},
  title = {{Effects of Semantic Context and Fundamental Frequency Contours on Mandarin Speech Recognition by Second Language Learners}},
  booktitle = {Frontiers in Psychology},
  year = {2016},
  doi = {10.3389/fpsyg.2016.00908},
}
Figure