Neurofunctional basis underlying audiovisual integration of print and speech sound in Chinese children

Zhichao Xia, Ting Yang, Xin Cui, Fumiko Hoeft, Hong Liu, Xianglin Zhang, Hua Shu, Xiangping LiubioRxiv 2020 · DOI

TL;DR

This study investigates how the brain integrates visual text (Chinese characters and Pinyin) with spoken sounds in children, revealing both universal brain mechanisms and language-specific adaptations. It highlights how different brain regions are involved in processing opaque (characters) versus transparent (Pinyin) scripts and how these processes relate to reading comprehension and fluency.

What Problem Does This Paper Address?

Reading acquisition relies heavily on the ability to link visual symbols (print) with their corresponding sounds. While research has extensively explored this "print-sound integration" in alphabetic languages, its neural underpinnings in non-alphabetic writing systems, like Chinese, remain less understood. Chinese presents a unique challenge due to its deep orthography (arbitrary character-sound mappings) and the use of Pinyin (a phonetic system) as a learning aid. This paper aims to bridge this gap by examining how typically developing Chinese children's brains integrate visual characters and Pinyin with speech sounds, and how these neural processes relate to their reading abilities.

Methods

The researchers employed functional Magnetic Resonance Imaging (fMRI) to study brain activity in children while they performed tasks involving visual characters and Pinyin presented alongside their corresponding sounds. Two key neuroimaging indicators were used:

  • Additive Enhancement: This measures whether the brain's response to combined auditory and visual stimuli is greater than the sum of responses to each stimulus presented alone. It's thought to reflect fundamental aspects of multisensory processing.
  • Neural Integration: This assesses differences in brain activity between congruent (matching print and sound) and incongruent (mismatching print and sound) audiovisual stimuli. This is believed to reflect deeper, more specific processing of the integrated information.

Additionally, behavioral measures of reading abilities (character recognition, oral reading fluency, silent reading comprehension) and cognitive-linguistic skills were collected to explore correlations with neural findings.

Datasets

The study used a dataset of 54 typically developing Chinese children. Behavioral data was collected using standardized tests for IQ, character recognition, word list reading, silent reading comprehension, phoneme deletion, rapid naming, and morphological production. fMRI data was collected during the audiovisual integration tasks.

Key Insights and Ideas

  • Universal Print-Sound Integration: The superior temporal cortex (STC) shows a consistent pattern of "additive enhancement" for both Chinese characters and Pinyin, suggesting a shared, fundamental brain mechanism for integrating visual and auditory information across different writing systems.
  • Script-Specific Neural Integration: While the STC is involved universally, the way the brain integrates print and sound differs based on orthographic transparency.
  • Opaque Characters: Integration of characters with sounds led to a "congruent < incongruent" pattern in areas like the left middle frontal gyrus (MFG) and STC, suggesting a more complex processing that might involve conflict resolution.
  • Transparent Pinyin: For Pinyin, the expected "congruent > incongruent" pattern was not observed at the group level, but individual differences in this integration were linked to oral reading fluency.
  • Brain-Behavior Links:
    • Neural integration in the left MFG for characters was specifically linked to silent reading comprehension, suggesting its role in semantic processing.
  • Neural integration in the left STG for Pinyin was linked to oral reading fluency, highlighting its role in grapho-phonological mapping.
  • Pinyin's Role: The findings suggest that Pinyin, as a transparent phonetic system, might help children develop the neural circuits for grapho-phonological mapping, which is crucial for oral reading fluency.

Key Results

  • Additive Enhancement: Bilateral STCs showed additive enhancement for both character and Pinyin stimuli.
  • Neural Integration (Character): A "congruent < incongruent" neural integration was observed in the left STC, left inferior/middle frontal gyrus, and parietal regions.
  • Neural Integration (Pinyin): No significant group-level neural integration was found for Pinyin.
  • Brain-Behavior Correlations:
    • Neural integration in the left MFG (character) correlated with silent reading comprehension (r = 0.494, p = 0.004).
  • Neural integration in the left STG (Pinyin) correlated with oral reading fluency (r = 0.437, p = 0.006).
  • Cross-Script Overlap: The left STC showed spatial overlap in neural integration for both character and Pinyin processing, and the magnitude of this integration was significantly correlated between the two scripts (r = 0.512, p = 0.002).

Limitations and Future Work

  • Age Range: Participants were in grades 3-5. The role of Pinyin might differ in earlier stages of reading development, suggesting a need for longitudinal studies with preliterate children.
  • Stimulus Level: The study focused on syllable-level processing. Future research could explore letter-speech sound associations more directly.
  • Order Effect: Pinyin stimuli were always presented before characters to avoid priming. Future studies could counterbalance the order of presentation.
  • Statistical Thresholds: The use of a liberal voxel threshold in whole-brain analysis might inflate false positives. Larger sample sizes are recommended for replication.
  • Reading Disorder (RD): The study focused on typically developing children. Investigating whether neural processes are altered in children with RD is a crucial next step.
  • Longitudinal Research: A comprehensive picture would require longitudinal studies incorporating preliterate predictors and neural substrates of print-sound integration.

Summary updated

Abstract

Abstract Effortless print-sound integration is essential to reading development, and the superior temporal cortex (STC) is the most critical brain region. However, to date, the conclusion is almost restricted to alphabetic orthographies. To examine the neural basis in non-alphabetic languages and its relationship with reading abilities, we conducted a functional magnetic resonance imaging study in typically developing Chinese children. Two neuroimaging-based indicators of audiovisual processing—additive enhancement (higher activation in the congruent than the average activation of unimodal conditions) and neural integration (different activations between the congruent versus incongruent conditions)—were used to investigate character-sounds (opaque) and pinyin-sounds (transparent) processing. We found additive enhancement in bilateral STCs processing both character and pinyin stimulations. Moreover, the neural integrations in the left STC for the two scripts were strongly correlated. In terms of differentiation, first, areas beyond the STCs showed additive enhancement in processing pinyin-sounds. Second, while the bilateral STCs, left inferior/middle frontal and parietal regions manifested a striking neural integration (incongruent &gt; congruent) for character-sounds, no significant clusters were revealed for pinyin-sounds. Finally, the neural integration in the left middle frontal gyrus for characters was specifically associated with silent reading comprehension proficiency, indicating automatic semantic processing during implicit character-sound integration. In contrast, the neural integration in the left STC for pinyin was specifically associated with oral reading fluency that relies on grapho-phonological mapping. To summarize, this study revealed both script-universal and script-specific neurofunctional substrates of print-sound integration as well as their processing- and region-dependent associations with reading abilities in typical Chinese children. Highlights The bilateral superior temporal cortices display additive enhancement in processing print-sound information regardless of orthographies. The location, direction, and strength of the neural audiovisual integration are modulated by orthographic transparency. The neural integration in the left middle frontal gyrus in processing character-sounds is correlated to reading comprehension. The neural integration in the left superior temporal gyrus for pinyin is correlated with oral reading fluency. The neural integrations of character-sounds and pinyin-sounds show spatial overlap and inter-subject correlation in the left superior temporal cortex.
BibTeX
@misc{xia2020neurofunctional,
  author = {Zhichao Xia and Ting Yang and Xin Cui and Fumiko Hoeft and Hong Liu and Xianglin Zhang and Hua Shu and Xiangping Liu},
  title = {{Neurofunctional basis underlying audiovisual integration of print and speech sound in Chinese children}},
  year = {2020},
  doi = {10.1101/2020.05.31.126128},
  howpublished = {bioRxiv},
}
Figure