A Computational Model of Early Language Acquisition from Audiovisual Experiences of Young Infants

Okko Räsänen, Khazar KhorramiInterspeech 2019

TL;DR

This paper introduces a novel computational model that learns word segmentation and meaning from noisy, real-world audiovisual infant-caregiver interactions. By treating visual attention as a weak label for speech, the model demonstrates that infants can begin to acquire lexical knowledge and even develop phonetic selectivity without explicit supervision.

What Problem Does This Paper Address?

Understanding how infants learn language is a fundamental question in cognitive science and AI. A key challenge is the "segmentation problem": how do infants learn to identify individual words within a continuous stream of speech, and how do they then map these words to their meanings? While previous research has explored statistical learning and audiovisual cues, it often relied on highly controlled laboratory settings or simplified datasets. This paper tackles the gap of testing these theories on more ecologically valid data – real infant-caregiver interactions – which are inherently noisy, unconstrained, and lack perfect alignment between speech and visual events. The authors aim to show that a computational model can learn early language skills from such realistic experiences, potentially revealing how infants bootstrap their language acquisition without pre-existing linguistic knowledge.

Methods

The core of the paper is a Convolutional Recurrent Neural Network (CRNN) designed for weakly-supervised audiovisual learning. The model takes log-Mel spectrograms of speech as input.

  • Speech Encoding: Convolutional layers extract increasingly complex spectrotemporal patterns from the audio. A recurrent GRU layer then models the temporal evolution of these patterns.
  • Weak Supervision: The model is trained using two loss functions:
  • Visual Prediction (VP) Loss: This is the primary learning signal. It encourages the model to predict visual objects that the infant attended to during an utterance. Crucially, the visual labels are only aligned at the utterance level, not at the word level, making it a weakly-supervised task.
  • Autoencoder (AE) Loss: An auxiliary loss function that reconstructs the original audio input. This helps to ensure that the learned representations are rich and capture general acoustic features, and it also affects all parameters of the network, unlike the temporal max-pooling used for VP.
  • Temporal Bottleneck: A max-pooling layer is used to downsample the GRU output, enforcing temporal contiguity and helping to resolve temporal ambiguity.
  • Phone Selectivity Analysis: After training, the authors analyze the activations in the hidden layers of the CRNN to see if they develop selectivity towards phonetic categories, mimicking supervised phone recognition models.

What's novel here is the combination of a CRNN architecture with utterance-level visual labels from real infant-caregiver interactions, and the use of an auxiliary AE loss to improve learning robustness.

Datasets

  • SEEDLingS Corpus: This is the primary dataset used for training and evaluation. It consists of audio and video recordings from head-mounted cameras worn by infants interacting with their caregivers in their homes. The data used was from 6- and 7-month-old infants.
  • Annotation: Utterances containing concrete, imageable nouns were hand-labeled. Crucially, labels indicated whether the mentioned object was present and attended by the infant (yes/no). For cases where the infant did not attend to an object, random visual labels were sampled from a set of 60 object classes.
  • Data Split: 80% of utterances were used for training, 20% for validation, and the remaining 20% for testing.
  • CAREGIVER Y2 UK Corpus: This is a studio-recorded corpus of enacted child-directed speech. It was used to verify the model's performance on more idealized data and to compare against established benchmarks. Utterances come with unordered bags of visual labels.
  • Data Split: 80% for training, 20% for validation, and 20% for testing.
  • TIMIT Corpus: Used for analyzing the phonetic selectivity of the trained CRNN model.

Key Insights and Ideas

  • Real-world data is challenging but learnable: The model demonstrates that even with noisy, unconstrained, and imperfectly aligned audiovisual data from real infant interactions, early lexical learning is possible. This moves beyond idealized lab settings.
  • Audiovisual bootstrapping works: The results strongly support the hypothesis that infants can use concurrent speech and visual information to bootstrap language acquisition, even when the visual cues are weak and only aligned at the utterance level.
  • The AE loss is crucial for real-world data: While the CAREGIVER corpus showed good performance even without the AE loss, the SEEDLingS results highlighted its necessity for successful learning on more challenging, naturalistic data. This suggests the AE loss helps the model learn more robust and generalizable representations.
  • Phonetic selectivity emerges implicitly: The model, trained solely for word learning and meaning acquisition, shows a gradual increase in phonetic selectivity in its deeper layers. This suggests that learning word meanings from audiovisual input can implicitly lead to the development of phonetic representations, potentially reducing the need for a separate, supervised phonetic learning module.

Key Results

  • CAREGIVER Corpus: The CRNN model achieved state-of-the-art performance, with F-scores of ≥ 0.96 on word detection, significantly outperforming the CM baseline (F-score of 0.81). The AE loss was beneficial but not strictly essential for high performance on this cleaner dataset.
  • SEEDLingS Corpus: Performance was more modest, reflecting the data's complexity. The CRNN with the AE loss achieved the best results, outperforming variants without the AE loss or temporal bottleneck. Word detection F-scores for individual words ranged up to 0.4-0.5.
  • Confusions: The model exhibited some interesting confusions, including semantic-like confusions (e.g., "nose" with "mouth"), suggesting it was learning more than just arbitrary acoustic-visual pairings.
  • Phonetic Selectivity: The analysis of hidden layer activations showed a pattern of increasing phonetic selectivity in deeper convolutional layers, similar to supervised phone recognition models. This selectivity was highest in the decoder part of the autoencoder.

Limitations and Future Work

  • Simplified Input: The SEEDLingS data used for experiments was limited to utterances with concrete nouns and did not include all types of linguistic input (e.g., verbs, abstract concepts). The visual input was also treated as categorical and invariant.
  • Limited Age Range: The study focused on 6- and 7-month-old infants, a relatively early stage of language development.
  • Future Work: The authors propose substituting the categorical visual labels with actual visual input captured by head-mounted cameras. This would provide a more complete and realistic simulation of infant experience and allow for a deeper understanding of how infants learn from the richness of their multimodal environment. They also suggest the model's applicability to adult speech perception and phonemic categorization.

Summary updated

BibTeX
@inproceedings{rasanen2019computational,
  author = {Okko Räsänen and Khazar Khorrami},
  title = {{A Computational Model of Early Language Acquisition from Audiovisual Experiences of Young Infants}},
  booktitle = {Interspeech},
  year = {2019},
}
Figure