Abstract: Nonverbal vocalizations, such as sighs, grunts, and yells, are informative expressions within typical verbal speech. Likewise, individuals who produce 0-10 spoken words or word approximations ("minimally speaking" individuals) convey rich affective and communicative information through nonverbal vocalizations even without verbal speech. Yet, despite their rich content, little to no data exists on the vocal expressions of this population. Here, we present ReCANVo: Real-World Communicative and Affective Nonverbal Vocalizations - a novel dataset of non-speech vocalizations labeled by function from minimally speaking individuals. The ReCANVo database contains over 7000 vocalizations spanning communicative and affective functions from eight minimally speaking individuals, along with communication profiles for each participant. Vocalizations were recorded in real-world settings and labeled in real-time by a close family member who knew the communicator well and had access to contextual information while labeling. ReCANVo is a novel database of nonverbal vocalizations from minimally speaking individuals, the largest available dataset of nonverbal vocalizations, and one of the only affective speech datasets collected amidst daily life across contexts.
Introduction: Addressing the Data Gap in Nonverbal Communication
For the approximately 1–2 million individuals in the United States who are "minimally speaking"—defined as producing fewer than 10 spoken words or word approximations—communication is a rich, nonverbal process. These individuals, often diagnosed with autism, cerebral palsy, or various genetic disorders, rely heavily on vocalizations such as sighs, grunts, and yells to convey their needs, emotions, and social intents.
Despite the communicative depth of these sounds, they have been largely overlooked by the fields of speech recognition and affective computing. Most existing vocalization datasets rely on professional actors in controlled laboratory settings, pre-verbal infants, or incidental sounds from typical speakers. There is a profound lack of "in the wild" data representing how minimally speaking individuals (referred to as mv* individuals) communicate in their daily lives. ReCANVo (Real-World Communicative and Affective Nonverbal Vocalizations) was developed to bridge this gap, providing a foundation for assistive technologies that can interpret these unique vocal signatures.
Methodology: Longitudinal Participatory Design
The researchers employed a longitudinal, participatory design to ensure the collection of ecologically valid data. This approach prioritized capturing vocalizations during regular daily activities rather than simulated tasks.
Data Collection Hardware and Software
Participants wore a lightweight Sony audio recorder attached to their clothing or placed in close proximity. To capture the intent behind the sounds, the study leveraged the expertise of those who know the communicators best: their family members. Caregivers used a custom smartphone application to label vocalizations in real-time as they occurred.
Labeling Schema
The labeling app featured six fixed categories: Frustration, Delight, Dysregulation, Self-talk, Request, and Social. Additionally, four customizable slots allowed families to include idiosyncratic labels specific to the participant, such as "Hungry," "Tablet," or "Yes." This flexibility was crucial for capturing the highly personalized nature of mv* communication.
Post-Processing and Alignment
To synchronize continuous audio recordings with discrete app "taps," the team utilized volume-based filters to isolate vocal segments. A set of five heuristic rules was then applied to ensure accurate alignment. For instance, an audio segment was assigned a specific label if it concluded during the duration of the label press on the smartphone app.
The ReCANVo Database
The primary output of this research is the ReCANVo database, which represents a significant expansion of available resources for this population.
- Scope: The dataset contains 7,077 labeled vocalizations from eight participants ranging in age from 6 to 23 years.
- Duration: Data collection was extensive, spanning between 4 and 64 weeks per participant to capture a wide variety of contexts and emotional states.
- Public Availability: To foster further research in the machine learning and assistive technology communities, the dataset—including audio files and detailed communication profiles—is hosted publicly on Zenodo.
ReCANVo distinguishes itself from benchmarks like OxVoc (infant/animal sounds) or IEMOCAP (acted emotional speech) by its focus on naturalistic, real-world data from the mv* population.
Key Insights: The Importance of Context and Heterogeneity
A central theme of the study is that context is paramount. Vocalizations are often ambiguous; a yell might signify excitement in a familiar living room but distress in a crowded, noisy environment. By utilizing real-time labels from familiar partners who have access to the immediate context, the researchers established a "ground truth" that would be impossible to achieve through retrospective lab analysis.
Furthermore, the data highlights extreme heterogeneity among participants. For example, while one participant (P08) contributed 781 clips labeled as "frustrated," another (P02) had none. This variability underscores the necessity for personalized, rather than generalized, AI models in the development of future communication aids.
Results and Validation
The scale of ReCANVo—over 7,000 samples—makes it the largest dataset of its kind. The results demonstrate that nonverbal vocalizations cover a full spectrum of expression, not just negative outbursts. "Self-talk" (1,882 clips) and "Delight" (1,296 clips) were among the most frequent categories recorded.
To validate the caregiver labels, researchers manually reviewed every clip, using surrounding audio context (such as a parent’s verbal response) to confirm the accuracy of the real-time intent labels.
Limitations and Future Directions
While ReCANVo is a landmark dataset, the authors acknowledge several challenges inherent in "wild" data collection. The sample size of eight individuals, while deep in terms of longitudinal data, needs to be expanded to a larger cohort to improve generalization. Additionally, the labels remain third-party interpretations, and real-world recordings are often subject to technical noise (e.g., wind, background media).
The ultimate goal for ReCANVo is to serve as a springboard for the development of "vocalization-to-intent" systems. By providing high-quality, labeled data, the researchers hope to empower the machine learning community to create tools that give a digital voice to those who do not use traditional speech.
Summary updated