NISP: A Multi-lingual Multi-accent Dataset for Speaker Profiling

Shareef Babu Kalluri, Deepu Vijayasenan, Sriram Ganapathy, Ragesh Rajan M, Prashant Krishnan V

TL;DR

Researchers from NITK Surathkal and IISc Bangalore have introduced the NISP (NITK-IISc Multilingual Multi-accent Speaker Profiling) dataset, a rich collection of speech from 345 speakers across five Indian languages and Indian-accented English. Unlike previous datasets, NISP includes detailed physical metadata—such as height, weight, and even shoulder width—enabling more robust "speaker profiling" systems that can estimate a person's physical traits from their voice alone.

What Problem Does This Paper Address?

"Speaker profiling" is the art of extracting a speaker's characteristics—like age, gender, native language, and physical traits—from a speech signal. This has massive implications for voice forensics (identifying a caller) and commercial applications (smart speakers tailoring content to a specific user).

However, the field faces a major data bottleneck. Most existing speech datasets are either:

  1. Monolingual: Primarily focused on English.
  2. Metadata-Poor: They might have age and gender but lack physical parameters like height or weight.
  3. Accent-Blind: They don't capture the nuances of how native languages (L1) influence a second language (L2), like English.

The authors tackle this by creating a dataset specifically designed for the linguistic and physical diversity of the Indian subcontinent.

Methods

To demonstrate the utility of the NISP dataset, the authors established baseline results for estimating physical parameters using Support Vector Regression (SVR).

They extracted three main types of features from the audio:

  • Fstat: First-order statistics from 40 Mel filter bank features (using a GMM-UBM framework).
  • Formants: Spectral peaks that represent the resonance of the vocal tract.
  • Harmonics: Features including frequency locations and amplitude, which relate to the source of the voice (the vocal folds).

By combining these features (a method they call Comb-3), they aimed to see if a machine could predict a speaker's height, weight, age, and shoulder width more accurately than a simple "mean predictor" (which just guesses the average of the training data).

Datasets

The core contribution is the NISP dataset:

  • Speakers: 345 individuals (219 male, 126 female).
  • Languages: Hindi, Kannada, Malayalam, Telugu, Tamil, and Indian-accented English.
  • Content: ~57 hours of total speech, including unique sentences, continuous stories, and standard TIMIT sentences.
  • Metadata: Native language, schooling language, regional info, age, gender, height, weight, and shoulder width (measured between acromion bones).

The paper compares NISP to existing benchmarks like TIMIT, SRE, and Fisher English, noting that NISP is unique in providing both multilingual data and a wide array of physical traits.

Key Insights and Ideas

  • The "Shoulder Width" Factor: While height and weight are common in some niche datasets, NISP includes shoulder width. This is an interesting proxy for body build that could provide unique acoustic cues related to the chest cavity and resonance.
  • Multilingual Robustness: By providing the same speaker's voice in both their native language and English, the dataset allows researchers to study how "speaker-specific" traits (like height) persist across different languages.
  • Feature Fusion is Key: The results show that spectral features (formants) and source features (harmonics) are complementary. Using them together significantly reduces the Mean Absolute Error (MAE) compared to using them in isolation.

Key Results

The baseline experiments proved that voice-based physical estimation is viable:

  • Body Build Estimation: For height, weight, and shoulder width, the proposed model improved error rates by 22–29% compared to the mean predictor baseline.
  • Age Estimation: The model showed a 14% improvement in MAE for age prediction.
  • Specific Performance: For the "All" category (male and female combined), the system achieved a Mean Absolute Error of 5.11 cm for height and 7.06 kg for weight.

Limitations and Future Work

While NISP is a significant step forward, it has some limitations:

  • Scale: With 345 speakers, it is a "mid-sized" dataset. While excellent for profiling, it is smaller than massive ASR (Automatic Speech Recognition) datasets.
  • Geographic Focus: The speakers are primarily from Southern India and Hindi-speaking regions. Expanding to other Indian states would further increase the "multi-accent" utility.
  • Environment: The recordings were made in quiet environments. Future work will likely need to address how these physical trait estimations hold up in noisy, real-world "in-the-wild" conditions.

The authors suggest that NISP will be a valuable resource for fine-tuning large-scale speaker recognition models to be more robust against multi-accent and multi-lingual variations.

Summary updated

BibTeX
@inproceedings{kalluri2021nisp,
  author = {Shareef Babu Kalluri and Deepu Vijayasenan and Sriram Ganapathy and Ragesh Rajan M and Prashant Krishnan V},
  title = {{NISP: A Multi-lingual Multi-accent Dataset for Speaker Profiling}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2021},
  doi = {10.1109/icassp39728.2021.9414349},
}
Figure