NISP: A Multi-lingual Multi-accent Dataset for Speaker Profiling

Shareef Babu Kalluri, Deepu Vijayasenan, Sriram Ganapathy, Ragesh Rajan M, Prashant Krishnan V

Introduction

Speaker profiling involves extracting a speaker's characteristics—such as age, gender, native language, and physical traits—from a speech signal. This field has significant implications for voice forensics and commercial applications like personalized smart assistants. However, existing datasets often lack linguistic diversity and detailed physical metadata. Researchers from NITK Surathkal and IISc Bangalore have introduced the NISP (NITK-IISc Multilingual Multi-accent Speaker Profiling) dataset to address these gaps.

The NISP Dataset

The NISP dataset is designed to capture the linguistic and physical diversity of the Indian subcontinent. It includes speech from 345 individuals (219 male, 126 female) across five Indian languages (Hindi, Kannada, Malayalam, Telugu, Tamil) and Indian-accented English.

Distribution of utterances and speech duration across languages in the NISP dataset.

The corpus contains approximately 57 hours of total speech, featuring unique sentences, continuous stories, and standard TIMIT sentences. Crucially, NISP provides extensive metadata, including native language, schooling language, regional information, age, gender, height, weight, and shoulder width. This inclusion of physical parameters like shoulder width—a proxy for body build—enables the study of acoustic cues related to chest cavity resonance.

Methodology

To establish baseline results for physical parameter estimation, the authors utilized Support Vector Regression (SVR). They extracted three primary types of audio features:

  • Fstat: First-order statistics from 40 Mel filter bank features.
  • Formants: Spectral peaks representing vocal tract resonance.
  • Harmonics: Frequency locations and amplitudes related to the vocal folds.

The study explored the fusion of these features (referred to as Comb-3) to determine if combining spectral and source features could improve prediction accuracy for height, weight, age, and shoulder width compared to a simple mean predictor baseline.

Key Results and Insights

The baseline experiments demonstrated that voice-based physical estimation is a viable task. The proposed model achieved significant improvements over the mean predictor baseline:

  • Physical Traits: Error rates for height, weight, and shoulder width were reduced by 22–29%.
  • Age Estimation: The model showed a 14% improvement in Mean Absolute Error (MAE).
Mean Absolute Error (MAE) for height and weight prediction across different gender categories.

For the combined "Both" category (male and female), the system achieved an MAE of 5.11 cm for height and 7.06 kg for weight. A key insight was that spectral features (formants) and source features (harmonics) are complementary; their fusion significantly reduces prediction error.

Conclusion and Future Work

While NISP is a substantial contribution, it is currently a mid-sized dataset with a geographic focus primarily on Southern India and Hindi-speaking regions. Future expansions to other Indian states would enhance its multi-accent utility. Additionally, while the current recordings were made in controlled environments, future research must address the robustness of these estimations in noisy, "in-the-wild" conditions. NISP serves as a valuable resource for developing speaker recognition models that are more resilient to multi-accent and multi-lingual variations.

Summary updated

BibTeX
@inproceedings{kalluri2021nisp,
  author = {Shareef Babu Kalluri and Deepu Vijayasenan and Sriram Ganapathy and Ragesh Rajan M and Prashant Krishnan V},
  title = {{NISP: A Multi-lingual Multi-accent Dataset for Speaker Profiling}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2021},
  doi = {10.1109/icassp39728.2021.9414349},
}
Figure