Temporally Aligning Long Audio Interviews with Questions: A Case Study in Multimodal Data Integration

Piyush Singh Pasi, Karthikeya Battepati, Preethi Jyothi, Ganesh Ramakrishnan, Tanmay Mahapatra, Manoj Singh

Abstract

The problem of audio-to-text alignment has seen significant research using complete supervision. However, this is typically not in the context of long audio recordings where the text being queried does not appear verbatim. This work, a collaboration with CARE India, focuses on long audio health surveys from rural Bihar, India. We aim to locate where a question from a questionnaire is asked within a long audio recording. Our proposed framework, INDENT, uses a cross-attention-based model and prior information on temporal ordering to learn speech embeddings capturing underlying semantics. These embeddings retrieve audio segments based on text queries at inference time. We demonstrate a 3% improvement in R-avg over text-based heuristics and show that noisy ASR can enhance results. INDENT, trained on Hindi, generalizes to 11 Indic languages via a shared semantic space.

TL;DR

Researchers have developed INDENT, a weakly-supervised framework designed to automatically locate where specific questions are asked within long, noisy audio interviews. By leveraging the fixed temporal order of survey questionnaires and cross-modal embeddings, the system helps NGOs quickly navigate hours of field recordings to extract vital health and social data.

What Problem Does This Paper Address?

In many developing regions, NGOs like CARE India conduct extensive audio surveys to collect health data from rural populations. These interviews are often long (40+ minutes), recorded in noisy home environments, and conducted in local dialects.

The core challenge is temporal alignment: given a written question from a master questionnaire, how do you find the exact moment it was asked in the audio? This is difficult because:

  1. Non-verbatim speech: Interviewers often paraphrase questions rather than reading them word-for-word.
  2. Noise and Dialects: Background noise and regional linguistic variations make standard speech-to-text difficult.
  3. Label Scarcity: Manually annotating thousands of hours of audio with precise timestamps is prohibitively expensive.
The challenge of aligning non-verbatim spoken text with fixed questionnaire items.

Methods

The authors propose INDENT (alIgning loNg auDio intErviews aNd quesTions), which uses a multimodal approach to bridge the gap between text queries and raw audio.

The INDENT architecture, featuring a Speech Encoder and a Question Encoder linked by Gaussian-weighted cross-attention.
  • Speech Encoder: Uses a pretrained wav2vec 2.0 model (specifically IndicWav2vec) to extract features from audio chunks. These chunks are processed through self-attention layers to capture local context.
  • Question Encoder: Uses LaBSE (Language-agnostic BERT Sentence Embedding), which maps sentences from different languages into a shared semantic space.
  • Gaussian-weighted Cross-Attention: This is the "secret sauce." Since the survey follows a fixed questionnaire, the model knows the order of questions. It uses a Gaussian distribution to "anchor" specific audio chunks to specific questions based on their relative position in the sequence, allowing the model to learn alignments without needing exact timestamps during training (weak supervision).
  • Contrastive Learning: The model is trained to minimize the distance between corresponding audio-text pairs while pushing unrelated pairs apart in the embedding space.

Datasets

The study introduces the CARE India Dataset, consisting of health surveys from young mothers in rural Bihar, India.

  • Training set: 34 interviews (~24 hours) with "coarse-grained" annotations (knowing which set of 5 questions appears in a specific 50-second segment).
  • Evaluation: Small, "fine-grained" development and test sets where every question is manually timestamped.
  • Multilingual Test: The model was tested on translations of the questionnaire in 11 Indic languages (e.g., Bengali, Tamil, Telugu) to verify its cross-lingual capabilities.

Key Insights and Ideas

  • Temporal Priors Matter: By assuming that questions in an interview roughly follow the order of the questionnaire, the authors could use a moving Gaussian mean to guide the attention mechanism. This significantly simplifies the learning process for long-form audio.
  • Cross-Lingual "Freebies": Because the model uses LaBSE for text, a model trained only on Hindi audio/text pairs can successfully retrieve segments using queries in other languages like Punjabi or Malayalam without any additional training.
  • Weak Supervision is Viable: You don't need second-by-second labels. Knowing that "these five questions happen somewhere in this one-minute block" is enough for the model to learn the underlying semantic mappings.

Key Results

  • Performance Gain: INDENT outperformed a "NO-TRAIN" baseline (simple dot-product matching) by an absolute 6.4% in R-avg (average Recall@k).
  • Audio vs. ASR: The authors tested a variant called INDENT-T, which uses ASR (Automatic Speech Recognition) transcripts instead of raw audio. While INDENT-T performed better (63.1% R-avg vs. 43.1% for raw audio), the raw audio model remains crucial for low-resource languages where high-quality ASR doesn't exist yet.
  • Robustness: The system proved effective across 11 different Indic languages, demonstrating that the semantic alignment holds even when the query language changes.

Limitations and Future Work

  • The "Verbatim" Gap: While the model handles paraphrasing, there is still a significant performance gap between using raw audio and using ASR transcripts. Bridging this "modality gap" remains a primary challenge.
  • Fixed Ordering: The current model relies heavily on the interviewer following the questionnaire's sequence. If an interviewer skips around or revisits questions out of order, the Gaussian temporal prior might struggle.
  • Future Extensions: The authors suggest extending this work to "audio-driven form filling," where the system not only finds the question but also automatically extracts the answer to populate a database.

Summary updated

Abstract

The problem of audio-to-text alignment has seen significant amount of research using complete supervision during training. However, this is typically not in the context of long audio recordings wherein the text being queried does not appear verbatim within the audio file. This work is a collaboration with a non-governmental organization called CARE India that collects long audio health surveys from young mothers residing in rural parts of Bihar, India. Given a question drawn from a questionnaire that is used to guide these surveys, we aim to locate where the question is asked within a long audio recording. This is of great value to African and Asian organizations that would otherwise have to painstakingly go through long and noisy audio recordings to locate questions (and answers) of interest. Our proposed framework, INDENT, uses a cross-attention-based model and prior information on the temporal ordering of sentences to learn speech embeddings that capture the semantics of the underlying spoken text. These learnt embeddings are used to retrieve the corresponding audio segment based on text queries at inference time. We empirically demonstrate the significant effectiveness (improvement in R-avg of about 3%) of our model over those obtained using text-based heuristics. We also show how noisy ASR, generated using state-of-the-art ASR models for Indian languages, yields better results when used in place of speech. INDENT, trained only on Hindi data is able to cater to all languages supported by the (semantically) shared text space. We illustrate this empirically on 11 Indic languages.
BibTeX
@inproceedings{pasi2023temporally,
  author = {Piyush Singh Pasi and Karthikeya Battepati and Preethi Jyothi and Ganesh Ramakrishnan and Tanmay Mahapatra and Manoj Singh},
  title = {{Temporally Aligning Long Audio Interviews with Questions: A Case Study in Multimodal Data Integration}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2023},
  url = {https://openreview.net/forum?id=IO8Vj3yfrE},
}
Figure