Abstract
The problem of audio-to-text alignment has seen significant research using complete supervision. However, this is typically not in the context of long audio recordings where the text being queried does not appear verbatim. This work, a collaboration with CARE India, focuses on long audio health surveys from rural Bihar, India. We aim to locate where a question from a questionnaire is asked within a long audio recording. Our proposed framework, INDENT, uses a cross-attention-based model and prior information on temporal ordering to learn speech embeddings capturing underlying semantics. These embeddings retrieve audio segments based on text queries at inference time. We demonstrate a 3% improvement in R-avg over text-based heuristics and show that noisy ASR can enhance results. INDENT, trained on Hindi, generalizes to 11 Indic languages via a shared semantic space.
TL;DR
Researchers have developed INDENT, a weakly-supervised framework designed to automatically locate where specific questions are asked within long, noisy audio interviews. By leveraging the fixed temporal order of survey questionnaires and cross-modal embeddings, the system helps NGOs quickly navigate hours of field recordings to extract vital health and social data.
What Problem Does This Paper Address?
In many developing regions, NGOs like CARE India conduct extensive audio surveys to collect health data from rural populations. These interviews are often long (40+ minutes), recorded in noisy home environments, and conducted in local dialects.
The core challenge is temporal alignment: given a written question from a master questionnaire, how do you find the exact moment it was asked in the audio? This is difficult because:
- Non-verbatim speech: Interviewers often paraphrase questions rather than reading them word-for-word.
- Noise and Dialects: Background noise and regional linguistic variations make standard speech-to-text difficult.
- Label Scarcity: Manually annotating thousands of hours of audio with precise timestamps is prohibitively expensive.
Methods
The authors propose INDENT (alIgning loNg auDio intErviews aNd quesTions), which uses a multimodal approach to bridge the gap between text queries and raw audio.
- Speech Encoder: Uses a pretrained wav2vec 2.0 model (specifically IndicWav2vec) to extract features from audio chunks. These chunks are processed through self-attention layers to capture local context.
- Question Encoder: Uses LaBSE (Language-agnostic BERT Sentence Embedding), which maps sentences from different languages into a shared semantic space.
- Gaussian-weighted Cross-Attention: This is the "secret sauce." Since the survey follows a fixed questionnaire, the model knows the order of questions. It uses a Gaussian distribution to "anchor" specific audio chunks to specific questions based on their relative position in the sequence, allowing the model to learn alignments without needing exact timestamps during training (weak supervision).
- Contrastive Learning: The model is trained to minimize the distance between corresponding audio-text pairs while pushing unrelated pairs apart in the embedding space.
Datasets
The study introduces the CARE India Dataset, consisting of health surveys from young mothers in rural Bihar, India.
- Training set: 34 interviews (~24 hours) with "coarse-grained" annotations (knowing which set of 5 questions appears in a specific 50-second segment).
- Evaluation: Small, "fine-grained" development and test sets where every question is manually timestamped.
- Multilingual Test: The model was tested on translations of the questionnaire in 11 Indic languages (e.g., Bengali, Tamil, Telugu) to verify its cross-lingual capabilities.
Key Insights and Ideas
- Temporal Priors Matter: By assuming that questions in an interview roughly follow the order of the questionnaire, the authors could use a moving Gaussian mean to guide the attention mechanism. This significantly simplifies the learning process for long-form audio.
- Cross-Lingual "Freebies": Because the model uses LaBSE for text, a model trained only on Hindi audio/text pairs can successfully retrieve segments using queries in other languages like Punjabi or Malayalam without any additional training.
- Weak Supervision is Viable: You don't need second-by-second labels. Knowing that "these five questions happen somewhere in this one-minute block" is enough for the model to learn the underlying semantic mappings.
Key Results
- Performance Gain: INDENT outperformed a "NO-TRAIN" baseline (simple dot-product matching) by an absolute 6.4% in R-avg (average Recall@k).
- Audio vs. ASR: The authors tested a variant called INDENT-T, which uses ASR (Automatic Speech Recognition) transcripts instead of raw audio. While INDENT-T performed better (63.1% R-avg vs. 43.1% for raw audio), the raw audio model remains crucial for low-resource languages where high-quality ASR doesn't exist yet.
- Robustness: The system proved effective across 11 different Indic languages, demonstrating that the semantic alignment holds even when the query language changes.
Limitations and Future Work
- The "Verbatim" Gap: While the model handles paraphrasing, there is still a significant performance gap between using raw audio and using ASR transcripts. Bridging this "modality gap" remains a primary challenge.
- Fixed Ordering: The current model relies heavily on the interviewer following the questionnaire's sequence. If an interviewer skips around or revisits questions out of order, the Gaussian temporal prior might struggle.
- Future Extensions: The authors suggest extending this work to "audio-driven form filling," where the system not only finds the question but also automatically extracts the answer to populate a database.
Summary updated