Abstract
The problem of audio-to-text alignment has seen significant research using complete supervision during training. However, this is typically not in the context of long audio recordings wherein the text being queried does not appear verbatim within the audio file. This work is a collaboration with CARE India, an NGO that collects long audio health surveys from mothers in rural Bihar, India. Given a question from a questionnaire, we aim to locate its occurrence within a long audio recording. Our proposed framework, INDENT, uses a cross-attention-based model and prior information on the temporal ordering of sentences to learn speech embeddings that capture the semantics of spoken text. We demonstrate a significant effectiveness (improvement in R-avg of ~3-6%) over text-based heuristics. Furthermore, INDENT, trained only on Hindi data, caters to 11 Indic languages via a semantically shared text space.
Introduction
Non-governmental organizations (NGOs) operating in the Global South frequently rely on extensive audio surveys to monitor health and education outcomes. In rural Bihar, India, CARE India conducts maternal health surveys where interviews often exceed 40 minutes. These recordings are vital for data collection but present a massive bottleneck: finding the answer to a specific question requires staff to manually listen through hours of noisy, informal audio.
Existing audio-to-text alignment tools generally assume that the text query appears verbatim in the audio. In real-world field surveys, however, interviewers often paraphrase questions, use local dialects, and record in high-noise environments. Furthermore, the lack of timestamped "gold standard" data makes training traditional supervised models prohibitively expensive. This paper introduces INDENT (alIgning loNg auDio intErviews aNd quesTions), a framework designed to bridge this gap using weak supervision and temporal constraints.
The INDENT Framework
INDENT addresses the alignment problem by leveraging the structured nature of survey questionnaires. While we do not have exact timestamps for every question, we do know the order in which questions are asked.
Model Architecture
The framework utilizes a dual-encoder architecture to map audio and text into a shared embedding space:
- Audio Encoder: A frozen IndicWav2vec-Hindi model extracts features from 2-second audio segments.
- Text Encoder: A frozen LaBSE (Language-agnostic BERT Sentence Embedding) model encodes the formal text of the survey questions.
Gaussian-weighted Cross-Attention
The core innovation of INDENT is its use of temporal ordering as a supervisory signal. Since the sequence of questions is fixed, the model employs a moving Gaussian distribution to "anchor" audio chunks to specific questions. This mechanism allows the model to learn which audio segments likely correspond to which question based on their relative position in the interview, effectively bypasses the need for fine-grained manual labels.
Weakly-Supervised Contrastive Learning
The model is trained to minimize the distance between an audio segment and its corresponding question in the embedding space while maximizing the distance from "negative" samples (segments from unrelated parts of the interview). Notably, INDENT only requires "weak" labels—knowing which group of five questions appeared in a larger audio segment—rather than precise per-word timestamps.
Dataset: The CARE India Case Study
The study utilizes a unique dataset of health surveys from rural Bihar. The training set consists of 34 interviews, each approximately 42 minutes long, with segment-level annotations. For evaluation, the researchers used a development set of 3 interviews and a test set of 5 interviews, all featuring fine-grained, per-question timestamps. These recordings are characterized by high ambient noise, informal speech patterns, and the use of local dialects, representing a significant challenge for standard ASR and alignment systems.
Key Results and Insights
Retrieval Performance
INDENT demonstrated a substantial improvement over baseline methods. It outperformed a "No-Train" baseline (which uses simple dot-product matching) by an absolute 6.4% in average recall (R-avg). This confirms that the temporal anchoring and contrastive learning approach successfully captures the semantic relationship between formal text and informal speech.
The Role of ASR
The researchers also explored a variant called INDENT-T, which first converts audio to text using state-of-the-art Automatic Speech Recognition (ASR) models for Indian languages before performing alignment. INDENT-T achieved an R-avg of 63.1%, compared to 43.1% for the raw audio version. This suggests that even noisy transcripts provide a more reliable anchor for semantic retrieval than raw audio features alone.
Zero-Shot Cross-Lingual Capabilities
By leveraging the multilingual LaBSE text encoder, INDENT exhibits remarkable cross-lingual flexibility. Although trained exclusively on Hindi audio, the model can be queried using text in 11 different Indic languages (e.g., Bengali, Telugu, Punjabi) and English. The model successfully matches the meaning of the query to the audio, regardless of the language used for the search, making it a versatile tool for diverse linguistic regions.
Conclusion and Future Work
INDENT provides a scalable solution for NGOs to process large volumes of audio data without the need for exhaustive manual transcription. By using the inherent structure of surveys as a signal, the model overcomes the limitations of noisy data and non-verbatim speech.
Future work aims to address the "faint voice" problem, where the interviewee's response is significantly quieter than the interviewer's question. Ultimately, this research paves the way for "audio-driven form filling," where systems could automatically extract and categorize answers from interviews to populate digital databases, further reducing the administrative burden on health workers.
Summary updated