TL;DR
This paper introduces VoxRAG, a novel system for spoken question answering (QA) that bypasses traditional speech-to-text transcription. By directly retrieving relevant audio segments from spoken queries and using them to generate answers, VoxRAG demonstrates the feasibility of transcription-free RAG systems, opening new avenues for audio-native information access.
What Problem Does This Paper Address?
Traditional Retrieval-Augmented Generation (RAG) systems rely on text-based retrieval to find relevant information for answering user queries. This approach breaks down when dealing with spoken audio, as it necessitates an intermediate transcription step. Automatic Speech Recognition (ASR) systems, while improving, can still introduce errors, especially in challenging audio environments like podcasts with informal language, overlapping speakers, and background noise. These errors can propagate and negatively impact the retrieval and subsequent answer generation. VoxRAG aims to address this by creating a system that operates directly on audio, from query to retrieval to answer generation, thereby avoiding the pitfalls of transcription.
Methods
VoxRAG is a modular system designed for full speech-to-speech retrieval. Its core components include:
- Silence-Aware Segmentation and Speaker Diarization: To break down audio into manageable segments and identify individual speakers, improving the granularity of retrieval.
- CLAP (Contrastive Language-Audio Pre-training) Embeddings: This is a key innovation. CLAP maps audio and text into a shared embedding space, allowing for semantic similarity search between spoken queries and audio segments without relying on exact word matches. This makes it robust to informal language and variations in speech.
- FAISS Retrieval: The CLAP embeddings are stored in a FAISS index for efficient similarity search. The system retrieves the top ten most similar segments based on L2-normalized cosine similarity.
- LLM-based Answer Generation: Once relevant audio segments are retrieved, their transcripts (generated by an optional, separate Whisper model) are used along with the transcribed query to prompt a large language model (GPT-4) for answer generation.
The system is designed to keep audio representations throughout the retrieval process, only converting to text for the final answer generation stage.
Datasets
- Source Corpus: Twenty episodes from the "Trash Taste" podcast were used as the source for retrieval.
- Evaluation Queries: A custom test set of 50 diverse, high-variance spoken queries was created. This set included:
- 11 organic queries from a Tokyo Weekender article and a live Trash Taste QA session.
- 205 synthetic queries generated by GPT-4.
- 294 synthetic queries generated by GPT-01.
- 50 non-duplicative synthetic queries were randomly sampled from the generated sets.
- Benchmark: The paper introduces a new 50-query test set recorded as spoken input by a native English speaker, specifically designed for evaluating transcription-free spoken QA.
Key Insights and Ideas
- Transcription-Free Retrieval is Feasible: The most significant insight is that direct speech-to-speech retrieval is achievable and can yield meaningful results, even with current embedding technology.
- CLAP's Strength in Semantic Alignment: CLAP embeddings are crucial for bridging the gap between spoken queries and audio content, enabling retrieval based on semantic meaning rather than just lexical overlap. This is particularly effective for noisy, informal audio like podcasts.
- Topical Alignment vs. Factual Precision: While VoxRAG excels at retrieving topically aligned segments, it struggles with fine-grained factual precision. This suggests that current audio embeddings capture broad themes well but lack the detail for precise answers.
- Modularity is Key: The modular design of VoxRAG allows for easy experimentation with different segmentation, embedding, and retrieval strategies, facilitating future research.
Key Results
- Retrieval Quality:
- For "very relevant" segments, cosine similarity achieved a Recall@10 of 0.34.
- For "somewhat relevant" segments, Recall@10 rose to 0.60 and nDCG@10 to 0.27, indicating strong topical alignment.
- The significant gap between "very relevant" and "somewhat relevant" scores highlights the challenge of precise matching.
- Answer Quality (rated on a 0-2 scale by an LLM judge):
- Relevance: Mean score of 0.84 (highest).
- Accuracy: Mean score of 0.58.
- Completeness: Mean score of 0.56.
- Precision: Mean score of 0.46 (lowest).
- The results show that while answers are often topically relevant, they frequently lack factual specificity and grounding.
- Perfect Scores: 20% of queries achieved perfect scores across all dimensions, demonstrating the system's potential when retrieval alignment is successful.
Limitations and Future Work
- Limitations:
- Lack of Baselines: The study lacks direct comparisons with transcript-based or hybrid retrieval baselines (e.g., BM25), making it hard to quantify the exact benefits of avoiding transcription.
- Hybrid Generation: While retrieval is audio-only, answer generation still relies on Whisper transcripts, reintroducing potential ASR noise.
- Evaluation Bias: Using GPT-4 for both answer generation and evaluation might lead to overestimation of performance.
- Limited Dataset: Evaluation was primarily on a single podcast episode, limiting generalizability.
- Topical vs. Precise Retrieval: A persistent gap exists between retrieving topically relevant segments and those that provide precise, factual answers.
- Future Work:
- Explore audio-native generation methods to achieve a fully transcription-free pipeline.
- Investigate improved embedding fine-tuning and more effective segmentation/reranking strategies.
- Conduct broader evaluations across multiple episodes and speakers.
- Develop methods for improving fine-grained alignment to bridge the gap between topical relevance and factual precision.
Summary updated