Spoken document retrieval for oral presentations integrating global document similarities into local document similarities

Hiroaki Nanjo, Yusuke Iyonaga, Takehiko YoshimiInterspeech 2010

TL;DR

This paper tackles the challenge of searching within long oral presentations by proposing a novel method that integrates information from broader presentation segments (global documents) into the search for shorter, specific segments (local documents). This approach significantly improves retrieval accuracy for finding precise moments within lectures.

What Problem Does This Paper Address?

The increasing availability of audio and video content, particularly lectures and presentations, presents a significant challenge for information retrieval. Unlike short, self-contained news stories, oral presentations are often long (12 minutes to an hour) and have a hierarchical topic structure. Simply searching for entire presentations isn't granular enough for users who want to find specific information within them. Existing Spoken Document Retrieval (SDR) methods, often tested on broadcast news, struggle with these longer, more complex documents. The authors aim to bridge this gap by developing a method to effectively search for short, semantically meaningful segments (local documents) within these longer oral presentations, treating them as structured documents.

Methods

The core innovation lies in integrating "global document similarities" into "local document similarities."

  • Local Documents: These are short segments of the oral presentation, defined as 15-utterance units (approximately 1-2 minutes). These are the primary targets for retrieval.
  • Global Documents: These are longer segments that encompass the local documents. The paper explores using 30-utterance units, 60-utterance units, and the entire presentation as global documents.
  • Similarity Integration: The system uses a log-linear interpolation to combine the similarity score between a query and a local document with the similarity score between the query and its corresponding global document(s). This means that if a query is relevant to a broader topic (global document), it can boost the relevance score of a specific segment (local document) within that topic.
  • Hierarchical Integration: For more sophisticated integration, the method extends to using multiple global documents hierarchically. Similarities from larger global documents (e.g., whole presentation) are first integrated, then combined with similarities from medium-sized global documents (e.g., 60-utterance unit), and finally with the local document similarity. This allows for a more nuanced understanding of context.
  • Standard IR System: The underlying retrieval mechanism uses a vector space model with noun and verb indexing, employing the SMART measure for similarity calculation.

Datasets

The research utilizes the Corpus of Spontaneous Japanese (CSJ), a large test collection for spoken document retrieval. This collection includes:

  • 2702 oral presentations, totaling approximately 604 hours of speech.
  • A set of 39 queries specifically designed for searching within presentations.
  • Relevance judgments indicating which segments are relevant to each query.

The CSJ collection is noted as being constructed by the spoken document processing working group in Japan and differs from conventional SDR test collections (like TREC SDR) by focusing on lectures/oral presentations rather than broadcast news.

Key Insights and Ideas

  • Context is King for Granular Search: The most significant insight is that leveraging the broader context of an oral presentation (global documents) dramatically improves the ability to find specific, short segments (local documents). This is because local segments alone might lack sufficient keywords or be more susceptible to ASR errors.
  • Hierarchical Structure Matters: Oral presentations naturally have a hierarchical structure (topics, subtopics). The paper demonstrates that explicitly modeling and utilizing this hierarchy in the retrieval process leads to better results than treating all segments independently.
  • Global Context Boosts Local Relevance: Even if a local segment is only partially relevant on its own, its relevance score can be significantly boosted if it falls within a globally relevant part of the presentation. This is akin to how humans might understand a specific sentence better when they know the overall topic of the paragraph or chapter.
  • Diminishing Returns with Longer Local Segments: While the integration method is effective, the degree of improvement tends to decrease as the "local document" itself becomes longer. This is intuitive, as longer segments inherently contain more information and are less prone to the limitations of very short segments.

Key Results

The proposed method consistently outperformed the baseline (retrieval without global document integration).

  • 15-utterance unit retrieval: The baseline achieved an 11ptAP of 0.172. Integrating a whole presentation as a global document improved this to 0.211.
  • Hierarchical integration (15-utterance unit): Using multiple global documents hierarchically (30-utterance, 60-utterance, and whole presentation) further boosted performance to 0.220, a statistically significant improvement over the baseline.
  • 30-utterance unit retrieval: The baseline was 0.235. Using a whole presentation as a global document improved this to 0.261, and using both 60-utterance and whole presentation units achieved 0.262.
  • 60-utterance unit retrieval: The baseline was 0.281. Using a whole presentation as a global document improved this to 0.296.

The paper notes that while improvements become smaller with longer local documents, there are no harmful effects.

Limitations and Future Work

  • Limitations:
    • The paper focuses solely on speech processing and does not incorporate visual cues (like slide changes) which could also aid in topic segmentation.
  • The definition of "local document" is based on a fixed number of utterances (15, 30, or 60), which might not always align perfectly with semantic topic boundaries.
  • Future Work:
    • The authors suggest exploring more sophisticated methods for defining semantic segments beyond fixed utterance counts.
  • Integrating visual information (e.g., slide detection) could further enhance retrieval performance.
  • Investigating different interpolation strategies or weighting schemes for combining global and local similarities.

Summary updated

BibTeX
@inproceedings{nanjo2010spoken,
  author = {Hiroaki Nanjo and Yusuke Iyonaga and Takehiko Yoshimi},
  title = {{Spoken document retrieval for oral presentations integrating global document similarities into local document similarities}},
  booktitle = {Interspeech},
  year = {2010},
}
Figure