Alignment of spoken utterances with slide content for easier learning with recorded lectures using structured support vector machine (SVM)

Han Lu, Sheng-syun Shen, Sz-Rung Shiang, Hung-yi Lee, Lin-Shan LeeInterspeech 2014 · OpenReview

Introduction

With the rise of Massive Open Online Courses (MOOCs), recorded lectures have become a staple of modern education. However, navigating these recordings remains a challenge. Students often struggle to find the exact moment a lecturer discusses a specific bullet point or diagram on a slide. While video exists, a "clickable" index linking speech to slide sub-sections is rarely available.

This research addresses the technical difficulty of aligning short, often noisy spoken utterances with specific sections of a presentation slide. The challenges are two-fold: the "sparse/noisy word problem," where short sentences lack keywords or contain distracting filler words, and the "granularity problem," which requires aligning speech to sub-sections rather than entire slides.

Proposed Methodology

The researchers propose a multi-stage pipeline to bridge the gap between speech and slide content, centered around a Structured Support Vector Machine (SVM).

System architecture for aligning lecture utterances with slide sections, featuring pre-processing and structured SVM stages.

The process begins with Pre-processing, which includes:

  • Utterance Clustering: Consecutive utterances are grouped into clusters based on vocabulary similarity to increase the available text data per segment.
  • Entropy-based Word Filtering: Words that appear uniformly across all clusters (high entropy) are discarded as noise, while words concentrated in specific clusters are retained as meaningful keywords.

Following pre-processing, the system employs Reliability-propagated Matching to identify high-confidence matches and propagate that confidence to neighboring clusters. The core of the alignment is the Structured SVM, which evaluates the entire sequence of a slide. Unlike standard classifiers, it considers both local features (word overlap) and global features (sequential flow, such as the "top-down" assumption of typical presentations) to determine the most likely alignment.

Experimental Evaluation

The system was evaluated using a corpus of 45 hours of Mandarin-English code-switching lectures from National Taiwan University. The dataset included 193 slides, with 38 slides manually labeled for ground truth. Tests were conducted on both perfect manual transcripts and noisy Automated Speech Recognition (ASR) output.

Alignment accuracy comparison between baseline and proposed approaches for manual and ASR transcripts.

As shown in the results table, the proposed approaches significantly outperformed the tf-idf similarity baseline. The "Score Integration" method, which combines word-based matching with the Structured SVM, achieved the highest accuracies, reaching over 77% for manual transcripts.

Key Insights and Discussion

  • Sequential Smoothness: The "top-down" assumption proved to be a powerful global feature. By penalizing out-of-order jumps, the model could correct local errors caused by ambiguous keyword matches.
  • Unsupervised Scalability: The researchers found that the Structured SVM could be trained effectively using the output of heuristic matching rather than human labels. This unsupervised version performed nearly as well as the supervised one, suggesting the system can scale to new courses without manual intervention.
  • Robustness to Noise: The performance gap between manual transcripts and ASR was remarkably small (approximately 4%), indicating that the system's filtering and structural modeling effectively mitigate speech recognition errors.

Conclusion

The proposed framework demonstrates that structured modeling of presentation flow can successfully align speech with slide content. Future work may involve incorporating visual features, such as laser pointer tracking, and testing the model on more diverse, non-linear presentation styles to further enhance generalizability.

Summary updated

BibTeX
@inproceedings{lu2014alignment,
  author = {Han Lu and Sheng-syun Shen and Sz-Rung Shiang and Hung-yi Lee and Lin-Shan Lee},
  title = {{Alignment of spoken utterances with slide content for easier learning with recorded lectures using structured support vector machine (SVM)}},
  booktitle = {Interspeech},
  year = {2014},
  url = {https://openreview.net/forum?id=vJYri5iqXZp},
}
Figure