Towards Comprehensive Lecture Slides Understanding: Large-scale Dataset and Effective Method

Enming Zhang, Yuzhe Li, Yuliang Liu, Yingying Zhu, Xiang BaiICCV 2025

TL;DR

This paper introduces LecSlides-370K, the largest and most diverse dataset to date for understanding lecture slides, featuring over 370,000 slides across 15 disciplines. They also propose SlideParser, a novel method that effectively models complex text relationships within slides to improve lecture summary and question answering tasks, outperforming existing methods.

What Problem Does This Paper Address?

Online education relies heavily on lecture slides, which are dense with information crucial for learning. However, understanding these slides presents unique challenges: the text is often concise and headline-like, making it difficult to grasp detailed context, and the relationships between different text elements can be complex and non-linear. Existing research has lacked a large-scale, comprehensive dataset and effective methods to tackle these issues. Previous datasets are often limited in scale, diversity, or the types of understanding tasks they support, hindering progress in areas like generating lecture summaries or answering specific questions about slide content.

Methods

The core of the proposed solution is SlideParser, a method designed to address the complex text relations in lecture slides. SlideParser builds upon a unified text-image multi-modal Transformer Encoder. The key innovation lies in its Text Relation Prediction Module, which comprises two components:

  • Relation Prediction Head (RPH): This module uses both text position (bounding box information) and semantic similarity to predict relationships between text elements on a slide. It leverages a weak supervision pipeline called DL-Cluster to generate high-quality pseudo relation labels. DL-Cluster dynamically determines the optimal number of clusters for text boxes and combines spatial and semantic information to create these labels.
  • Relation Attention Layer (RAL): This layer uses the predicted text relations to enhance attention between related text segments and suppress attention between unrelated ones. This effectively helps the model focus on the logical flow and connections within the slide content.

The updated text embeddings from this module are then combined with image embeddings and fed into a Text Decoder (a T5 decoder) to generate the final output, such as a lecture summary or an answer to a question.

Datasets

The paper introduces LecSlides-370K, a new large-scale dataset for lecture slide understanding. It contains:

  • 25,542 lectures
  • 370,078 slides
  • Across 15 diverse academic disciplines (e.g., business, computer science, medicine)

This dataset is significantly larger and more diverse than previous benchmarks like SlideVQA. It supports two key tasks:

  • Lecture Summary: Generating a concise summary of a lecture chunk.
  • Lecture Question Answering (QA): Answering questions based on the content of a lecture chunk.

LecSlides-370K includes 114,472 summary annotations and 259,461 QA pairs. The dataset was also used to evaluate SlideParser on the existing SlideVQA dataset.

Key Insights and Ideas

  • Text relations are paramount: The paper strongly argues that effectively modeling the complex, often non-linear, relationships between text elements on lecture slides is crucial for deep understanding, more so than just processing individual text snippets or images in isolation.
  • Weak supervision for relations: Directly predicting text relations is challenging. The DL-Cluster pipeline provides a novel way to generate useful pseudo-labels for training the relation prediction module, outperforming traditional clustering methods by incorporating both spatial and semantic cues.
  • Scale and diversity matter: The creation of LecSlides-370K highlights the need for larger, more diverse datasets to push the boundaries of lecture slide understanding, moving beyond limited domains and scales.
  • Pre-training potential: The large-scale LecSlides-370K dataset can serve as a valuable pre-training resource for downstream tasks, as demonstrated by the performance boost when fine-tuning SlideParser on SlideVQA.

Key Results

  • LecSlides-370K Performance: SlideParser achieved state-of-the-art results on the LecSlides-370K dataset for both Lecture Summary and Lecture QA tasks. For Lecture Summary, it reported ROUGE-1 scores of 42.2, ROUGE-L of 25.3, and METEOR of 31.3. For Lecture QA, it achieved Exact Matching (EM) of 23.7 and F1 of 33.0.
  • SlideVQA Performance: SlideParser also outperformed existing methods on the SlideVQA benchmark, achieving an F1 score of 47.7.
  • Ablation Studies: Removing the Text Relation Prediction Module significantly degraded performance, underscoring its importance. Combining both spatial and semantic relations in DL-Cluster yielded the best results for pseudo-label generation.

Limitations and Future Work

  • Complexity of relations: While SlideParser significantly improves text relation modeling, the paper acknowledges that the inherent complexity of relations in slides can still pose challenges.
  • Generalization: The paper suggests that LecSlides-370K can be used for large-scale pre-training, implying future work could explore more extensive pre-training strategies and their impact on various downstream tasks.
  • Task expansion: The dataset and method lay the groundwork for further research into other lecture slide understanding tasks beyond summarization and QA.

Summary updated

BibTeX
@inproceedings{zhang2025towards,
  author = {Enming Zhang and Yuzhe Li and Yuliang Liu and Yingying Zhu and Xiang Bai},
  title = {{Towards Comprehensive Lecture Slides Understanding: Large-scale Dataset and Effective Method}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2025},
}
Figure