TL;DR
This paper introduces "Textbook," a novel multimodal dataset derived from over 22,000 hours of instructional videos, designed to improve Vision-Language Models (VLMs) by providing richer, more coherent, and temporally aligned visual and textual information. VLMs pre-trained on Textbook show significant gains in knowledge-intensive tasks and improved in-context learning capabilities.
What Problem Does This Paper Address?
Vision-Language Models (VLMs) have made impressive strides, largely due to advancements in large language models (LLMs) and the availability of diverse training data. Current training corpora often consist of image-text pairs or webpage-scraped interleaved data. While image-text pairs are good for alignment, they can lack naturalness. Webpage-scraped interleaved data, though more natural, suffers from loose text-image relations, poor logical coherence between images, and low knowledge density, often including irrelevant content like ads. This limits VLMs' ability to perform complex reasoning and in-context learning. The authors identify a gap in high-quality, structured multimodal data that can provide foundational knowledge and coherent context, similar to how humans learn from textbooks.
Methods
The authors propose a "multimodal textbook" corpus constructed from instructional videos. Their approach involves a multi-stage pipeline:
- LLM-Powered Knowledge Taxonomy: An LLM is used to generate a hierarchical taxonomy of subjects, courses, sub-courses, and knowledge points (e.g., Mathematics -> Elementary Mathematics -> Rational and Irrational Numbers -> Definition of Irrational Numbers). This taxonomy guides the collection of relevant videos.
- Video Collection and Filtering: Videos are collected from YouTube using keywords derived from the taxonomy. Initial filtering is done using video metadata and LLM analysis to remove irrelevant or low-quality content.
- Video-to-Textbook Pipeline:
- Video-Level Extraction (ASR): Audio is extracted from videos and transcribed into text using Whisper. This raw ASR is then refined by an LLM (Qwen2-72B-Instruct) to improve fluency and coherence while preserving semantics.
- Clip-Level Extraction: Long videos are segmented into shorter clips (10-20 seconds) based on ASR timestamps. Incomplete ASR segments are merged, and clips are filtered to ensure they contain visual knowledge and are not just static shots of the speaker.
- Keyframe-Level Extraction (Keyframes & OCR): Keyframes are extracted from video clips by detecting significant visual changes between consecutive frames using the Structural Similarity Index (SSIM) algorithm. On-screen text, symbols, and formulas are extracted from these keyframes using OCR (InternVL2).
- Filtering: Keyframes with occlusions and redundant OCR text are removed.
The final output is an interleaved image-text corpus where keyframes and OCR text are aligned with corresponding ASR segments, organized chronologically.
Datasets
- Training Data: The primary dataset introduced is Textbook, a multimodal corpus curated from over 22,000 class hours of instructional videos, resulting in approximately 6.5 million keyframes and 0.75 billion text tokens, organized into 610K interleaved samples.
- Evaluation Benchmarks: The paper evaluates VLMs on several standard benchmarks:
- VQA: TextVQA, OKVQA
- Visual Reasoning: MathVista, MathVision, MathVerse
- General: ScienceQA-IMG, MMMU (Math, Finance, Clinical Medicine subsets)
Key Insights and Ideas
- Instructional Videos as Textbooks: The core idea is that instructional videos, with their structured explanations and visual demonstrations, serve as a natural "textbook" for VLMs, offering a richer and more coherent learning experience than scraped web data.
- Temporal Coherence is Crucial: The video-centric approach naturally provides temporally aligned image-text sequences, which is shown to be vital for VLMs to effectively attend to interleaved context and improve in-context learning.
- ASR Refinement is Key: Raw ASR transcripts from videos are noisy and can impair language abilities. LLM-based refinement significantly improves their quality and usefulness.
- OCR Adds Domain-Specific Expertise: Extracting on-screen text, formulas, and symbols via OCR complements ASR by providing precise domain-specific knowledge, boosting performance on benchmarks like TextVQA and MathVista.
- SSIM for Keyframe Extraction: The SSIM algorithm proves effective for extracting semantically relevant keyframes from instructional videos, balancing visual change detection with efficiency.
Key Results
- Significant Performance Gains: VLMs pre-trained on Textbook (Textbook-6.5M) show substantial improvements across seven benchmarks, with average gains of +3.2% to +8.3% compared to models trained on other datasets like MMC4 and OBELICS.
- Knowledge-Intensive Tasks Excel: Textbook yields particularly large improvements on knowledge-oriented and reasoning benchmarks like ScienceQA (over 20% improvement) and MathVista (+5.3% to +6.4% improvement).
- Enhanced In-Context Learning: The "Cheat Test" reveals that models trained on Textbook are better at identifying and utilizing relevant information from interleaved contexts, outperforming other datasets significantly, especially in few-shot scenarios.
- Robustness to Image Order: Unlike webpage-based datasets, the performance of models trained on Textbook drops more significantly when image order is shuffled, highlighting the importance of its inherent sequential coherence.
Limitations and Future Work
- Limitations:
- The paper notes that while raw ASR can be noisy, the refined ASR still has a higher perplexity than standard corpora.
- Low-quality OCR can introduce noise and degrade performance, emphasizing the need for reliable OCR tools.
- The "Cheat Test" shows that while Textbook improves in-context learning, there's still room for improvement, especially as context length increases.
- Future Work: The authors suggest that their multimodal textbook can be further expanded by incorporating more subjects and knowledge points. They also highlight the potential for further research into more advanced keyframe extraction and OCR techniques. The work also opens avenues for exploring how VLMs can better leverage the rich, sequential information present in video data for more sophisticated reasoning and understanding.
Summary updated