Abstract
Compared to image-text pair data, interleaved corpora enable Vision-Language Models (VLMs) to understand the world more naturally like humans. However, such existing datasets are crawled from webpages, facing challenges like low knowledge density, loose image-text relations, and poor logical coherence between images. On the other hand, the internet hosts vast instructional videos (e.g., online geometry courses) that are widely used by humans to learn foundational subjects, yet these valuable resources remain underexplored in VLM training. In this paper, we introduce a high-quality multimodal textbook corpus with richer foundational knowledge for VLM pretraining. It collects over 2.5 years of instructional videos, totaling 22,000 class hours. We first use an LLM-proposed taxonomy to systematically gather instructional videos. Then we progressively extract and refine visual (keyframes), audio (ASR), and textual knowledge (OCR) from the videos, and organize as an image-text interleaved corpus based on temporal order. Experiments demonstrate its superb pretraining performance, particularly in knowledge- and reasoning-intensive tasks like ScienceQA and MathVista. Moreover, VLMs pre-trained on our textbook exhibit outstanding interleaved context awareness, leveraging visual and textual cues in their few-shot context for task solving.
Introduction
Most Vision-Language Models (VLMs) are currently trained on two primary data types: isolated image-text pairs and interleaved web data. While massive, web-crawled interleaved data (such as MMC4 or OBELICS) is often "noisy." These datasets frequently suffer from loose relations between images and text—such as news articles interrupted by unrelated advertisements—and a lack of logical flow between consecutive images. This leads to low "knowledge density," which hinders a model's ability to learn complex reasoning.
To master foundational subjects, models require data where visuals and text are tightly coupled and follow a clear pedagogical progression. The internet already hosts an ideal source for this: instructional videos. In a geometry tutorial, for instance, the instructor explains exactly what they are drawing as they draw it, creating a high-density stream of multimodal knowledge that follows a logical, step-by-step sequence.
The Video-to-Textbook Pipeline
To transform raw video into a structured, high-quality dataset, the researchers developed an automated Video-to-Textbook Pipeline consisting of five key stages:
- Knowledge Taxonomy: GPT-4o was utilized to create a hierarchy of 3,915 knowledge points across six core subjects (Math, Physics, Chemistry, Biology, Engineering, and Computer Science) to guide systematic video collection.
- Video-Level Extraction: The team collected 159,000 videos from YouTube, extracted audio using Whisper-large-v3, and employed Qwen2-72B to refine raw transcripts into fluent, textbook-style "tutorial text."
- Clip & Keyframe Filtering: To ensure visual relevance, VideoLlama2 was used to caption clips, and segments where the visual content did not match the audio were discarded. The Structural Similarity Index (SSIM) was then applied to extract only the most informative keyframes, avoiding redundant shots of static backgrounds.
- OCR Integration: InternVL2 performed Optical Character Recognition (OCR) on keyframes to capture formulas and diagrams that instructors might not have spoken aloud but are vital for conceptual understanding.
- Interleaving: Finally, these elements—refined ASR, keyframes, and OCR—were organized into a chronological, interleaved sequence that mimics a high-quality digital textbook.
Dataset Scale and Composition
The resulting Multimodal Textbook dataset is a massive leap in educational data for AI. It encompasses 75,000 instructional videos totaling 22,697 hours of content. Structurally, it contains 6.5 million images interleaved with 0.75 billion text tokens.
Compared to standard web-crawled datasets like OBELICS, this textbook corpus offers significantly higher "logical coherence." The authors measured this using an InSI-SIM score, finding their dataset scored 0.686 compared to 0.345 for OBELICS. This indicates that the model spends more time learning actual concepts and less time attempting to filter out irrelevant noise.
Key Results and Insights
Pretraining LLaVA-1.5-7B and Idefics2-8B on this textbook dataset yielded substantial performance gains across several benchmarks, including ScienceQA, OKVQA, TextVQA, MathVista, MathVision, and MathVerse.
- Broad Improvements: Models saw average accuracy gains of +3.2% to +4.6% across all benchmarks compared to those trained on standard web data.
- Reasoning Powerhouse: The impact was most pronounced in reasoning-intensive tasks. On ScienceQA, the textbook-trained model outperformed MMC4-trained models by over 20%.
- Enhanced In-Context Learning: The structured nature of the data improved "few-shot" learning capabilities. In a "Cheat Test"—where an identical example was hidden in the model's prompt—the textbook-trained model reached nearly 100% accuracy on MathVision by successfully identifying the pattern, while standard models often failed to notice the identical example.
Conclusion
By repurposing 2.5 years of instructional video content into a multimodal textbook, this research provides a blueprint for high-density, logically coherent VLM pretraining. While some redundancy remains in the 6.5-million-image corpus, the results clearly demonstrate that pedagogical structure is as important as data scale. Future work may expand this approach to "omnimodal" models capable of both understanding and generating complex educational content simultaneously.
Summary updated