A Dataset for Programming-based Instructional Video Classification and Question Answering

Sana Javaid Raja, Adeel Zafar, Aqsa ShoaibCOLING 2025

TL;DR

This paper introduces two novel datasets, CodeVidQA and CodeVidCL, designed to advance AI's ability to understand and interact with programming tutorial videos. By providing annotated question-answer pairs and categorized programming/non-programming videos, these datasets pave the way for more intelligent systems that can help developers learn more effectively from online resources.

What Problem Does This Paper Address?

The internet is awash with programming tutorials, offering a powerful avenue for self-learning. However, finding specific information within these videos can be a tedious process for developers, who often spend a significant portion of their time searching for solutions. Existing video question-answering (VideoQA) systems, while promising, often struggle with the nuances of instructional content, particularly in specialized domains like programming. Furthermore, classifying programming videos from general instructional content is a challenge that lacks dedicated datasets. This paper aims to bridge this gap by creating resources that enable AI to not only answer questions about programming videos but also to accurately classify them.

Methods

The authors propose a multi-step methodology for creating their datasets.

For CodeVidQA, the process involves:

  1. Crawling Stack Overflow: Gathering a large number of programming-related questions.
  2. Filtering Questions: Identifying instructional questions that can be demonstrated visually.
  3. Querying YouTube: Searching for relevant instructional videos based on these questions.
  4. Expert Annotation: Programming experts meticulously identify answer segments within videos, formulate precise questions, and mark start/end timestamps.

For CodeVidCL, the approach focuses on video classification:

  1. Collecting Videos: Gathering programming videos (from CodeVidQA) and a diverse set of non-programming videos from sources like HowTo100M.
  2. Subtitle Extraction: Obtaining subtitles for all videos, a crucial step for text-based analysis.
  3. Ensemble Classifier: Building a hybrid classifier combining the strengths of BigBird (a transformer model adept at handling long sequences like subtitles) and SVM (a robust statistical classifier). This ensemble aims to leverage the contextual understanding of BigBird and the statistical power of SVM for accurate classification.
  4. Validation: Programming experts validate the classifier's predictions to ensure high-quality data.

Datasets

The paper introduces two new datasets:

  • CodeVidQA: Contains 2,104 expert-curated, timestamped question-answer pairs derived from programming instructional videos. This dataset is specifically designed for programming-based VideoQA tasks.
  • CodeVidCL: Comprises 4,291 videos, with 2,540 classified as non-programming and the rest as programming. This dataset is intended for programming video classification.

Key Insights and Ideas

  • The Need for Domain-Specific Datasets: The research highlights that general VideoQA datasets are insufficient for the specialized needs of programming education. Programming requires understanding visual demonstrations, code execution, and specific terminology, which are not well-represented in broader datasets.
  • Leveraging Stack Overflow and YouTube: The paper effectively demonstrates how to combine community-driven Q&A platforms like Stack Overflow with video platforms like YouTube to create rich instructional datasets.
  • The Power of Ensemble Models: Combining BigBird and SVM for video classification proved highly effective, suggesting that hybrid approaches can capture both deep contextual understanding and robust statistical patterns.
  • Importance of Expert Annotation: The meticulous work of programming experts in annotating timestamps and formulating questions is crucial for the quality and utility of the CodeVidQA dataset.

Key Results

The proposed ensemble classifier for video classification achieved a remarkable 99.61% accuracy. This high performance underscores the effectiveness of the BigBird and SVM combination for distinguishing between programming and non-programming videos. The paper also details the creation of the CodeVidQA dataset with 2,104 question-answer pairs, providing a valuable resource for VideoQA research in this domain.

Limitations and Future Work

The authors acknowledge that the current datasets cover a limited set of programming languages. They suggest that a larger and more diverse dataset spanning a broader range of programming topics is needed. Additionally, they identify result explainability as a significant limitation, noting that current models do not clearly identify the features contributing to their predictions. Future work could focus on expanding language coverage, improving model interpretability, and exploring more advanced VideoQA techniques tailored for programming content.

Summary updated

BibTeX
@inproceedings{raja2025dataset,
  author = {Sana Javaid Raja and Adeel Zafar and Aqsa Shoaib},
  title = {{A Dataset for Programming-based Instructional Video Classification and Question Answering}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2025},
}
Figure