TL;DR
This paper introduces a novel, project-based curriculum for teaching data labeling to future machine learning engineers. By shifting from traditional lectures to hands-on, data-centric competitions focused on quality, budget, and time, the authors demonstrate a significant improvement in student engagement and perceived career relevance.
What Problem Does This Paper Address?
While much research in AI focuses on developing new ML algorithms, a critical bottleneck remains the creation of high-quality, timely annotated datasets. Machine learning engineers and data scientists often spend a disproportionate amount of their time on data processing, labeling, and infrastructure, yet dedicated training for these skills is scarce. Existing university courses often cover crowdsourcing theory but lack practical, engineering-focused application. This paper addresses the need for a curriculum that equips future ML professionals with the practical skills to build scalable and sustainable data labeling processes, moving beyond the perception of crowdsourcing as mere worker management.
Methods
The authors developed a semester-long course titled "Data Collection and Labeling for Machine Learning." Initially, the course followed a traditional structure with weekly lectures and practical assignments. However, based on feedback indicating monotony, they revamped it into a project-based curriculum organized around three rounds, each focusing on a key aspect of data labeling: Quality, Budget, and Time.
Each round involved students working in small groups on real-world projects, such as image classification (license plates) or natural language processing (search query paraphrasing). These projects were framed as "data-centric competitions," where the quality of the data collected and its impact on a pre-trained model's performance determined the students' scores. This approach aimed to demonstrate the direct link between data quality and model performance. The curriculum also incorporated elements of the Kolb experiential learning cycle, encouraging students to gain experience, reflect, conceptualize, and test their new approaches.
Datasets
The paper does not explicitly list specific public datasets used for training or evaluation. Instead, it describes the types of data used in the projects:
- Computer Vision Task: 8.5k images for training a MobileNetV2 model to classify car license plates. A public testing dataset was provided for evaluation during training, and a private testing set was used for final assessment.
- Natural Language Processing Task: 15k anonymized search user sessions for training a model to recognize paraphrases. Ground truth estimations were used for quality evaluation.
- Content Moderation Task: An unlabeled text corpus and two labeled test sets with imbalanced classes (positive/negative comments) were used.
The focus was on students collecting or labeling data themselves, rather than relying on pre-existing, large-scale datasets.
Key Insights and Ideas
- Data Labeling as an Engineering Skill: The core idea is to position data labeling not as a peripheral task or simple worker management, but as a crucial engineering skill for ML professionals.
- Project-Based Learning for Engagement: Shifting to a project-based, competitive format significantly increased student engagement and satisfaction compared to traditional lectures.
- Data-Centric Competitions Drive Mindset Change: Framing projects as competitions where data quality directly impacts model performance proved highly effective in changing students' perception of data's importance.
- Iterative Experience is Key: Going through multiple project rounds (Quality, Budget, Time) allowed students to continually refine their approach to data labeling.
- Real-World Constraints are Crucial: Incorporating budget and time limitations into the projects mirrored real-world challenges and fostered more conscious decision-making.
Key Results
The paper reports positive feedback from students, particularly third-year students, after the implementation of the project-based curriculum:
- Usefulness for Career: Increased from 3.16 to 3.8 (out of 5) for third-year students.
- Broadening Outlook: Increased from 3.69 to 4.3 (out of 5) for third-year students.
- Novelty of Knowledge: Increased from 4.0 to 4.5 (out of 5) for third-year students.
- Complexity: Increased from 3.0 to 3.8 (out of 5) for third-year students.
Fourth-year students showed less dramatic improvements, with some metrics even decreasing slightly, which the authors attribute to their prior experience with ML models making the content less novel. Qualitative feedback highlighted that students felt the practical focus was the most valuable aspect, leading to a clearer understanding of data's role in the ML lifecycle and a tangible skill addition to their CVs.
Limitations and Future Work
Limitations:
- Curriculum Placement: The course's prerequisites (advanced programming, ML/deep learning) limit its integration into programs for students entirely new to these fields. The authors suggest targeting it towards students with some ML background.
- Industrial Data Constraints: Transforming some real-world business cases into learning tasks was challenging due to the need to protect proprietary information.
- Lecturer Management: Coordinating multiple industry practitioners as lecturers required significant effort to ensure a cohesive narrative and avoid content duplication.
Future Work:
- Exporting the Syllabus: Adapting the syllabus for use in other universities or educational projects.
- Integrating into ML Courses: Incorporating data-centric competition elements into existing ML, deep learning, computer vision, or NLP courses to demonstrate the value of data labeling pipelines.
- Shorter Public Version: Developing a condensed version of the curriculum for a public audience interested in industrial data labeling.
Summary updated