Abstract
This paper presents a course on neural networks based on the Transformer architecture targeted at diverse groups of people from academia and industry with experience in Python, Machine Learning, and Deep Learning but little or no experience with Transformers. The course covers a comprehensive overview of the Transformers NLP applications and their use for other data types. The course features 15 sessions, each consisting of a lecture and a practical part, and two homework assignments organized as CodaLab competitions. The first six sessions of the course are devoted to the Transformer and the variations of this architecture (e.g., encoders, decoders, encoder-decoders) as well as different techniques of model tuning. Subsequent sessions are devoted to multilingualism, multimodality (e.g., texts and images), efficiency, event sequences, and tabular data. We ran the course for different audiences: academic students and people from industry. The first run was held in 2022. During the subsequent iterations until 2024, it was constantly updated and extended with recently emerged findings on GPT-4, LLMs, RLHF, etc. Overall, it has been ran six times (6 times in industry and 3 times in academia) and received positive feedback from academic and industry students.
Introduction
While Transformers have become the backbone of modern AI, a significant knowledge gap exists between "using" a model and "understanding" it. Computer science students often receive a theoretical foundation but lack exposure to industrial constraints, while industry engineers frequently rely on high-level libraries (like Hugging Face) without grasping the underlying mechanisms.
This paper details the design and execution of a comprehensive 15-session course on Transformer architectures, successfully delivered to both university students and industry professionals. The authors sought to build a curriculum rigorous enough for a Master’s program but practical enough to help a software engineer at a major IT company deploy a model effectively.
Curriculum Design
The course is structured into 15 sessions, each pairing a theoretical lecture with a hands-on practical seminar using Jupyter Notebooks. The curriculum is divided into two logical halves: Foundations (Vanilla Transformer, BERT, GPT) and Advanced Multi-domain applications.
Beyond standard NLP, the course demonstrates the versatility of the architecture by utilizing a variety of datasets, including the WMT14 translation set, Russian-language drug reviews for NER, and even bank marketing data to demonstrate "TabTransformer" models for tabular data.
Pedagogical Approaches: The "Two Setups"
A key contribution of this work is the adaptation of delivery based on the audience. In academic runs, instructors emphasized mathematical formulas and original scientific papers. In industrial runs, the focus shifted toward business use cases, practical implementation "gotchas," and localized materials.
Students were evaluated through a final quiz and two major homework assignments organized as CodaLab competitions. This gamified approach encouraged students to iterate on their models to climb a leaderboard, mimicking real-world ML development cycles.
Key Insights and Results
The course has been run nine times with high success, yielding an average student rating of 8.5 out of 10. Several key insights emerged from these iterations:
- Generalization: Teaching Transformers as a general-purpose architecture helps students generalize the technology beyond chatbots to areas like financial transactions.
- Motivation Differences: Industry professionals showed significantly higher attendance and were more likely to ask questions related to specific job-related projects, whereas academic students were more comfortable with dense, theoretical lectures.
- Localization: A surprising finding was that industry professionals specifically requested materials translated into their native language, whereas academic students were comfortable with English-only resources.
Conclusion
The primary challenge in teaching this subject is the velocity of the field. The authors constantly updated the course between 2022 and 2024 to include GPT-4, LLaMA-2, and new Parameter-Efficient Fine-Tuning (PEFT) techniques. Future iterations plan to expand the "Multimodality" section and develop more specialized modules for "Efficiency," as deploying these massive models remains a significant hurdle for industrial applications.
Summary updated