Abstract
The rapid advancement of Large Language Models (LLMs) has created a "moving target" for computer science education. This paper describes the design and implementation of a new course at Charles University that balances theoretical foundations with hands-on experience. By leveraging local GPU clusters and open-source models, the curriculum provides students with deep technical insights into model inference, benchmarking, and research evaluation without the financial constraints of commercial APIs.
Introduction
Teaching Large Language Models (LLMs) in a university setting presents two primary challenges: the extreme pace of field evolution and the high cost of infrastructure. To address these, researchers at Charles University developed a 13-session course that moves students from being passive users of AI to becoming critical engineers. The course emphasizes the nuances of model behavior, the limitations of current benchmarks, and the ethical implications of deployment.
Infrastructure and Technical Setup
A core objective of the course was to democratize LLM education by bypassing the "pay-to-play" model of commercial providers like OpenAI. The technical backbone consists of a local GPU cluster running the text-generation-webui (oobabooga) package. This setup provides a unified API that allows students to access a variety of open-source models, including Mistral 7B, Phi-2, and Aya, through a simplified "starter code" interface. This approach ensures that students can experiment with model parameters at scale without incurring prohibitive costs.
Pedagogical Methods and Assignments
The curriculum blends traditional lectures with interactive labs designed to foster active learning through practical application.
Parameter Exploration
Students are tasked with exploring the impact of decoding parameters on model output. Rather than focusing solely on prompt engineering, the assignment requires students to manipulate temperature, top-$k$, top-$p$, and beam size. This exercise helps students understand the trade-offs between output diversity and quality, providing a more granular view of model inference mechanics.
Competitive Benchmarking
For machine translation (MT) tasks, the instructors implemented a custom web application featuring a real-time leaderboard. Students work with text paragraphs in 21 different languages, aiming to optimize their prompts and model selection. Performance is measured using the chrF score (a character n-gram F-score), providing immediate, gamified feedback that encourages iterative refinement and competition.
Research Literacy and Role-Playing
To develop critical evaluation skills, the course includes a "best paper" committee session. Students are assigned recent high-impact research papers and must act as advocates or critics in a simulated peer-review environment. This method shifts the focus from passive reading to active analysis of model descriptions and analytical methodologies, simulating the professional academic environment.
Datasets and Evaluation
The course utilizes a diverse range of datasets to ground theoretical concepts in real-world data:
- Data-to-Text: JSON files retrieved from the OpenWeatherMap.org API serve as the ground truth for generating factual weather reports.
- Multilingual MT: A collection of paragraphs across 21 different languages is used for the translation leaderboard.
- Student-Sourced Tasks: For downstream task activities (e.g., hate speech detection or code generation), students are responsible for identifying and justifying their own datasets, fostering independence in research design.
Key Insights
The inaugural run of the course yielded several significant findings regarding the current state of open-source LLMs:
- The Factual Gap: Despite the capabilities of modern models, students observed persistent difficulties in generating factually accurate reports from structured JSON data, highlighting the ongoing need for grounding techniques.
- Multilingual Disparity: While open-source models like Tower Instruct and ALMA-R performed well for medium-resource languages, a significant performance gap remains for lower-resource languages such as Georgian or Serbian.
- Active Learning Efficacy: The "Best Paper" session was particularly successful in engaging students with primary literature, proving more effective than traditional passive reading assignments.
Results and Future Work
The 2024 course saw an enrollment of 51 students across BSc and MSc levels. The local API setup successfully supported concurrent access for small teams without significant latency or scaling issues. However, scaling to larger cohorts will require more robust hardware scheduling and automated evaluation methods to reduce the labor-intensive nature of grading open-ended reports. Future iterations of the course aim to transition it from an optional to an elective status within the formal degree program, potentially integrating more automated feedback loops to support larger student populations.
Summary updated