Shared Tasks as Tutorials: A Methodical Approach

Theresa Elstner, Frank Loebe, Yamen Ajjour, Christopher Akiki, Alexander Bondarenko, Maik Fröbe, Lukas Gienapp, Nikolay Kolyada, Janis Mohr, Stephan Sandfuchs, Matti Wiegmann, Jörg Frochte, Nicola Ferro, Sven Hofmann, Benno Stein, Matthias Hagen, Martin PotthastAAAI 2023 · DOI

Introduction: Bridging the Gap Between Classroom and Lab

In fields like Artificial Intelligence (AI) and Information Retrieval (IR), a significant gap often exists between the simplified "toy" problems of the classroom and the complex, open-ended challenges of active research. While "shared tasks"—competitive scientific challenges using standardized datasets—are common in the research community, they are rarely integrated into formal higher education curricula. This paper proposes a structured framework to bridge this gap, transforming shared tasks into a scalable and effective teaching tool.

The authors identify a lack of structured approaches for faculty to integrate these competitions into their curricula, often leaving such efforts as isolated, non-reproducible initiatives. By providing a didactically grounded "process model," this research seeks to make the integration of research competitions into university tutorials seamless and effective.

A Synchronized Process Model

The core of the proposed approach is a semi-formal process model that synchronizes two parallel tracks: the external Shared Task Event (e.g., organized by conferences like CLEF) and the internal Shared Task Tutorial (the university course).

UML process model synchronizing the external shared task event with the internal university tutorial track.

As shown in the diagram, the tutorial track provides the necessary instruction, tutoring, and grading, while the event track handles data release and objective evaluation. This synchronization allows students to engage with state-of-the-art research workflows while receiving the pedagogical support required in an academic setting. A key feature is the use of "synchronization bars" to ensure that student progress aligns with external competition milestones, such as data releases and submission deadlines.

Pedagogical Principles

The framework is grounded in several key pedagogical strategies designed to foster a "research-teaching nexus":

  • Project-Based Learning (PBL): Students progress through phases of technology selection, literature review, and iterative prototyping (Coding I and II).
  • Evaluation-as-a-Service: By using platforms like TIRA or CodaLab, students submit software for automated evaluation against hidden test sets, mirroring professional research standards and reducing the manual grading burden on teaching staff.
  • Decoupled Grading: To encourage risk-taking and collaborative learning, grades are based on scientific reports and presentations rather than leaderboard rank. This ensures that learning success is measured by scientific rigor rather than just competitive performance.
  • Interdisciplinary Teams: By forming teams with diverse skills (e.g., pairing Computer Science and Digital Humanities students), the tutorial mimics real-world research environments.

Empirical Evaluation and Results

The methodology was evaluated using feedback from 129 students across 13 Information Retrieval courses at Leipzig University and Martin-Luther-Universität Halle-Wittenberg, primarily utilizing tasks from the Touché lab at CLEF.

Student evaluation metrics across seven terms, highlighting high satisfaction and balanced workload.

Key findings from the student evaluations include:

  • High Student Satisfaction: Satisfaction scores consistently exceeded 4.0 out of 5, with an average of 4.33 in the most recent terms.
  • Manageable Workload: Students reported that the effort required (averaging 3.3/5) was appropriate for the credit hours, avoiding the burnout often associated with high-stakes competitions.
  • Research Impact: In several cases, student groups surpassed official baselines in professional competitions, demonstrating their ability to contribute to the state-of-the-art when provided with the right framework.
  • Resilience: The method proved highly adaptable to online-only teaching during the COVID-19 pandemic, as the shared task infrastructure is inherently cloud-based.

Challenges and Future Outlook

While successful, the model faces challenges such as synchronizing university terms with competition deadlines—often requiring a 4-week buffer at the start of the term for literature review. Additionally, bachelor-level students often require more "starter kits" to bridge coding skill gaps compared to master-level students.

Future work focuses on improving the integration between learning management systems (like Moodle) and shared task platforms to further reduce administrative overhead. By formalizing this approach, the authors aim to turn students from passive consumers of knowledge into active producers of scientific insight, preparing the next generation of researchers for the complexities of the field.

Summary updated

BibTeX
@inproceedings{elstner2023shared,
  author = {Theresa Elstner and Frank Loebe and Yamen Ajjour and Christopher Akiki and Alexander Bondarenko and Maik Fröbe and Lukas Gienapp and Nikolay Kolyada and Janis Mohr and Stephan Sandfuchs and Matti Wiegmann and Jörg Frochte and Nicola Ferro and Sven Hofmann and Benno Stein and Matthias Hagen and Martin Potthast},
  title = {{Shared Tasks as Tutorials: A Methodical Approach}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i13.26877},
}
Figure