Findings of the Third Automatic Minuting (AutoMin) Challenge

Kartik Shinde, Laurent Besacier, Ondrej Bojar, Thibaut Thonet, Tirthankar GhosalarXiv 2025 · DOI · arXiv

Abstract

The Third Automatic Minuting (AutoMin) Challenge benchmarks the current state of the art in transforming unstructured meeting transcripts into professional, bulleted minutes and introduces a new task for meeting-based Question Answering (QA). While large language models (LLMs) like GPT-4o currently lead performance across these tasks, the challenge highlights significant difficulties in evaluation, particularly in distinguishing between content adequacy and relevance. This paper summarizes the methods, datasets, and key findings from the 2025 challenge, emphasizing the gap between specialized research pipelines and general-purpose frontier models.

Introduction

Meetings are a fundamental component of professional collaboration, yet the manual creation of structured minutes remains a tedious and time-consuming process. Automatic minuting presents unique challenges: transcripts are often long (exceeding 20,000 tokens), involve multiple participants, and follow non-linear conversational paths. Traditional summarization techniques often fail to produce the actionable, bulleted format required for professional use. The AutoMin 2025 challenge addresses these gaps by evaluating systems on structured minuting and the emerging need for personalized access via meeting-based QA.

Methodology

The challenge was organized into two primary tracks:

  • Task A (Minuting): Generating structured, bulleted minutes from transcripts in both English and Czech.
  • Task B (Meeting QA): Answering specific questions based on transcripts, including a cross-lingual component where Czech questions were posed against English meeting content.

Evaluation Framework

For Task A, organizers introduced a novel LLM-based evaluation framework using GPT-4 with a Chain-of-Thought (CoT) prompt. This approach required the model to ground its evaluation by mapping summary units back to the original transcript before scoring metrics such as Adequacy, Fluency, Grammaticality, and Relevance on a 1-5 scale. Task B utilized an "LLM-as-a-judge" approach (GPT-4o) with a 10-point rubric to compare system outputs against human-crafted gold standards.

Example of conversational vs. standalone QA versions for meeting analysis.

Datasets

The challenge leveraged three distinct resources to test system robustness across domains:

  1. ELITR Minuting Corpus (ELMI): Technical project meetings with manual transcripts and multiple sets of minutes.
  2. EuroParlMin: A dataset curated from European Parliament sessions, offering a more formal, parliamentary domain.
  3. ELITR-Bench: A specialized benchmark for Task B containing 271 manually authored question-answer pairs.
Statistical overview of the ELMI and EuroParlMin datasets showing the scale of transcripts and reference minutes.

Key Insights and Results

The Evaluation Dilemma

A primary finding was the difficulty LLM judges face in distinguishing between "Adequacy" (completeness) and "Relevance" (conciseness). High correlations between these scores suggest that LLMs often perceive them as a single dimension of content quality. However, the CoT prompting strategy proved highly robust in adversarial scenarios, successfully identifying and penalizing summaries from incorrect meetings.

  • LLM Dominance: GPT-4o emerged as the top performer, achieving a mean score of 7.74/10 in monolingual QA. Its performance remained stable (7.69/10) in the cross-lingual setting.
  • Research Gap: The best participant system (GETALP) scored 4.55/10 in QA, highlighting a significant performance gap between specialized research models (like Flan-T5) and general-purpose frontier LLMs.
  • Metric Instability: Traditional model-based metrics like BARTScore were found to be highly unstable across different datasets and years, sometimes showing negative correlations with other quality indicators.

Conclusion

The Third AutoMin Challenge confirms that while LLMs have significantly advanced the quality of automatic minuting and QA, content selection remains a primary hurdle. Future work must focus on refining evaluation metrics to provide more granular feedback and developing adversarial benchmarks to further test the reasoning capabilities of these models beyond simple summarization.

Summary updated

Abstract

This paper presents the third edition of AutoMin, a shared task on automatic meeting summarization into minutes. In 2025, AutoMin featured the main task of minuting, the creation of structured meeting minutes, as well as a new task: question answering (QA) based on meeting transcripts. The minuting task covered two languages, English and Czech, and two domains: project meetings and European Parliament sessions. The QA task focused solely on project meetings and was available in two settings: monolingual QA in English, and cross-lingual QA, where questions were asked and answered in Czech based on English meetings. Participation in 2025 was more limited compared to previous years, with only one team joining the minuting task and two teams participating in QA. However, as organizers, we included multiple baseline systems to enable a comprehensive evaluation of current (2025) large language models (LLMs) on both tasks.
BibTeX
@misc{shinde2025findings,
  author = {Kartik Shinde and Laurent Besacier and Ondrej Bojar and Thibaut Thonet and Tirthankar Ghosal},
  title = {{Findings of the Third Automatic Minuting (AutoMin) Challenge}},
  year = {2025},
  eprint = {2509.13814},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.13814},
}
Figure