Abstract
The Third Automatic Minuting (AutoMin) Challenge benchmarks the current state of the art in transforming unstructured meeting transcripts into professional, bulleted minutes and introduces a new task for meeting-based Question Answering (QA). While large language models (LLMs) like GPT-4o currently lead performance across these tasks, the challenge highlights significant difficulties in evaluation, particularly in distinguishing between content adequacy and relevance. This paper summarizes the methods, datasets, and key findings from the 2025 challenge, emphasizing the gap between specialized research pipelines and general-purpose frontier models.
Introduction
Meetings are a fundamental component of professional collaboration, yet the manual creation of structured minutes remains a tedious and time-consuming process. Automatic minuting presents unique challenges: transcripts are often long (exceeding 20,000 tokens), involve multiple participants, and follow non-linear conversational paths. Traditional summarization techniques often fail to produce the actionable, bulleted format required for professional use. The AutoMin 2025 challenge addresses these gaps by evaluating systems on structured minuting and the emerging need for personalized access via meeting-based QA.
Methodology
The challenge was organized into two primary tracks:
- Task A (Minuting): Generating structured, bulleted minutes from transcripts in both English and Czech.
- Task B (Meeting QA): Answering specific questions based on transcripts, including a cross-lingual component where Czech questions were posed against English meeting content.
Evaluation Framework
For Task A, organizers introduced a novel LLM-based evaluation framework using GPT-4 with a Chain-of-Thought (CoT) prompt. This approach required the model to ground its evaluation by mapping summary units back to the original transcript before scoring metrics such as Adequacy, Fluency, Grammaticality, and Relevance on a 1-5 scale. Task B utilized an "LLM-as-a-judge" approach (GPT-4o) with a 10-point rubric to compare system outputs against human-crafted gold standards.
Datasets
The challenge leveraged three distinct resources to test system robustness across domains:
- ELITR Minuting Corpus (ELMI): Technical project meetings with manual transcripts and multiple sets of minutes.
- EuroParlMin: A dataset curated from European Parliament sessions, offering a more formal, parliamentary domain.
- ELITR-Bench: A specialized benchmark for Task B containing 271 manually authored question-answer pairs.
Key Insights and Results
The Evaluation Dilemma
A primary finding was the difficulty LLM judges face in distinguishing between "Adequacy" (completeness) and "Relevance" (conciseness). High correlations between these scores suggest that LLMs often perceive them as a single dimension of content quality. However, the CoT prompting strategy proved highly robust in adversarial scenarios, successfully identifying and penalizing summaries from incorrect meetings.
Performance Trends
- LLM Dominance: GPT-4o emerged as the top performer, achieving a mean score of 7.74/10 in monolingual QA. Its performance remained stable (7.69/10) in the cross-lingual setting.
- Research Gap: The best participant system (GETALP) scored 4.55/10 in QA, highlighting a significant performance gap between specialized research models (like Flan-T5) and general-purpose frontier LLMs.
- Metric Instability: Traditional model-based metrics like BARTScore were found to be highly unstable across different datasets and years, sometimes showing negative correlations with other quality indicators.
Conclusion
The Third AutoMin Challenge confirms that while LLMs have significantly advanced the quality of automatic minuting and QA, content selection remains a primary hurdle. Future work must focus on refining evaluation metrics to provide more granular feedback and developing adversarial benchmarks to further test the reasoning capabilities of these models beyond simple summarization.
Summary updated