Introduction
We have all been in meetings that spiral into off-topic tangents, leaving the actual agenda untouched. While the natural language processing (NLP) community has spent years perfecting meeting summarization—generating a retrospective "TL;DR"—there has been surprisingly little research into agenda relevance. This is the ability to determine if a live conversation actually matches the pre-planned topics.
To address this, researchers at Microsoft have introduced the Topic-Conversation Relevance (TCR) dataset. As the first major benchmark designed to measure how well a meeting’s transcript aligns with its intended agenda, TCR aggregates over 1,500 meetings and 22 million words. It provides a foundational stepping stone for building "virtual moderators" that can keep professional discussions on track and quantify meeting effectiveness.
The "Agenda Gap"
The primary challenge in measuring topic relevance is that most existing datasets are "post-hoc." They provide summaries of what was said, rather than the agenda of what should have been said. Using a summary as a "topic" is essentially cheating, as real-world agendas are often vague and aspirational.
To bridge this "Agenda Gap," the authors created a dataset where conversation snippets are mapped to specific, pre-defined topics. This allows machine learning models to detect when a group has drifted off-course.
Methodology
The authors employed a multi-pronged approach to build and validate the TCR dataset:
- Data Aggregation & Re-annotation: The team pulled from several public datasets, including ICSI and MeetingBank. Crucially, they used GPT-4 to rewrite detailed, retrospective meeting minutes into a "pre-meeting agenda" style. This ensures models are tested against the brief, goal-oriented topics found in real-world calendar invites.
- New Data Collection (SIM): They collected the Speech Interruption Meetings (SIM) dataset, consisting of 84 multi-party online meetings where participants actively debated specific topics.
- Synthetic Augmentation: To test a model's ability to detect "off-topic" segments, researchers created scripts to inject irrelevant conversation snippets into meetings or remove topics entirely. This created a robust test bed for binary classification (Discussed vs. Not Discussed).
- Benchmarking: Using GPT-4-32k, they established baseline performance across different meeting lengths (5, 10, and 15-minute snippets) to evaluate how well the LLM could identify which agenda items were actively being addressed.
The TCR Dataset
The resulting TCR dataset is a massive collection of 1,506 unique meetings and over 15,000 topics (combining unique meetings and re-annotated variations).
The corpus spans highly diverse environments:
- SIM & SIM_syn100: Newly collected and synthetic meetings focusing on natural, high-interaction debates.
- MeetingBank: 1,100 highly structured city council meetings.
- ICSI & QMSum: Academic and multi-domain meetings.
- NCPC: US National Capital Planning Commission meetings.
- ELITR: A corpus of European parliament and technical meetings.
Key Findings and Benchmarks
When evaluating GPT-4's ability to detect if a topic was "Not Discussed" (the positive class for their off-topic detection task), the researchers uncovered several critical insights about meeting structure and model performance.
- The "Structure" Premium: Model performance is highly dependent on the style of the meeting. In highly structured environments with clear speaker transitions like city council meetings (MeetingBank) and the NCPC, GPT-4 achieved near-perfect F1 scores (between 0.96 and 0.98).
- Struggles in Informal Settings: In messy, brainstorming-heavy environments (like ICSI) or technical discussions (ELITR), performance dropped significantly, with F1 scores hovering between 0.72 and 0.86.
- Precision vs. Recall: Across almost all datasets, the model showed very high precision (often >0.90). When the model flags that a topic is being ignored, it is usually correct. However, recall was lower in unstructured meetings, suggesting the model sometimes misses subtle, organic transitions into new topics.
- The Multi-Topic Challenge: Models struggle heavily when a single transcript snippet contains blurred boundaries—the exact moments where one agenda item ends and another begins.
Limitations and Future Work
The authors acknowledge a few primary hurdles moving forward. First is the privacy wall: it is notoriously difficult to obtain high-quality data from real corporate meetings due to sensitive business information, forcing reliance on public government or academic recordings.
Second is granularity. Currently, the benchmarks look at 5- to 15-minute snippets. Future work aims to move toward sentence-level relevance, which would allow a virtual moderator to nudge a conversation back on track in real-time. Finally, while the current focus is English, the team plans to expand the TCR framework to other languages using reliable translation services.
Summary updated