When Cohesion Lies in the Embedding Space: Embedding-Based Reference-Free Metrics for Topic Segmentation

Iacopo Ghinassi, Lin Wang, Chris Newell, Matthew PurverCOLING 2024

TL;DR

This paper introduces a novel framework for evaluating topic segmentation systems without relying on human-annotated ground truth. They propose new "embedding-based" metrics that leverage sentence embeddings from language models, showing they correlate well with traditional metrics and outperform existing reference-free methods, especially with smaller, fine-tuned models.

What Problem Does This Paper Address?

Topic segmentation, the task of breaking down a text into coherent thematic units, is crucial for many NLP applications like summarization and search. However, evaluating these systems is challenging due to the scarcity and cost of creating expert-annotated datasets. Traditional evaluation metrics (like Pk and WindowDiff) require this human-annotated "ground truth." To overcome this, researchers have explored "reference-free" metrics that assess segmentation quality without ground truth. While some attempts exist, there's a lack of a formal framework and comprehensive evaluation of these methods, especially concerning their reliance on modern language model embeddings. This paper aims to fill this gap by proposing a unified framework and new metrics for reference-free topic segmentation evaluation using sentence embeddings.

Methods

The core contribution is a new framework for embedding-based reference-free topic segmentation metrics. The general idea is to use sentence embeddings to measure the "cohesion" within proposed segments. If sentences within a segment are highly similar (cohesive), and segments are distinct from each other, it suggests a good segmentation.

The authors propose several scoring functions within this framework:

  • Clustering-based Baselines: They adapt traditional clustering evaluation metrics like Davies-Bouldin Index (SegReFree) and Silhouette Score. These metrics compare segment centroids and intra-segment distances.
  • Average Relative Proximity (ARP) Methods: This is their novel contribution. They propose three variants (ARPstd, ARPcos, ARPpair) that measure the dispersion or similarity of embeddings between and within segments.
  • ARPstd/ARPcos: These focus on the variance (standard deviation for ARPstd, cosine similarity for ARPcos) of embeddings within a segment versus across segment boundaries.
  • ARPpair: This method directly computes pairwise distances between embeddings within and across segments, aiming to be more robust to variations in segment size.

They also evaluate different embedding models:

  • ROBERTa: A standard large language model.
  • MPNET: Known for strong performance on sentence similarity tasks.
  • Falcon: A more recent, large-parameter model, tested in its smaller version.

Datasets

The paper uses four diverse datasets to evaluate their methods:

  • en_city: Wikipedia articles about cities, using headings as topic shift markers.
  • en_disease: Similar to en_city but focused on medical articles, offering a more specialized lexicon.
  • QMSum: A collection of conversational datasets from meeting transcripts (ICSI, AMI, Canadian parliamentary meetings). This dataset is noted for its longer, more blurred segments and disfluencies, making it a challenging test case.
  • SBBC-RadioNews: A lightweight dataset for multimodal topic segmentation from BBC Sound collection news shows.

Key Insights and Ideas

  • Embedding Space as a Cohesion Measure: The fundamental idea that semantic similarity captured by embeddings can directly reflect topic cohesion is powerful. This moves beyond simple lexical overlap.
  • Smaller Models Can Outperform Larger Ones: Surprisingly, fine-tuned smaller models like MPNET often outperform much larger models like Falcon in terms of correlation with traditional metrics. This suggests that task-specific optimization is more critical than sheer model size for this task.
  • ARP Methods are Robust: The proposed ARP-based metrics, particularly ARPpair, show consistently strong correlations with traditional metrics across various datasets and encoders, indicating their robustness.
  • Conversational Data is Challenging: The QMSum dataset, with its conversational nature, disfluencies, and blurred boundaries, presents a significant challenge for all reference-free metrics, including the proposed ones.
  • Synthetic Evaluation Limitations: While synthetic experiments (boundary removal/transposition) are useful for understanding metric behavior, they can sometimes overestimate performance and may not fully reflect real-world complexities.

Key Results

  • High Correlation with Traditional Metrics: The proposed ARP-based metrics achieved high Pearson correlation coefficients (often over 90%) with traditional reference-based metrics (Pk, WindowDiff, Boundary Similarity) across most datasets.
  • ARPpair Outperforms Others: ARPpair generally showed the strongest and most consistent correlations, especially when compared to SegReFree and Silhouette.
  • MPNET as a Strong Encoder: MPNET consistently yielded better results than ROBERTa and Falcon, reinforcing the idea that specialized encoders are beneficial.
  • Domain Sensitivity: Performance varied across domains, with conversational data (QMSum) being the most difficult, leading to lower correlations.
  • Robustness: ARP methods demonstrated lower standard deviations in correlation coefficients compared to SegReFree and Silhouette, indicating more stable performance.

Limitations and Future Work

  • Conversational Data: The paper acknowledges that conversational data remains a challenge for their metrics, and future work could focus on developing solutions for these cases.
  • Synthetic Evaluation Nuances: The authors note that synthetic experiments might be too lenient and should be used in conjunction with real system evaluations.
  • Ethical Considerations: The paper raises ethical concerns about the potential misuse of metrics for critical decisions, job displacement for annotators, and the environmental cost of large language models.
  • Broader Applicability: The authors suggest their framework could extend to other tasks like topic modeling and text clustering.

Summary updated

BibTeX
@inproceedings{ghinassi2024when,
  author = {Iacopo Ghinassi and Lin Wang and Chris Newell and Matthew Purver},
  title = {{When Cohesion Lies in the Embedding Space: Embedding-Based Reference-Free Metrics for Topic Segmentation}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2024},
}
Figure