Improving Context Modeling in Neural Topic Segmentation

Linzi Xing, Brad Hackinen, Giuseppe Carenini, Francesco Trebbi

Abstract

Topic segmentation—the task of automatically dividing a document into topically consistent sections—is often hindered by neural models that get "distracted" by long-distance information. This paper introduces a hierarchical BiLSTM model that prioritizes local context through a coherence-based auxiliary task and a restricted self-attention mechanism. The result is a state-of-the-art segmenter that is more robust across different domains and languages including English, German, and Chinese.

Introduction

Topic segmentation is a fundamental task in natural language processing, essential for document summarization, information retrieval, and navigation. While humans can intuitively identify shifts in topic (e.g., from "History" to "Geography" in a Wikipedia article), automated systems often struggle. Modern neural approaches typically rely on Recurrent Neural Networks (RNNs), such as LSTMs, to process text. However, the "long memory" of LSTMs, designed to capture long-distance dependencies, can be a liability in topic segmentation. A boundary between two sections depends primarily on the immediate transition between adjacent sentences; information from distant paragraphs often acts as noise. This research addresses the need for models to focus on local context and semantic coherence.

Methodology

The proposed architecture builds upon a Hierarchical Attention BiLSTM (HAN), enhanced with three primary components:

  1. BERT Integration: Task-specific BiLSTM sentence embeddings are concatenated with pre-trained BERT embeddings, combining structural awareness with deep semantic knowledge.
  2. Coherence-Related Auxiliary Task (AUX): During training, the model performs a secondary task: predicting whether adjacent sentences belong to the same segment. This forces internal representations to capture the "semantic glue" holding a topic together.
  3. Restricted Self-Attention (RSA): Unlike standard self-attention which considers the entire document, RSA limits the attention window to immediate neighbors (e.g., 3 sentences). This mimics human perception of topic shifts through local changes in vocabulary and tone.
The proposed model architecture featuring Restricted Self-Attention and the Auxiliary Task Module for coherence modeling.

The final model is optimized using a joint loss function that balances the main segmentation task and the coherence auxiliary task, as shown in the architecture diagram above.

Experimental Setup

The model was evaluated across three scenarios:

  • Intra-Domain: Using CHOI (synthetic), RULES (U.S. Federal Register), and WIKI-SECTION (English Wikipedia).
  • Domain Transfer: Training on Wikipedia and testing on specialized datasets like WIKI-50, CITIES, ELEMENTS (chemistry), and CLINICAL (medical textbooks).
  • Multilingual: Evaluation on German (SECTION-DE) and Chinese (SECTION-ZH) Wikipedia.

Results

Performance was measured using the $P_k$ error score, where a lower score indicates better performance. The proposed model (+AUX+RSA) consistently outperformed baselines and previous state-of-the-art methods.

Comparison of P_k scores across different datasets. The combined model (+AUX+RSA) achieves the lowest error rates.

On the English Wikipedia (SECTION) dataset, the model achieved a $P_k$ of 9.7, a significant improvement over the previous state-of-the-art (TextSeg) at 12.6. In domain transfer tests, the model demonstrated superior robustness, particularly on the challenging Clinical textbook dataset, where it achieved a $P_k$ of 30.5 compared to the basic model's 33.8.

Discussion and Conclusion

The results highlight that for topic segmentation, "forgetting" distant information is a feature. By restricting the attention window, the model avoids noise from unrelated segments. Furthermore, the auxiliary task and RSA are complementary; while RSA limits input scope, the auxiliary task refines the training objective, leading to a more specialized and effective model.

One limitation noted was that the auxiliary task provided no benefit for the Chinese dataset, likely due to shorter, more fragmented sentence structures in Chinese Wikipedia making the coherence signal too weak. Future work will explore applying these local-context strategies to dialogue segmentation and integrating more complex discourse dependencies.

Summary updated

BibTeX
@inproceedings{xing2020improving,
  author = {Linzi Xing and Brad Hackinen and Giuseppe Carenini and Francesco Trebbi},
  title = {{Improving Context Modeling in Neural Topic Segmentation}},
  booktitle = {AACL/IJCNLP},
  year = {2020},
  url = {https://openreview.net/forum?id=H8IuVv6ENSm},
}
Figure