Putting Back the Stops: Integrating Syntax with Neural Topic Models

Mayank Nagda, Sophie FellenzIJCAI 2024 · OpenReview

Abstract

Syntax and semantics are two key concepts for language understanding. Topic models typically represent the semantics of a text corpus, while removing syntactic information during preprocessing. Without preprocessing, the generated topics become uninterpretable because the syntactic words dominate generated topics. To learn interpretable topics while keeping valuable syntactic information, we propose a novel framework that can simultaneously learn both syntactic and semantic topics from the corpus without requiring any preprocessing. A context network leverages textual dependencies to distinguish between syntactic and semantic words, while a composite VAE topic model learns two sets of topics. We demonstrate on seven datasets that our proposed method effectively captures both syntactic and semantic representations of a corpus while outperforming state-of-the-art neural topic models and statistical topic models in terms of topic quality.

The Challenge of Stop Words in Topic Modeling

In Natural Language Processing (NLP), topic models are essential for discovering underlying themes in large document collections. However, these models face a persistent "stop word" problem. High-frequency words like a, is, and to—the syntactic "glue" of language—often drown out meaningful content. If a model is trained on raw, unprocessed text, the resulting topics are frequently dominated by grammar, rendering them useless for semantic analysis.

Traditionally, researchers solve this by manually removing stop words or using frequency filters. This approach is flawed for several reasons:

  1. Inconsistency: Stop-word lists vary between researchers, making results difficult to replicate or compare.
  2. Information Loss: Some "stop words" carry stylistic or domain-specific importance (e.g., the word "ingredients" in a culinary dataset).
  3. Manual Effort: It requires human intervention for every new language or specialized domain.

Methodology: The SyConNTM Framework

To address these issues, we introduce SyConNTM (Syntactic Context Neural Topic Model). This framework is built on the linguistic intuition that syntax is local (dependent on immediate neighbors), while semantics is global (related to the document's overall theme).

The architecture of SyConNTM, illustrating the integration of the Context Network and the Composite VAE to separate syntactic and semantic information.

As shown in the architecture above, the model consists of three primary components:

  • Context Network: A language model that predicts a word based on its immediate neighbors. If a word is highly predictable from its local context, it is likely a syntactic "stop word."
  • Decision Module: This module uses the context network's output to categorize words. Words that rank in the "Top-M" predicted words for a slot are labeled as syntactic; others are labeled as semantic.
  • Composite VAE: A Variational Auto-Encoder with two distinct latent subspaces. One subspace reconstructs the Bag-of-Words (BoW) for syntactic words, while the other handles semantic words. This results in dual decoders: one for syntactic topics (grammar, style) and one for semantic topics (themes).

Qualitative Analysis: Disentangling Syntax and Semantics

SyConNTM enables automatic stop-word discovery without manual lists. It identifies corpus-specific syntactic words that form a "template" for writing. For instance, in a review dataset, a syntactic topic might provide the structural bones ("very [semantic word] service..."), leaving the semantic topics to focus on specific cuisines or atmospheres.

A comparison of topics generated by SyConNTM versus baseline models, demonstrating the clear separation of syntactic and semantic words in our approach.

As illustrated in the comparison table, SyConNTM (Ours) successfully isolates syntactic elements (punctuation, pronouns, common verbs) into a dedicated space, allowing semantic topics to remain "pure" and meaningful. In contrast, baseline models trained on raw text produce topics heavily contaminated by syntactic noise.

Key Results and Performance

We evaluated SyConNTM across seven diverse datasets, including 20 Newsgroups, Amazon Reviews, and Government Reports. The model demonstrated several key advantages:

  • Superior Topic Quality: SyConNTM outperformed state-of-the-art models like D-VAE and ETM in Topic Quality (TQ), a metric that balances coherence and diversity.
  • Enhanced Feature Extraction: When the semantic words identified by our model were used as features for a Naive Bayes classifier, they achieved significantly higher accuracy (e.g., 89.7% on 20NG) compared to standard manual stop-word removal (75.2%).
  • Robustness to Scale: The model maintains high performance even when scaled to a large number of topics (e.g., 200), where traditional models often suffer from redundancy.

Conclusion and Future Work

SyConNTM represents a significant step toward automated, preprocessing-free topic modeling. By "putting back the stops," we can capture both the structural template and the thematic content of a corpus. Future research may explore finer-grained categories beyond the binary syntax/semantics split, such as specific parts of speech, and investigate how these disentangled representations can improve the generation of grammatically structured, coherent text.

Summary updated

Abstract

Syntax and semantics are two key concepts for language understanding. Topic models typically represent the semantics of a text corpus, while removing syntactic information during preprocessing. Without preprocessing, the generated topics become uninterpretable because the syntactic words dominate generated topics. To learn interpretable topics while keeping valuable syntactic information, we propose a novel framework that can simultaneously learn both syntactic and semantic topics from the corpus without requiring any preprocessing. A context network leverages textual dependencies to distinguish between syntactic and semantic words, while a composite VAE topic model learns two sets of topics. We demonstrate on seven datasets that our proposed method effectively captures both syntactic and semantic representations of a corpus while outperforming state-of-the-art neural topic models and statistical topic models in terms of topic quality.
BibTeX
@inproceedings{nagda2024putting,
  author = {Mayank Nagda and Sophie Fellenz},
  title = {{Putting Back the Stops: Integrating Syntax with Neural Topic Models}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=uAo7f3QHBN},
}
Figure