TL;DR
CobwebTM is a novel neuro-symbolic topic modeling framework that combines the power of pretrained text embeddings with a classic incremental clustering algorithm. It dynamically builds and updates a hierarchical topic tree from streaming text data in real time, completely avoiding catastrophic forgetting without needing a fixed topic capacity or expensive retraining.
What Problem Does This Paper Address?
Traditional topic modeling methods like Latent Dirichlet Allocation (LDA) are rigid: they require you to predefine the number of topics ($K$) and rely on bag-of-words representations that ignore word semantics.
Modern neural topic models fix the semantic issue by leveraging dense document embeddings, but they introduce new headaches. They are highly parameterized, sensitive to hyperparameters, and typically require batch training on the entire corpus. When applied to streaming data (lifelong learning), neural models suffer from catastrophic forgetting—as they learn new topics, their ability to represent older topics degrades. Furthermore, while real-world topics are naturally hierarchical (e.g., Technology $\rightarrow$ Operating Systems $\rightarrow$ Windows), most streaming models only output a flat list of topics, or require a separate, post-hoc step to build a hierarchy.
The authors introduce CobwebTM to solve this dilemma, creating a model that is simultaneously incremental, hierarchical, and semantically rich.
Methods
CobwebTM uses a two-step neuro-symbolic pipeline:
- Incremental Clustering via Continuous Cobweb: Instead of learning topic-word distributions directly, the model processes documents one by one. It converts text into dense vectors using a pretrained transformer (like RoBERTa Large). It then feeds these embeddings into a modified version of the classic Cobweb algorithm adapted for continuous-valued data.
As each document arrives, the model performs a top-down search of a topic tree, guided by an information-theoretic metric called Category Utility (CU). At each node, Cobweb evaluates four operators to update the tree structure:
- Insert: Place the document into the best-matching existing child node.
- New: Create a new singleton child node for this document.
- Merge: Combine the two best-matching sibling nodes.
- Split: Dissolve a child node, promoting its sub-children up one level.
Each node in the tree models its document distribution as a multivariate Gaussian with a diagonal covariance matrix.
- Topic Representation (c-TF-IDF): To make the clusters human-readable, the model applies class-based TF-IDF (c-TF-IDF) post-hoc. It treats all documents within a node's subtree as a single text class to extract the most descriptive keywords for that specific topic level.
To allow fair comparison with flat topic models, the authors also introduce a dynamic tree-cutting procedure based on user-defined thresholds (max_clusters and leaf_ratio) to extract a flat partition from the evolving hierarchy.
Datasets
The authors evaluated CobwebTM across two distinct experimental setups:
- Lifelong Topic Modeling (Streaming):
- Spatiotemporal News Dataset: ~1.2 million news articles (5k sampled); temporally ordered to simulate real-world topic shifts.
- TweetNER7 Dataset: ~2.8k short tweets from X/Twitter; temporally ordered.
- Stack Overflow Dataset: 5k technical forum posts; randomly shuffled to simulate a uniform incremental distribution.
- Hierarchical Topic Modeling (Batch/Static Benchmarks):
- 20 Newsgroups: 18,846 documents evenly distributed across 20 categories.
- AG News: News articles categorized into four high-level classes (50k train, 7.6k test).
- Stack Overflow: Used here to evaluate static hierarchical quality.
Key Insights and Ideas
- Variance Tracking as Feature Weighting: Because each node maintains variance estimates across the embedding dimensions, CobwebTM implicitly weights feature importance differently for different clusters. This allows it to cleanly separate ambiguous terms (e.g., keeping the word "drive" under a computing context rather than letting it drift toward automotive meanings).
- True Lifelong Learning without Replay: Unlike neural lifelong models that require experience replay buffers or complex regularization (like Elastic Weight Consolidation) to mitigate forgetting, CobwebTM’s symbolic tree structure naturally preserves past knowledge while dynamically spawning new branches for novel concepts.
- Logarithmic Efficiency: CobwebTM updates an average of $O(\log N)$ nodes per document insertion. This makes it highly scalable for long document streams where retraining a neural network from scratch would be computationally prohibitive.
Key Results
CobwebTM consistently matched or outperformed competitive baselines (including Online LDA, Lifelong NTM, and BERTopic variants) across multiple metrics:
- Topic Coherence ($C_v$): CobwebTM achieved the highest final topic coherence on all streaming datasets, showing steady growth as more documents were observed.
- Extreme Stability: The model demonstrated near-perfect temporal stability, scoring an Adjusted Rand Index (ARI) of 0.997 on Stack Overflow and 0.984 on Spatiotemporal News, indicating that topic assignments remain highly consistent over time.
- Superior Hierarchical Quality: On static benchmarks, CobwebTM achieved the highest Topic Coherence (NPMI) and Parent-Child Coherence (PCC), proving that its vertically generated subtopics are highly meaningful specializations of their parent topics.
Limitations and Future Work
Despite its strong performance, the authors note a few limitations:
- Post-Hoc Word Extraction: Topic keywords are generated via c-TF-IDF after clustering, rather than being natively integrated into the generative process.
- Encoder Dependency: The quality of the hierarchy is fundamentally bounded by the representational capacity of the underlying pretrained embedding model.
- Order Sensitivity: Like most incremental clustering algorithms, the final tree structure can be sensitive to the order in which documents arrive.
- Memory Growth: Maintaining statistics over massive, long-running hierarchies will eventually incur high memory costs, necessitating future research into tree-pruning or compression strategies.
Future Directions: The authors propose extending CobwebTM to multimodal topic modeling (integrating images and audio via vision-language models) and using node-level statistics to automatically generate abstractive topic summaries.
Summary updated