Introduction
In Natural Language Processing (NLP), modeling "in-domain" text streams—such as niche product reviews or specialized news feeds—presents a significant challenge. These data "chunks" are often too small to provide the global context necessary for identifying broad topics or the local context required to train accurate word embeddings.
Existing approaches typically suffer from two main issues: a reliance on generic pre-trained embeddings (like GloVe) that fail to capture domain-specific nuances (e.g., "apple" in tech vs. cooking), and isolated learning processes that treat topic modeling and word embedding as separate tasks. Furthermore, many models experience "catastrophic forgetting," where learning a new domain causes the model to lose insights gained from previous ones.
The Lifelong Collaborative Model (LCM)
The Lifelong Collaborative Model (LCM) addresses these challenges by jointly learning coherent topics and domain-specific word embeddings from a continuous stream of text. Built on Non-negative Matrix Factorization (NMF), LCM is designed for flexibility and interpretability, producing sparse results that are easier for humans to analyze.
As shown in the architecture above, the model utilizes two specialized structures:
- Knowledge Graph (KG): This serves as the model's long-term memory, accumulating word relationships discovered in previous domains to combine topical (global) and contextual (local) information.
- Subword Graph (SG): To handle rare words or typos common in small datasets, LCM uses Byte Pair Encoding (BPE) to break words into sub-units. This graph captures the internal structure of the domain's vocabulary, allowing the model to infer meanings for words with low frequency.
LCM optimizes a single objective function that factorizes the word-document matrix (for topics) and the word co-occurrence matrix (for embeddings) simultaneously, constrained by the insights stored in the KG and SG.
Experimental Setup
The researchers evaluated LCM using the Amazon Review dataset, selecting 9 distinct in-domain corpora, including "Pet Behavior Center," "SIM Cards & Prepaid Minutes," and "Cult Movies." The first seven domains were used to build the lifelong knowledge base, while the final two ("Science Education" and "Cult Movies") served as test cases to evaluate how well the model adapted its accumulated knowledge to new tasks.
Results and Discussion
LCM demonstrated superior performance across multiple dimensions. In terms of Topic Coherence, the model achieved significantly higher Normalized Pointwise Mutual Information (NPMI) and Topic Uniqueness (TU) scores compared to baselines like LDA-LTM and NMF-LTM. This indicates that the topics identified were both more logical and more distinct.
Key findings include:
- Synergy through Collaboration: Learning topics and embeddings together proved more effective than isolated learning. Topical information refined the embeddings, while the local context from embeddings improved topic coherence.
- Knowledge Retention: Unlike neural networks that overwrite old data, LCM’s Knowledge Graph allows it to retain and even improve its performance on older domains using insights from newer ones.
- Interpretability: The model produced highly sparse embeddings (90-94% zeros). This sparsity allows specific dimensions to correspond to human-understandable features, such as distinguishing between the biological and technological meanings of the word "cell."
Conclusion and Future Work
LCM offers a robust framework for lifelong learning in niche domains by leveraging subword information and a collaborative learning objective. While the model currently faces computational complexity challenges due to matrix operations proportional to vocabulary size ($M^2$), future work aims to implement "divide-and-conquer" strategies and incorporate contextualized representations like BERT to further enhance performance on polysemous words.
Summary updated