In natural language processing, a word’s meaning is often treated as a function of its immediate neighbors. If you see the word "bank" surrounded by "water" and "flow," you know it refers to a river. However, if the local context is sparse—such as "I went to the bank"—the meaning becomes ambiguous.
To solve this, researchers have developed a family of techniques that fuse local context (the immediate syntactic and semantic cues of nearby words) with global context (the overarching thematic or document-level information). This dual-scale approach allows models to resolve polysemy, handle noisy short texts, and maintain thematic consistency across long documents.
The Resolution Gap
Standard word embeddings like Word2vec or early versions of BERT primarily rely on local windows or attention mechanisms. While excellent at capturing syntax and immediate semantics, they often struggle with "global" disambiguation. As Eric H. Huang and colleagues argued in Improving Word Representations via Global Context and Multiple Word Prototypes, local context is sufficient for determining parts of speech, but global document-level features are essential for resolving semantic ambiguity across different topics.
By combining local n-gram context with document-wide features, models can learn multiple "prototypes" for a single word. This ensures that "star" in a document about astronomy is represented differently than "star" in a document about Hollywood, even if the immediate sentence structure is similar.
Jointly Learning Words and Topics
One powerful way to integrate these signals is to learn word embeddings and latent topics simultaneously rather than in a two-step pipeline. In Jointly Learning Word Embeddings and Latent Topics, Bei Shi and Wai Lam introduced the Skip-gram Topical word Embedding (STE). This framework allows word representations to vary based on the topic, ensuring that fine-grained semantic relationships are disambiguated by the global context.
Similarly, Yu Meng and colleagues proposed Spherical Text Embedding, which jointly learns word and document-level embeddings on the surface of a unit sphere. This joint optimization forces the model to capture both local co-occurrence patterns and global thematic context in the same vector space, improving the quality of both representations.
Injecting Global Signals into Pre-trained Models
With the rise of Large Language Models, the focus has shifted toward "injecting" global signals into architectures like BERT. Nicole Peinelt and her team developed tBERT in tBERT: Topic Models and BERT Joining Forces for Semantic Similarity Detection. Instead of just fine-tuning BERT, they injected global topic distributions directly into the model. This hybrid approach leverages the broad thematic overview of topic models to clarify ambiguities that local syntax cannot resolve, which is particularly effective in domain-specific contexts like community forums.
Other researchers have used similar "injection" strategies for specialized tasks:
- Clinical Diagnostics: Akshay Budhkar demonstrated in Augmenting word2vec with latent Dirichlet allocation within a clinical application that treating topic labels as "pseudo-words" during training helps embeddings capture the subtle linguistic shifts indicative of Alzheimer’s disease.
- Sentiment Analysis: Peng Fu introduced a Global Sentiment Representation (GSR) in Learning Sentiment-Specific Word Embedding via Global Sentiment Representation, which treats sentiment as a document-wide property to prevent words with opposite polarities (like "good" and "bad") from having identical local embeddings.
Handling Noisy and Length-Varying Text
The fusion of local and global signals is especially critical when comparing texts of vastly different lengths, such as a short summary and a full research paper. Standard models often fail here because they lack a shared "concept" space.
In Unsupervised Concept Representation Learning for Length-Varying Text Similarity, Xuchao Zhang and colleagues proposed CGNet. This model constructs document representations by combining high-level corpus concepts with specific local phrases that don't fit into broad categories. This ensures that unique, distinguishing details are preserved alongside abstract thematic information.
For noisy data like social media, Payam Karisani used a "dual-view" approach in Contextual Multi-View Query Learning for Short Text Classification in User-Generated Data. By combining a global document embedding with a local keyword-specific embedding, the model can resolve the meaning of a tweet even when the text is fragmented or slang-heavy.
Why the Hybrid Approach Works
The intuition behind these models is that language operates on multiple scales simultaneously.
- Local streams (n-grams, dependencies, attention windows) capture the "how" of language—the immediate grammatical and functional roles of words.
- Global streams (topic distributions, document summaries, corpus concepts) capture the "what"—the broader thematic intent.
As shown in Document Neural Autoregressive Distribution Estimation by Stanislas Lauly and others, maintaining separate embedding spaces for these two scales allows a model to simultaneously capture broad semantics and immediate grammatical structure, leading to significantly more robust representations than single-scale models.