When you push a topic model to extract hundreds of niche topics from a corpus—a regime known as high-granularity topic modeling—you often run into the "over-clustering" problem. You end up with a mess of redundant subtopics that make the model hard to interpret.
The standard fix is to merge topics that are "too similar" based on vector distance or to let a Large Language Model (LLM) clean up the list. However, a new paper from researchers at Heriot-Watt University and STMicroelectronics argues that these common fixes are fundamentally flawed for high-granularity tasks.
TL;DR
The authors propose Embedding-Gated Iterative Merge (EGIM), a hybrid method that uses embedding similarity to find candidate topic pairs and then uses an LLM—grounded in actual document evidence—to decide whether to merge them. They find that 83% of "highly similar" topics are actually meaningfully distinct, and their method prevents the "semantic snowballing" that causes other models to collapse into a few generic mega-topics.
What Problem Does This Paper Address?
In fields like patent informatics or legal mapping, researchers need to find the "niche" topics. Standard benchmarks usually stop at 100 topics, but real-world tasks often require 300 or more.
The core challenge is distinguishing between genuine redundancy (two topics describing the same thing) and semantic similarity (two topics that are related but distinct).
Current approaches fail here:
- Threshold-based methods: These merge topics based on a cosine similarity score. The problem? As Figure 1 shows, there is a massive overlap in similarity scores between topics that should be merged and those that should be kept separate.
- LLM-guided methods: These ask an LLM to look at a list of topics and merge them. This often leads to "semantic snowballing," where a merged topic becomes slightly broader, which then attracts more merges, eventually collapsing the entire distribution into a few uselessly broad categories.
Methods
The authors introduce EGIM, which treats refinement as a "merge-or-keep" loop:
- Rich Representation: Instead of just keywords, they use an LLM to generate a descriptive label and summary for every topic.
- Coverage-Based Sampling: They cluster these topic embeddings using k-means. Instead of just picking the top-N most similar pairs globally, they pick the most similar pair from each cluster. This ensures they are cleaning up the whole topic space, not just one dense area.
- Gated LLM Judgment: For each candidate pair, they send the labels, summaries, and—crucially—representative documents to an LLM. The LLM makes a binary choice: Merge or Keep.
- Iterative Convergence: The process repeats until no more merges occur for a set number of rounds.
To measure success, they also introduced two new diagnostics:
- Topic Dispersion: A metric that penalizes "mega-topics" and rewards a balanced distribution.
- Topic Discriminability: A measure of how well a topic's representation actually matches its assigned documents compared to the rest of the corpus.
Datasets
The researchers evaluated EGIM on three specialized corpora where fine-grained distinctions matter:
- SPAD Patents: 1,102 documents on Single-Photon Avalanche Diodes.
- Laser Defense Patents: 3,302 documents on defense-related laser technology.
- Legal Case Reports: 3,890 Australian Federal Court cases.
Key Insights and Ideas
- Similarity is not Redundancy: The most striking finding is that the LLM rejected 83% of the candidates proposed by embedding similarity. In high-granularity regimes, "similar" usually just means "related," and merging them destroys the very niche distinctions the model was built to find.
- The "Snowball" Effect: Without the "gate" of document-level evidence, iterative merging tends to create "gravity wells." For example, in the Laser corpus, a baseline model collapsed distinct topics like LIDAR, explosive detection, and laser guidance into a single generic "Laser Targeting" topic. EGIM correctly kept them separate.
- Document Grounding is Non-Negotiable: By providing the LLM with actual text snippets from the topics, the decision moves from "do these labels sound the same?" to "does the evidence show these are the same concept?"
Key Results
EGIM significantly outperformed the iterative LLM baseline (LLM-Iter) across the board:
- Discriminability: EGIM improved topic discriminability by 25–55% over the baseline (see Table 3).
- Preserving the Distribution: EGIM preserved over 94% of the original topic dispersion. In contrast, the baseline collapsed the SPAD corpus dispersion down to just 34%, effectively ruining the model's granularity.
- Efficiency: In an ablation study, their k-means sampling strategy found 56% more true duplicates than a simple greedy similarity ranking.
Limitations and Future Work
- Computational Cost: EGIM is more expensive than simple thresholding, requiring roughly 2.9x more LLM calls than the baseline because it is more selective.
- Model Sensitivity: The study used a specific embedding model (Qwen3-0.6B) and LLM (GPT-4o). The authors note that different models might shift the similarity distributions.
- Language: The current study is limited to English-language corpora.
- Future Directions: The authors suggest exploring how domain-specific fine-tuning of the embedding models might further improve the initial candidate selection.
Summary updated