When we read a long document, we intuitively sense when the author shifts from one subtopic to another. These transitions are rarely random; they are marked by a shift in the "vocabulary of the moment." A discussion about the engine of a car uses a different set of nouns and verbs than a discussion about its interior design.
Researchers have spent decades formalizing this intuition into algorithms that can automatically partition unstructured text into coherent segments. The core mechanism across this research is the detection of lexical cohesion valleys: points in a text where the semantic connection between adjacent blocks of words reaches a local minimum.
The Sliding Window Intuition
The foundational approach to this problem, pioneered by Marti A. Hearst in the TextTiling algorithm, treats a document as a sequence of vocabulary distributions. To find a boundary, the algorithm moves a sliding window across the text. This window is divided into two adjacent blocks.
At each step, the system calculates the similarity between these two blocks. If the blocks are part of the same subtopic, they will share many words (high lexical cohesion). As the window slides across a transition point, the first block contains words from the old topic while the second block introduces words from the new one. This mismatch causes the similarity score to drop.
By plotting these scores across the entire document, we get a "cohesion profile." The "valleys" or "troughs" in this graph represent the most likely locations for topic boundaries. Hideki Kozima expanded this by using a Lexical Cohesion Profile (LCP), which uses spreading activation on semantic networks to find these valleys even when the exact words don't repeat, but related concepts do Text Segmentation Based on Similarity between Words.
Measuring the Depth of a Shift
Not every dip in similarity is a true topic change; some are just minor fluctuations in prose. To filter out the noise, researchers like Stefan Kaufmann 0001 and Marti A. Hearst introduced depth scores.
A depth score measures how deep a valley is relative to the "peaks" (high similarity areas) on either side of it. A shallow dip might just be a transition between paragraphs within the same topic, while a deep trough indicates a significant shift in the underlying vocabulary. These troughs are then aligned with natural breaks, like paragraph markers, to finalize the segment Multi-Paragraph Segmentation of Expository Text.
From Word Counts to Semantic Embeddings
Early methods relied on simple word repetition or cosine similarity of term frequency vectors. However, language is flexible; an author might use "automobile" in one sentence and "car" in the next.
To solve this, the field moved toward semantic representations. Stefan Kaufmann 0001 developed VecTile, which replaced raw word counts with "context vectors"—an early precursor to modern word embeddings Cohesion and Collocation: Using Context Vectors in Text Segmentation. Similarly, Gaël Dias introduced InfoSimba, a measure that accounts for word co-occurrences to detect thematic shifts even without direct word overlap Topic Segmentation Algorithms for Text Summarization and Passage Retrieval: An Exhaustive Evaluation.
In the era of Large Language Models (LLMs), this "valley-finding" logic remains central to semantic chunking. For example, Renyi Qu explores how calculating embedding distances between adjacent sentences can identify optimal split points for Retrieval-Augmented Generation (RAG), ensuring that retrieved "chunks" are topically self-contained Is Semantic Chunking Worth the Computational Cost?.
Local Context is Often Enough
A recurring discovery in this field is that topic shifts are "local" phenomena. While it might seem intuitive to look at the entire document to understand its structure, Boris Dadachev and colleagues found that models focusing only on the immediate context around a potential boundary often perform as well as, or better than, complex hierarchical models Text Segmentation by Cross Segment Attention.
This suggests that the "signal" of a topic change is concentrated at the transition itself. Whether it's a shift in phoneme distributions in a spoken lecture Recent progress in the MIT spoken lecture processing project or a change in the formatting of a PDF table Table Header Detection and Classification, the most reliable indicator of a boundary is a sudden, local drop in the similarity of the content stream.