Abstract
We propose in this paper a new, hybrid document embedding approach in order to address the problem of document similarities with respect to the technical content. To do so, we employ a state-of-the-art graph techniques to first extract the keyphrases (composite keywords) of documents and, then, use them to score the sentences. Using the ranked sentences, we propose two approaches to embed documents and show their performances with respect to two baselines. With domain expert annotations, we illustrate that the proposed methods can find more relevant documents and outperform the baselines up to 27% in terms of NDCG.
Introduction
Finding similar documents is a core machine learning task, but standard methods often struggle when the goal is to match specific technical content rather than broad topical overlap. For instance, a document describing a novel medical device might be matched with general medical literature rather than documents discussing similar technical mechanisms.
Standard approaches like tf-idf or doc2vec treat documents holistically, often diluting specific technical terminology with general domain vocabulary and irrelevant boilerplate. This paper introduces Terminology-based Document Embedding (TDE) to solve this by selectively weighting document segments that contain dense technical keyphrases, effectively filtering noise to focus on core technology.
Methods
The TDE approach combines graph-based keyword extraction with neural text embeddings in a three-step pipeline:
- Graph-of-Words & k-core Extraction: Documents are represented as graphs where words are nodes and edges represent co-occurrences within a sliding window. A k-core decomposition identifies cohesive, interconnected subgraphs representing composite technical keyphrases.
- Sentence Scoring: Sentences are scored based on the presence of these keyphrases. Sentences containing phrases from the most central "cores" receive higher weights.
- Document Embedding: The final embedding is a weighted average of sentence embeddings.
Two variants are evaluated: TDE_s2v (using pre-trained sent2vec) and TDE_iw (using tf-idf weighted averages of word2vec embeddings).
Experimental Setup
The models were trained on a custom dataset of 2.8 million pages from ~43,000 startups. Evaluation was conducted on a curated set of 100 startup documents across four domains (medical, agriculture, energy, biology). Domain experts manually evaluated the top 5 similar startups retrieved by each method to calculate Normalized Discounted Cumulative Gain (NDCG).
Results
The proposed TDE methods significantly outperformed the baselines. TDE_s2v achieved the best performance with an NDCG@1 of 0.69, representing a 27% improvement over the strongest baseline (TWA).
The doc2vec (D2V) baseline performed poorly, likely due to the fragmented nature of web-crawled text which lacks the long-form coherence doc2vec typically requires.
Conclusion
Weighting sentences by technical keyphrase density effectively filters "marketing fluff" and general noise, leading to more precise technical matching. The use of unsupervised k-core graph decomposition provides a robust way to identify terminology without external knowledge bases. While the current pipeline relies on proprietary pre-filtering, the core TDE methodology demonstrates substantial gains in technical document retrieval.
Summary updated