Abstract
We propose a new, hybrid document embedding approach, Terminology-based Document Embedding (TDE), to address the challenge of computing document similarities based on technical content. By employing state-of-the-art graph techniques, we extract keyphrases to score sentences within a document. We propose two embedding methods and demonstrate that they outperform standard baselines by up to 27% in terms of Normalized Discounted Cumulative Gain (NDCG), as validated by domain expert annotations.
Introduction
Calculating document similarity is fundamental to search engines and recommendation systems. However, technical content presents unique challenges. Standard embedding methods often struggle to distinguish between specific technical nuances and general domain "noise." For instance, a document describing a "novel insulin pump for diabetes" might be incorrectly flagged as similar to a generic medical disclaimer simply because both contain medical terminology.
Technical documents frequently contain "noise"—such as privacy policies, legal disclaimers, or generic filler—that dilutes specific technical signal. The goal of this research is to create a numerical representation (an embedding) that prioritizes technical terminology to identify truly relevant matches.
Methodology: Terminology-based Document Embedding (TDE)
The proposed TDE framework utilizes a hybrid approach that progresses from words to graphs to sentences:
- Graph-of-Words Representation: Instead of a "bag of words," the document is modeled as a graph where nodes represent terms and edges represent word co-occurrences within a sliding window.
- K-Core Extraction: The "k-core" algorithm is applied to identify the most cohesive subgraphs. The "main core" represents the most interconnected and essential keyphrases (e.g., "artificial intelligence").
- Sentence Scoring: Sentences are ranked based on the importance of the keyphrases they contain. Sentences belonging to the "main core" receive the highest weights.
- Hybrid Embedding: The final document embedding is a weighted average of its sentence embeddings. Two variants were tested:
- TDE_iw: Uses IDF-weighted word embeddings to construct sentence vectors.
- TDE_s2v: Utilizes the Sent2Vec model to generate sentence vectors directly.
Experimental Setup
The researchers constructed a dataset by crawling the websites of approximately 68,000 startups, resulting in 3.4 million webpages. After filtering for English content and removing "noisy" pages (like privacy policies) using SVM classifiers, the final dataset comprised 43,000 startups and 2.8 million pages. For evaluation, 100 startups were selected from four domains: medical, agriculture, energy, and biology.
Key Results
The performance was evaluated using NDCG (Normalized Discounted Cumulative Gain) based on rankings provided by domain experts. The results compare the proposed TDE methods against standard baselines: Doc2Vec (D2V) and a Term Weighting Approach (TWA).
As shown in the table, TDE_s2v significantly outperformed the baselines. It achieved an NDCG@1 of 0.69, representing a 27% improvement over the strongest baseline (TWA) for the top-ranked result. This suggests the model is highly effective at identifying the most relevant technical documents.
Discussion and Insights
The success of the TDE approach highlights several key insights:
- Cohesion over Centrality: The k-core approach captures composite technical terms better than frequency-based or centrality-based algorithms like PageRank.
- Sentences as Semantic Units: Weighting at the sentence level allows the model to naturally ignore "filler" text that often degrades the performance of global models like Doc2Vec.
- Contextual Importance: The relatively poor performance of Doc2Vec in this context suggests that for technical similarity, capturing specific semantic context is more critical than modeling global document structure.
Limitations and Future Work
The TDE method relies on effective initial noise filtering. Its performance is sensitive to the quality of the preprocessing stage. Additionally, the computational complexity of building graph-of-words and calculating k-cores is higher than simpler methods like TF-IDF. Future work will focus on optimizing these processes for production environments and testing the scalability of TDE on longer technical documents, such as patents or whitepapers.
Summary updated