Research on Similarity Detection of Massive Text Based on Semantic Fingerprint

Xiaolin Jin, Shuwu Zhang, Jie Liu, Hu Guan2018 · DOI

Abstract

In order to find the required information quickly and efficiently in massive texts, this paper proposes a method of combining semantic fingerprint with cosine distance. After text preprocessing for Chinese texts, the Term Frequency-Inverse Document Frequency algorithm is used to extract feature words of the text, and then screen the text initially by the Simhash algorithm, finally compare these candidate texts tby using the cosine distance for the second similarity to extract the most similar texts. Based on a single Simhash algorithm, the proposed method can greatly improve the accuracy and recall under the modified textual environment, and can also meet the needs of massive texts' similarity testing requirements. Therefore, this method of combining semantic fingerprint with cosine distance can effectively make up for the problem of high false positive rate of Simhash algorithm and is more suitable for the similarity detection of massive texts in fact.

Introduction

In the era of the "information explosion," finding similar documents in massive datasets is like looking for a needle in a haystack. Traditional methods like the Vector Space Model (VSM) are highly accurate but computationally expensive because they require comparing every word vector in a document. Conversely, hashing algorithms like Google’s Simhash are incredibly fast and memory-efficient, but they suffer from high false-positive rates.

The specific challenge addressed is the "fuzziness" of Simhash. While Simhash is excellent at finding near-duplicates, it often fails or misjudges similarity when a text has been slightly modified—such as through deletions, additions, or word-order swaps. This research seeks a "best of both worlds" solution: a system fast enough for massive data but precise enough to handle modified content.

Methodology

The proposed multi-step pipeline enhances standard feature extraction and adds a secondary verification layer to balance speed and precision.

  1. Enhanced Feature Extraction (TF-IDF + POS): Instead of just using standard Term Frequency-Inverse Document Frequency (TF-IDF), the authors incorporate Part-of-Speech (POS) weighting. Nouns carry the most "feature" weight, followed by verbs and adjectives, ensuring the semantic fingerprint is built on the most meaningful words.
  2. Initial Screening (Simhash): The system generates a binary fingerprint for each document. It uses the Hamming distance to quickly filter out documents that are clearly different, leaving a small pool of "candidate" texts.
  3. Secondary Verification (Cosine Distance): To solve the false-positive problem of Simhash, the candidate texts undergo a second, more rigorous comparison using Cosine Similarity. This calculates the actual angle between word frequency vectors, providing a much more granular measure of similarity.
Flowchart of the proposed hybrid similarity detection pipeline

Experimental Setup

The researchers evaluated their method using the THUCNews dataset, comprising 219,173 news items across domains like finance, real estate, and stocks. To test robustness, they created a subset of 500 news items modified by approximately 20% (including deletions, additions, and position exchanges) to simulate real-world textual variations.

Results and Discussion

The hybrid method outperformed a standalone Simhash approach across the board, particularly on modified texts. Through experimentation, the authors found that a Hamming distance threshold of 16 provided the optimal balance for the initial Simhash screen.

  • Accuracy: On modified text, the proposed method achieved 97.6% accuracy, compared to only 86.4% for the standard Simhash algorithm.
  • Recall: The recall rate for modified text was 96%, significantly higher than Simhash’s 83.4%.
Comparison of recall and accuracy between the proposed method and standard Simhash

While the hybrid method is technically slower than pure Simhash due to the second stage of calculation, the time consumption remains well within acceptable limits for massive text processing.

Conclusion

This research demonstrates that a coarse-to-fine filtering approach effectively mitigates the high false-positive rate of Simhash. Future work will focus on moving toward sentence-level similarity for even more refined detection and further optimizing the candidate selection process to reduce computational overhead. Additionally, while the current preprocessing is tailored for Chinese text, adapting this to other languages remains an area for further exploration.

Summary updated

Abstract

In order to find the required information quickly and efficiently in massive texts, this paper proposes a method of combining semantic fingerprint with cosine distance. After text preprocessing for Chinese texts, the Term Frequency-Inverse Document Frequency algorithm is used to extract feature words of the text, and then screen the text initially by the Simhash algorithm, finally compare these candidate texts tby using the cosine distance for the second similarity to extract the most similar texts. Based on a single Simhash algorithm, the proposed method can greatly improve the accuracy and recall under the modified textual environment, and can also meet the needs of massive texts' similarity testing requirements. Therefore, this method of combining semantic fingerprint with cosine distance can effectively make up for the problem of high false positive rate of Simhash algorithm and is more suitable for the similarity detection of massive texts in fact.
BibTeX
@misc{jin2018research,
  author = {Xiaolin Jin and Shuwu Zhang and Jie Liu and Hu Guan},
  title = {{Research on Similarity Detection of Massive Text Based on Semantic Fingerprint}},
  year = {2018},
  doi = {10.22323/1.300.0009},
}

Loading related content...

Figure