Abstract In such an era of informatization and big data where the Internet has greatly enriched the sources of information of people and changed the way they get information, search engine is widely used and it results from the combination of text retrieval and Web. Based on the research on Boolean model, vector model and probability model, this paper deduces a text retrieval algorithm using keyword retrieval, which can realize keyword retrieval of text documents in text database.
1. Introduction
In the era of "big data," the sheer volume of information makes finding specific, relevant content a daunting task. While search engines are ubiquitous in daily life, the underlying mechanics of how a system decides which document matches a user's query remains a complex field of study. This paper addresses the challenge of "surface-based matching"—the process of taking a natural language query, segmenting it into keywords, and efficiently locating documents within a database that contain those terms. The goal is to bridge the gap between theoretical retrieval models and practical implementation for text-based databases.
2. Traditional Retrieval Models
The paper begins by reviewing the three pillars of traditional information retrieval:
- Boolean Model: Utilizes logic operators (AND, OR, NOT). While precise, it often suffers from "all-or-nothing" results and can be difficult for average users to format correctly.
- Vector Space Model (VSM): Represents documents and queries as vectors in a multi-dimensional space. It employs IDF (Inverse Document Frequency) for weighting and cosine similarity to calculate the relevance based on the angle between vectors.
- Probability Model: Based on the statistical distribution of terms, calculating the likelihood that a document is relevant given the presence of specific keywords.
3. Proposed Keyword Retrieval Algorithm
The authors implement a Keyword Retrieval Algorithm that focuses on direct character matching. The process involves opening the target database, segmenting the user's input query into discrete keywords, and performing a character-by-character comparison between these keywords and the document content. A counter tracks matches, and the system ranks the output based on these frequencies.
4. Experimental Setup
The study utilizes a custom-built dataset referred to as "The text retrieval library." This dataset is composed of 300 individual text files, labeled mydoc001.txt through mydoc300.txt. The content covers a variety of topics ranging from social issues and politics to economics. Unlike modern NLP papers that use massive benchmarks like SQuAD or GLUE, this study uses a controlled, small-scale environment to validate the fundamental logic of the keyword matching algorithm.
5. Key Results and Discussion
The authors conducted two primary experiments to test their implementation:
- Experiment 1 (Content Retrieval): Using five keywords (China, people, Jintao Hu, environment, politics) across the 300 documents, the system successfully ranked the top matches. The most relevant document (No. 274) contained 63 keyword instances, followed by Doc 150 (52 instances) and Doc 79 (37 instances).
- Experiment 2 (Title Retrieval): Using the keyword "human," the system successfully identified three specific documents (No. 69, 201, and 276) based on their titles.
One of the primary insights is how the Vector Space Model transforms a messy linguistic problem into a clean geometric one. However, keyword retrieval remains "surface-based"; it relies on the statistical occurrence of characters rather than deep semantic understanding.
6. Conclusion and Future Work
The authors identify a significant "bottleneck" in the current approach: high time complexity. Because the method relies on repetitive character comparisons, retrieval speed decreases significantly as the database size increases. This makes the current implementation impractical for massive, real-world datasets. Future work will focus on more efficient retrieval methods, likely involving advanced indexing or data structures to maintain performance at scale.
Summary updated