Abstract
Traditional search engines struggle with the velocity and volume of modern text streams, such as social media and news feeds. This paper presents an efficient information collection system that inverts the standard search paradigm by indexing user queries instead of documents. By employing a dynamic analyzer based on a Patricia-trie (PAT), the proposed "Stream Search" method reduces processing latency and memory overhead. Experimental results using Chinese patent data demonstrate a 50% reduction in time cost compared to batch-incremental search approaches.
Introduction
In "Big Data" environments, the continuous flow of text data necessitates real-time monitoring systems. Traditional indexing methods face bottlenecks as the document base grows, leading to excessive search times and re-indexing requirements due to dynamic vocabularies. For prospective search—where users have standing queries for new information—a more agile approach is required. This study proposes a system that treats incoming documents as search terms against a pre-indexed set of user queries.
Methodology
Dynamic Analyzer (DA)
To address the "new word" problem in text streams, a Dynamic Analyzer was developed. Unlike static dictionaries, the DA uses a Trie tree to store user-defined keywords. It employs dynamic N-Grams, where the value of "N" is determined by the maximum length of terms currently in the Trie. This allows the system to adapt instantly as users modify their queries.
The choice of data structure is critical for performance. The researchers implemented a Patricia-trie (PAT) to optimize memory and speed.
As shown in the figure above, the PAT implementation significantly outperforms standard concurrent trees (CT) in both time cost and memory consumption as the keyword count increases.
Query-as-Index Paradigm
The core innovation lies in indexing queries rather than documents. When a document arrives, it is tokenized by the DA and matched against the query index using a Vector Space Model (VSM). Because query sets are typically smaller and more stable than the document stream, the index is easier to maintain and update in real-time.
Experimental Evaluation
The system was evaluated using a dataset of Chinese patent records from 1985 to 2011, with a specific focus on 219,144 patents from 2012 to simulate real-time streaming.
Performance Results
The "Stream Search" method was compared against a traditional "Batch-Incremental" (BI) search.
The results indicate that Stream Search consistently maintains a lower time cost—approximately 50% less than BI Search—across varying numbers of records. Furthermore, the Dynamic Analyzer showed superior efficiency compared to the ICTCLAS baseline by focusing only on relevant tokens, thereby reducing index size and processing time.
Conclusion
By flipping the search paradigm and utilizing a dynamic, PAT-based analyzer, this research provides a scalable solution for high-speed text stream monitoring. The system achieves significant gains in latency and memory efficiency, making it suitable for production environments requiring real-time information collection. Future work will focus on bridging the semantic gap through synonym handling and exploring distributed hardware acceleration.
Summary updated