Collecting Valuable Information from Fast Text Streams

Baoyuan Qi, Gang Ma, Zhongzhi Shi, Wei WangLecture Notes in Computer Science 2014 · DOI · OpenAlex

Abstract

Traditional search engines struggle with the velocity and volume of modern text streams, such as social media and news feeds. This paper presents an efficient information collection system that inverts the standard search paradigm by indexing user queries instead of documents. By employing a dynamic analyzer based on a Patricia-trie (PAT), the proposed "Stream Search" method reduces processing latency and memory overhead. Experimental results using Chinese patent data demonstrate a 50% reduction in time cost compared to batch-incremental search approaches.

Introduction

In "Big Data" environments, the continuous flow of text data necessitates real-time monitoring systems. Traditional indexing methods face bottlenecks as the document base grows, leading to excessive search times and re-indexing requirements due to dynamic vocabularies. For prospective search—where users have standing queries for new information—a more agile approach is required. This study proposes a system that treats incoming documents as search terms against a pre-indexed set of user queries.

Methodology

Dynamic Analyzer (DA)

To address the "new word" problem in text streams, a Dynamic Analyzer was developed. Unlike static dictionaries, the DA uses a Trie tree to store user-defined keywords. It employs dynamic N-Grams, where the value of "N" is determined by the maximum length of terms currently in the Trie. This allows the system to adapt instantly as users modify their queries.

The choice of data structure is critical for performance. The researchers implemented a Patricia-trie (PAT) to optimize memory and speed.

Performance comparison of Patricia-trie (PAT) and Concurrent Tree (CT) in terms of time and memory cost.

As shown in the figure above, the PAT implementation significantly outperforms standard concurrent trees (CT) in both time cost and memory consumption as the keyword count increases.

Query-as-Index Paradigm

The core innovation lies in indexing queries rather than documents. When a document arrives, it is tokenized by the DA and matched against the query index using a Vector Space Model (VSM). Because query sets are typically smaller and more stable than the document stream, the index is easier to maintain and update in real-time.

Experimental Evaluation

The system was evaluated using a dataset of Chinese patent records from 1985 to 2011, with a specific focus on 219,144 patents from 2012 to simulate real-time streaming.

Performance Results

The "Stream Search" method was compared against a traditional "Batch-Incremental" (BI) search.

Comparison of time cost between Stream Search and Batch-Incremental (BI) Search.

The results indicate that Stream Search consistently maintains a lower time cost—approximately 50% less than BI Search—across varying numbers of records. Furthermore, the Dynamic Analyzer showed superior efficiency compared to the ICTCLAS baseline by focusing only on relevant tokens, thereby reducing index size and processing time.

Conclusion

By flipping the search paradigm and utilizing a dynamic, PAT-based analyzer, this research provides a scalable solution for high-speed text stream monitoring. The system achieves significant gains in latency and memory efficiency, making it suitable for production environments requiring real-time information collection. Future work will focus on bridging the semantic gap through synonym handling and exploring distributed hardware acceleration.

Summary updated

BibTeX
@inproceedings{qi2014collecting,
  author = {Baoyuan Qi and Gang Ma and Zhongzhi Shi and Wei Wang},
  title = {{Collecting Valuable Information from Fast Text Streams}},
  booktitle = {Lecture Notes in Computer Science},
  year = {2014},
  doi = {10.1007/978-3-662-44980-6\_11},
}
Figure