TL;DR
This paper introduces "BonTen," a powerful web-based corpus concordance system designed to query the massive NINJAL Web Japanese Corpus (NWJC), which contains 25 billion words. BonTen allows researchers to search not only by exact strings but also by complex morphological and syntactic dependency structures, offering unprecedented flexibility for linguistic analysis of Japanese.
What Problem Does This Paper Address?
Analyzing large-scale linguistic corpora, especially for morphologically rich languages like Japanese, presents significant challenges. Traditional search methods often rely on simple string matching, which is insufficient for capturing the nuances of language. Researchers need tools that can handle complex queries involving word forms, grammatical information, and sentence structure to uncover deeper linguistic patterns. The NINJAL Web Japanese Corpus (NWJC), a monumental collection of 25 billion words, requires a sophisticated system to make its vast linguistic data accessible and searchable for research. Existing systems often struggle with the scale of such corpora or lack the flexibility to perform detailed linguistic queries.
Methods
BonTen is built upon the foundation of existing web-based search systems, drawing inspiration from "Shonagon" for its string search mechanisms and "ChaKi.NET" for its user interface design. The system employs a three-tiered query approach:
- String Search: A basic but essential function that finds exact string matches. It omits regular expression capabilities like concatenation and Kleene star, likely due to the shorter nature of Japanese words and the performance demands of a 25-billion-word corpus.
- Short Unit Search: This method allows queries based on sequences of morphological information. Users can specify word forms, parts of speech (POS tags), lemma readings, and their relative positions within a sentence. This is a significant step beyond simple string matching, enabling more precise linguistic queries.
- Dependency Search: This advanced function enables users to query based on syntactic dependency structures, specifically "Bunsetsu" (Japanese base phrases). Researchers can define subtrees of dependency structures, including morphological information, relative positions of Bunsetsus, and dependency relations. This allows for the analysis of grammatical relationships and sentence structures.
The backend search system, "Sedue for Bigdata," developed by Retrieva Inc., is designed to handle the massive scale of the NWJC efficiently.
Datasets
The primary dataset is the NINJAL Web Japanese Corpus (NWJC), a collection of 25 billion words. The paper focuses on data crawled from October to December 2014 (2014-4Q) for evaluation purposes, which comprised 83,992,556 URLs, 3,885,889,575 tokens of sentences, and 1,463,142,939 types of sentences. No new benchmarks were introduced; instead, BonTen leverages the existing NWJC.
Key Insights and Ideas
- Scalability is Paramount: The core challenge is making a 25-billion-word corpus searchable at practical speeds. BonTen's architecture, particularly the "Sedue for Bigdata" backend, is crucial for this.
- Beyond String Matching: The true power of BonTen lies in its ability to query based on morphological and syntactic information. This unlocks deeper linguistic analysis than simple keyword searches.
- User Interface Matters: Drawing from established systems like ChaKi.NET ensures a familiar and usable interface for researchers, reducing the learning curve.
- Trade-offs for Scale: The omission of complex regular expressions in string search highlights the performance considerations when dealing with such a massive dataset.
Key Results
Evaluations were conducted using the Firefox 40.0.1 browser.
- String Search: Simple display mode response times were generally fast (under 1 second for most queries), with network latency being a factor. Rich display mode, which visualizes linguistic annotations, took longer, with the longest being 11.387 seconds for "cherry blossoms."
- Short Unit Search: For a query yielding 412,763 hits, the rich display response time was 16.567 seconds. A query with 2,067 hits took 10.702 seconds.
- Dependency Search: A complex query involving two Bunsetsus and a dependency relation took 47.904 seconds for 15,641 hits. A query that reduced query time by using a lexicalized entry took 9.304 seconds for 2,704 hits. The paper notes that dependency subtree searches on the ten-billion-word corpus can take less than one minute, including rich annotation information.
Limitations and Future Work
- History Function: The history function for listing previous user queries was not implemented at the time of writing but was planned for implementation by the end of 2016.
- Regular Expressions: Complex regular expressions are omitted in the string search due to performance considerations and the nature of Japanese word lengths.
- Sentence Boundaries: Queries cannot span across sentence boundaries, as the data is stored in sentence form.
Future work is implicitly suggested by the planned implementation of the history function and the ongoing development of corpus linguistics tools. The success of BonTen also paves the way for further exploration of advanced search functionalities on even larger corpora.
Summary updated