Abstract
We describe an investigation into the identification and extraction of unrecorded potential lexical items in Japanese text by detecting text passages containing selected language patterns typically associated with such items. We identified a set of suitable patterns, then tested them with two large collections of text drawn from the WWW and Twitter. Samples of the extracted items were evaluated, and it was demonstrated that the approach has considerable potential for identifying terms for later lexicographic analysis.
Introduction
Language evolves faster than lexicographers can work. Traditional dictionaries and even modern wordnets struggle to keep pace with the constant influx of slang, technical terms, and social media neologisms. In Japanese, this challenge is compounded by the writing system's complexity. However, Japanese writers often use predictable linguistic "hooks" when they suspect a reader might not know a term. For example, they might follow a new word with to iu no wa (meaning "as for that which is called...") or place the term in specific types of brackets. This paper investigates whether these social and linguistic cues can be programmatically exploited to "harvest" new words at scale.
Methodology
The researchers employed a pattern-based extraction pipeline starting with the identification of linguistic patterns known to signal definitions or highlights.
Pattern Identification and Filtering
Initially, 37 patterns were considered. Using the Google Japanese n-gram Corpus, the team filtered patterns based on frequency. Patterns that were too rare lacked utility, while those that were too common (like the topic marker to wa) lacked precision.
Extraction Logic
The extraction process relied on two main strategies:
- Parenthetical Capture: Terms appearing in parentheses or brackets (e.g., 「...」 or 『...』) near a pattern were extracted directly.
- Morphological Heuristics: For non-bracketed text, the MeCab morphological analyzer grouped adjacent morphemes (nouns, prefixes, suffixes) until reaching a boundary like a particle or verb.
The set was eventually narrowed to 18 high-performing patterns for detailed testing. Extracted terms were manually audited and sorted into categories ranging from "definitely a neologism" to "corrupted text" or "existing word variant."
Datasets
The study utilized three primary data sources:
- Kyoto WWW Corpus: Approximately 500 million Japanese sentences crawled from the web in 2004.
- Twitter Dataset: A massive collection of 870 million Japanese tweets from 2014 and 2015, providing a look at contemporary slang and "bursty" trends.
- Google Japanese n-gram Corpus: Used to check the general frequency of patterns and candidate terms across the broader web.
Key Results and Insights
The system successfully extracted 234,733 unique terms from the Kyoto WWW Corpus. A manual evaluation of the 50 most frequent extractions from the web corpus revealed that 36% (18 terms) were classified as Category C ("definitely of interest"/new lexical items), and another 42% were variants or inflections of existing words.
Frequency and Quality
A critical finding was the correlation between extraction frequency and term quality. Terms appearing multiple times were significantly more likely to be legitimate neologisms than those appearing only once, which were often "noise."
Pattern Performance
Despite testing dozens of patterns, the majority of useful extractions came from variations of the pattern to iu kotoba ("the word called..."). While other patterns were more "precise," their low frequency made them less practical for large-scale harvesting. Additionally, brackets proved to be high-confidence "containers" for neologisms.
Twitter vs. Web
The "hit rate" for these patterns on Twitter was about 30% of that on the WWW corpus. This is likely due to the brevity of tweets and a more informal style. Twitter extractions were also heavily dominated by proper nouns (anime characters, singers, etc.), which differ from the typical targets of general-purpose lexicons.
Conclusion and Future Work
This approach demonstrates considerable potential for identifying terms for lexicographic analysis. Future work could involve integrating this technique into a live "lexicographer's dashboard" for real-time scanning of social media feeds. While the current heuristic struggles with distinguishing between common and proper nouns, the high precision of frequent terms suggests a viable path for automated dictionary maintenance.
Summary updated