TL;DR
This paper proposes using large language models (LLMs) like BERT and BART as context-sensitive word search engines, offering a more flexible and powerful alternative to traditional n-gram based systems. By fine-tuning these models, they achieve comparable or better performance in answering cloze-style queries, especially for queries that are too sparse for n-gram models to handle.
What Problem Does This Paper Address?
Writing assistants often include tools to help users find the right words or phrases. One such tool is a context-sensitive word search engine, which allows users to input a sentence with a blank (a "cloze query") and get suggestions for what word or phrase fits best. Traditional engines rely on large collections of n-grams (sequences of n words) and their frequencies. However, these systems struggle when the context becomes longer (n ≥ 5) because the n-grams become too sparse, meaning they haven't been observed often enough in the training data to provide reliable suggestions. This limitation restricts the amount of context users can provide, hindering the engine's ability to understand the user's intent.
Methods
The authors propose two main strategies for adapting LLMs to act as context-sensitive word search engines:
- Masked Language Modeling (MLM): This approach leverages bidirectional transformer models like BERT. The core idea is that MLM, which is used to pre-train BERT, is inherently similar to wildcard word search. The model predicts a masked token based on its surrounding context. The authors fine-tune a BERT model (specifically DistilBERT for efficiency) on individual n-grams to estimate the probability distribution of potential words for a given cloze query. The top-scoring predictions form the search results.
- Conditional Language Modeling (CDLM): This strategy uses sequence-to-sequence models like BART. CDLM frames the problem as generating a word or phrase given a context. They explore two CDLM techniques:
- Denoising: The model takes the context and predicts the original sequence, with the blank being the target. Similar to MLM, the output logits at the masked position are used to derive the search results.
- Infilling: The model is trained to generate arbitrary-length sequences to fill in blanks. While BART is fine-tuned using infilling, the authors use denoising to predict the results for the search task.
Datasets
The paper uses two primary datasets for evaluation:
- Original Token (OT) Dataset: This dataset consists of queries where the "correct" answer is the single original token from the source text. It's derived from:
- Wikitext-103: A large corpus of English Wikipedia articles.
- CLOTH: A dataset of cloze tests created by teachers for middle and high school learners.
- Ranked Answers (RA) Dataset: This dataset is derived from the OT dataset, but instead of a single original token, the answers are the top 30 results retrieved by Netspeak (a state-of-the-art n-gram based engine). This dataset is used to evaluate the ranking quality of the proposed models.
Key Insights and Ideas
- LLMs as a natural fit for cloze queries: The inherent nature of masked language modeling (predicting missing words) and conditional generation (filling in blanks) makes LLMs a strong candidate for context-sensitive word search.
- Fine-tuning is crucial: Pre-trained LLMs perform reasonably well, but fine-tuning them specifically for the word search task significantly improves their performance, especially for BERT.
- Handling sparsity: The biggest advantage of LLMs is their ability to handle sparse contexts where n-gram models fail. This allows users to provide more context, leading to more accurate suggestions.
- Closed-class words benefit most from fine-tuning: Fine-tuning shows a greater performance boost for closed-class words (like prepositions, determiners) compared to open-class words (nouns, verbs, adjectives). This is likely because closed-class words have more constrained usage patterns.
- CLOTH dataset is more challenging: Models perform worse on the CLOTH dataset compared to Wikitext. This is attributed to CLOTH queries often involving lower-scoring open-class words, whereas Wikitext queries more frequently involve high-scoring closed-class words.
Key Results
- Competitive performance: The fine-tuned LLM models (dBERT ft and BART ft) achieve mean reciprocal ranks (MRR) within 0.03-0.07 of Netspeak on queries that Netspeak can answer.
- Answering more queries: The LLM-based engines can answer 4-5 times more queries than Netspeak, particularly for queries with uncommon n-grams that are not present in Google's n-gram index.
- Improved ranking: The normalized discounted cumulative gain (nDCG) scores show that the fine-tuned models outperform their pre-trained counterparts, with dBERT ft performing best.
- Speed: Netspeak and dBERT are equally fast (around 5 ms per query), while BART is twice as slow. However, all models are considered fast enough for real-time use.
Limitations and Future Work
- Bias in data: The models and evaluation procedures may inherit biases present in the training data (Wikipedia, Google n-grams). The authors did not take extra steps to de-bias the models or data.
- Privacy concerns: As LLMs are trained on large text corpora, there's a risk of them "leaking" sensible data. The authors attempted to mitigate this by avoiding training models to predict proper nouns, which could be used to search for personal information.
- Future work: The paper suggests that LLMs are a suitable extension to context-sensitive word search engines, especially for scenarios where n-gram indices provide no results due to data sparsity. Further research could explore more advanced de-biasing techniques and investigate the potential for LLMs to infer user intent more broadly.
Summary updated