Introduction
The field of Large Language Models (LLMs) is advancing at an unprecedented pace, with hundreds of new papers released weekly. This rapid growth creates a "firehose" problem for researchers and engineers, making it difficult to distinguish between solved problems and active research frontiers. This paper provides a comprehensive "state of the union" as of mid-2023, systematically categorizing technical bottlenecks and mapping out the diverse landscape of real-world applications.
Taxonomy of Challenges
To navigate the complexities of LLM development, the authors propose a framework that groups open challenges into three distinct categories: Design, Behavior, and Science.
- Design: This category encompasses decisions made before deployment, including dataset curation, subword tokenization strategies, and the massive compute costs associated with pre-training.
- Behavior: These issues emerge during model deployment, such as hallucinations, "brittle" sensitivity to prompt phrasing, and the ongoing struggle to align model outputs with human values.
- Science: This refers to hurdles in academic progress, specifically the lack of reproducibility in closed-source models and the widespread "contamination" of evaluation sets.
Data and Benchmarking
Modern models are built on "unfathomable" datasets. Pre-training corpora like The Pile, C4, and RefinedWeb have grown into the trillions of tokens, making it nearly impossible for humans to audit them for bias or personally identifiable information (PII).
Furthermore, standard benchmarks like MMLU and GLUE are reaching saturation, leading to the rise of more complex evaluations like BIG-bench and AGIEval. A critical concern highlighted by the authors is "data contamination"—the leaking of test questions into training data—which artificially inflates performance scores and obscures true progress.
Key Insights
The Tokenizer Bottleneck
A significant hidden flaw in most LLMs is their reliance on subword tokenization. This process can lead to "glitch tokens" that the model cannot process correctly and creates pricing inequities for non-English languages that require more tokens to convey the same meaning. It also hinders the model's ability to perform character-level tasks like spelling.
Inverse Scaling and the Evaluation Crisis
While scaling usually improves performance, "inverse scaling" occurs when models learn to follow common but incorrect patterns in the training data more strongly as they grow. Additionally, we are reaching an "evaluation crisis" where models are becoming better at certain tasks than the humans writing the tests. This has led to the problematic trend of using LLMs to evaluate other LLMs, which introduces its own set of biases.
Results and Applications
Despite these challenges, LLMs have achieved remarkable milestones across various domains:
- Efficiency: Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA allow users to fine-tune models with less than 1% of the original parameters, making it possible to run 65B parameter models on consumer-grade hardware.
- Medicine & Law: GPT-4 has shown performance strikingly close to human-level on professional exams like the USMLE and the Bar Exam, though it still fails on basic safety-critical consistency checks.
- Coding: Models like Codex have transformed programming, but "long-range dependencies" (understanding a whole codebase versus a single function) remain a major hurdle due to limited context windows.
Limitations and Future Outlook
The "Science" of LLMs is currently in a state of flux. The dominance of closed-source APIs (like GPT-4) hinders reproducibility, and the extreme cost of training often precludes rigorous ablation studies. Future work must focus on multimodality (moving beyond text to video and audio) and on-device efficiency, as the current trajectory of compute costs is unsustainable for global accessibility.
Summary updated