Abstract
While large language models (LLMs) excel at factual recall, the real challenge lies in knowledge application. A gap persists between their ability to answer complex questions and their effectiveness in performing tasks that require that knowledge. We investigate this gap using a patent classification problem that requires deep conceptual understanding to distinguish semantically similar but objectively different patents written in dense, strategic technical language. We find that LLMs often struggle with this distinction. To diagnose the source of these failures, we introduce a framework that decomposes model errors into two categories: missing knowledge and unused knowledge. Our method prompts models to generate clarifying questions and compares three settings -- raw performance, self-answered questions that activate internal knowledge, and externally provided answers that supply missing knowledge (if any). We show that most errors stem from failures to deploy existing knowledge rather than from true knowledge gaps. We also examine how models differ in constructing task-specific question-answer databases. Smaller models tend to generate simpler questions that they, and other models, can retrieve and use effectively, whereas larger models produce more complex questions that are less effective, suggesting complementary strengths across model scales. Together, our findings highlight that shifting evaluation from static fact recall to dynamic knowledge application offers a more informative view of model capabilities.
Introduction
Large Language Models (LLMs) have demonstrated remarkable capabilities in factual recall, yet they frequently falter when tasked with expert-level conceptual reasoning. This research investigates a critical paradox: models that appear to "know everything" often struggle to apply that knowledge to solve complex problems. We focus on patent understanding—a domain characterized by dense, legalistic, and strategically obfuscated language—to explore whether these failures stem from missing knowledge (the model never learned the underlying science) or unused knowledge (the model cannot retrieve the science during the task).
Methodology
We introduce a diagnostic framework centered on a "Differentiation Task." Models are presented with pairs of patent descriptions and must determine if they represent the same invention or two distinct ones. To isolate the source of errors, we compare three experimental conditions:
- Baseline: The model answers the question directly.
- Self-QA (Self-Talk): The model is prompted to generate its own clarifying questions about the patents and then answer them using its internal knowledge before making a final judgment.
- Scientific RAG: The model generates questions, but the answers are retrieved from external, high-quality scientific papers (arXiv).
Our dataset comprises over 1.3 million USPTO patents granted after 2015, specifically selected for high surface similarity to prevent models from relying on simple keyword matching.
Key Results
Our findings indicate that the primary bottleneck in patent understanding is the deployment of existing knowledge rather than a lack of information. As shown in Figure 1, the "Self-QA" condition significantly reduces the error rate of misclassifying different patents as identical, often performing nearly as well as the "Scientific RAG" condition. This suggests that the necessary scientific knowledge is often present within the model's weights but remains "latent" or "unused" without explicit scaffolding.
Furthermore, we observed significant logical inconsistencies in smaller models. While larger models like GPT-4o mini and Llama 3.3-70B maintained consistency when the framing of the question was flipped (e.g., "Are these the same?" vs. "Are these different?"), smaller models like Mistral-7B and Qwen-2.5-7B exhibited strong response biases. Figure 2 illustrates these variations in accuracy and confidence across different model scales, highlighting that smaller models often struggle with logical stability in complex differentiation tasks.
Discussion and Conclusion
The "deployment bottleneck" suggests that LLM knowledge is stored in a highly compressed state that requires active "unpacking" through techniques like self-talk. Interestingly, smaller models (e.g., Llama 8B) often generated more effective "fundamental" questions that served as better scaffolds than the overly complex questions produced by massive models. This suggests a potential for hybrid strategies using small models for conceptual scaffolding and large models for high-level reasoning.
Future work could involve agentic retrieval, where models iteratively refine their understanding. Ultimately, shifting evaluation from static recall to dynamic knowledge application provides a more accurate measure of true model capability in specialized domains like patent law.
Summary updated