Abstract
Model scaling is becoming the default choice for many language tasks due to the success of large language models (LLMs). However, it can fall short in specific scenarios where simple customized methods excel. In this paper, we delve into the patent approval prediction task and unveil that simple domain-specific graph methods outperform enlarging the model, using the intrinsic dependencies within the patent data. We propose a novel Fine-grained cLAim depeNdency (FLAN) Graph that captures inherent dependencies across segments of patent text. Extensive experiments prove that incorporating FLAN Graph via various graph models consistently outperforms all LLM baselines significantly.
Introduction
Predicting whether a patent application will be approved by the USPTO is a high-stakes task involving billions of dollars in intellectual property. Patent "novelty" is time-sensitive, knowledge-intensive, and crucially, structure-dependent. A single patent contains multiple "claims" forming a complex web where "dependent claims" refer back to "independent claims." Standard LLMs, which process text as a linear sequence, struggle to capture these non-linear, structural dependencies.
The Scaling Wall in Patent Law
The researchers tested a wide array of modern LLMs—including LLaMA2 (up to 70B), Mistral, Vicuna, GPT-3.5, and GPT-4—using both fine-tuning and sophisticated prompting techniques like Chain-of-Thought (CoT). Surprisingly, scaling from a 7B to a 70B model did not guarantee better results. In some tests, larger models performed worse, often hovering near random guessing (50.0 Macro-F1). This "scaling wall" suggests that the structure of the legal argument is as important as the content.
The FLAN Graph Approach
To address these limitations, the authors proposed the Fine-grained cLAim depeNdency (FLAN) Graph. This method decomposes each claim into informative segments (e.g., "an authentication component") and builds edges based on two types of dependencies:
- Inner-claim: The hierarchy within a single complex claim.
- Inter-claim: References between different claims.
These graphs are then processed by cost-effective Graph Neural Networks (GNNs) like GraphSage, GCN, and GAT. This approach explicitly models how components of an invention relate to one another, allowing the model to "reason" about the scope of the claim in a way that transformer-based self-attention cannot easily replicate.
Experimental Results
The study utilized the PATENTAP dataset, containing ~2 million real-world patent claims. The FLAN Graph approach consistently outperformed all LLM baselines.
As shown in the radar charts, GraphSage using the FLAN Graph emerged as the top performer, achieving an AUC of 66.04 and a Macro-F1 of 58.22. This represents a significant improvement over previous BERT-based SOTA models and massive LLMs. The results demonstrate that domain-specific structural modeling is more effective and cost-efficient than raw compute for specialized fields like patent law.
Conclusion
While model scaling is a powerful trend, this research highlights its limitations in domains requiring deep structural understanding. The FLAN Graph provides a robust, cost-effective alternative for patent approval prediction. Future work may explore "GraphLLMs" to combine the linguistic power of LLMs with the structural reasoning of graph models.
Summary updated