Beyond Scaling: Predicting Patent Approval with Domain-specific Fine-grained Claim Dependency Graph

Xiaochen Gao, Feng Yao, Kewen Zhao, Beilei He, Animesh Kumar, Vish Krishnan, Jingbo Shang

Abstract

Model scaling is becoming the default choice for many language tasks due to the success of large language models (LLMs). However, it can fall short in specific scenarios where simple customized methods excel. In this paper, we delve into the patent approval prediction task and unveil that simple domain-specific graph methods outperform enlarging the model, using the intrinsic dependencies within the patent data. We propose a novel Fine-grained cLAim depeNdency (FLAN) Graph that captures inherent dependencies across segments of patent text. Extensive experiments prove that incorporating FLAN Graph via various graph models consistently outperforms all LLM baselines significantly.

Introduction

Predicting whether a patent application will be approved by the USPTO is a high-stakes task involving billions of dollars in intellectual property. Patent "novelty" is time-sensitive, knowledge-intensive, and crucially, structure-dependent. A single patent contains multiple "claims" forming a complex web where "dependent claims" refer back to "independent claims." Standard LLMs, which process text as a linear sequence, struggle to capture these non-linear, structural dependencies.

Overview of the proposed approach comparing LLM scaling with the FLAN Graph model.

The Scaling Wall in Patent Law

The researchers tested a wide array of modern LLMs—including LLaMA2 (up to 70B), Mistral, Vicuna, GPT-3.5, and GPT-4—using both fine-tuning and sophisticated prompting techniques like Chain-of-Thought (CoT). Surprisingly, scaling from a 7B to a 70B model did not guarantee better results. In some tests, larger models performed worse, often hovering near random guessing (50.0 Macro-F1). This "scaling wall" suggests that the structure of the legal argument is as important as the content.

The FLAN Graph Approach

To address these limitations, the authors proposed the Fine-grained cLAim depeNdency (FLAN) Graph. This method decomposes each claim into informative segments (e.g., "an authentication component") and builds edges based on two types of dependencies:

  1. Inner-claim: The hierarchy within a single complex claim.
  2. Inter-claim: References between different claims.

These graphs are then processed by cost-effective Graph Neural Networks (GNNs) like GraphSage, GCN, and GAT. This approach explicitly models how components of an invention relate to one another, allowing the model to "reason" about the scope of the claim in a way that transformer-based self-attention cannot easily replicate.

Experimental Results

The study utilized the PATENTAP dataset, containing ~2 million real-world patent claims. The FLAN Graph approach consistently outperformed all LLM baselines.

Performance comparison (AUC and Macro-F1) of different graph models using the FLAN Graph vs. baselines.

As shown in the radar charts, GraphSage using the FLAN Graph emerged as the top performer, achieving an AUC of 66.04 and a Macro-F1 of 58.22. This represents a significant improvement over previous BERT-based SOTA models and massive LLMs. The results demonstrate that domain-specific structural modeling is more effective and cost-efficient than raw compute for specialized fields like patent law.

Conclusion

While model scaling is a powerful trend, this research highlights its limitations in domains requiring deep structural understanding. The FLAN Graph provides a robust, cost-effective alternative for patent approval prediction. Future work may explore "GraphLLMs" to combine the linguistic power of LLMs with the structural reasoning of graph models.

Summary updated

Abstract

Model scaling is becoming the default choice for many language tasks due to the success of large language models (LLMs). However, it can fall short in specific scenarios where simple customized methods excel. In this paper, we delve into the patent approval pre-diction task and unveil that simple domain-specific graph methods outperform enlarging the model, using the intrinsic dependencies within the patent data. Specifically, we first extend the embedding-based state-of-the-art (SOTA) by scaling up its backbone model with various sizes of open-source LLMs, then explore prompt-based methods to harness proprietary LLMs' potential, but find the best results close to random guessing, underlining the ineffectiveness of model scaling-up. Hence, we propose a novel Fine-grained cLAim depeNdency (FLAN) Graph through meticulous patent data analyses, capturing the inherent dependencies across segments of the patent text. As it is model-agnostic, we apply cost-effective graph models to our FLAN Graph to obtain representations for approval prediction. Extensive experiments and detailed analyses prove that incorporating FLAN Graph via various graph models consistently outperforms all LLM baselines significantly. We hope that our observations and analyses in this paper can bring more attention to this challenging task and prompt further research into the limitations of LLMs. Our source code and dataset can be obtained from http://github.com/ShangDataLab/FLAN-Graph.
BibTeX
@inproceedings{gao2024beyond,
  author = {Xiaochen Gao and Feng Yao and Kewen Zhao and Beilei He and Animesh Kumar and Vish Krishnan and Jingbo Shang},
  title = {{Beyond Scaling: Predicting Patent Approval with Domain-specific Fine-grained Claim Dependency Graph}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=MBiag4IdPa},
}
Figure