Beyond Scaling: Predicting Patent Approval with Domain-specific Fine-grained Claim Dependency Graph

Xiaochen Gao, Feng Yao, Kewen Zhao, Beilei He, Animesh Kumar, Vish Krishnan, Jingbo Shang

Abstract

Model scaling is becoming the default choice for many language tasks due to the success of large language models (LLMs). However, it can fall short in specific scenarios where simple customized methods excel. In this paper, we delve into the patent approval prediction task and unveil that simple domain-specific graph methods outperform enlarging the model, using the intrinsic dependencies within the patent data. We propose a novel Fine-grained cLAim depeNdency (FLAN) Graph that captures inherent dependencies across segments of patent text, providing a cost-effective and high-performing alternative to massive LLMs.

Introduction

In the current era of "bigger is better," scaling up Large Language Models (LLMs) to 70B parameters is often seen as the solution to complex NLP tasks. However, predicting patent approval—a high-stakes task involving intricate intersections of technology and law—presents a unique challenge where scaling hits a performance ceiling. Patent text, or "patentese," is highly structured and hierarchical, often eluding the linear processing capabilities of standard LLMs.

Overview of the proposed approach vs. standard LLM scaling.

Our research reveals that scaling from 7B to 70B parameters or using proprietary models like GPT-4 yields results barely better than random guessing. Instead, we demonstrate that explicitly modeling the structural dependencies of patent claims via a domain-specific graph representation significantly outperforms even the most powerful LLMs.

The Challenge of "Patentese"

Predicting whether the USPTO will approve a patent is traditionally a manual, labor-intensive process. LLMs struggle with this task because a single patent claim might depend on multiple sub-components or reference previous claims in a complex hierarchy. Standard LLMs process text linearly and fail to capture these non-linear dependencies. Furthermore, the computational cost of massive models makes them impractical for real-world legal tech if they cannot surpass well-tuned baselines.

The FLAN Graph Approach

To address these limitations, we propose the Fine-grained cLAim depeNdency (FLAN) Graph. Rather than treating a patent claim as a flat string, we decompose it into a structured graph through three key steps:

  1. Node Construction: Claims are segmented based on punctuation (colons/semicolons) and specific keywords like "comprising" or "whereby."
  2. Edge Construction: We identify inner-claim relationships (between sub-components) and inter-claim relationships (referencing "ancestor" claims).
  3. Identity Matching: We use heuristic-based identity extraction (via POS tagging) to link nodes accurately, ensuring the graph reflects the logical flow of the patent application.
Visualization of a fine-grained claim dependency graph showing hierarchical relationships.

By applying Graph Neural Networks (GNNs) such as GraphSage, GCN, and GAT to this FLAN Graph, we generate embeddings that capture the essential structural logic required for approval prediction.

Experimental Results

We evaluated our approach using the PATENTAP dataset, which contains approximately 2 million real-world USPTO claims. Our findings underscore the superiority of structural modeling over simple scaling:

  • Scaling Failure: LLM performance remained near random in many prompt-based settings, often "evading" the question with generic legal disclaimers.
  • GNN Superiority: All models utilizing the FLAN Graph consistently outperformed LLM baselines.
  • Top Performance: GraphSage, combined with handcrafted features (e.g., citation counts), achieved an AUC of 66.04 and a Macro-F1 of 58.22, marking a significant improvement over previous BERT-based SOTA models.
  • Efficiency: The GNN approach is highly cost-effective, requiring a fraction of the compute power needed for LLM fine-tuning or inference.

Conclusion

Our work demonstrates that domain knowledge and structural representation can be more powerful than model size. The FLAN Graph provides a robust, efficient framework for specialized legal reasoning tasks. Future research may explore "GraphLLMs" to combine the semantic depth of transformers with the structural precision of GNNs, further pushing the boundaries of automated patent analysis.

Summary updated

Abstract

Model scaling is becoming the default choice for many language tasks due to the success of large language models (LLMs). However, it can fall short in specific scenarios where simple customized methods excel. In this paper, we delve into the patent approval prediction task and unveil that simple domain-specific graph methods outperform enlarging the model, using the intrinsic dependencies within the patent data. Specifically, we first extend the embedding-based state-of-the-art (SOTA) by scaling up its backbone model with various sizes of open-source LLMs, then explore prompt-based methods to harness proprietary LLMs’ potential, but find the best results close to random guessing, underlining the ineffectiveness of model scaling-up. Hence, we propose a novel Fine-grained cLAim depeNdency (FLAN) Graph through meticulous patent data analyses, capturing the inherent dependencies across segments of the patent text. As it is model-agnostic, we apply cost-effective graph models to our FLAN Graph to obtain representations for approval prediction. Extensive experiments and detailed analyses prove that incorporating FLAN Graph via various graph models consistently outperforms all LLM baselines significantly. We hope that our observations and analyses in this paper can bring more attention to this challenging task and prompt further research into the limitations of LLMs.
BibTeX
@inproceedings{gao2024beyond,
  author = {Xiaochen Gao and Feng Yao and Kewen Zhao and Beilei He and Animesh Kumar and Vish Krishnan and Jingbo Shang},
  title = {{Beyond Scaling: Predicting Patent Approval with Domain-specific Fine-grained Claim Dependency Graph}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=MBiag4IdPa},
}
Figure