Abstract
Model scaling is becoming the default choice for many language tasks due to the success of large language models (LLMs). However, it can fall short in specific scenarios where simple customized methods excel. In this paper, we delve into the patent approval prediction task and unveil that simple domain-specific graph methods outperform enlarging the model, using the intrinsic dependencies within the patent data. We propose a novel Fine-grained cLAim depeNdency (FLAN) Graph that captures inherent dependencies across segments of patent text, providing a cost-effective and high-performing alternative to massive LLMs.
Introduction
In the current era of "bigger is better," scaling up Large Language Models (LLMs) to 70B parameters is often seen as the solution to complex NLP tasks. However, predicting patent approval—a high-stakes task involving intricate intersections of technology and law—presents a unique challenge where scaling hits a performance ceiling. Patent text, or "patentese," is highly structured and hierarchical, often eluding the linear processing capabilities of standard LLMs.
Our research reveals that scaling from 7B to 70B parameters or using proprietary models like GPT-4 yields results barely better than random guessing. Instead, we demonstrate that explicitly modeling the structural dependencies of patent claims via a domain-specific graph representation significantly outperforms even the most powerful LLMs.
The Challenge of "Patentese"
Predicting whether the USPTO will approve a patent is traditionally a manual, labor-intensive process. LLMs struggle with this task because a single patent claim might depend on multiple sub-components or reference previous claims in a complex hierarchy. Standard LLMs process text linearly and fail to capture these non-linear dependencies. Furthermore, the computational cost of massive models makes them impractical for real-world legal tech if they cannot surpass well-tuned baselines.
The FLAN Graph Approach
To address these limitations, we propose the Fine-grained cLAim depeNdency (FLAN) Graph. Rather than treating a patent claim as a flat string, we decompose it into a structured graph through three key steps:
- Node Construction: Claims are segmented based on punctuation (colons/semicolons) and specific keywords like "comprising" or "whereby."
- Edge Construction: We identify inner-claim relationships (between sub-components) and inter-claim relationships (referencing "ancestor" claims).
- Identity Matching: We use heuristic-based identity extraction (via POS tagging) to link nodes accurately, ensuring the graph reflects the logical flow of the patent application.
By applying Graph Neural Networks (GNNs) such as GraphSage, GCN, and GAT to this FLAN Graph, we generate embeddings that capture the essential structural logic required for approval prediction.
Experimental Results
We evaluated our approach using the PATENTAP dataset, which contains approximately 2 million real-world USPTO claims. Our findings underscore the superiority of structural modeling over simple scaling:
- Scaling Failure: LLM performance remained near random in many prompt-based settings, often "evading" the question with generic legal disclaimers.
- GNN Superiority: All models utilizing the FLAN Graph consistently outperformed LLM baselines.
- Top Performance: GraphSage, combined with handcrafted features (e.g., citation counts), achieved an AUC of 66.04 and a Macro-F1 of 58.22, marking a significant improvement over previous BERT-based SOTA models.
- Efficiency: The GNN approach is highly cost-effective, requiring a fraction of the compute power needed for LLM fine-tuning or inference.
Conclusion
Our work demonstrates that domain knowledge and structural representation can be more powerful than model size. The FLAN Graph provides a robust, efficient framework for specialized legal reasoning tasks. Future research may explore "GraphLLMs" to combine the semantic depth of transformers with the structural precision of GNNs, further pushing the boundaries of automated patent analysis.
Summary updated