TL;DR
This paper introduces Patentformer, a novel method for automatically generating patent specifications using Large Language Models (LLMs). By fine-tuning LLMs on a new dataset combining patent claims and drawing descriptions, Patentformer significantly improves the generation of legally sound and technically accurate patent text, showing potential to assist patent attorneys in a time-consuming and costly process.
What Problem Does This Paper Address?
Drafting patent specifications is a complex and expensive process, typically requiring patent attorneys to spend significant time understanding technical details and legal nuances. While LLMs have shown great promise in natural language processing, their application to generating patent specifications has been largely unexplored. Existing LLMs struggle with the highly technical and legally precise language of patents, the need to accurately reference drawings, and the sheer length of patent documents, which often exceed typical LLM context windows. This paper aims to bridge this gap by developing a method to automate patent specification generation, making the process more efficient and accessible.
Methods
The core of this work is Patentformer, a method that fine-tunes LLMs on a specially constructed dataset. The authors introduce two new tasks:
- Claim-to-Specification (C→S): This task focuses on generating specification text solely from patent claims. To handle the length limitations of LLMs, they break down claims into smaller "claim features" and match them to corresponding specification paragraphs.
- Claim+Drawing-to-Specification (T→S): This more comprehensive task incorporates patent claims, drawing descriptions, and component names/numbers extracted from drawings.
Patentformer enhances the T→S task by introducing an enriched training data format (T'→S'). This format includes:
- Contextual claims: Providing related claim features (e.g., parent claims for dependent claims) as context.
- Special tags: Marking the presence of figure numbers, component names, and component numbers.
- Previous paragraph context: Including the preceding paragraph and its number to help maintain coherence.
This enriched context aims to improve the LLM's understanding of the intricate relationships between claims, drawings, and the specification.
Datasets
The authors created a new dataset called Patent-2015-2023-G06N. This dataset was constructed from 13,725 patents granted by the USPTO between 2015 and 2023, specifically focusing on the CPC code G06N (Artificial Intelligence, Neural Networks, etc.). The dataset consists of approximately 284,531 quadruplets of <claim_feature, drawing_description, component_info, specification_paragraph>. This dataset is publicly released.
Key Insights and Ideas
- Context is King for Patent Generation: Simply fine-tuning LLMs on patent text isn't enough. The real breakthrough comes from providing rich, structured context, including related claims, drawing details, and even previous paragraphs, to guide the generation process.
- Claims and Drawings are Crucial: While claims define the invention's boundaries, drawings provide essential visual context. Combining these inputs significantly improves the quality and accuracy of generated specifications compared to using claims alone.
- LLMs Can Mimic Legal Style: The generated specifications, when evaluated by patent experts, exhibit a legal writing style and often demonstrate correctness, sometimes even surpassing human-written examples in specific aspects.
- Human Evaluation is Paramount: Due to the specialized nature of patent examination, automatic metrics like BLEU and ROUGE are insufficient. Human evaluation by patent experts is critical for assessing the true correctness and quality of generated patent text.
Key Results
- Improved Performance: Patentformer, by leveraging enriched context (T'→S'), significantly outperforms baselines (pretrained LLMs and models trained on simpler tasks like C→S or T→S) in terms of perplexity. For instance, Pat_T5* (Patentformer with T5 model) achieved a perplexity of 3.790, notably lower than Pat_T5 (T→S) at 4.649.
- High Correctness Rates: Human evaluations by patent experts showed that Patentformer (specifically Pat_T5*) generated correct specifications in 66% of cases for neural processor patents and 20% for system-on-chip patents. While the latter is lower, it's still competitive and highlights the complexity of the task.
- Quality Beyond Correctness: In cases where specifications were deemed correct, patent experts sometimes found the quality of Patentformer's output to be better than the actual human-written specifications.
- Ablation Study Highlights Context: Removing any component of the input (claims, drawings, or component names) or context (like previous paragraphs or claim context) negatively impacted performance, underscoring the importance of the proposed method's design.
Limitations and Future Work
- Not Fully Automated: The authors acknowledge that patent drafting cannot be entirely automated. Patent attorneys still need to thoroughly review and verify the generated specifications for quality and correctness.
- Specialized Diagrams: The current method does not handle specialized diagrams like block diagrams or flow charts.
- Paragraph-to-Claim/Drawing Mapping: The assumption that each specification paragraph corresponds to a single claim feature and drawing is a simplification; real patents can have more complex relationships.
- Future Directions: The authors suggest leveraging more advanced LLMs with larger context windows, exploring multimodal models that can directly process patent images, and redesigning the training dataset to handle more complex relationships between claims, drawings, and specifications. They also plan to improve the system's ability to extract component names and numbers directly from drawing images.
Summary updated