Patent Response System Optimised for Faithfulness: Procedural Knowledge Embodiment with Knowledge Graph and Retrieval Augmented Generation

Jung-Mei Chu, Hao-Cheng Lo, Jieh Hsiang, Chun-Chieh ChoACL 2024

TL;DR

This paper introduces PRO, a novel framework that enhances the faithfulness of AI-generated patent responses by incorporating procedural knowledge used by patent agents. By combining a domain-specific Large Language Model (LLM) with a knowledge graph of patent precedents and retrieval-augmented generation, PRO significantly reduces errors and outperforms state-of-the-art models like GPT-4.

What Problem Does This Paper Address?

Obtaining a patent is a complex process, and responding to an Office Action (OA) from a patent examiner is a critical step. While general-purpose LLMs like GPT-4 have been explored for this task, they often struggle to generate responses that are not only accurate but also faithful to the patent's details and legal nuances. This can lead to responses that are biased, factually incorrect, or fail to effectively address the examiner's concerns, potentially jeopardizing the patent application. The core challenge lies in imbuing LLMs with the implicit, procedural knowledge that experienced patent agents use to analyze rejections, identify key inventive concepts, and construct persuasive arguments.

Methods

The authors propose the Patent Response System Optimized for Faithfulness (PRO) framework, which addresses the identified problem through several key components:

  • PRLLM (Patent Response LLM): A specialized LLM continuously pre-trained and fine-tuned on a vast dataset of patent-related documents. This ensures it possesses domain-specific knowledge, including legal terminology and patent structure.
  • PPNet (Patent Precedents Knowledge Graph): A novel knowledge graph that encodes legal interpretations and relationships between technical components derived from judicial sources. This graph goes beyond simple definitions to capture legal logic and specificity.
  • Retrieval-Augmented Generation (RAG): PRO employs multiple RAG processes. Retrievers identify relevant information from the patent text and PPNet based on points of contention or prior art. The generator then uses this retrieved information, along with the PPNet and PRLLM outputs, to construct faithful responses.
  • Procedural Knowledge Embodiment: The framework explicitly integrates the step-by-step reasoning and strategic decision-making that patent agents follow when responding to OAs. This includes identifying rejection points, finding core inventive concepts, and searching for supporting precedents.

Datasets

  • Continual Pretraining Dataset: A comprehensive dataset comprising patent documents and OA records from 2003-2022 (over 950,000 patents and 1.2 million OAs), supplemented with publicly available online resources like academic papers, websites, wikis, books, news articles, and judicial rulings.
  • Supervised Fine-Tuning (SFT) Dataset:
    • Dpr: Paired OA-response datasets from 2023 (10,000 instances).
  • Dg: A general dataset of 20,000 instances from sources like UltraChat, Databricks-dolly-15k, and Guanaco Dataset.
  • The final SFT dataset (D) is the union of Dpr and Dg.
  • PPNet QA Dataset: A dataset of 4,730 questions (3,000 for training, 300 for validation, 1,430 for testing) specifically designed for testing the PPNet QA pipeline on patent judgment-related questions.

Key Insights and Ideas

  • Domain-Specific LLMs are Crucial: General LLMs, while powerful, lack the specialized knowledge and reasoning capabilities required for complex domains like patent law. Tailoring models (PRLLM) is a significant step forward.
  • Procedural Knowledge is Key to Faithfulness: The implicit "how-to" knowledge of patent agents, which involves structured analysis and strategic argumentation, is vital for generating accurate and effective patent responses. PRO's success hinges on explicitly encoding this procedural knowledge.
  • Knowledge Graphs Enhance Legal Reasoning: PPNet, by capturing legal interpretations and relationships beyond simple definitions, provides a richer context for LLMs, enabling more nuanced and legally sound arguments.
  • RAG is More Effective with Domain-Specific Components: Integrating RAG with domain-specific retrievers (patent text, PPNet) and a specialized generator (PRLLM) yields superior results compared to using generic RAG approaches.
  • Arguments Over Amendments: The authors observed that patent agents often prefer arguments to amendments to avoid narrowing claim scope. Their domain-specific metrics (RA and PA) reflect this preference, showing the system's inclination towards argumentation.

Key Results

  • Reduced Unfaithfulness: PRO significantly reduces unfaithfulness across six defined error types (Intrinsic Entity, Extrinsic Entity, Intrinsic Event, Extrinsic Event, Reasoning Coherence, Irrelevant Evidence).
  • Superior to GPT-4: PRO outperforms GPT-4 by an average of 39% in terms of faithfulness.
  • Effectiveness of PRO Components:
    • Zero-shot PRLLM is better than general LLMs but still struggles.
  • CoT (Chain-of-Thought) improves performance over zero-shot.
  • PRO, by incorporating procedural knowledge, achieves the best results, demonstrating that procedural knowledge is more impactful than just CoT or zero-shot approaches.
  • Qualitative Improvements: PRO generates arguments that closely match ground truth in format and completeness, effectively enhancing the quality of patent responses.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that while PRO significantly reduces errors, it does not eliminate them entirely.
  • The study focused on novelty and non-obviousness rejections; other aspects of patent responses might require different approaches.
  • The effectiveness of using the same LLM for both retriever and generator in RAG was highlighted, but the paper also explored using GPT-4 for RAG and retriever with PRLLM for the generator, suggesting potential benefits from mixed-model approaches.
  • Future Work:
    • Exploring prompt tuning in CoT to further refine reasoning.
  • Addressing other types of patent responses beyond novelty and non-obviousness.
  • Incorporating the history trajectory of OAs to further improve response effectiveness.
  • Investigating the impact of different LLM architectures and sizes on the PRO framework.

Summary updated

BibTeX
@inproceedings{chu2024patent,
  author = {Jung-Mei Chu and Hao-Cheng Lo and Jieh Hsiang and Chun-Chieh Cho},
  title = {{Patent Response System Optimised for Faithfulness: Procedural Knowledge Embodiment with Knowledge Graph and Retrieval Augmented Generation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure