Teaching Language Models to Narrow Boolean Patent Searches from Search Feedback

Seiya Kawano, Koji Marusaki, Hirofumi Nonaka, Koichiro YoshinoarXiv 2026 · arXiv

TL;DR

Researchers have developed a method to teach Large Language Models (LLMs) the art of the "narrow search." By using a novel line-level narrowing credit signal, they trained models to not only find relevant patent citations but to refine Boolean queries until the results are small enough for a human to actually read—slashing the median result set size from 768 documents to under 50.

What Problem Does This Paper Address?

In the high-stakes world of patent law, "prior-art search" is a critical gatekeeper. If an examiner misses a document that proves an invention isn't new, it can lead to multi-million dollar invalidation trials years later.

Unlike web search, which relies on black-box ranking algorithms, professional patent searchers use Boolean queries (e.g., combining keywords and classification codes with AND/OR/NOT). This process is iterative: searchers start broad to ensure they don't miss anything (high recall) and then progressively narrow their queries until they have a manageable list of documents to screen manually.

While modern LLMs are surprisingly good at writing valid Boolean queries that find the right documents, they are "lazy" searchers. They tend to return thousands of results, leaving the human expert with an impossible reading list. This paper addresses how to bridge the gap between "finding the needle" and "finding the needle in a very small haystack."

Methods

The authors adapted a Qwen3.5-9B model using search feedback in two distinct ways:

  1. Strategy-Level Learning: The model is rewarded based on a global score called NetCapture, which balances high recall with a penalty for large result sets and invalid queries. This was implemented using standard techniques like Direct Preference Optimization (DPO) and Advantage-Weighted Regression (AWR).
  2. Line-Level Narrowing Credit (The Novel Approach): Instead of just scoring the final result, the authors proposed a "process supervision" signal. A specific line in a search strategy receives credit only if it keeps a previously found citation while significantly reducing the total number of hits.
Overview of the line-level credit system

As shown in Figure 1, if a query line $q_2$ narrows a set from 2,154 hits to 12 hits without losing the target citations, it receives a high training weight. If a line narrows too much and loses the citation ($q'_3$), it gets zero credit.

Datasets

The researchers created a massive new resource by distilling 38,397 Japan Patent Office (JPO) search reports.

  • Scale: The dataset includes 425,405 executable Boolean query lines linked to over 100,000 cited prior-art occurrences.
  • Real-World Complexity: These aren't toy examples; they include complex F-terms (Japanese patent classifications), proximity operators, and references to previous search steps.
  • Environment: They built a custom search environment to re-execute these expert strategies against a database of Japanese patent publications.

Key Insights and Ideas

  • Prompting is not enough: Even the most powerful models (like GPT-4o-mini or Qwen-35B) fail to narrow results effectively through prompting alone. They can find the citations, but the result sets remain massive.
  • The "Narrowing" Trade-off: There is a clear divergence in how models learn. Strategy-level DPO focuses on coverage first, eventually reaching near-100% recall but with larger result sets. Line-level credit focuses on screening first, producing incredibly tight result sets but sometimes sacrificing a bit of recall.
  • Expertise is Iterative: An expert strategy is essentially a small Boolean program. Figure 2 illustrates how experts use specific classification codes and text refinements to subtract already-inspected documents.
Example of an expert Boolean search strategy

Key Results

The results, summarized in Table 2, show that the adapted models significantly outperform both raw LLMs and, in some metrics, human experts.

Table 2. Test results on 356 reports. Min-hits is lower-is-better; all other columns are higher-is-better. Bold marks the best value among generated strategies. NetCapture- based AWR and DPO and narrowing-credit learning are the strategy-level and line-level forms of our search-feedback adaptation.
  • Efficiency Gain: The "Narrowing credit" model reduced the median number of hits per citation to 48.5, compared to 768 for the initial SFT model and 282 for the human expert reference.
  • Superior Recall: The "NetCapture-DPO" model achieved a staggering 99.23% recall, far exceeding the human expert reference of 96.55%.
  • Complementary Strengths: While DPO is better at finding every possible document, the narrowing credit method is better at placing those documents into sets of fewer than 100 results (Rec<100).

Limitations and Future Work

  • Strategy Repertoire: The narrowing-credit model learned to narrow primarily through text reformulation, often ignoring the complex patent classifications and set algebra (like subtracting previous results) that human experts favor.
  • Evaluation Nuance: The current evaluation looks at the smallest set containing a citation. In practice, searchers might read overlapping sets. The authors suggest that measuring the union of returned sets would be a more realistic next step.
  • Language and Scope: The study was limited to Japanese patent reports and a specific query grammar. Expanding this to global patent systems (USPTO, EPO) remains a future challenge.

Summary updated

Abstract

Patent prior-art searchers issue Boolean queries line by line, broadening to find relevant documents and narrowing until candidates can be screened. We distill 38,397 Japan Patent Office search reports into executable expert strategies, re-execute them, and evaluate generated strategies by citation recall and the smallest set containing each citation. Prompted models write valid, high-recall strategies but leave citations in sets too large to screen. We adapt a generator offline from scored rollouts in two forms: strategy-level learning from a screening-aware strategy score, and line-level learning from citation-preserving narrowing credit, which upweights the query lines keeping a found citation in a smaller set. On shared rollouts, line-level credit outperforms strategy-level scores on screening measures; iterated, it cuts the median citation-containing set from 768 to under 50 and places more citations in sets under 100 documents than experts, at lower recall. Strategy-level preference learning reaches near-complete recall; the two forms are complementary.
BibTeX
@misc{kawano2026teaching,
  author = {Seiya Kawano and Koji Marusaki and Hirofumi Nonaka and Koichiro Yoshino},
  title = {{Teaching Language Models to Narrow Boolean Patent Searches from Search Feedback}},
  year = {2026},
  eprint = {2611.22270},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.22270},
}
Figure