TL;DR
This paper introduces DeepPatent, a novel, large-scale dataset of over 350,000 patent drawings, designed to advance the field of technical drawing recognition and retrieval. They propose PatentNet, a deep learning model that achieves state-of-the-art performance on this challenging task, demonstrating the importance of within-domain fine-grained associations for effective retrieval.
What Problem Does This Paper Address?
Technical drawings, like those found in patents, are crucial for conveying complex information efficiently, often more so than text alone. However, existing computer vision approaches for image retrieval struggle with these drawings. Unlike natural images or simple sketches, patent drawings are detailed, often abstract, and lack the contextual information found in photographs. Furthermore, current retrieval methods often rely on text-based search or struggle with the unique characteristics of these drawings, leading to poor performance. The authors aim to bridge this gap by creating a dedicated dataset and developing a robust retrieval model for patent drawings.
Methods
The core of the paper's methodological contribution lies in the development of PatentNet, a deep learning model for patent drawing retrieval. PatentNet is built upon established best practices for training retrieval models on static images, leveraging architectures like ResNet18 and ResNet50. Key modifications include the use of Generalized Mean (GeM) pooling instead of standard average pooling, and 12 normalization of feature vectors for effective comparison.
The training process for PatentNet involves three stages:
- Weight Initialization: They explore self-supervised learning (RotNet) on the patent data versus using ImageNet pre-trained weights, finding the latter to be a better starting point.
- Classification Fine-tuning: The network is fine-tuned using either a classification loss on patent categories (each patent gets a unique label) or on sketch datasets (like Sketchy).
- Retrieval Fine-tuning: The classification layer is removed, and the network is further fine-tuned using contrastive loss or triplet loss to learn to distinguish between similar and dissimilar drawings.
Datasets
The primary contribution is the introduction of the DeepPatent dataset.
- DeepPatent: This dataset comprises over 350,000 design patent drawings, collected from USPTO bulk downloads. It's unique in providing fine-grained, within-domain associations by grouping drawings from the same patent, which is crucial for learning semantic similarity without relying on external labels. The dataset is split into training, validation, and test sets.
- ImageNet: Used for pre-training network weights.
- Sketchy: A sketch dataset used for comparison during the classification fine-tuning stage.
Key Insights and Ideas
- Patent drawings are a distinct domain: Despite similarities to sketches, patent drawings possess unique characteristics (detail, abstraction, multiple views) that require specialized approaches.
- Within-domain fine-grained associations are key: Simply using natural image pre-training or even sketch-based fine-tuning isn't as effective as training directly on patent drawings with their inherent relationships.
- Deep learning outperforms traditional methods: Deep learning models, particularly those based on ResNet architectures, significantly outperform classic computer vision descriptors like SIFT, HOG, and LBP on patent drawing retrieval.
- ImageNet pre-training is a strong baseline: Even though patent drawings differ from natural images, ImageNet pre-trained weights provide a better starting point for retrieval models than self-supervised learning on the target domain alone.
- The network learns higher-level semantics: t-SNE visualizations show that PatentNet can cluster drawings based on visual similarity (e.g., photo-like renderings, display objects) and perspective, indicating it learns more than just low-level features.
Key Results
- PatentNet achieves state-of-the-art performance: In quantitative comparisons (Table 1 and 2), PatentNet (specifically ResNet50 Tri) significantly outperforms traditional methods and other deep learning approaches on the DeepPatent dataset. For instance, PatentNet achieves a Mean Average Precision (mAP) of 0.379 and an Acc@1 of 0.701, far exceeding classic methods which are below 0.1 mAP.
- Fine-tuning on patents is superior: Fine-tuning a network on patent drawings directly yields better retrieval performance than fine-tuning on sketch datasets.
- Scalability is a challenge: As the database size increases from ~38,000 to ~350,000 images, mAP drops from 0.376 to 0.262, highlighting the difficulty of retrieving relevant drawings from a large collection.
Limitations and Future Work
- Limitations:
- The paper acknowledges that while PatentNet shows promise, patent drawing retrieval remains a challenging problem, as evidenced by the performance drop with increasing database size and qualitative failure cases.
- Cross-domain retrieval (e.g., sketch to patent) is shown to be difficult, suggesting that sketches and drawings, despite visual similarities, are distinct domains.
- Future Work:
- The authors hope their dataset and benchmark will spur further research into methods that effectively analyze abstract drawings.
- They suggest exploring models that can learn image representations at different levels of abstraction, potentially bridging the gap between sketches and natural images.
- Developing models that are more sensitive to shape and robust to image type, rather than relying heavily on color and texture, is another avenue.
- Achieving cross-modal understanding (e.g., relating drawings to text descriptions) is a long-term goal.
Summary updated