Abstract
We tackle object category discovery, which is the problem of discovering and localizing novel objects in a large unlabeled dataset. While existing methods show results on datasets with less cluttered scenes and fewer object instances per image, we present our results on the challenging COCO dataset. Moreover, we argue that, rather than discovering new categories from scratch, discovery algorithms can benefit from identifying what is already known and focusing their attention on the unknown. We propose a method that exploits prior knowledge about certain object types to discover new categories by leveraging two memory modules, namely Working and Semantic memory. We show the performance of our detector on the COCO minival dataset to demonstrate its in-the-wild capabilities.
Introduction: The Challenge of Open-World Discovery
Unsupervised object discovery aims to identify and group recurring objects in images without manual labels. Despite its potential to drastically reduce annotation costs, current approaches face significant hurdles in scalability and real-world applicability. Most existing methods are limited to small, curated datasets with simple scenes. When faced with "in-the-wild" data like COCO—characterized by cluttered backgrounds and numerous overlapping objects—these methods often fail.
Furthermore, most algorithms attempt to learn from a blank slate. In reality, we often possess prior knowledge of certain categories (e.g., "cars" or "dogs"). Failing to leverage this "known" information makes it significantly harder to isolate and learn truly novel "unknown" objects.
Methodology: A Dual-Memory Architecture
To address these challenges, we introduce a dual-memory framework inspired by human cognition. The system consists of three primary components:
- Encoding: A Faster R-CNN detector, pre-trained on a set of "known" classes (e.g., Pascal VOC), generates region proposals and extracts visual features.
- Dual Memory Storage:
- Working Memory ($M_w$): A short-term, volatile module that stores recently encountered patterns using feature centroids. This allows for rapid updates as new data arrives.
- Semantic Memory ($M_s$): A long-term module representing established knowledge. It stores both the initial "known" classes and newly discovered categories using robust LDA classifiers to maintain high semantic purity.
- Retrieval and Consolidation: New image regions are first compared against Semantic Memory. If no match is found, they are checked against Working Memory. Periodically, a "Memory Consolidation" step merges stable clusters in $M_w$ and promotes them to $M_s$ after refinement.
Experimental Setup and Results
We evaluated our framework using a rigorous benchmark: Pascal VOC 2007 served as the "known" set, while the remaining 60 non-overlapping categories in COCO 2014 were treated as "unknowns."
Our method demonstrated superior discovery capabilities, identifying 46 novel categories on the full COCO dataset. In terms of localization, we achieved a Correct Localization (CorLoc) of 43.00%, a substantial improvement over the previous state-of-the-art (OSD), which achieved only 6.6%. Qualitative results highlight the system's ability to group complex categories like traffic lights and bears in diverse environments.
Conclusion and Future Directions
By using the "known" as a filter, our dual-memory approach effectively isolates novel signals in complex datasets. However, a primary bottleneck remains the Region Proposal Network (RPN). Since the RPN is trained on known classes, its recall for novel objects is significantly lower (35% vs. 71%). Future work will focus on developing class-agnostic proposal generators to ensure that the memory system can "see" all potential objects, regardless of prior exposure.
Summary updated