Object-conditioned Bag of Instances for Few-Shot Personalized Instance Recognition

Umberto Michieli, Jijoong Moon, Daehyun Kim, Mete Ozay

Abstract

Nowadays, users demand for increased personalization of vision systems to localize and identify personal instances of objects (e.g., my dog rather than dog) from a few-shot dataset only. Despite outstanding results of deep networks on classical label-abundant benchmarks (e.g., those of the latest YOLOv8 model for standard object detection), they struggle to maintain within-class variability to represent different instances rather than object categories only. We construct an Object-conditioned Bag of Instances (OBoI) based on multi-order statistics of extracted features, where generic object detection models are extended to search and identify personal instances from the OBoI's metric space, without need for backpropagation. By relying on multi-order statistics, OBoI achieves consistent superior accuracy in distinguishing different instances. In the results, we achieve 77.1% personal object recognition accuracy in case of 18 personal instances, showing about 12% relative gain over the state of the art.

Introduction

Standard object detection models, such as YOLOv8, have reached high levels of precision in identifying object categories. However, these models often fail at personalization—the ability to distinguish between specific instances within a category (e.g., "my dog" vs. "your dog"). This failure is largely due to a phenomenon called neural collapse, where the training process encourages the model to ignore individual differences in favor of category-level generalization.

For practical applications like smart home robotics, instance-level recognition is critical. However, users typically cannot provide the thousands of labeled images required for traditional deep learning. The challenge lies in achieving this personalization in a "few-shot" scenario—using only one or a few examples—without the computational burden of full-network retraining.

Methodology

To address these challenges, we propose a modular, backpropagation-free addition to existing object detectors. Our pipeline consists of three core components:

  1. Object-Level Conditioning (OBoI): When a base detector identifies a category (e.g., "scissors"), the system restricts its search for the specific instance to a "bag" containing only prototypes related to that category. This conditioning significantly reduces the search space and eliminates cross-category distractors.
  2. Augmented Encoder Embeddings (AEE): Standard embeddings often lose the fine-grained texture information needed for instance recognition. We introduce multi-order statistics (such as mean and higher-order central moments) calculated from the features within the detected bounding box. This "re-expands" the feature space, capturing the unique distribution of an object's characteristics.
  3. Backpropagation-free Learning: We utilize Prototypes-based Few-Shot Learners (PFSLs). Instead of updating weights via gradient descent, the model stores an average feature map (a "prototype") for each new instance. Recognition is performed by finding the nearest prototype in a metric space, making the system ideal for edge devices.
The OBoI architecture integrates a frozen object detection backbone with a trainable Bag of Instances module using Augmented Encoder Embeddings (AEE) to identify specific object instances.

Experimental Evaluation

We evaluated our approach on two primary datasets:

  • CORe50: Features 45 personal instances across 9 categories, recorded in 11 different background sequences to test environmental robustness.
  • iCubWorld Transformations (iCWT): Includes 9 object classes with 10 instances each, captured under various rotations and scales.

Testing focused on two scenarios: 1SAS (1-Shot from All Sequences), where training data includes various environments, and 1S1S (1-Shot from 1 Sequence), a more difficult cross-domain task where the model is trained on one background and tested on others.

Results and Discussion

Our OBoI approach achieved a 77.1% accuracy for 18 personal instances on the CORe50 dataset, a 12% relative gain over current state-of-the-art methods. The inclusion of AEE proved vital, providing consistent performance boosts across nearly all object categories.

Accuracy comparison across various object categories for (a) 1SAS (same domain) and (b) 1S1S (other domain) scenarios, demonstrating the consistent superiority of Augmented Encoder Embeddings (AEE).

In the challenging 1S1S cross-domain setup, the method showed relative gains between 7% and 18%, proving its robustness to changes in lighting and background. Furthermore, the computational overhead is minimal; the AEE module increases inference time by only 0.8%, making it highly suitable for real-time applications on resource-constrained hardware.

Conclusion

The Object-conditioned Bag of Instances (OBoI) provides an efficient, scalable solution for personalized instance recognition. By fighting neural collapse through multi-order statistics and leveraging category conditioning, we achieve significant accuracy gains without the need for expensive backpropagation. Future work will explore extending this framework to more complex multi-object scenes and advanced continual learning environments.

Summary updated

Abstract

Nowadays, users demand for increased personalization of vision systems to localize and identify personal instances of objects (e.g., my dog rather than dog) from a few-shot dataset only. Despite outstanding results of deep networks on classical label-abundant benchmarks (e.g., those of the latest YOLOv8 model for standard object detection), they struggle to maintain within-class variability to represent different instances rather than object categories only. We construct an Object-conditioned Bag of Instances (OBoI) based on multi-order statistics of extracted features, where generic object detection models are extended to search and identify personal instances from the OBoI's metric space, without need for backpropagation. By relying on multi-order statistics, OBoI achieves consistent superior accuracy in distinguishing different instances. In the results, we achieve 77.1% personal object recognition accuracy in case of 18 personal instances, showing about 12% relative gain over the state of the art.
BibTeX
@inproceedings{michieli2024object,
  author = {Umberto Michieli and Jijoong Moon and Daehyun Kim and Mete Ozay},
  title = {{Object-conditioned Bag of Instances for Few-Shot Personalized Instance Recognition}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2024},
  doi = {10.1109/icassp48485.2024.10446073},
}
Figure