Object-Conditioned Bag of Instances for Few-Shot Personalized Instance Recognition

Umberto Michieli, Jijoong Moon, Daehyun Kim, Mete Ozay

TL;DR

Researchers from Samsung have developed a lightweight, backpropagation-free method called Object-conditioned Bag of Instances (OBoI) to help AI tell the difference between specific items, like "my dog" versus "your dog." By using "multi-order statistics" to capture the unique textures of individual objects, their approach achieves up to a 22% improvement in recognition accuracy with only a handful of training images.

What Problem Does This Paper Address?

Standard object detection models (like YOLOv8) are incredibly good at identifying categories—they can spot a "dog" or a "bottle" with high precision. However, they struggle with personalization. To a standard model, all dogs look like the "average dog" because the training process encourages the model to ignore individual differences to better generalize the category. This phenomenon is known as neural collapse, where within-class variability is lost.

For a user with a smart vacuum cleaner, "stay away from the dog" isn't enough if one pet is brave and the other is terrified of the robot. We need instance-level recognition. The challenge is that users won't provide thousands of labeled photos of their pets; they might only provide one or two. Traditional fine-tuning fails in these "few-shot" scenarios and is too computationally expensive for most home devices.

Methods

The authors propose a modular addition to existing detectors that doesn't require retraining the entire network. Their pipeline consists of three main steps:

  1. Object-Level Conditioning (OBoI): When the base detector (e.g., YOLOv8) identifies a "dog," the system restricts its search for the specific instance to a "bag" containing only dog-related prototypes. This drastically simplifies the search space.
  2. Augmented Encoder Embeddings (AEE): This is the "secret sauce." Instead of just using the standard feature vector (the "embedding") from the neural network, they calculate multi-order statistics (like mean and higher-order central moments) of the features within the bounding box. This captures the unique "distribution" of an object's features, making it easier to distinguish between two very similar-looking items.
  3. Backpropagation-free Learning: They use Prototypes-based Few-Shot Learners (PFSLs) like ProtoNet or SimpleShot. Instead of updating weights via gradient descent, the model simply stores a "prototype" (an average feature map) of the new instance. Recognition is then just a matter of finding the nearest prototype in a metric space.

Datasets

The researchers evaluated their method on two primary datasets designed for continuous and instance-level recognition:

  • CORe50: Contains 45 personal instances across 9 object categories, filmed across 11 different background sequences to test how well the model handles environment changes.
  • iCubWorld Transformations (iCWT): Features 9 object classes with 10 instances each, acquired under various affine transformations (rotations, scaling, etc.).

They tested two scenarios: 1SAS (1-Shot from All Sequences), where the model sees the object in various environments during training, and 1S1S (1-Shot from 1 Sequence), a much harder "cross-domain" task where the model is trained on one background and tested on others.

Key Insights and Ideas

  • Fighting Neural Collapse: The paper demonstrates that standard deep learning features are too "collapsed" to distinguish individuals. By adding multi-order statistics, they effectively "re-expand" the feature space to recover the details necessary for personalization.
  • Efficiency is King: Because the method is backpropagation-free, it is ideal for edge devices (smartphones, robots). It avoids the need for "dynamic compilers" or heavy GPU usage during the personalization phase.
  • Conditioning Matters: Simply knowing the category (e.g., "this is a ball") before trying to identify the specific instance (e.g., "this is ball #2") significantly boosts accuracy by removing "distractor" instances from other categories.

Key Results

  • Superior Accuracy: On the CORe50 dataset, the OBoI approach achieved 77.1% accuracy for 18 personal instances, representing a 12% relative gain over previous state-of-the-art methods.
  • Robustness to Change: In the challenging "other domain" (1S1S) setup, the method showed relative gains between 7% and 18%, proving it can recognize objects even when the background or lighting changes.
  • Negligible Overhead: Adding the AEE module increased the inference time by only 0.8%, making it a "practically free" upgrade for existing detection pipelines.
  • Scalability: The method works better as the base model gets larger (e.g., moving from YOLOv8n to YOLOv8x), but even the smallest "nano" models see massive benefits from the OBoI module.

Limitations and Future Work

The paper notes a few areas for improvement:

  • Dependency on the Base Detector: If the initial detector fails to categorize the object correctly (e.g., it thinks a dog is a cat), the OBoI module will look in the wrong "bag" and fail to identify the instance.
  • Single Object Assumption: For simplicity, the math in the paper assumes one object per image, though the authors mention the logic scales to multiple objects.
  • Future Directions: The authors suggest that this backpropagation-free approach could be a stepping stone for more advanced "continual learning" systems, where robots learn about new objects in their environment on the fly without ever needing a cloud-based update.

Summary updated

Abstract

Nowadays, users demand for increased personalization of vision systems to localize and identify personal instances of objects (e.g., my dog rather than dog) from a few-shot dataset only. Despite outstanding results of deep networks on classical label-abundant benchmarks (e.g., those of the latest YOLOv8 model for standard object detection), they struggle to maintain within-class variability to represent different instances rather than object categories only. We construct an Object-conditioned Bag of Instances (OBoI) based on multi-order statistics of extracted features, where generic object detection models are extended to search and identify personal instances from the OBoI's metric space, without need for backpropagation. By relying on multi-order statistics, OBoI achieves consistent superior accuracy in distinguishing different instances. In the results, we achieve 77.1% personal object recognition accuracy in case of 18 personal instances, showing about 12% relative gain over the state of the art.
BibTeX
@inproceedings{michieli2024object,
  author = {Umberto Michieli and Jijoong Moon and Daehyun Kim and Mete Ozay},
  title = {{Object-Conditioned Bag of Instances for Few-Shot Personalized Instance Recognition}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2024},
  doi = {10.1109/icassp48485.2024.10446073},
}

Loading related content...

Figure