Meta-Tuning Loss Functions and Data Augmentation for Few-Shot Object Detection

Berkan Demirel, Orhun Bugra Baran, Ramazan Gokberk Cinbis

Abstract

Few-shot object detection, the problem of modelling novel object detection categories with few training instances, is an emerging topic in the area of few-shot learning and object detection. Contemporary techniques can be divided into two groups: fine-tuning based and meta-learning based approaches. While meta-learning approaches aim to learn dedicated meta-models for mapping samples to novel class models, fine-tuning approaches tackle few-shot detection in a simpler manner, by adapting the detection model to novel classes through gradient based optimization. Despite their simplicity, fine-tuning based approaches typically yield competitive detection results. Based on this observation, we focus on the role of loss functions and augmentations as the force driving the fine-tuning process, and propose to tune their dynamics through meta-learning principles. The proposed training scheme, therefore, allows learning inductive biases that can boost few-shot detection, while keeping the advantages of fine-tuning based approaches. In addition, the proposed approach yields interpretable loss functions, as opposed to highly parametric and complex few-shot meta-models. The experimental results highlight the merits of the proposed scheme, with significant improvements over the strong fine-tuning based few-shot detection baselines on benchmark Pascal VOC and MS-COCO datasets, in terms of both standard and generalized few-shot performance metrics.

Introduction

Few-Shot Object Detection (FSOD) is the challenge of teaching a model to recognize new objects using only a handful of examples (e.g., 1 to 30 images). Current approaches usually fall into two camps: simple fine-tuning, which is effective but often lacks the "inductive bias" needed to learn efficiently from very little data, and meta-learning, which uses complex systems to "learn how to learn" but is prone to overfitting and difficult to interpret.

This paper introduces Meta-Tuning, a framework that bridges this gap. Instead of building a complex, "black-box" meta-model, the authors propose using Reinforcement Learning to optimize the fine-tuning process itself. By automatically discovering the best loss function dynamics and data augmentation magnitudes, the method achieves state-of-the-art performance while maintaining the simplicity and interpretability of standard fine-tuning.

Methods: The Meta-Tuning Framework

The authors propose a framework that uses a Reinforcement Learning (RL) agent (specifically the REINFORCE algorithm) to search for optimal training hyperparameters.

1. Meta-Tuning Loss Functions

The core innovation is making the classification loss flexible. They introduce:

  • Dynamic Temperature Scaling: Instead of a static Softmax, they use a temperature parameter that changes over time based on a learned polynomial function. This allows the model to "sharpen" its predictions as training progresses.
  • Novel Class Scaling: A learned coefficient balances the scores between well-known "base" classes and the "novel" few-shot classes, preventing the model from ignoring new categories.

2. Meta-Tuning Augmentations

The RL agent also learns the optimal shared magnitude for photometric augmentations (brightness, contrast, saturation, hue). This ensures the data pipeline is perfectly tailored for the specific few-shot task at hand.

3. The Proxy Task Strategy

To train the RL agent without "cheating" on actual novel classes, the authors divide the base dataset into "proxy-base" and "proxy-novel" sets. The agent practices on these proxy tasks to find parameters that generalize well, which are then applied to the real few-shot classes.

The Meta-Tuning framework using a proxy task strategy to optimize augmentations and loss functions via REINFORCE.

Experimental Results

The approach was evaluated on the Pascal VOC and MS-COCO benchmarks, consistently outperforming strong baselines:

  • MS-COCO (10-shot): Improved the MPSR baseline from 9.1 to 12.5 mAP.
  • Pascal VOC: When combined with the state-of-the-art DeFRCN model, it achieved top results across almost all k-shot settings, hitting 61.3 mAP on 5-shot.
  • Generalized FSOD: The method significantly improved "Harmonic Mean" scores, proving it helps the model learn new classes without degrading performance on original base classes.
Qualitative detection results on Pascal VOC across various splits and shot settings, demonstrating the model's ability to localize and classify novel objects.

Discussion and Insights

  • Sharpening the Focus: The learned dynamic temperature curves show that the model prefers "sharper" probability distributions toward the end of fine-tuning, forcing more confident predictions for novel classes.
  • Interpretability: Unlike meta-networks with millions of parameters, a temperature curve or an augmentation magnitude is human-readable. We can actually see what the meta-learner discovered.
  • Transferability: Interestingly, loss parameters meta-tuned on one base model (like MPSR) improved performance when transferred to a completely different model (like DeFRCN), suggesting these "learned biases" are fundamental to the few-shot problem.

Limitations and Future Work

The paper focuses primarily on classification loss and photometric augmentations. The authors note that meta-tuning other components—like localization loss or initial model weights—remains unexplored. Additionally, while final fine-tuning is fast, the initial RL search for optimal parameters requires running many training episodes on proxy tasks, adding offline computational overhead.

Summary updated

Abstract

Few-shot object detection, the problem of modelling novel object detection categories with few training instances, is an emerging topic in the area of few-shot learning and object detection. Contemporary techniques can be divided into two groups: fine-tuning based and meta-learning based approaches. While meta-learning approaches aim to learn dedicated meta-models for mapping samples to novel class models, fine-tuning approaches tackle few-shot detection in a simpler manner, by adapting the detection model to novel classes through gradient based optimization. Despite their simplicity, fine-tuning based approaches typically yield competitive detection results. Based on this observation, we focus on the role of loss functions and augmentations as the force driving the fine-tuning process, and propose to tune their dynamics through meta-learning principles. The proposed training scheme, therefore, allows learning inductive biases that can boost few-shot detection, while keeping the advantages of fine-tuning based approaches. In addition, the proposed approach yields interpretable loss functions, as opposed to highly parametric and complex few-shot meta-models. The experimental results highlight the merits of the proposed scheme, with significant improvements over the strong fine-tuning based few-shot detection baselines on benchmark Pascal VOC and MS-COCO datasets, in terms of both standard and generalized few-shot performance metrics.
BibTeX
@inproceedings{demirel2023meta,
  author = {Berkan Demirel and Orhun Bugra Baran and Ramazan Gokberk Cinbis},
  title = {{Meta-Tuning Loss Functions and Data Augmentation for Few-Shot Object Detection}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  url = {https://openreview.net/forum?id=FQnY2JZ61X},
}
Figure