Multi-Task Learning with Multi-Annotation Triplet Loss for Improved Object Detection

Meilun Zhou, Aditya Dutt, Alina ZarearXiv 2025 · DOI · arXiv

Abstract

Traditional object detection frameworks often struggle to reconcile the distinct requirements of classification and localization within a Multi-Task Learning (MTL) paradigm. While Triplet Loss is highly effective for learning discriminative feature embeddings for classification, it typically ignores the spatial characteristics essential for precise localization. This paper discusses the development of Multi-Annotation Triplet Loss (MATL), a framework that incorporates spatial metadata—such as bounding box size and symmetry—directly into the metric learning process. By supervising the latent space with both semantic and geometric labels, the MATL approach significantly enhances both classification accuracy and Intersection over Union (IoU) metrics in aerial wildlife detection tasks.

Introduction

In computer vision, Multi-Task Learning (MTL) seeks to optimize a single model for multiple objectives, most notably identifying an object's class and determining its precise spatial coordinates. A common challenge in MTL is the disconnect between feature representations: the features that distinguish a species (e.g., a deer from a cow) are not necessarily the same features required to define its boundaries.

Metric learning, specifically through Triplet Loss, has become a standard for improving classification by clustering similar instances and pushing dissimilar ones apart in a latent space. However, standard Triplet Loss treats all instances of a class as identical targets, regardless of their orientation, scale, or shape within the frame. This research introduces Multi-Annotation Triplet Loss (MATL) to bridge this gap by integrating bounding box characteristics into the embedding logic, ensuring the latent space is informed by the physical geometry of the targets.

Methodology

The core innovation of this work is the generation of "box labels" ($y_{box}$) to complement traditional class labels ($y_{class}$). This dual-supervision forces the model to organize its latent space based on both what an object is and how it appears spatially.

Spatial Feature Extraction and Clustering

To create meaningful spatial annotations, the authors extract two primary features from every ground-truth bounding box:

  1. Area: The total pixel count, representing the scale of the object.
  2. Symmetric Squareness: A measure of how closely the bounding box approaches a perfect square, which serves as a proxy for object orientation and posture.

These continuous variables are normalized and processed via K-means clustering to produce discrete spatial categories (e.g., "Small-Square" vs. "Large-Elongated"). These categories serve as the secondary labels for the triplet loss function.

The MATL Loss Function

The total loss is calculated as a weighted combination of the class-based triplet loss and the box-based triplet loss:

$$L_{MATL} = (1 - \lambda)L_{class} + \lambda L_{box}$$

The hyperparameter $\lambda$ controls the influence of spatial metadata on the embedding. Through empirical testing, the researchers identified $\lambda = 0.25$ as the optimal balance for maximizing performance across both tasks.

Model Architecture

The framework utilizes an autoencoder-based backbone. An Encoder extracts multi-scale features, which are then branched into two heads: a Decoder that reconstructs a binary mask for localization and a Classifier Head for species identification. The MATL is applied to the bottleneck features of the encoder to ensure the learned representations are robust for both downstream tasks.

Experimental Setup

The model was evaluated using the Animal Wildlife Image Repository (AWIR), which contains RGB aerial imagery of three species: Deer, Horse, and Cow. To prevent positional bias and ensure the model learned shape rather than location, images were processed into $300 \times 300$ pixel tiles with random cropping, ensuring the animals were not always centered.

Results and Discussion

The MATL approach was benchmarked against models trained without triplet loss (WTL) and models using standard Class-Label Triplet Loss (CLTL).

Quantitative Performance

Method Classification Accuracy Localization (IoU)
Without Triplet Loss (WTL) 67.8% -
Class-Label Triplet Loss (CLTL) 78.1% 0.149
MATL ($\lambda=0.25$) 83.2% 0.189

The results demonstrate that MATL not only improves classification by creating more nuanced sub-clusters within the latent space (e.g., grouping horses by orientation) but also provides a superior foundation for the localization head, leading to a significant increase in IoU.

Qualitative Insights

By utilizing "Symmetric Squareness," the model learns to differentiate between animals based on their posture. This is particularly effective for larger animals like cows; while standard methods often produce loose or inaccurate boundaries, MATL-trained models generate tighter, more precise bounding boxes that better reflect the animal's actual footprint in the aerial view.

Conclusion and Future Work

Multi-Annotation Triplet Loss represents a significant step forward in aligning metric learning with the requirements of object detection. By incorporating spatial metadata into the loss function, the model develops a more sophisticated understanding of the relationship between an object's identity and its physical manifestation.

Future research will focus on expanding the dataset to include higher intra-class variance and exploring multi-modal inputs. The integration of thermal imagery is a primary interest, as heat signatures could provide a third annotation layer to improve detection in low-light or high-canopy environments. Additionally, the team plans to investigate more complex fusion strategies, such as attention-based mechanisms, to further refine the integration of class and spatial data.

Summary updated

Abstract

Triplet loss traditionally relies only on class labels and does not use all available information in multi-task scenarios where multiple types of annotations are available. This paper introduces a Multi-Annotation Triplet Loss (MATL) framework that extends triplet loss by incorporating additional annotations, such as bounding box information, alongside class labels in the loss formulation. By using these complementary annotations, MATL improves multi-task learning for tasks requiring both classification and localization. Experiments on an aerial wildlife imagery dataset demonstrate that MATL outperforms conventional triplet loss in both classification and localization. These findings highlight the benefit of using all available annotations for triplet loss in multi-task learning frameworks.
BibTeX
@misc{zhou2025multi,
  author = {Meilun Zhou and Aditya Dutt and Alina Zare},
  title = {{Multi-Task Learning with Multi-Annotation Triplet Loss for Improved Object Detection}},
  year = {2025},
  eprint = {2504.08054},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2504.08054},
}
Figure