Interactive Instance Annotation with Siamese Networks

TL;DR

SiamAnno is a novel interactive instance annotation framework that adapts Siamese networks—typically used in video object tracking—to predict object boundaries from a simple bounding box. By framing annotation as a few-shot similarity matching problem, SiamAnno achieves exceptional zero-shot generalization on entirely unseen datasets, significantly outperforming existing contour-based annotation tools in cross-domain scenarios.

What Problem Does This Paper Address?

Creating high-quality datasets with pixel-level instance masks is notoriously slow and labor-intensive, taking human annotators an average of 20–30 seconds per object. While interactive deep learning tools can accelerate this by predicting initial boundaries that humans can refine, existing models suffer from a major flaw: poor generalization.

Most current tools rely on standard, one-pass convolutional backbones that overfit to the specific object categories and environments present in their training data. When deployed on a new dataset (a cross-domain scenario), their performance drops sharply. The authors address this gap by designing an annotation network specifically engineered for zero-shot, cross-domain generalization, minimizing the need for costly retraining or fine-tuning.

Methods

The core innovation of SiamAnno is treating instance annotation similarly to Video Object Tracking (VOT). In VOT, a Siamese network tracks a user-defined target across frames by matching features, even if that object was never seen during training.

Expand

SiamAnno's architecture consists of three main components:

1. Siamese Feature Extraction: The network splits into two weight-sharing branches: a target branch (focusing strictly on the object inside the bounding box) and a search branch (capturing the object and its surrounding context).
2. Pixel-wise Correlation & U-Net Fusion: Instead of naive feature concatenation, the model uses pixel-wise correlation to fuse the two branches, preserving precise spatial information. This fused map is then enhanced using a U-Net-style coarse-to-fine mechanism that injects low-level structural details (like edges and color changes) back into the high-level semantic features.
3. Contour Prediction Head: The model samples a set of initial vertices along the bounding box and applies dilated circular convolutions to predict how these vertices should deform to tightly hug the object's actual boundary.

Additionally, SiamAnno incorporates an attentive deformation mechanism to handle interactive human refinement: when an annotator drags an incorrect vertex, the model adaptively re-weights and updates the surrounding contour.

Datasets

The authors evaluated SiamAnno across several diverse datasets to rigorously test both in-domain and cross-domain performance:

• Cityscapes: Urban street-view dataset used as the primary training environment.
• COCO: A massive, general-object dataset used to train one of the baseline variants.
• KITTI: A smaller urban driving dataset used to evaluate performance under environmental and city shifts.
• ADE20k: A general scene dataset; a subset of categories (e.g., cars, persons, bicycles) was used to test cross-domain category generalization.
• Rooftop: An aerial imagery dataset of rural buildings, presenting a radical shift in both viewpoint (top-down) and geometry compared to Cityscapes.

Key Insights and Ideas

• Annotation as Tracking: The fundamental "aha" moment is realizing that isolating an object from its background within a static bounding box is functionally identical to how a Siamese tracker isolates a target frame-by-frame.
• Intrinsic Similarity Over Explicit Classification: By learning how to match features rather than how to classify specific objects, the Siamese architecture inherently possesses one-shot learning capabilities. This allows it to successfully outline objects it has never encountered before.
• The Power of Per-Component Labeling: The authors observed that contour-based models naturally struggle with fragmented or occluded objects (e.g., a person blocked by a pole). By introducing a "per-component" mode—where annotators box separate visible parts of an object individually—performance metrics skyrocketed, offering a more pragmatic blueprint for real-world annotation workflows.

Key Results

SiamAnno demonstrates state-of-the-art (SOTA) capabilities in cross-domain tasks, outperforming all previous contour-based methods without any fine-tuning on the target datasets:

• Cross-Domain SOTA: When trained on Cityscapes and tested zero-shot on other domains, SiamAnno achieved a mean Intersection over Union (mIoU) of 86.41% on KITTI, 74.90% on ADE20k, and 78.04% on Rooftop, substantially beating established baselines like Curve-GCN and Polygon-RNN++.
• In-Domain Performance: On the Cityscapes validation set, SiamAnno achieved a highly competitive 72.33% mIoU in per-instance mode, which rose to 76.69% mIoU in per-component mode. It also achieved a massive leap in Average Precision (mAP), hitting 39.6% compared to Split-GCN's 29.6%.

Limitations and Future Work

Because SiamAnno models object boundaries as a single continuous cycle graph (a closed loop), it inherently struggles with topological breaks—such as an object split into two disconnected pieces by an foreground obstruction.

For future work, the authors aim to develop a tighter, "annotator-in-the-loop" correction loop. This would allow the model to dynamically re-predict the entire boundary in real-time based on a user's manual vertex adjustments, further reducing human labor.

@misc{xu2025interactive,
  author = {Xiang Xu and Ruotong Li and Mengjun Yi and Baile XU and Furao Shen and Jian Zhao},
  title = {{Interactive Instance Annotation with Siamese Networks}},
  year = {2025},
  eprint = {2505.03184},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.03184},
}