GetNet: Get Target Area for Image Pairing

Henry H. Yu, Jiang Liu, Hao Sun, Ziwen Wang, Haotian Zhang

TL;DR

This paper introduces GetNet, a novel image pairing method that combines Spatial Transformer Networks (STNs) with Siamese networks to accurately identify and extract target objects within image pairs. This approach significantly improves image matching and retrieval accuracy, especially in cluttered scenes or with small objects, by focusing on relevant subparts rather than whole images.

What Problem Does This Paper Address?

Finding image pairs that contain objects of the same category is a fundamental task in computer vision, crucial for applications like object tracking, person re-identification, and image retrieval. Existing methods, both traditional and deep learning-based, often struggle with speed and accuracy, particularly when dealing with small objects, cluttered backgrounds, or large, unstructured datasets. While Siamese networks have shown promise, they can be less effective when the relevant object is not the sole focus of the image. This paper aims to bridge this gap by developing a method that can precisely locate and extract the target object of interest, thereby improving the robustness and accuracy of image pairing.

Methods

The core innovation of GetNet lies in its integration of Spatial Transformer Networks (STNs) with a Siamese CNN architecture.

  • Spatial Transformer Networks (STNs): STNs are used to dynamically learn an explicit spatial transformation of the input image. This allows the network to identify and extract a Region of Interest (ROI) that contains the target object. The STN learns to apply an affine transformation (initially translation, with potential for rotation and scaling in future work) to the input image, effectively "cropping" or focusing on the most relevant part.
  • Siamese CNN Architecture: After the STN extracts the target region from both images in a pair, a Siamese CNN architecture is employed. This architecture uses shared weights to process each image independently, extracting feature vectors. These feature vectors are then compared to determine the similarity between the two extracted regions.
  • Alternating Training Strategy: A novel training approach is proposed to address the challenge of training the STN effectively, especially with limited supervision. The training alternates between:
  • Training STN alone: Freezing the Siamese network's weights and training only the STN. This allows the STN to learn to accurately extract the target object.
  • Training the entire network: Training both the STN and the Siamese network. This allows the Siamese network to learn more robust features based on the STN's output. This strategy is crucial because traditional end-to-end training can be less effective for the STN when only simple 0/1 labels are available.

Datasets

The proposed GetNet method was evaluated on three datasets:

  • MNIST: A dataset of handwritten digits. For this paper, a "distorted MNIST" was created by placing original MNIST images onto a larger background with random noise, making it more challenging than the standard MNIST for pairing.
  • "Shelf & Tote" Benchmark Dataset: This dataset, originally created for the Amazon Picking Challenge, contains images of objects from multiple viewpoints. It was used for self-supervised deep learning for 6D pose estimation. Images were resized to 160x160 pixels for experiments.
  • Caltech Leaves Dataset: This dataset comprises images of three species of leaves against various backgrounds. Images were also resized to 160x160 pixels for the experiments.

For training and evaluation, image pairs were created by taking two images of the same object category (labeled '1' for matching) and one image from a different category (labeled '0' for non-matching).

Key Insights and Ideas

  • Focusing on the Target: The core idea that focusing on specific object regions, rather than the entire image, significantly boosts image pairing performance is a key insight. This mimics human visual attention.
  • STN for ROI Extraction: The clever use of STNs to automatically learn and extract the relevant object region is a novel contribution. This bypasses the need for manual annotation of ROIs, enabling effective training with simple 0/1 labels.
  • Alternating Training for Robustness: The proposed alternating training strategy is critical for effectively training the STN component, especially in scenarios with weak supervision. It allows for more precise learning of spatial transformations.
  • Improved Performance in Challenging Scenarios: The method demonstrates a clear advantage in scenarios where traditional Siamese networks struggle, such as with small objects or cluttered backgrounds.

Key Results

GetNet achieved competitive results in both speed and accuracy compared to traditional Siamese networks.

  • Accuracy Improvements:
    • MNIST: 98.2% (Siamese) vs. 99.3% (GetNet)
    • "Shelf & Tote" Dataset: 80.4% (Siamese) vs. 87.1% (GetNet)
    • Caltech Leaves Dataset: 84.3% (Siamese) vs. 88.6% (GetNet)
  • Qualitative Improvements: The paper shows qualitative examples where GetNet successfully pairs images that traditional Siamese networks failed to match, particularly when the target object was small or in a complex background. The STN output clearly demonstrates its ability to focus on the relevant object.

Limitations and Future Work

  • Limitations: The authors acknowledge that the current STN implementation primarily handles translation. They also note that performance can degrade when the image background is highly complex, making it difficult for the STNs to detect the correct object. Physical factors like light and noise can also affect results.
  • Future Work:
    • Expanded Transformations: Extend the STN to handle more complex affine transformations, including rotation and scaling, to further improve its ability to extract ROIs.
  • Network Optimization: Simplify the network structure and training process to improve efficiency and ease of use.
  • Noise Robustness: Develop strategies to overcome the effects of noise and complex backgrounds.
  • Medical Imaging Application: The authors suggest that this method could be highly beneficial for medical image research, particularly for lesion detection and outcome prediction, by providing a way to automatically locate regions of interest.

Summary updated

Abstract

Image pairing is an important research task in the field of computer vision. And finding image pairs containing objects of the same category is the basis of many tasks such as tracking and person re-identification, etc., and it is also the focus of our research. Existing traditional methods and deep learning-based methods have some degree of defects in speed or accuracy. In this paper, we made improvements on the Siamese network and proposed GetNet. The proposed method GetNet combines STN and Siamese network to get the target area first and then perform subsequent processing. Experiments show that our method achieves competitive results in speed and accuracy.
BibTeX
@inproceedings{yu2019getnet,
  author = {Henry H. Yu and Jiang Liu and Hao Sun and Ziwen Wang and Haotian Zhang},
  title = {{GetNet: Get Target Area for Image Pairing}},
  booktitle = {Ivcnz},
  year = {2019},
  doi = {10.1109/ivcnz48456.2019.8960995},
}
Figure