Adaptive Deep Learning Through Visual Domain Localization

Gabriele Angeletti, Barbara Caputo, Tatiana Tommasi

TL;DR

The LoAd (Local Adaptive) network improves robot vision by spatially localizing "domain shift"—identifying which parts of an image are specific to an environment (like a specific background) versus which are universal (the object itself). Unlike standard domain adaptation methods, LoAd remains highly effective even when the robot only sees a subset of object categories in its new environment during training.

What Problem Does This Paper Address?

In robotics, a "domain shift" occurs when a robot trained in a factory or lab (the source domain) is deployed in a real home or warehouse (the target domain). Changes in lighting, background clutter, and camera angles can cause deep learning models to fail.

Most existing Domain Adaptation (DA) techniques are "transductive," meaning they assume the robot has access to unlabeled images of all possible object classes in the new environment during the adaptation phase. In the real world, this is unrealistic; a robot might only encounter a few types of objects initially. When standard DA methods (like DANN or Auto-DIAL) encounter this "sub-target" scenario, their performance often collapses because they try to align global distributions that don't actually match.

Methods

The authors propose the LoAd (Local Adaptive) architecture, which focuses on where the domain shift is happening rather than just that it is happening.

  1. Domain Localization (Domainness Maps): Inspired by Grad-CAM, the network uses gradients to create "domainness maps." These maps highlight regions that are domain-specific (e.g., a specific lab wall) versus domain-generic (e.g., the object of interest).
  2. Multiplicative Fusion: The network duplicates the feature maps from the final convolutional layer. One set is kept as-is, while the other is element-wise multiplied by the domainness maps. This "attends" the network's focus toward shared, domain-invariant features.
  3. Adaptive Max Pooling: Instead of standard fixed pooling, they use an adaptive window size (reducing the final feature map to 3x3). This reduces the number of parameters in the fully connected layers and increases invariance to the object's exact spatial location.
  4. Modular Architecture: The "clean" features and "domain-aware" features are concatenated before passing through the final classification layers.

Datasets

The researchers evaluated their method using the iCubWorld Transformations dataset. This dataset is specifically designed for robotics, featuring 15 object categories (like cups, remote controls, and toys) filmed by a robot under different conditions:

  • Translation: The object moves in a semi-circle, causing significant changes in background and lighting.
  • Scale: The object moves closer to or further from the camera, changing its size and the amount of visible background.

Key Insights and Ideas

  • Domain Shift is Local: In many robotics tasks, the "shift" isn't global; the object remains the same while the background or lighting changes. By localizing these roots of the shift, the model can "ignore" the distracting environment-specific noise.
  • Robustness to Missing Classes: Because LoAd learns to identify "domain-generic" regions (the objects) regardless of their specific class, the maps learned on a few categories generalize well to unseen categories. This solves the "sub-target" problem where other DA methods fail.
  • Explainability: The domainness maps provide a visual rationale for the robot's decisions. You can literally see if the robot is focusing on the object or getting distracted by the background.
  • Parameter Efficiency: By using adaptive pooling to shrink the feature maps before the fully connected layers, the authors significantly reduced the number of parameters to tune, making the model easier to train with limited data.

Key Results

The authors compared LoAd against state-of-the-art methods like DANN (Domain-Adversarial Neural Networks) and Auto-DIAL.

  • Translation Task: LoAd achieved an average accuracy of 64.55%, significantly outperforming DANN (51.08%) and Auto-DIAL (59.59%).
  • The "Sub-Target" Test: When only 8 of the 15 classes were available in the target domain during training, DANN's performance dropped. LoAd, however, remained stable, dropping only slightly from 64.55% to 63.82%.
  • Scale Task: This proved more difficult for localization. While DANN performed better on the "whole-target" version of this task, LoAd was more robust in the "sub-target" scenario, where DANN's accuracy fell from 43% to 28%.

Limitations and Future Work

  • Scale Challenges: The method struggled more with scale changes than translation. When an object is very close, it's harder for the network to distinguish between "object" and "shared background" because the object occupies most of the frame.
  • Open Set Adaptation: The authors suggest extending this to "open set" scenarios, where the target domain might contain entirely new categories of objects not seen in the source domain at all.
  • Multi-Modal Data: Future work involves applying this spatial localization logic to RGB-D (color + depth) data, which is common in modern robotics.

Summary updated

Abstract

A commercial robot, trained by its manufacturer to recognize a predefined number and type of objects, might be used in many settings, that will in general differ in their illumination conditions, background, type and degree of clutter, and so on. Recent computer vision works tackle this generalization issue through domain adaptation methods, assuming as source the visual domain where the system is trained and as target the domain of deployment. All approaches assume to have access to images from all classes of the target during training, an unrealistic condition in robotics applications. We address this issue proposing an algorithm that takes into account the specific needs of robot vision. Our intuition is that the nature of the domain shift experienced mostly in robotics is local. We exploit this through the learning of maps that spatially ground the domain and quantify the degree of shift, embedded into an end-to-end deep domain adaptation architecture. By explicitly localizing the roots of the domain shift we significantly reduce the number of parameters of the architecture to tune, we gain the flexibility necessary to deal with subset of categories in the target domain at training time, and we provide a clear feedback on the rationale behind any classification decision, which can be exploited in human-robot interactions. Experiments on two different settings of the iCub World database confirm the suitability of our method for robot vision.
BibTeX
@inproceedings{angeletti2018adaptive,
  author = {Gabriele Angeletti and Barbara Caputo and Tatiana Tommasi},
  title = {{Adaptive Deep Learning Through Visual Domain Localization}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2018},
  url = {https://openreview.net/forum?id=bU4E2duFThlm},
}

Loading related content...

Figure