Invariant Feature Mappings for Generalizing Affordance Understanding Using Regularized Metric Learning

Martin Hjelm, Carl Henrik Ek, Renaud Detry, Danica KragicarXiv 2019 · arXiv

Abstract

Robotic agents require a functional understanding of objects—affordances—to operate autonomously in unstructured environments. We propose a regularized metric learning approach, Regularized Large Margin Component Analysis (LMCA-R), to identify "invariant features" that define these functional relationships. By learning a sparse linear transformation, our system maps high-dimensional RGBD data into a low-dimensional latent space, effectively performing feature selection to isolate properties like shape or dimensions over irrelevant visual cues. Experimental results on a 22-category affordance dataset demonstrate that LMCA-R achieves an average accuracy of 90.75%, outperforming standard baselines and providing interpretable grounding of functional object parts.

Introduction

In robotics, the concept of "affordance" describes the potential actions an environment or object offers an agent (e.g., a handle affords grasping). While traditional computer vision excels at object recognition (labeling a "cup"), autonomous robots must understand why an object is suitable for a task to generalize to novel items. For instance, a spoon and a screwdriver may both afford "stirring" despite belonging to different semantic classes.

Existing deep learning models often act as "black boxes," making it difficult to extract the underlying physical principles driving a classification. This paper addresses the need for interpretable affordance understanding by identifying relevant sensory features, grounding this knowledge in the robot's sensory input, and discovering functional similarities between different affordances.

Overview of the metric learning process and feature grounding.

Methodology: LMCA-R

The authors introduce Regularized Large Margin Component Analysis (LMCA-R) to learn a functional mapping from sensory input to affordance space.

Feature Representation

Objects are represented by a 234-dimensional vector combining global features (volume, shape primitives, material) and local features (histograms of gradients, color statistics, and Fast Point Feature Histograms).

Metric Learning and Regularization

The core of the approach is learning a linear transformation matrix, L, that projects high-dimensional features into a low-dimensional latent space. In this space, objects sharing the same affordance are clustered together, while dissimilar objects are separated by a large margin. To ensure interpretability, the authors apply an $l_{1/2}$-norm penalty during learning. This regularization forces the model to become sparse, effectively discarding irrelevant features (e.g., ignoring color for "hammering") and focusing on the invariant properties that define the affordance.

Interpretability and Grounding

Because the transformation is linear, the magnitude of the columns in L directly corresponds to feature importance. These weights can be projected back onto the 3D point cloud of an object, creating "heatmaps" that highlight functional parts, such as the rim of a cup for "drinking."

Experimental Setup

The model was evaluated using a dataset of 265 RGBD images captured via a Kinect sensor. The dataset includes various everyday objects labeled with a binary vector across 22 affordance categories, such as Containing, Cutting, Drinking, and Stirring. This dataset presents a significant challenge due to "vague" labels where objects may have secondary affordances.

Results and Discussion

LMCA-R demonstrated strong performance across the board, achieving an average F1-score of 0.7 and an accuracy of 90.75%.

Performance comparison of LMCA-R against baseline models across 22 affordance categories.

Key Findings

  • Dimensionality Reduction: The model successfully projected 234-dimensional data down to just 3 dimensions with minimal loss in accuracy, representing a 99% reduction in dimensionality.
  • Feature Invariance: On average, the model discarded 60% of input features, proving that affordance understanding relies on a specific subset of invariant properties. For example, "Stirring" was consistently associated with thin, elongated geometries.
  • Semantic Discovery: By analyzing the learned transformation matrices, the system discovered functional hierarchies, such as the relationship between Rolling and Lifting Top, or Tool Use and Handle Grasping.
  • Comparison with Deep Learning: LMCA-R performed comparably to an SVM trained on features from a pre-trained GoogLeNet (CNN), while offering significantly higher interpretability and using far fewer parameters.

Conclusion and Future Work

LMCA-R provides a robust framework for grounding affordance understanding in physical object properties. While the system is highly effective, depth noise from the Kinect sensor occasionally hindered the recognition of flat surfaces. Future research will explore learning features directly from raw data (end-to-end) while preserving the sparsity and interpretability of the current metric learning approach.

Summary updated

Abstract

This paper presents an approach for learning invariant features for object affordance understanding. One of the major problems for a robotic agent acquiring a deeper understanding of affordances is finding sensory-grounded semantics. Being able to understand what in the representation of an object makes the object afford an action opens up for more efficient manipulation, interchange of objects that visually might not be similar, transfer learning, and robot to human communication. Our approach uses a metric learning algorithm that learns a feature transform that encourages objects that affords the same action to be close in the feature space. We regularize the learning, such that we penalize irrelevant features, allowing the agent to link what in the sensory input caused the object to afford the action. From this, we show how the agent can abstract the affordance and reason about the similarity between different affordances.
BibTeX
@misc{hjelm2019invariant,
  author = {Martin Hjelm and Carl Henrik Ek and Renaud Detry and Danica Kragic},
  title = {{Invariant Feature Mappings for Generalizing Affordance Understanding Using Regularized Metric Learning}},
  year = {2019},
  eprint = {1901.10673},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1901.10673},
}
Figure