Is the U-Net Directional-Relationship Aware?

Mateus Riva, Pietro Gori, Florian Yger, Isabelle Bloch

Abstract

While Convolutional Neural Networks (CNNs) are widely assumed to utilize spatial context for scene understanding, empirical evidence of their ability to learn specific directional relationships remains sparse. This paper investigates whether a standard U-Net architecture can implicitly learn and utilize relative spatial configurations—such as "object A is above object B"—to perform segmentation. Using a novel synthetic benchmark, the Cloud of Structured Objects (CSO), we demonstrate that while U-Nets can learn these relationships without architectural modifications, doing so requires significantly more data than standard object recognition tasks.

Introduction

In computer vision, researchers often attribute a model's success to its ability to "understand" the scene. For instance, if a network identifies a person holding a racket on a tennis court, it is assumed the network uses the context of the court and the racket to inform its classification. However, distinguishing between true spatial reasoning and reliance on local textures or spurious dataset biases is a significant challenge in Explainable AI (XAI).

This study specifically targets directional relationships. We ask: can a standard U-Net learn to identify a target object based solely on its spatial relationship to other "anchor" objects? This question is fundamental to building robust models that do not fail when objects appear in unusual or out-of-context configurations.

Methodology

The researchers employed a standard U-Net architecture, a staple in medical and general image segmentation, trained using standard cross-entropy loss. To isolate the network's reasoning capabilities, they designed a specific segmentation task:

  1. The Task: The network must segment a specific "target" object (a shirt).
  2. The Challenge: The image contains multiple "noise" objects of the same class (other shirts). To identify the correct target, the network must determine which shirt sits at the vertex of a specific right-angled triangle formed by two reference objects: a pair of pants and a bag.
  3. Probing via Perturbation: To verify that the model relied on these relationships, the authors conducted a "sliding reference" analysis. By moving the reference objects (the pants or the bag) after training, they measured how the network's performance on the target changed. If the network truly learned the relationship, displacing the anchors should result in a failure to segment the target.

The Cloud of Structured Objects (CSO) Dataset

To facilitate this study, the authors introduced the Cloud of Structured Objects (CSO) dataset. This synthetic benchmark utilizes icons from the Fashion-MNIST dataset arranged in controlled spatial configurations. The dataset is categorized into three difficulty levels:

  • Easy: Noise objects belong to a different class than the target, allowing the model to rely on simple class recognition.
  • Hard: Noise objects are the same class as the target, forcing the model to use spatial context to distinguish the target from the background noise.
  • Strict: The absolute position of the object group is randomized, and noise is concentrated. This configuration renders absolute coordinates useless, forcing the model to learn relative directional relations to succeed.

Key Insights and Results

Implicit Reasoning and the Receptive Field

The study finds that a standard U-Net does not require specialized "spatial memory" modules or Graph Neural Networks to understand relative positioning. If the spatial relationship falls within the network's receptive field, it can be learned implicitly through standard backpropagation.

The "Data Hunger" of Relational Reasoning

A primary finding is the massive disparity in data requirements between simple recognition and relational reasoning. While the "Easy" task converged with as few as 100 samples, the "Strict" relational task required 50,000 samples to achieve high reliability.

In the most difficult "Strict" configuration, the model converged only 24% of the time (6 out of 25 seeds) when trained on 1,000 images. However, when the training set was increased to 50,000 images, the convergence rate jumped to 88% (22 out of 25 seeds). Once converged, the U-Net achieved high performance with an average Precision of 0.91 and Recall of 0.90.

Spatial Sensitivity

Heatmap analysis provided evidence of the network's reliance on directional context. When a reference object (e.g., the bag) was moved out of its expected position, the network's ability to recall the target shirt dropped significantly. This confirms that the network was not merely identifying the shirt's pixels but was actively searching for the presence of the bag in a specific relative direction to validate the target.

Discussion and Future Work

While these results demonstrate that U-Nets are capable of directional-relationship awareness, the study's reliance on synthetic data remains a limitation. Fashion-MNIST icons lack the occlusion, lighting variations, and perspective shifts found in real-world imagery.

Future research should focus on:

  1. Efficiency: Developing methods to reduce the high data requirements for relational learning.
  2. Architectural Constraints: Investigating how receptive field size limits the distance over which a network can reason.
  3. Real-world Application: Applying these findings to complex domains like medical imaging, where the structural relationship between organs is critical for accurate diagnosis.

Summary updated

BibTeX
@inproceedings{riva2022is,
  author = {Mateus Riva and Pietro Gori and Florian Yger and Isabelle Bloch},
  title = {{Is the U-Net Directional-Relationship Aware?}},
  booktitle = {ICIP},
  year = {2022},
  doi = {10.1109/icip46576.2022.9897715},
}
Figure