Image interpretation by iterative bottom-up top-down processing

Shimon Ullman, Liav Assif, Alona Strugatski, Ben Zion Vatashsky, Hila Levy, Aviv Netanyahu, Adam Yaari

TL;DR

This paper introduces a bidirectional "counter-streams" neural network architecture that mimics human vision by iteratively combining bottom-up feature extraction with top-down cognitive guidance. By executing top-down instructions as dynamic "visual routines," the model avoids the capacity limitations of standard multi-task networks and achieves remarkable combinatorial generalization on novel scenes and object-property pairings.

What Problem Does This Paper Address?

Most state-of-the-art computer vision models are predominantly feed-forward, bottom-up systems. While they excel at localized tasks like object detection or segmentation, they struggle with comprehensive scene understanding—the ability to parse objects, their sub-parts, their individual properties, and the complex interactions between them all at once.

When standard deep networks are trained to perform multiple tasks simultaneously, they hit a wall. They suffer from capacity limitations and task interference, where learning a new task degrades performance on an old one. More critically, they fail at combinatorial generalization: if a model learns what a "red bag" and a "blue shoe" look like, it often fails to correctly identify a "blue bag" in a novel, cluttered scene without extensive retraining. Human vision solves this by using massive top-down feedback to sequentially shift attention and dynamically execute "visual routines." This paper bridges that gap.

Methods

The authors propose the BU-TD (Bottom-Up Top-Down) counter-streams network.

Vision-Cognition Loop

The architecture consists of two parallel, structurally identical streams running in opposite directions:

  1. Bottom-Up (BU) Stream: A standard deep convolutional network (e.g., ResNet-18) that extracts visual features from the input image.
  2. Top-Down (TD) Stream: A reversed network that propagates a task instruction down through the hierarchy, modulating the feature maps of the BU stream via bidirectional cross-stream lateral connections (using additive or multiplicative gating).

The Iterative Process and Visual Routines

Instead of outputting everything in a single pass, the model operates in cycles. It receives a symbolic top-down instruction formatted as a <task, argument> pair (e.g., <segment, person-5> or <classify, location>).

To handle complex queries like "Is the woman facing the girl holding a paper bag?", the model automatically composes a sequence of instructions—a visual routine—by combining two specialized sub-networks:

  • The Expansion Network: Responsible for discovering and segmenting new entities based on spatial or semantic relations (e.g., finding the next closest object).
  • The Elaboration Network: Responsible for evaluating specific properties or binary relations of already discovered entities.

Datasets

The model was evaluated across several distinct datasets designed to test multi-task capacity and relational reasoning:

  • Multi-MNIST: Images containing multiple, partially overlapping handwritten digits to evaluate spatial attention and multi-task scaling.
  • Persons Dataset: A synthetic avatar dataset generated to test fine-grained property extraction (hair style, glasses type, etc.) and combinatorial generalization.
  • The Actors Dataset: A complex, custom-generated 3D dataset using Makehuman and Blender. It features multiple characters and objects interacting via complex relations like facing, touching, holding, and obstructing.
  • CUB200 & EMNIST: Standard benchmarks used to validate object-part relations (bird anatomy) and abstract spatial configurations (character arrays).

Key Insights and Ideas

  • Task Selection Over Multi-Branching: Standard multi-task networks use a shared backbone with dedicated branches for each task. The BU-TD model proves that executing one instructed task at a time completely bypasses task interference, allowing the network to scale to an arbitrary number of tasks without performance degradation.
  • Emergence of Compositionality: When the model is given a single, unified symbolic instruction vector (blending task and argument together), the learned internal vector embedding naturally differentiates into a compositional structure (separating "what" from "who/where"). This internal compositionality is what unlocks generalization.
  • Generalization via Location: Deep networks typically fail to generalize spatial relations (like "right-of") to untrained object classes (e.g., learning on digits doesn't transfer to letters). The authors show that breaking the relation down into a location-based routine (find object location $\rightarrow$ predict neighbor location $\rightarrow$ classify at neighbor location) allows abstract spatial concepts to generalize perfectly to entirely novel shapes.

Key Results

  • Multi-Task Scaling: On a highly challenging 9-digit Multi-MNIST classification task, the BU-TD model achieved 88.07% accuracy, significantly outperforming standard multi-objective branching networks (80.52%) and channel modulation techniques (76.56%).
  • Shattering the Generalization Ceiling: In experiments where specific person-property combinations (e.g., a specific avatar wearing a specific type of glasses) were strictly hidden during training, the BU-TD model maintained an accuracy of 89% on sufficient data. In contrast, the unguided baseline model collapsed to near-chance levels (31%).
Generalization Performance
  • Complex Scene Parsing: The model successfully demonstrated full scene-graph extraction on the Actors dataset, sequentially building a structured graph of components, properties, and interactions matching the ground truth.

Limitations and Future Work

  • The Cognitive Augmentation Oracle: The paper introduces a "cognitive augmentation" phase to inject non-visual world knowledge (e.g., inferring "ownership" or "stealing" based on gaze and context). However, this component is largely treated as an external module and is not the focus of the computational implementation.
  • Unguided Exploration: The current routine generation relies heavily on having a goal or a target query graph. Developing a robust "default mode" for open-ended, unguided scene exploration—deciding what is interesting to look at when no question is asked—remains an open challenge.
  • Real-World Scaling: While validated on complex 3D renderings (Actors dataset), scaling this iterative, instruction-driven approach to massive, unannotated real-world image datasets is a major next step.

Summary updated

Abstract

Scene understanding requires the extraction and representation of scene components together with their properties and inter-relations. We describe a model in which meaningful scene structures are extracted from the image by an iterative process, combining bottom-up (BU) and top-down (TD) networks, interacting through a symmetric bi-directional communication between them (counter-streams structure). The model constructs a scene representation by the iterative use of three components. The first model component is a BU stream that extracts selected scene elements, properties and relations. The second component (cognitive augmentation) augments the extracted visual representation based on relevant non-visual stored representations. It also provides input to the third component, the TD stream, in the form of a TD instruction, instructing the model what task to perform next. The TD stream then guides the BU visual stream to perform the selected task in the next cycle. During this process, the visual representations extracted from the image can be combined with relevant non-visual representations, so that the final scene representation is based on both visual information extracted from the scene and relevant stored knowledge of the world. We describe how a sequence of TD-instructions is used to extract from the scene structures of interest, including an algorithm to automatically select the next TD-instruction in the sequence. The extraction process is shown to have favorable properties in terms of combinatorial generalization, generalizing well to novel scene structures and new combinations of objects, properties and relations not seen during training. Finally, we compare the model with relevant aspects of the human vision, and suggest directions for using the BU-TD scheme for integrating visual and cognitive components in the process of scene understanding.
BibTeX
@misc{ullman2020image,
  author = {Shimon Ullman and Liav Assif and Alona Strugatski and Ben Zion Vatashsky and Hila Levy and Aviv Netanyahu and Adam Yaari},
  title = {{Image interpretation by iterative bottom-up top-down processing}},
  year = {2020},
  eprint = {2105.05592},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2105.05592},
}
Figure