Image interpretation by iterative bottom-up top-down processing

Shimon Ullman, Liav Assif, Alona Strugatski, Ben-Zion Vatashsky, Hila Levi, Aviv Netanyahu, Adam Yaari

TL;DR

This paper introduces a novel "counter-streams" neural network architecture that combines bottom-up and top-down processing for image interpretation. By using iterative top-down instructions to guide bottom-up feature extraction, the model achieves strong combinatorial generalization, enabling it to understand novel scene structures and combinations of objects and relations not seen during training.

What Problem Does This Paper Address?

Understanding complex visual scenes requires not just identifying objects, but also their properties, relations, and how they interact. Traditional computer vision models often struggle with this by either focusing on a single task or requiring extensive training for every possible combination of objects and relations. This paper addresses the challenge of building a system that can flexibly extract rich, structured scene representations, generalize to novel situations, and mimic aspects of human visual cognition, particularly the ability to focus attention and process information in a goal-directed manner.

Methods

The core of the proposed solution is the BU-TD (Bottom-Up, Top-Down) counter-streams network. This architecture features two interconnected streams:

  • Bottom-Up (BU) stream: Similar to standard deep networks (e.g., ResNet), it extracts visual features from the input image.
  • Top-Down (TD) stream: This stream receives "TD instructions" from a cognitive part of the system. These instructions guide the BU stream on what specific information to extract next.

The interaction is iterative and bi-directional:

  1. The BU stream processes the image and sends high-level representations to the TD stream.
  2. A "cognitive augmentation" component (not detailed in this paper) enriches these representations with non-visual knowledge.
  3. The TD stream, informed by this augmented representation and a goal, generates a specific TD instruction.
  4. This instruction is fed back to the BU stream, guiding its next processing cycle to focus on the instructed task.

This iterative process allows the model to compose complex visual tasks from sequences of simpler instructions, forming "visual routines." The paper also highlights the importance of compositional instruction representation, where tasks and their arguments are specified separately, leading to better generalization.

Datasets

The paper utilizes several datasets for training and evaluation:

  • Persons dataset: A synthetic dataset generated using "avatarmaker.com" with variations in facial features, clothing, and accessories.
  • Actors dataset: Computer-generated scenes created with "Makehuman" featuring actors, objects, and relations.
  • EMNIST dataset: Used for tasks involving spatial relations between characters.
  • Multi-MNIST dataset: Used to test multi-task learning capabilities with overlapping digits.
  • COCO train2017 and val2017: Used for object detection and classification tasks.

Key Insights and Ideas

  • Top-down guidance is crucial for flexible scene understanding: By allowing a "cognitive" component to issue instructions, the system can dynamically focus its processing, leading to more efficient and targeted information extraction.
  • Iterative "counter-streams" enable complex task composition: The bi-directional flow of information allows for the step-by-step construction of complex visual routines, akin to visual programs.
  • Compositional instruction representation unlocks combinatorial generalization: Separating the "task" from the "argument" in TD instructions is key to learning how to combine known components in novel ways, a critical aspect of human-like intelligence.
  • Generalization is not just about seeing more data, but about structured learning: The model demonstrates that learning structured representations and using guided instructions leads to better generalization than simply increasing dataset size or using unguided models.

Key Results

  • Strong Combinatorial Generalization: The BU-TD model achieved significantly higher accuracy in combinatorial generalization tests (understanding novel combinations of objects, properties, and relations) compared to unguided models. For instance, in the Persons dataset, the BU-TD model reached 97% accuracy on generalization tests with extended data, while the unguided model achieved only 71%.
  • Task Selectivity: The model demonstrated high selectivity for the instructed task, achieving over 90% accuracy for the selected task location while remaining close to chance level for non-selected tasks.
  • Efficient Learning: The BU-TD model often achieved comparable or better performance with fewer training examples or epochs compared to other approaches, especially in combinatorial generalization scenarios.
  • Human Vision Parallels: The counter-streams architecture is inspired by the reciprocal connections observed in the primate visual cortex, suggesting a potential link to biological vision mechanisms.

Limitations and Future Work

  • Cognitive Augmentation Not Detailed: The paper mentions a "cognitive augmentation" component that enriches visual representations with non-visual stored knowledge, but this aspect is not elaborated upon.
  • Scalability to Real-World Complexity: While the model shows promise, dealing with the vast complexity of natural images and the potentially enormous number of objects, properties, and relations remains a significant challenge.
  • Future Directions: The authors suggest exploring the "counter-tree" structure for hierarchical processing, further investigating the integration of visual and cognitive components, and studying combinatorial generalization in early human development to understand innate vs. learned representations.

Summary updated

Abstract

Scene understanding requires the extraction and representation of scene components together with their properties and inter-relations. We describe a model in which meaningful scene structures are extracted from the image by an iterative process, combining bottom-up (BU) and top-down (TD) networks, interacting through a symmetric bi-directional communication between them (counter-streams structure). The model constructs a scene representation by the iterative use of three components. The first model component is a BU stream that extracts selected scene elements, properties and relations. The second component (cognitive augmentation) augments the extracted visual representation based on relevant non-visual stored representations. It also provides input to the third component, the TD stream, in the form of a TD instruction, instructing the model what task to perform next. The TD stream then guides the BU visual stream to perform the selected task in the next cycle. During this process, the visual representations extracted from the image can be combined with relevant non-visual representations, so that the final scene representation is based on both visual information extracted from the scene and relevant stored knowledge of the world. We describe how a sequence of TD-instructions is used to extract from the scene structures of interest, including an algorithm to automatically select the next TD-instruction in the sequence. The extraction process is shown to have favorable properties in terms of combinatorial generalization, generalizing well to novel scene structures and new combinations of objects, properties and relations not seen during training. Finally, we compare the model with relevant aspects of the human vision, and suggest directions for using the BU-TD scheme for integrating visual and cognitive components in the process of scene understanding.
BibTeX
@misc{ullman2020image,
  author = {Shimon Ullman and Liav Assif and Alona Strugatski and Ben-Zion Vatashsky and Hila Levi and Aviv Netanyahu and Adam Yaari},
  title = {{Image interpretation by iterative bottom-up top-down processing}},
  year = {2020},
  eprint = {2105.05592},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2105.05592},
}
Figure