The Representation and Use of a Visual Lexicon for Automated Graphics Generation

IJCAI 1997

TL;DR

This paper introduces IMPROVISE, a system that combines constructive and parametric approaches for automated graphics generation. The core contribution is a comprehensive, general, and extensible formalism for representing a "visual lexicon" – a collection of parametrized primitive visual objects that act as building blocks for complex visualizations. This lexicon, along with a constraint-based selection and instantiation mechanism, enables more efficient and flexible automated graphics generation.

What Problem Does This Paper Address?

Automated graphics generation is a complex task. Existing systems typically fall into two categories: constructive (building visuals from scratch based on design principles) and parametric (instantiating pre-defined, parametrized visual models). While effective for specific domains (like scientific visualization), these approaches often lack the generality and expressiveness to handle heterogeneous information or qualitative data. This paper addresses the need for a more robust and flexible system that can generate a wider variety of visual presentations by combining the strengths of both approaches, specifically focusing on how to represent and utilize a "visual lexicon" for parametric synthesis.

Methods

The paper proposes a visual lexicon as a collection of "visual words," where each word is a parametrized primitive visual object. This lexicon is represented using an object-oriented paradigm, allowing for a comprehensive capture of syntactic, semantic, and pragmatic features of each visual word.

  • Representation: Each visual word is associated with a domain object and has multiple "senses," each with its own syntax, semantics, and pragmatics.
  • Syntax: Describes the structure and type of the visual word (category, subcategory, media format like graphical model, file, image, or video).
  • Semantics: Abstracts the meaning, role (e.g., Identify, Locate), and scope (e.g., Naming, SpatialPosition) of the visual word.
  • Pragmatics: Captures contextual information like domain specifics (application type, audience), hardware requirements, and performance characteristics.
  • Lexeme: The actual graphical expression of the visual word, which can be a procedural definition or a reference to a file.
  • Usage: The system uses a constraint-based approach for selecting and instantiating visual words.
  • Visual Lexicon Selection: Involves matching an object description against object-patterns in the lexicon. This matching considers inheritance and specificity. A set of syntactic, semantic, and pragmatic constraints then filters and ranks the candidates.
  • Visual Lexicon Instantiation: Once a visual word is selected, its parameters are instantiated. This process also uses a "least-commitment" algorithm, deferring instantiation until sufficient information is available. Variables are used to represent parameters, allowing for co-reference and consistency across different visual words.

Datasets

Not discussed in the paper. The paper focuses on the formalism and methodology for representing and using a visual lexicon, rather than on training or evaluating on specific datasets.

Key Insights and Ideas

  • Hybrid Approach: Combining constructive and parametric graphics synthesis offers a powerful way to achieve both flexibility and efficiency in automated visualization.
  • Visual Lexicon as Building Blocks: Representing primitive visual objects as "visual words" in a lexicon provides a structured and reusable set of components for building complex presentations.
  • Comprehensive Representation: Capturing syntactic, semantic, and pragmatic features for each visual word is crucial for intelligent selection and instantiation, moving beyond purely structural or data-mapping approaches.
  • Constraint-Driven Generation: Using a multi-faceted constraint system (syntactic, semantic, pragmatic) allows for nuanced decision-making in selecting the most appropriate visual representation for a given context.
  • Least-Commitment Algorithm: Deferring decisions until sufficient information is available (for both selection and instantiation) avoids unnecessary backtracking and leads to a more efficient generation process.

Key Results

The paper presents a formalism and a system architecture (IMPROVISE) that demonstrates the effective representation and use of a visual lexicon. While specific quantitative results are not detailed, the paper illustrates the system's capabilities through examples, such as generating a patient information summary for a nurse (Figure 5). The key outcome is a robust framework for automated graphics generation that can handle heterogeneous information and adapt to various contexts.

Limitations and Future Work

  • Constraint Modeling: The authors acknowledge that a more systematic and comprehensive approach could be achieved by using better computational models (e.g., probability models) to describe constraints more accurately, moving from qualitative to quantitative measurements.
  • Probabilistic Reasoning: Incorporating probabilistic reasoning could further aid the generation process and constraint modeling.
  • Visual Word Capabilities: Further study is needed to fully understand and leverage the capabilities of each visual word for a more comprehensive understanding of various visual forms.
  • Future Work: The authors suggest exploring more sophisticated constraint models and further research into the diverse capabilities of visual words to enhance the system's power and flexibility.

Summary updated

BibTeX
@inproceedings{1997representation,
  title = {{The Representation and Use of a Visual Lexicon for Automated Graphics Generation}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {1997},
}

Loading related content...

Figure