Introduction
In his 1987 paper presented at the AAAI proceedings, Fred Lakin introduced a foundational approach to how computers can "read" diagrams and sketches. This research marked a significant shift away from the restrictive, menu-heavy interfaces of the era toward a more natural "digital blackboard" experience. By enabling computers to understand the spatial relationships between drawn elements, Lakin sought to bridge the gap between human graphical communication and machine processing.
The Challenge: Syntax-Driven vs. Agile Interfaces
In the late 1980s, most visual interfaces were strictly "syntax-driven." To create a flowchart or bar chart, users were forced to select specific object types from a menu before drawing them. Lakin argued that this approach "substitutes restriction for understanding," severely limiting a user's freedom to spontaneously arrange text and graphics.
Humans communicate effectively using "informal conversational graphics"—the kind of doodling and sketching seen on whiteboards. The challenge Lakin addressed was providing "graphical agility"—the freedom to draw anything—while maintaining the "interpretative power" to act on those drawings as if they were formal language expressions.
Methods: Spatial Parsing and Visual Grammars
The system proposed by Lakin is built on two primary pillars:
- Spatial Parsing: Unlike traditional textual parsing, which processes linear strings of characters, spatial parsing recovers structure from 2D arrangements. It treats the relative positions of text and lines as the "syntax" of a visual language.
- Visual Grammar Notation (VGN): To make the system extensible, Lakin developed a method for writing grammars that are themselves visual. A VGN rule consists of a non-terminal symbol followed by a spatial template showing where its components (terminals and other non-terminals) should be located.
The system was implemented in PAM (PAttern Manipulation), a LISP-based environment that treats graphic objects as symbolic expressions, and vmacs, a specialized graphics editor. The parser, parse-appropriately, iteratively tests different grammars against a region of the screen until it finds a match, effectively "recognizing" the user's intent.
Evaluation Across Visual Languages
The author evaluated the system using four distinct formal visual languages, demonstrating its versatility across different domains:
- Bar Charts: A simple language for data representation.
- Feature Structures: A notation used by linguists for directed acyclic graphs.
- SIBTRAN: A system of graphic devices (arrows, bullets) used to organize textual sentence fragments.
- Visual Grammar Notation: The system is self-describing, using its own logic to parse the rules that define other languages.
Key Insights and Results
Lakin's work provided several crucial insights into visual language processing:
- Finessing Recognition: By having users create objects in a digital editor rather than scanning physical paper, the system avoids low-level OCR issues and focuses on high-level arrangement and meaning.
- Heterogeneous Embedding: The general-purpose parser can handle "mixed" languages, such as a bar chart nested within a SIBTRAN organizational arrow.
- Automated Cleanup: Once the system understands a drawing's structure, it can automatically "prettify" it—aligning bars and labels perfectly without manual intervention.
- Linguistic Integration: The system successfully converted hand-drawn "Feature Structures" into LISP code compatible with natural language processing systems like PATR.
Limitations and Future Work
Despite its successes, the system faced challenges in "machine grouping"—the ability to automatically identify related objects without human assistance. Lakin also acknowledged that while formal languages were handled well, truly "informal conversational graphics" remained a much harder theoretical and practical problem. Additionally, the recursive nature of spatial parsing suggested potential performance bottlenecks as diagrams grew in complexity. Nevertheless, this work remains a cornerstone in the evolution of intelligent visual interfaces.
Summary updated