Computational Linguistic Techniques in an On-Line System for Textual Analysis

Donald E. WalkerCOLING 1969

TL;DR

This paper introduces an on-line text-processing system designed for information analysts, integrating linguistic techniques to analyze and manipulate textual data. The system, named SAFARI, aims to bridge the gap between complex linguistic theory and practical user needs by offering capabilities for syntactic analysis, text editing, and sophisticated text searching.

What Problem Does This Paper Address?

The paper addresses the challenge of making advanced linguistic analysis accessible and useful to non-expert users, specifically information analysts working with personal text files. At the time of writing (late 1960s), accessing and manipulating information within data bases often required specialized programming knowledge. The authors sought to develop a system that would allow users to interact with their data using more natural language-like queries and to perform sophisticated text analysis without needing to be computer scientists or linguists. This involved bridging the gap between theoretical linguistic models, particularly Chomsky's transformational grammar, and the practical demands of information retrieval and text manipulation.

Methods

The system, SAFARI, is built around three core components:

  • Linguistically Based Procedure (SAFARI): This is the heart of the system, designed for analyzing textual information content. It takes sentences as input, analyzes them syntactically, and stores them as tree structures in a data base. Queries are also analyzed syntactically and matched against these stored structures to retrieve relevant information. The analysis process involves:
  • Lexical Categorization and Morphological Analysis: Identifying word categories and breaking down words into stems and affixes.
  • Context-Free Parsing: Analyzing the structure of sentences using phrase structure rules.
  • Transformational Rules: Applying rules to convert surface structures into a canonical "base" form, which helps in standardization and facilitates searching. This stage also includes error checking and ambiguity reduction.
  • Storage/Searching: Declarative sentences are stored as canonical trees, while interrogative sentences are used to search the data base.
  • Editing Operations: These are more traditional text editing functions, allowing users to insert, delete, and move lines, annotate text, create files and subfiles, and format data for reports. While less linguistically sophisticated, they are crucial for user interaction and data preparation.
  • Text-Searching Procedures: These procedures allow users to search for patterns within texts. Two methods are offered:
  • Direct Scan: A character-by-character scan of the text, allowing for complex pattern matching including words, strings, and disjunctions. Synonym sets can be used to broaden search criteria.
  • Index Search: This method uses an index of words in the text to narrow down the search to relevant sentences, which are then scanned directly. This is more efficient for larger, stable files.

The system is implemented in TREET, a list processing language, on an IBM 360/50 computer with IBM 2260 display consoles.

Datasets

Not discussed in the paper. The system is designed to work with user-provided text files.

Key Insights and Ideas

  • Bridging Theory and Practice: The core idea is to make powerful linguistic analysis tools accessible to end-users by abstracting away much of the complexity of linguistic theory.
  • User Control: The system emphasizes user control, allowing analysts to define how material is analyzed and searched, and to interpret results within their own problem context.
  • SAFARI as a Testbed: The system is envisioned as a platform for testing theoretical hypotheses in computational linguistics by observing how analysts use and interact with the linguistic features.
  • Linguistic Insights for Practical Tools: The authors argue that linguistic considerations, even in seemingly non-linguistic tasks like text searching, can significantly improve the effectiveness and utility of these tools. For example, understanding semantic and syntactic relations can inform the creation of synonym sets and the organization of indexes.

Key Results

The paper focuses on the design and capabilities of the system rather than presenting quantitative results from user studies or performance benchmarks. The authors state that the system is not yet in full production status, so definitive statements about user satisfaction or program parameters are not yet possible. However, they describe the intended functionality and the linguistic underpinnings of the SAFARI procedure, highlighting its ability to perform syntactic analysis and support sophisticated text retrieval.

Limitations and Future Work

  • Production Status: None of the components are in full production status at the time of writing.
  • Grammar Limitations: The authors acknowledge that no grammar written within any linguistic theory is currently adequate for processing random text passages automatically. The grammar used in SAFARI was primarily developed for testing the system and its utility for specific data bases needs further investigation.
  • Hardware Limitations: The flexibility of the editing procedures is limited by the display hardware.
  • Future Work: The authors express a desire to introduce additional options to broaden the system's capabilities and to refine existing components for increased efficiency and better interfaces. They also aim to conduct further studies on user experience to derive procedures with general value and validity, and to explore how linguistic insights can further enhance text-searching procedures and indexing.

Summary updated

BibTeX
@inproceedings{walker1969computational,
  author = {Donald E. Walker},
  title = {{Computational Linguistic Techniques in an On-Line System for Textual Analysis}},
  booktitle = {International Conference on Computational Linguistics},
  year = {1969},
}
Figure