TL;DR
This paper presents a novel empirical approach to understanding how programmers actually use cognitive agent programming languages, focusing on the GOAL language. By analyzing real-world agent code, the authors identify practical challenges and propose improvements to language design, programming guidelines, and development tools to foster better agent development.
What Problem Does This Paper Address?
While many agent programming languages and frameworks exist, there's a significant gap in systematic studies of how these languages are used in practice. This lack of empirical understanding hinders the development of effective tools and methods for creating high-quality agent programs. The authors aim to bridge this gap by empirically studying the GOAL language, a high-level language for programming rational agents, to inform the design of better programming languages, guidelines, and development environments.
Methods
The authors propose a qualitative research approach to empirically study how programmers use agent programming languages. This involves:
- Defining Analysis Dimensions: They establish four key dimensions for analyzing agent programs: functional, structural, software quality (readability), and run-time behavior.
- Case Studies: Two case studies were conducted:
- Dynamic Blocks World: A simpler, single-agent domain with limited dynamics.
- UNREAL TOURNAMENT 2004 (UT2004): A complex, real-time, multi-agent environment requiring cooperation.
- Qualitative Analysis of Code: The core of the research involved analyzing the source code of agent programs written by subjects with varying levels of GOAL experience. This included identifying code patterns, structural metrics, and functional usage of language constructs.
- Subject Selection: For the Blocks World study, subjects included researchers and a programmer with different GOAL experience levels. For the UT2004 study, first-year BSc students who had recently learned GOAL were involved.
Datasets
Not explicitly discussed as "datasets" in the traditional machine learning sense. The "datasets" are the agent programs themselves, developed by human subjects for specific tasks:
- Dynamic Blocks World: Agent programs designed to solve goal configurations in a dynamic Blocks World environment.
- UNREAL TOURNAMENT 2004 (UT2004): Agent programs designed to control bots in the CTF game mode, requiring cooperation and coordination.
Key Insights and Ideas
- Experience Matters: More experienced programmers tend to utilize a wider range of language constructs, suggesting that education and training are crucial for programmers to fully leverage a language's features.
- Knowledge Base vs. Action Rules: Programmers can either leverage the expressive power of the knowledge base (like Prolog) to encode complex strategies, or focus more on action rules. This choice can be influenced by domain simplicity and programmer familiarity with the underlying knowledge representation.
- Goal Dynamics are Key: In programs that use goals, adopting and dropping goals constitute a significant portion of the agent's reasoning and executed actions, highlighting the importance of goal management in dynamic environments.
- Preference for Explicit Control: Programmers, especially novices, tend to prefer explicit control mechanisms (like linear rule evaluation and explicit module exit actions) over built-in semantics with delayed effects, possibly due to a desire for predictability and a lack of understanding of more abstract features.
- Code Duplication is a Problem: Students struggled to avoid code duplication, indicating a need for language features or tools that promote code reuse (e.g., modules, import functionalities).
- Macros for Readability: The lack of macro usage by students suggests that more emphasis on their benefits and practical examples is needed to improve code readability, especially when dealing with complex conditions.
Key Results
- Blocks World Case Study:
- Program A (least experienced programmer) heavily relied on the knowledge base.
- Programs B and C (more experienced) used more action rules, particularly for goal adoption/dropping.
- Program C demonstrated the most extensive use of GOAL's constructs.
- Readability experiments showed Program A was easiest to understand, suggesting training is needed for more complex constructs.
- Run-time analysis showed program A was more efficient in terms of total actions, but programs B and C executed a considerable portion of adopt/drop actions.
- UT2004 Case Study:
- Observed patterns in knowledge base usage, goal base usage (limited), and communication.
- Identified code duplication as a significant issue.
- Found that modules were used to code roles and tasks.
- Noted a preference for explicit control mechanisms over built-in semantics.
- Observed that teams with higher performance tended to have more action rules, suggesting more elaborate strategies.
Limitations and Future Work
- Generalizability: While the authors believe their findings generalize to other cognitive agent programming languages, they acknowledge that the specific observations might depend on the language's features (e.g., the expressive power of its knowledge representation). More research is needed to confirm this across different languages.
- Quantitative Validation: The study is primarily qualitative. Future work should involve more quantitative analysis to validate the hypotheses formed and to provide more robust evidence.
- Overlooked Observations: The authors acknowledge the possibility of having overlooked important observations and suggest explicitly addressing this in future research approaches.
- Language and Tool Improvement: The findings directly inform improvements to the GOAL language itself, programming guidelines, and the development environment. This includes adding features for specifying single-instance goals, enhancing module support, and developing tools for detecting code duplication.
- Teaching and Training: The results strongly suggest the need for better teaching materials and programming guidelines, especially for novice programmers, to ensure they can effectively utilize the full range of language features.
- Domain Exploration: Further research is recommended in other domains like social robotics and organization-aware agents to assess the language's suitability and identify context-specific extensions.
Summary updated