TL;DR
This paper argues that "shaped rewards," commonly used to speed up reinforcement learning, can inadvertently bias the emergent languages learned by AI agents. By introducing a simple navigation game, the authors demonstrate that shaped rewards can distort the meaning of learned words, alter the complexity of the language, and hide the influence of other environmental factors, undermining the goal of studying truly emergent linguistic phenomena.
What Problem Does This Paper Address?
Emergent language research aims to understand how language arises from basic principles, often by training agents using reinforcement learning (RL). A common practice in RL is to use "shaped rewards" – intermediate rewards designed to guide the agent towards a solution, especially when the primary task reward is sparse. However, the authors contend that these shaped rewards introduce an "inductive bias" that conflicts with the core idea of emergent phenomena. Instead of language emerging purely from the environment and task, it becomes influenced by the human-designed reward structure. This bias can obscure the very linguistic properties researchers are trying to study, such as the semantics and complexity of the emergent language. The paper seeks to quantify and illustrate these detrimental effects of shaped rewards in the context of emergent language.
Methods
The researchers introduce a simple 2D navigation game where a "sender" agent observes a "receiver" agent's location and sends a one-word message. The "receiver" agent, acting "blindly," uses this message to navigate towards a goal. They compare three scenarios: one with no shaped rewards (only a sparse base reward for reaching the goal), one with a standard shaped reward (rewarding proximity to the goal), and one with a "trivially biased" shaped reward (specifically favoring horizontal movement).
To analyze the learned languages, they use "sea star plots" to visualize the learned meanings of messages (i.e., the actions taken by the receiver in response to each message). They also measure the Shannon entropy of the learned languages, which quantifies their complexity and information-carrying capacity. Additionally, they propose a mathematical model based on the Chinese restaurant process to explain how the size of the RL agent's experience buffer (the memory of past experiences used for learning) can influence language entropy, and how shaped rewards might effectively alter this buffer size.
Datasets
The paper does not use pre-existing datasets. Instead, it introduces a novel, simple sender-receiver navigation game as its experimental environment. This environment is designed to be amenable to studying emergent language and the effects of shaped rewards, featuring continuous states and actions and allowing for both "centerward" (navigating to the center) and "edgeward" (navigating away from the center) tasks.
Key Insights and Ideas
- Shaped rewards are not neutral: Contrary to the goal of studying emergent phenomena from basic principles, shaped rewards introduce human-designed biases that directly influence the learned language.
- Semantic distortion: Even a seemingly innocuous shaped reward (like favoring horizontal movement) can cause the learned language to develop biased meanings, with certain messages consistently leading to actions in specific directions.
- Entropy manipulation: Shaped rewards can significantly alter the entropy (complexity/information content) of the emergent language, potentially shifting it upwards even when the reward doesn't explicitly bias semantics.
- Masking environmental effects: Shaped rewards can mask the influence of other environmental factors (like the size of the navigation area) on language complexity. Without shaped rewards, the environment's size has a more pronounced effect on language entropy.
- Experience buffer size matters: The size of the RL agent's experience buffer plays a crucial role in language entropy. The authors propose a model linking buffer size to language complexity, suggesting that shaped rewards might indirectly affect entropy by influencing how effectively the buffer is utilized.
Key Results
- In the "edgeward" navigation task, the "trivially biased" shaped reward (favoring horizontal movement) led to learned languages where messages predominantly mapped to horizontal actions, as visualized by "sea star plots."
- The presence of shaped rewards (even without explicit semantic bias) shifted the distribution of language entropies upwards compared to environments with only base rewards.
- In the "centerward" navigation task, shaped rewards masked the effect of world radius on language entropy. While entropy decreased with increasing world radius in both settings, the decrease was much more rapid in the absence of shaped rewards.
- The authors' mathematical model demonstrated a correlation between the size of the experience buffer and language entropy, which they used to explain the observed effects of shaped rewards and world radius.
Limitations and Future Work
The authors acknowledge that their study uses a simplified navigation environment. They suggest that future work should explore a wider variety of environments with increasing complexity to better characterize the biases introduced by shaped rewards. They also propose investigating more complex shaped reward strategies and designing experiments to further validate their explanatory model based on the Chinese restaurant process. The paper emphasizes that as emergent language research tackles more challenging environments, accounting for the biases of shaped rewards will be crucial for maintaining the integrity of experimental findings.
Summary updated