Shaped Rewards Bias Emergent Language

Brendon Boldt, Yonatan Bisk, David R MortensenICLR 2022 · OpenReview

TL;DR

This paper argues that "shaped rewards," commonly used to speed up reinforcement learning, can inadvertently bias the emergent languages learned by AI agents. By introducing a simple navigation game, the authors demonstrate that shaped rewards can distort the meaning of learned words, alter the complexity of the language, and hide the influence of other environmental factors, undermining the goal of studying truly emergent linguistic phenomena.

What Problem Does This Paper Address?

Emergent language research aims to understand how language arises from basic principles, often by training agents using reinforcement learning (RL). A common practice in RL is to use "shaped rewards" – intermediate rewards designed to guide the agent towards a solution, especially when the primary task reward is sparse. However, the authors contend that these shaped rewards introduce an "inductive bias" that conflicts with the core idea of emergent phenomena. Instead of language emerging purely from the environment and task, it becomes influenced by the human-designed reward structure. This bias can obscure the very linguistic properties researchers are trying to study, such as the semantics and complexity of the emergent language. The paper seeks to quantify and illustrate these detrimental effects of shaped rewards in the context of emergent language.

Methods

The researchers introduce a simple 2D navigation game where a "sender" agent observes a "receiver" agent's location and sends a one-word message. The "receiver" agent, acting "blindly," uses this message to navigate towards a goal. They compare three scenarios: one with no shaped rewards (only a sparse base reward for reaching the goal), one with a standard shaped reward (rewarding proximity to the goal), and one with a "trivially biased" shaped reward (specifically favoring horizontal movement).

To analyze the learned languages, they use "sea star plots" to visualize the learned meanings of messages (i.e., the actions taken by the receiver in response to each message). They also measure the Shannon entropy of the learned languages, which quantifies their complexity and information-carrying capacity. Additionally, they propose a mathematical model based on the Chinese restaurant process to explain how the size of the RL agent's experience buffer (the memory of past experiences used for learning) can influence language entropy, and how shaped rewards might effectively alter this buffer size.

Datasets

The paper does not use pre-existing datasets. Instead, it introduces a novel, simple sender-receiver navigation game as its experimental environment. This environment is designed to be amenable to studying emergent language and the effects of shaped rewards, featuring continuous states and actions and allowing for both "centerward" (navigating to the center) and "edgeward" (navigating away from the center) tasks.

Key Insights and Ideas

  • Shaped rewards are not neutral: Contrary to the goal of studying emergent phenomena from basic principles, shaped rewards introduce human-designed biases that directly influence the learned language.
  • Semantic distortion: Even a seemingly innocuous shaped reward (like favoring horizontal movement) can cause the learned language to develop biased meanings, with certain messages consistently leading to actions in specific directions.
  • Entropy manipulation: Shaped rewards can significantly alter the entropy (complexity/information content) of the emergent language, potentially shifting it upwards even when the reward doesn't explicitly bias semantics.
  • Masking environmental effects: Shaped rewards can mask the influence of other environmental factors (like the size of the navigation area) on language complexity. Without shaped rewards, the environment's size has a more pronounced effect on language entropy.
  • Experience buffer size matters: The size of the RL agent's experience buffer plays a crucial role in language entropy. The authors propose a model linking buffer size to language complexity, suggesting that shaped rewards might indirectly affect entropy by influencing how effectively the buffer is utilized.

Key Results

  • In the "edgeward" navigation task, the "trivially biased" shaped reward (favoring horizontal movement) led to learned languages where messages predominantly mapped to horizontal actions, as visualized by "sea star plots."
  • The presence of shaped rewards (even without explicit semantic bias) shifted the distribution of language entropies upwards compared to environments with only base rewards.
  • In the "centerward" navigation task, shaped rewards masked the effect of world radius on language entropy. While entropy decreased with increasing world radius in both settings, the decrease was much more rapid in the absence of shaped rewards.
  • The authors' mathematical model demonstrated a correlation between the size of the experience buffer and language entropy, which they used to explain the observed effects of shaped rewards and world radius.

Limitations and Future Work

The authors acknowledge that their study uses a simplified navigation environment. They suggest that future work should explore a wider variety of environments with increasing complexity to better characterize the biases introduced by shaped rewards. They also propose investigating more complex shaped reward strategies and designing experiments to further validate their explanatory model based on the Chinese restaurant process. The paper emphasizes that as emergent language research tackles more challenging environments, accounting for the biases of shaped rewards will be crucial for maintaining the integrity of experimental findings.

Summary updated

Abstract

One of the primary characteristics of emergent phenomena is that they are determined by the basic properties of the system whence they emerge as opposed to explicitly designed constraints. Reinforcement learning is often used to elicit such phenomena which specifically arise from the pressure to maximize reward. We distinguish two types of rewards. The first is the base reward which is motivated directly by the task being solved. The second is shaped rewards which are designed specifically to make the task easier to learn by introducing biases in the learning process. The inductive bias which reward shaping introduces is problematic for emergent language experimentation because it biases the object of study: the emergent language. The fact that shaped rewards are intentionally designed conflicts with the basic premise of emergent phenomena arising from basic principles. In this paper, we use a simple sender-receiver navigation game to demonstrate how reward shaping can 1) explicitly bias the semantics of the learned language, 2) significantly change the entropy of the learned communication, and 3) mask the potential effects of other environmental variables of interest.
BibTeX
@inproceedings{boldt2022shaped,
  author = {Brendon Boldt and Yonatan Bisk and David R Mortensen},
  title = {{Shaped Rewards Bias Emergent Language}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=057dxuWpfx},
}
Figure