Abstract
We analyze strategic communication when advice is generated by a reinforcement-learning (RL) algorithm rather than by a fully rational sender. Building on the cheap-talk framework of Crawford and Sobel (1982), an advisor adapts its messages based on payoff feedback, while a decision-maker best-responds. We provide a theoretical analysis of the long-run communication outcomes induced by such reward-driven adaptation. With aligned preferences, we establish that learning robustly leads to informative communication even from uninformative initial policies. With misaligned preferences, no stable outcome exists; instead, learning generates cycles that sustain highly informative communication and payoffs exceeding those of any static equilibrium.
Introduction
In modern digital economies, we increasingly rely on algorithmic advice. Platforms like Airbnb suggest pricing strategies to hosts, and recommendation engines like Netflix guide consumer choices. Traditionally, economists have modeled these interactions using "cheap talk" frameworks, where an informed sender (the advisor) communicates with an uninformed receiver (the decision-maker).
The central challenge in these models is strategic misalignment. If an advisor’s incentives differ from the decision-maker’s—for instance, if a platform prefers higher prices than a host—communication often collapses. In a "babbling equilibrium," the receiver ignores the advisor because the advice is deemed untrustworthy, and the advisor, in turn, sends meaningless messages. This paper investigates whether reinforcement learning algorithms, which learn purely through payoff feedback rather than through complex strategic reasoning, can overcome these traditional barriers to communication.
Methodology
The study embeds a tabular Q-learning algorithm into the seminal Crawford and Sobel (1982) cheap-talk model. The setup involves a sender who observes a state (a value between 0 and 1) and sends a message. A rational receiver observes this message and chooses an action.
The sender utilizes softmax (Boltzmann) Q-learning, updating its strategy based on the rewards generated by the receiver's actions. To analyze the long-term behavior of this system, the researchers employ stochastic approximation theory to transform discrete RL updates into an Ordinary Differential Equation (ODE). This mathematical framework allows for the characterization of "limiting policies"—the stable patterns of communication the algorithm eventually adopts. These theoretical findings are supplemented by 1,000 simulation runs to test the impact of various exploration rates and bias levels on the resulting "language."
Key Insights and Mechanisms
Reward-Driven Adaptation vs. Rational Expectation
Unlike rational agents who might remain stuck in a babbling state because they expect to be ignored, an RL algorithm is inherently experimental. If a minor adjustment in a message leads to a slightly more favorable action from the receiver, the algorithm reinforces that behavior. This bottom-up adaptation allows informative communication to "unravel" even from initially uninformative policies.
The "Escalator" Pattern in Biased Environments
In environments with conflicting interests, the authors discovered that no stable equilibrium exists. Instead, the algorithm enters a stable cycle. The algorithm attempts to "inflate" its messages to nudge the receiver toward a preferred action; the receiver eventually adjusts their interpretation, prompting the algorithm to shift to a new message. This "escalator" of shifting language keeps communication dynamic and prevents the total breakdown of information transfer.
The Friction of Exploration
While exploration is necessary for learning, it also acts as a friction. Because the receiver knows the algorithm occasionally sends random messages (due to the softmax exploration), they adopt more cautious actions. This caution, in turn, prevents the algorithm from achieving 100% truthfulness, as the rewards for full revelation are slightly dampened by the receiver's skepticism of the algorithm's "noise."
Middle State Fully Revealing (MSFR)
A robust finding across simulations is that the algorithm consistently learns to be fully revealing at the "middle" state (the prior mean). This MSFR property serves as an anchor for the communication system, ensuring that at least some portion of the state space is communicated with high precision.
Results
- Informative by Default: In scenarios with perfectly aligned interests, the RL algorithm reaches a welfare level equal to at least 98% of the potential surplus from full revelation, even when starting from a completely uninformative policy.
- Surpassing Rationality: In biased environments, the cyclical behavior of the RL algorithm results in payoffs for both the sender and the receiver that are strictly higher than the best possible static equilibrium predicted by traditional game theory. The dynamic nature of the RL language allows for more information transmission than a static rational model would permit.
- Robustness: These outcomes persist even if the receiver is also a learner, provided the receiver adapts at a faster rate than the sender—a common dynamic in digital platform interactions.
Limitations and Future Work
This research focuses on tabular Q-learning, which is mathematically transparent but simpler than the deep reinforcement learning (DRL) architectures used in industry. While the core logic of reward-driven adaptation is expected to hold, DRL may introduce unique artifacts or complexities.
Furthermore, the model assumes a "sophisticated but myopic" receiver who responds to the current algorithm without attempting to manipulate the algorithm's learning trajectory. Future research should explore "adversarial" receivers who might try to exploit the algorithm's learning curve. Finally, behavioral experiments are necessary to determine if human decision-makers can effectively interpret and navigate the cyclical, shifting languages generated by these algorithms.
Summary updated