Introduction
While Large Language Model (LLM) agents have excelled at specific tasks like coding or playing games, they often struggle with the open-ended, messy nature of human society. Making a sound financial or political decision requires more than just following a recipe; it requires tracking real-time events, understanding historical context, and performing complex reasoning across different domains (e.g., how a new tech regulation might impact a specific sector's stock price).
Existing benchmarks are often static or expensive to update. To address this, researchers introduce SocioDojo, a new open-ended "lifelong learning" environment designed to train and test AI agents on their ability to analyze human society and make complex decisions. By introducing the "hyperportfolio" task—where agents "invest" in thousands of real-world time series based on a constant stream of news—the researchers provide a scalable way to measure an agent's "societal intelligence" beyond simple question-answering.
Methodology: Architecture and Prompting
To help agents navigate this complex, continuous environment, the authors propose two major innovations:
1. Analyst-Assistant-Actuator (AAA) Architecture
Instead of relying on a single LLM to handle everything, the task is split into three specialized roles:
- The Analyst: The "brain" that filters incoming news, forms hypotheses, and requests specific information.
- The Assistant: The "researcher" equipped with access to a massive knowledge base and tools (like Google Search and Wikipedia) to find evidence for the Analyst.
- The Actuator: The "trader" that takes the final analysis and executes specific "investment" instructions through a simulated broker.
2. Hypothesis & Proof (H&P) Prompting
Inspired by the scientific method, this zero-shot prompting technique encourages the Analyst to move beyond simple fact-finding. The agent must propose a hypothesis (e.g., "This news will cause Tech Stock X to rise"), seek evidence to prove or disprove it, and then decide whether to act based on the strength of that proof.
Datasets and The "Hyperportfolio" Task
SocioDojo is built on a massive collection of real-world data (approximately 30GB):
- Information Sources: A continuous stream of news, social media posts, and reports spanning from October 2021 to August 2023.
- Knowledge Base: A curated library of books, journals, and encyclopedias (e.g., Fincyclopedia, Ballotpedia) to provide "timeless" background knowledge.
- Time Series: Over 30,000 high-quality time series across finance (stocks, ETFs), economics (GDP, unemployment), and society (political polls, Google search trends).
These time series serve as the "assets" the agents can invest in. By treating diverse societal metrics (like a politician's approval rating or search trends) as investable assets, the researchers created a quantifiable way to measure how well an agent understands the "inner state" of society.
To prevent agents from "gaming" the system with high-frequency trading or luck, the authors implemented strict "Broker" rules, including commission fees, overnight holding fees, and a "return bound" to limit the impact of extreme outliers.
Key Results and Insights
- Superior Performance: The H&P prompting combined with the AAA architecture significantly outperformed state-of-the-art methods (like Plan-and-Solve, Self-Ask, or ReAct). The proposed method achieved improvements of 32.4% in the "Standard" setting and 30.4% in the "Tracking" setting.
- Depth of Inquiry: The H&P agent made significantly more "ask" calls to its assistant than other models, indicating it was performing much deeper research before making a decision.
- The Analyst is the Bottleneck: Ablation studies showed that upgrading the "Analyst" role to a more powerful model (like GPT-4) yielded much higher gains than upgrading the Assistant or Actuator. The ability to form deep hypotheses is the most critical skill for societal analysis.
- Information Mismatch: The paper highlights a "real-world" challenge: news is often delayed, and its impact on time series isn't instantaneous. Agents must learn to predict long-term trends rather than just reacting to the immediate past.
Limitations and Future Work
While SocioDojo represents a significant step forward, the authors note a few limitations. The environment currently uses daily close prices for financial assets; future iterations may require more granular, minute-level data to avoid "information advantage" biases. Additionally, the "overnight fees" used to balance risk-reward are currently based on empirical rates, and factors like stock dividends or splits aren't yet accounted for. Future work could incorporate more advanced economic pricing theories to make the environment even more robust.
Source: SocioDojo: Building Lifelong Analytical Agents with Real-world Text and Time Series, ICLR 2024.
Summary updated