SocioDojo: Building Lifelong Analytical Agents with Real-world Text and Time Series

Junyan Cheng, Peter Chin

Introduction

While Large Language Model (LLM) agents have excelled at specific tasks like coding or playing games, they often struggle with the open-ended, messy nature of human society. Making a sound financial or political decision requires more than just following a recipe; it requires tracking real-time events, understanding historical context, and performing complex reasoning across different domains (e.g., how a new tech regulation might impact a specific sector's stock price).

Existing benchmarks are often static or expensive to update. To address this, researchers introduce SocioDojo, a new open-ended "lifelong learning" environment designed to train and test AI agents on their ability to analyze human society and make complex decisions. By introducing the "hyperportfolio" task—where agents "invest" in thousands of real-world time series based on a constant stream of news—the researchers provide a scalable way to measure an agent's "societal intelligence" beyond simple question-answering.

SocioDojo Environment Overview

Methodology: Architecture and Prompting

To help agents navigate this complex, continuous environment, the authors propose two major innovations:

1. Analyst-Assistant-Actuator (AAA) Architecture

Instead of relying on a single LLM to handle everything, the task is split into three specialized roles:

  • The Analyst: The "brain" that filters incoming news, forms hypotheses, and requests specific information.
  • The Assistant: The "researcher" equipped with access to a massive knowledge base and tools (like Google Search and Wikipedia) to find evidence for the Analyst.
  • The Actuator: The "trader" that takes the final analysis and executes specific "investment" instructions through a simulated broker.

2. Hypothesis & Proof (H&P) Prompting

Inspired by the scientific method, this zero-shot prompting technique encourages the Analyst to move beyond simple fact-finding. The agent must propose a hypothesis (e.g., "This news will cause Tech Stock X to rise"), seek evidence to prove or disprove it, and then decide whether to act based on the strength of that proof.

Datasets and The "Hyperportfolio" Task

SocioDojo is built on a massive collection of real-world data (approximately 30GB):

  • Information Sources: A continuous stream of news, social media posts, and reports spanning from October 2021 to August 2023.
  • Knowledge Base: A curated library of books, journals, and encyclopedias (e.g., Fincyclopedia, Ballotpedia) to provide "timeless" background knowledge.
  • Time Series: Over 30,000 high-quality time series across finance (stocks, ETFs), economics (GDP, unemployment), and society (political polls, Google search trends).

These time series serve as the "assets" the agents can invest in. By treating diverse societal metrics (like a politician's approval rating or search trends) as investable assets, the researchers created a quantifiable way to measure how well an agent understands the "inner state" of society.

To prevent agents from "gaming" the system with high-frequency trading or luck, the authors implemented strict "Broker" rules, including commission fees, overnight holding fees, and a "return bound" to limit the impact of extreme outliers.

Key Results and Insights

Performance Comparison of H&P against Baselines
  • Superior Performance: The H&P prompting combined with the AAA architecture significantly outperformed state-of-the-art methods (like Plan-and-Solve, Self-Ask, or ReAct). The proposed method achieved improvements of 32.4% in the "Standard" setting and 30.4% in the "Tracking" setting.
  • Depth of Inquiry: The H&P agent made significantly more "ask" calls to its assistant than other models, indicating it was performing much deeper research before making a decision.
  • The Analyst is the Bottleneck: Ablation studies showed that upgrading the "Analyst" role to a more powerful model (like GPT-4) yielded much higher gains than upgrading the Assistant or Actuator. The ability to form deep hypotheses is the most critical skill for societal analysis.
  • Information Mismatch: The paper highlights a "real-world" challenge: news is often delayed, and its impact on time series isn't instantaneous. Agents must learn to predict long-term trends rather than just reacting to the immediate past.

Limitations and Future Work

While SocioDojo represents a significant step forward, the authors note a few limitations. The environment currently uses daily close prices for financial assets; future iterations may require more granular, minute-level data to avoid "information advantage" biases. Additionally, the "overnight fees" used to balance risk-reward are currently based on empirical rates, and factors like stock dividends or splits aren't yet accounted for. Future work could incorporate more advanced economic pricing theories to make the environment even more robust.

Source: SocioDojo: Building Lifelong Analytical Agents with Real-world Text and Time Series, ICLR 2024.

Summary updated

Abstract

We introduce SocioDojo, an open-ended lifelong learning environment for developing ready-to-deploy autonomous agents capable of performing human-like analysis and decision-making on societal topics such as economics, finance, politics, and culture. It consists of (1) information sources from news, social media, reports, etc., (2) a knowledge base built from books, journals, and encyclopedias, plus a toolbox of Internet and knowledge graph search interfaces, (3) 30K high-quality time series in finance, economy, society, and polls, which support a novel task called "hyperportfolio", that can reliably and scalably evaluate societal analysis and decision-making power of agents, inspired by portfolio optimization with time series as assets to "invest". We also propose a novel Analyst-Assistant-Actuator architecture for the hyperportfolio task, and a Hypothesis & Proof prompting for producing in-depth analyses on input news, articles, etc. to assist decision-making. We perform experiments and ablation studies to explore the factors that impact performance. The results show that our proposed method achieves improvements of 32.4% and 30.4% compared to the state-of-the-art method in the two experimental settings.
BibTeX
@inproceedings{cheng2024sociodojo,
  author = {Junyan Cheng and Peter Chin},
  title = {{SocioDojo: Building Lifelong Analytical Agents with Real-world Text and Time Series}},
  booktitle = {International Conference on Learning Representations},
  year = {2024},
  url = {https://openreview.net/forum?id=s9z0HzWJJp},
}
Figure