IIRSim Studio: A Dashboard for User Simulation

Saber Zerhoudi, Adam Roegiest, Michael Granitzer

TL;DR

IIRSim Studio is an open-source, web-based workbench designed to make user simulation in Interactive Information Retrieval (IIR) accessible and strictly reproducible. By combining a visual pipeline composer with Git-backed component tracking and containerized Docker execution, it transforms fragmented, code-heavy simulation setups into a unified, verifiable workflow.

What Problem Does This Paper Address?

Evaluating how users interact with search engines is a fundamental challenge in Information Retrieval (IR). Traditional human user studies capture authentic behavior but are costly, time-consuming, and notoriously difficult to replicate. Conversely, standard automated evaluations (like Cranfield-style test collections) fail to model the dynamic feedback loop of a user querying, viewing snippets, clicking documents, and deciding when to stop.

User simulation offers a powerful, low-cost alternative, but existing simulation frameworks are primarily code-centric libraries. They require steep learning curves and substantial environment configuration, which severely hinders reproducibility. The authors argue that the bottleneck isn't the simulation engines themselves, but rather the lack of shared infrastructure connecting experiment design, execution, and sharing.

Methods

The authors present IIRSim Studio, an orchestration layer built on top of the established SimIIR 3 simulation framework. The system is split into three core layers:

  1. Frontend Workbench: A low-code visual interface where users can drag, drop, and connect simulation components (e.g., query generators, snippet classifiers, and stopping decision makers) to define a user's behavioral pipeline. It features an interactive tutorial, a browser-based code editor for authoring custom Python components, and a "Playground" for running short pilot simulations.
  2. Orchestration Backend: This layer translates visual pipelines into structured, human-readable "experiment bundles." To ensure code provenance, the backend integrates with GitHub OAuth; when a user saves a custom component, the system automatically commits it to the user's GitHub repository and logs the exact commit hash within the experiment bundle.
  3. Isolated Execution Layer: Simulations are executed inside coordinated Docker containers. This guarantees that the dataset, search backend (e.g., PyTerrier, Whoosh), and simulation engine remain identical across different machines.

Additionally, the platform supports Large Language Model (LLM) integration, allowing researchers to plug in remote APIs (like OpenAI) or local inference servers (like Ollama or vLLM) to drive advanced generative user models.

Figure 1: The IIRSim Studio architecture. The Frontend Workbench provides visual pipeline composition, tutorials, and a playground for prototyping. The Orchestration Backend translates pipelines into versioned experiment bundles that are executed in isolated Docker containers, either through the workbench or at scale through the API wrapper.
Figure 2: The visual pipeline composer. Each node represents a simulation component (e.g., query generator, stopping strategy) with its own configuration panel. Users connect components to define the simulator pipeline that will be executed.

Datasets

Rather than introducing a new dataset, IIRSim Studio introduces Environment Templates. These templates package dataset configurations, search backends, and baseline simulation pipelines into reusable modules.

To demonstrate the platform's utility for collaborative research, the authors re-deployed a micro-task from the Sim4IA Shared Task benchmark. This deployment proved that organizers can distribute a standardized evaluation environment, allowing participants to focus purely on modifying simulator configurations rather than wrestling with infrastructure code.

Key Insights and Ideas

  • Infrastructure Over Algorithms: The primary barrier to widespread user simulation adoption isn't the complexity of behavioral models, but the lack of standard tooling to connect design to execution.
  • Visual Prototyping vs. Programmatic Scaling: The architecture cleanly separates prototyping from execution. A researcher can visually pilot and debug a pipeline in the web playground, then export the versioned experiment bundle to run massive batch experiments via an API wrapper on private infrastructure.
  • Strict Git-Backed Provenance: By embedding specific Git commit hashes directly into the experiment configuration, the platform eliminates the ambiguity of versioning custom user models, making the exact scope of replication explicit.

Key Results

  • Shared Task Streamlining: The successful re-deployment of the Sim4IA micro-task demonstrated that IIRSim Studio can transform a complex coding challenge into a streamlined configuration submission process.
  • Community Adoption: The platform has already been selected as the technical backbone for an upcoming CHIIR 2026 tutorial.
  • Long-Term Availability: The authors have committed to maintaining a hosted version of the workbench for at least 5 years, alongside a fully containerized local deployment option for offline or privacy-restricted research.
Figure 4: The playground results interface displaying simula- tion logs and behavioral traces. Users inspect these outputs to validate whether a configuration behaves as expected be- fore exporting for large-scale runs.

Limitations and Future Work

  • Framework Lock-in: Currently, the platform only supports the SimIIR 3 engine. While the architecture is designed to be framework-agnostic, community contributions will be required to integrate other simulation toolkits.
  • Tooling Prerequisites: To get the most value out of the platform—particularly for local deployments and custom component authoring—users still need basic familiarity with Docker and GitHub.
  • The LLM Reproducibility Paradox: For simulations relying on external LLM APIs, strict deterministic reproducibility cannot be fully guaranteed, as remote model outputs can change over time despite identical local configurations.

Summary updated

Abstract

User simulation is a valuable methodology for evaluation in Information Retrieval (IR), enabling low-cost experimentation and counterfactual analysis. However, existing simulation frameworks are primarily code-centric libraries that require substantial setup effort, which limits adoption and hinders reproducibility. The bottleneck is not the simulation engines themselves, but the lack of infrastructure connecting experiment design, execution, and sharing into a single verifiable workflow. This paper introduces IIRSim Studio, a web-based workbench that addresses this gap through four contributions: (1) a visual environment for composing simulation pipelines on top of simulation frameworks, serving both novices learning simulation concepts and experts piloting large-scale experiments; (2) a component lifecycle that supports authoring, versioning, and sharing custom simulation components through Git-backed storage and runtime injection; (3) a provenance model based on experiment bundles and environment templates that makes the scope of replication explicit; and (4) a shared-task workflow, demonstrated through the re-deployment of a Sim4IA micro-task. IIRSim Studio is available as a hosted service and as a portable containerized deployment.
BibTeX
@inproceedings{zerhoudi2026iirsim,
  author = {Saber Zerhoudi and Adam Roegiest and Michael Granitzer},
  title = {{IIRSim Studio: A Dashboard for User Simulation}},
  booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
  year = {2026},
  doi = {10.1145/3805712.3808593},
}

Loading related content...

Figure