SDialog: A Python Toolkit for End-to-End Agent Building, User Simulation, Dialog Generation, and Evaluation

Sergio Burdisso, Séverin Baroudi, Yanis Labrak, David Grunert, Pawel Cyrta, Yiyang Chen, Srikanth R. Madikeri, Esaú Villatoro-Tello, Thomas Schaaf, Ricard Marxer, Petr Motlícek

TL;DR

This paper introduces SDialog, a comprehensive Python toolkit designed to streamline the entire lifecycle of building and analyzing conversational AI agents. It unifies dialog generation, user simulation, evaluation, and mechanistic interpretability into a single framework, addressing the fragmentation in current research workflows and enabling more systematic and reproducible experimentation.

What Problem Does This Paper Address?

The rapid advancement of Large Language Models (LLMs) has led to increasingly sophisticated conversational AI agents. However, the research and development process remains fragmented. Researchers often grapple with inconsistent data formats, limited control over synthetic data generation, varying evaluation practices, and a lack of tools to understand the internal mechanisms driving agent behavior. This fragmentation hinders the development of robust, transparent, and reproducible conversational systems. Existing tools tend to focus on specific aspects like data creation or model training, leaving a gap in integrated solutions for end-to-end agent development and analysis.

Methods

SDialog is built around a central Dialog object, which acts as a unified representation for various modules. Key components include:

  • Persona-driven Multi-Agent Simulation: SDialog allows for composable orchestration of agents and user simulations, enabling controlled, synthetic dialog generation.
  • Comprehensive Evaluation: It integrates linguistic metrics, LLM-as-a-judge evaluations, and functional correctness validators to assess dialog quality.
  • Mechanistic Interpretability Tools: The toolkit offers features for activation inspection and steering, allowing researchers to probe and influence LLM behavior.
  • Audio Generation: SDialog can generate synthetic audio for dialogs, including acoustic simulation with 3D room modeling and microphone effects.
  • Multi-Backend Support: It abstracts LLM interactions through a unified API, supporting major backends like OpenAI, vLLM, HuggingFace Transformers, Ollama, Google Gemini, and AWS Bedrock, facilitating mixed-backend experiments.

Datasets

The paper does not explicitly mention specific datasets used for training the core SDialog toolkit. However, it discusses the generation of synthetic dialog datasets for model training and evaluation. The interpretability module utilizes datasets like ADVBENCH, MALICIOUSINSTRUCT, HARMBENCH, JAILBREAKBENCH, and ALPACA for evaluating refusal behavior. The audio module can leverage Hugging Face datasets for voice databases and impulse responses.

Key Insights and Ideas

  • Unified Framework: The core innovation is the integration of dialog generation, simulation, evaluation, and interpretability into a single, coherent toolkit. This significantly reduces the friction in research workflows.
  • Dialog-Centric Architecture: Grounding all components around a Dialog object simplifies data handling, provenance tracking, and overall system coherence.
  • Mechanistic Interpretability for Dialogs: Applying interpretability techniques directly to dialog generation and steering opens new avenues for understanding and controlling conversational agents.
  • Composable Evaluation: The ability to combine diverse evaluation metrics (linguistic, LLM-as-a-judge, functional) provides a more holistic assessment of agent performance.
  • Multi-Backend Flexibility: Supporting a wide range of LLM backends allows researchers to easily compare models and leverage different LLMs for various tasks within the same framework.

Key Results

The paper demonstrates SDialog's utility through a call-center scenario comparing different sizes of the Qwen3 model.

  • Functional Correctness: In a scenario requiring verification, the 8B model showed a good balance of verification sensitivity (0.97) and tool sequencing (0.83), outperforming the 14B model (0.56 tool sequencing). In a scenario without verification, the 14B model performed best.
  • Linguistic Complexity: The Gunning Fog index increased systematically with model size, indicating more complex language in larger models, from 10.15 for the 0.6B model to 11.81 for the 14B model.
  • Interpretability: The interpretability module successfully replicated findings on refusal behavior, demonstrating the ability to steer LLMs to suppress or induce refusal tendencies with high accuracy (e.g., 99% bypass of refusal for harmful instructions).
  • Efficiency: The end-to-end workflow implementation for the use case required under 100 lines of code, highlighting SDialog's efficiency for rapid prototyping.

Limitations and Future Work

  • Computational Requirements: Large-scale dialog generation, evaluation, and interpretability analysis can be computationally expensive, especially with large models.
  • Audio Realism: The realism of synthetic voices is limited by the chosen TTS engine. Subjective evaluation and validation against real-world recordings are needed.
  • Evaluation Validity: LLM-as-a-judge evaluators can inherit biases from their underlying models and may not always align with human judgments. Combining multiple evaluation approaches is recommended.
  • Interpretability Scope: Activation analysis is currently limited to PyTorch models. API-based models do not provide direct activation access.
  • Room Model Limitations: The current room model is limited to rectangular geometries, and furniture acoustic properties are not yet modeled.
  • Future Work: The authors suggest extending the room model to more complex shapes, modeling furniture acoustics, and further developing the audio realism and evaluation validity. They also highlight the ethical considerations of automated content generation and impersonation.

Summary updated

Abstract

We present SDialog, an MIT-licensed open-source Python toolkit that unifies dialog generation, evaluation and mechanistic interpretability into a single end-to-end framework for building and analyzing LLM-based conversational agents. Built around a standardized Dialog representation, SDialog provides: (1) persona-driven multi-agent simulation with composable orchestration for controlled, synthetic dialog generation, (2) comprehensive evaluation combining linguistic metrics, LLM-as-a-judge and functional correctness validators, (3) mechanistic interpretability tools for activation inspection and steering via feature ablation and induction, and (4) audio generation with full acoustic simulation including 3D room modeling and microphone effects. The toolkit integrates with all major LLM backends, enabling mixed-backend experiments under a unified API. By coupling generation, evaluation, and interpretability in a dialog-centric architecture, SDialog enables researchers to build, benchmark and understand conversational systems more systematically.
BibTeX
@inproceedings{burdisso2026sdialog,
  author = {Sergio Burdisso and Séverin Baroudi and Yanis Labrak and David Grunert and Pawel Cyrta and Yiyang Chen and Srikanth R. Madikeri and Esaú Villatoro-Tello and Thomas Schaaf and Ricard Marxer and Petr Motlícek},
  title = {{SDialog: A Python Toolkit for End-to-End Agent Building, User Simulation, Dialog Generation, and Evaluation}},
  booktitle = {EACL (System Demonstrations)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-demo.23},
}
Figure