SDialog: A Python Toolkit for End-to-End Agent Building, User Simulation, Dialog Generation, and Evaluation

Sergio Burdisso, Séverin Baroudi, Yanis Labrak, David Grunert, Pawel Cyrta, Yiyang Chen, Srikanth Madikeri, Thomas Schaaf, Esaú Villatoro-Tello, Ahmed Hassoon, Ricard Marxer, Petr Motlicek

TL;DR

SDialog is an open-source Python toolkit that unifies conversational AI generation, evaluation, and mechanistic interpretability into a single, cohesive framework. By centering its architecture around a standardized Dialog object, it allows researchers to seamlessly simulate multi-agent interactions, apply fine-grained behavioral controls, and inspect or steer internal model activations using intuitive code.

What Problem Does This Paper Address?

Building and analyzing Large Language Model (LLM)-based conversational agents is currently a highly fragmented process. Researchers typically have to stitch together disparate tools: multi-agent frameworks for generating synthetic data, separate evaluation libraries (like LLM-as-a-judge) that only look at final text outputs, and isolated mechanistic interpretability (MI) libraries that aren't designed to handle the turn-by-turn structure of a dialogue.

This fragmentation introduces non-determinism, makes controlled experiments difficult to replicate, and completely decouples evaluation from model introspection—leaving researchers in the dark about why an agent behaves the way it does. SDialog bridges these gaps by providing an integrated, reproducible ecosystem built specifically for dialogue workflows.

Methods

The core innovation of SDialog is its dialogue-centric architecture, which organizes eight distinct modules into auxiliary, core, and output components around a single, metadata-rich Dialog abstraction.

Figure 1: SDialog architecture overview showing eight modules organized into auxiliary, core, and output com- ponents.

Key methodological features include:

  • Composable Orchestration: SDialog allows developers to dynamically inject instructions or enforce constraints (like conversation length) onto agents using a clean Python pipe operator (|) syntax.
  • Integrated Mechanistic Interpretability: The toolkit attaches PyTorch forward hooks directly to model layers to capture per-token activations. Remarkably, it maps activation steering operations directly to intuitive arithmetic operators: subtracting a steering vector (-) ablates a behavior, while adding it (+) induces it.
  • 3D Acoustic Simulation: For speech-based applications, SDialog features an audio module that pairs Text-to-Speech (TTS) with full acoustic environment modeling (via pyroomacoustics and dScaper), accounting for room geometry, wall materials, air absorption, and microphone impulse responses.
  • Multi-Backend Support: A unified configuration layer abstracts LLM interactions, allowing users to seamlessly mix and match backends like OpenAI, vLLM, Hugging Face, Ollama, Gemini, and AWS Bedrock within the same experiment.

Datasets

While SDialog is primarily a toolkit for generating and transforming synthetic dialogue data rather than a static dataset, the authors utilize several established safety and instruction-following datasets to demonstrate its mechanistic interpretability capabilities:

  • ADVBench
  • MaliciousInstruct
  • HarmBench
  • JailbreakBench
  • Alpaca

These datasets are split into training and testing sets to isolate and validate "refusal directions" within the models.

Key Insights and Ideas

  • The Model Size vs. Tool Sequencing Paradox: Bigger isn't always better for complex agent workflows. While the largest models excel at straightforward tasks, they can struggle with strict conditional logic (such as verifying an account before executing an address change) compared to well-balanced, mid-sized models.
  • Inherent Linguistic Shifts: Model size inherently dictates communication style. Even when given identical system prompts, larger models naturally generate text with significantly higher linguistic complexity, which has direct implications for user accessibility.
  • Dialog-Centric Interpretability: Mechanistic interpretability shouldn't be an isolated, post-hoc analysis. Integrating activation steering directly into the agent's runtime allows for powerful combinations, such as conditional steering mixed with behavioral orchestrators.

Key Results

The authors evaluated Qwen3 model sizes (0.6B, 1.7B, 8B, and 14B) on a simulated call-center scenario to balance functional correctness against linguistic accessibility.

Functional correctness across Qwen3 sizes.
  • Operational Trade-offs: In complex scenarios requiring identity verification (Case A), the Qwen3-8B model provided the best operational balance, achieving a high verification rate (0.97) and superior tool sequencing correctness (0.83), actually outperforming the larger 14B model on sequencing.
  • Linguistic Complexity: Readability analysis showed a systematic upward trend in text complexity as model size increased. Gunning Fog scores climbed from 10.15 (0.6B) to 11.81 (14B), spanning nearly two full grade levels despite identical prompting.
Figure 2: Average Gunning Fog scores increase with model size, indicating more complex language in larger models.
  • Activation Steering Fidelity: In an interpretability case study on Llama-3-8B-Instruct, SDialog successfully isolated the model's internal refusal direction. Ablating this direction allowed the model to bypass safety guardrails for 99% of harmful prompts, while inducing it caused the model to refuse 100% of harmless prompts.

Limitations and Future Work

  • LLM Dependency: Advanced features like function calling and deterministic seeding are heavily reliant on the underlying LLM backend capabilities.
  • Interpretability Scope: Activation inspection and steering are currently restricted to local PyTorch models via Hugging Face Transformers; closed-source, API-based models (OpenAI, Anthropic) do not expose the necessary internal activation layers.
  • Audio Realism Evaluation: The 3D audio generation pipeline currently lacks subjective human listening validation or downstream testing against real-world acoustic recordings to verify its impact on tasks like Automatic Speech Recognition (ASR).

Summary updated

Abstract

We present SDialog, an MIT-licensed open-source Python toolkit that unifies dialog generation, evaluation and mechanistic interpretability into a single end-to-end framework for building and analyzing LLM-based conversational agents. Built around a standardized Dialog representation, SDialog provides: (1) persona-driven multi-agent simulation with composable orchestration for controlled, synthetic dialog generation, (2) comprehensive evaluation combining linguistic metrics, LLM-as-a-judge and functional correctness validators, (3) mechanistic interpretability tools for activation inspection and steering via feature ablation and induction, and (4) audio generation with full acoustic simulation including 3D room modeling and microphone effects. The toolkit integrates with all major LLM backends, enabling mixed-backend experiments under a unified API. By coupling generation, evaluation, and interpretability in a dialog-centric architecture, SDialog enables researchers to build, benchmark and understand conversational systems more systematically.
BibTeX
@inproceedings{burdisso2026sdialog,
  author = {Sergio Burdisso and Séverin Baroudi and Yanis Labrak and David Grunert and Pawel Cyrta and Yiyang Chen and Srikanth Madikeri and Thomas Schaaf and Esaú Villatoro-Tello and Ahmed Hassoon and Ricard Marxer and Petr Motlicek},
  title = {{SDialog: A Python Toolkit for End-to-End Agent Building, User Simulation, Dialog Generation, and Evaluation}},
  booktitle = {EACL (System Demonstrations)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-demo.23},
}
Figure