Inclusive Leadership in the Age of AI: A Dataset and Comparative Study of LLMs vs. Real-Life Leaders in Workplace Action Planning

Vindhya Singh, Sabine Schulte im Walde, Ksenia KeplingerEMNLP 2025

TL;DR

This paper introduces a novel dataset of workplace action plans generated by over 250 real-life leaders and compares them to plans generated by seven state-of-the-art Large Language Models (LLMs). The study reveals that while LLMs can produce structured plans, real-life leaders excel in readability, conciseness, and a more human-centric, action-oriented communication style, highlighting areas for LLM improvement in leadership contexts.

What Problem Does This Paper Address?

The increasing integration of LLMs into professional workflows raises questions about their efficacy in complex, human-centric tasks like leadership and strategic planning. Specifically, translating abstract concepts like "inclusion" into actionable, effective workplace plans is a critical leadership function. While LLMs are being explored for HR and communication support, their ability to generate practical, inclusive action plans comparable to those of human leaders remains underexplored. This paper aims to bridge this gap by providing a direct, empirical comparison between LLM-generated and human-generated action plans, focusing on the crucial aspect of inclusive leadership.

Methods

The researchers developed a conversational chatbot, the "Leader Success Bot," to guide over 250 real-life leaders in generating daily inclusion action plans over a two-week period. This bot was designed with domain experts to ensure it facilitated the creation of SMART (Specific, Measurable, Actionable, Relevant, Time-bound) goals. Simultaneously, seven state-of-the-art LLMs were prompted using the same socio-demographic data of the leaders and instructions co-developed with experts. The study then employed a multi-faceted analysis, including:

  • Structural Variations: Analyzing response length, words per sentence, and Part-of-Speech distribution.
  • Readability and Lexical Diversity: Using the Flesch Reading Ease Score, Type-Token Ratio (TTR), and Moving Average TTR (MATTR).
  • Similarity Metrics: Employing BLEU, ROUGE-L, and sentence cosine similarity to compare LLM outputs to human benchmarks.
  • Sentiment and Emotion Analysis: Using valence-arousal-dominance models and the NRC Lexicon to assess emotional tone.
  • Human Evaluation: A survey of 300 participants rated action plans on dimensions like accuracy, relevance, clarity, empathy, bias, and trust.

Datasets

The primary dataset introduced in this paper consists of workplace action plans generated by over 250 real-life leaders. This dataset is publicly released to enable direct comparison. The LLMs were evaluated using prompts that incorporated socio-demographic data of these leaders.

Key Insights and Ideas

  • Readability is King: Real-life leaders produce significantly more readable and concise action plans than LLMs, suggesting LLMs need to adapt to more direct, human-friendly communication styles for leadership tasks.
  • LLMs Tend to Be Verbose and Abstract: LLMs often generate longer, more descriptive plans with a higher proportion of nouns and determiners, while leaders favor verbs and pronouns, indicating a focus on action and interpersonal engagement.
  • Emotional Nuance is a Differentiator: While LLMs can project idealized, positive leadership personas, real-life leaders, particularly female leaders, exhibit a broader emotional range, suggesting LLMs could be fine-tuned to better capture these nuances for more authentic communication.
  • LLMs Show "Helpful" Bias: LLMs tend to overemphasize confidence and positivity, potentially a design bias to appear helpful, which might not always reflect the nuanced reality of leadership.
  • Three-Shot Prompting Improves LLM Performance: Using examples in prompts significantly enhances LLM performance across various metrics, highlighting the importance of context and guidance.

Key Results

  • Response Length: Real-life leaders' plans averaged 19 words, while LLMs ranged from 70 to over 400 words.
  • Readability: Real-life leaders achieved a Flesch Reading Ease score of 61.7 (standard readable range), significantly higher than all LLMs, with most LLM scores below 35.
  • Semantic Similarity: Llama-3.3-70b and Qwen-Plus showed the highest semantic similarity to real-life leaders' plans.
  • Human Evaluation: Gemini-2.0-Flash was rated highly for relevance, accuracy, and coherence, while Llama-3.3-70b was best for empathy. Command-a-03-2025 and Llama-3.3-70b received high ratings for bias.
  • Sentiment: LLMs like Llama-3.3-70b, GPT-40 mini, and Command-a-03-2025 expressed higher dominance than real-life leaders.

Limitations and Future Work

The authors acknowledge several limitations:

  • Language Barrier: The study was conducted solely in English, and future work should aim to adapt the chatbot and analysis for other languages.
  • Accessibility: The current chatbot is not designed for blind users, and future iterations should consider greater accessibility.
  • Rapid LLM Development: Benchmarking LLMs is an ongoing, iterative process due to their rapid evolution.
  • Demographic Focus: While gender was analyzed, future work could explore other intersecting identities like age and leadership experience more deeply.

Future work should focus on fine-tuning LLMs based on real-life communication patterns, emphasizing readability and domain-specific language. Further research should also explore how LLMs can outperform human leaders in specific areas, like emotional consistency, and assess action plans against each SMART criterion in greater detail.

Summary updated

BibTeX
@inproceedings{singh2025inclusive,
  author = {Vindhya Singh and Sabine Schulte im Walde and Ksenia Keplinger},
  title = {{Inclusive Leadership in the Age of AI: A Dataset and Comparative Study of LLMs vs. Real-Life Leaders in Workplace Action Planning}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure