A Note on the Level of Customer Support by State Governments: A Mystery-Shopping Approach

Oeindrila Dube, Sendhil Mullainathan, Devin G. Pope2021 · DOI

Abstract

Many government services are provided at the state level such as unemployment insurance, Medicaid, and SNAP. Given the lack of competition, a natural worry is that customer support provided by states for these services is less than adequate. While there are many different measures of how a state can support beneficiaries, we focus on just one in this short and applied report: the ability to get a live representative on the phone to help with an application question. To do this, we take a "mystery shopping" approach and make 2,000 phone calls to state government offices. We find substantial heterogeneity in the availability of live phone representatives across states and types of service (UI, Medicaid, etc.). For example, live representatives in New Jersey and Georgia were reached less than 20% of the time while representatives in New Hampshire and Wisconsin were reached more than 80% of the time. We hope that this report provides a simple example for how academics, investigative reporters, and watch groups can help states be more accountable for their customer support systems.

TL;DR

Researchers conducted a large-scale "mystery shopping" audit of state government agencies to see how difficult it is to reach a human for help. They found massive disparities in accessibility: while some states answer over 80% of calls, others fail to connect callers to a live representative more than 80% of the time, with no evidence that poor phone support is offset by better digital tools.

What Problem Does This Paper Address?

During the COVID-19 pandemic, over 68 million Americans applied for unemployment benefits. For many, this process was a bureaucratic nightmare. Unlike private businesses, which face competitive pressure to provide quality customer support, state governments often operate as monopolies with little incentive or oversight to ensure their systems are efficient.

The authors tackle a fundamental question of government "friction": how much of a hurdle does the administrative process itself create for vulnerable populations? Specifically, they focus on the "last mile" of support—the ability to reach a live human being on the phone to answer application questions for essential services like Unemployment Insurance (UI), Medicaid, and SNAP.

Methods

The researchers employed a "mystery shopping" methodology, a technique common in retail but less frequently applied to public policy at this scale.

  • The Team: Ten research assistants (RAs) were hired to act as "shoppers."
  • The Protocol: Each RA was tasked with making 200 independent calls (covering 50 states across 4 different government offices).
  • The Rules: Calls were made during local business hours. RAs were required to stay on hold for up to 45 minutes before recording a failure. If they reached a representative, they politely disengaged to avoid wasting the staff's time.
  • Data Collection: Beyond just "success or failure," RAs recorded the number of steps (phone menu buttons) required to reach a person, the hold time, and a subjective rating of the agency’s website and live chat availability.

Datasets

The study generated a primary dataset of 2,000 phone calls made between September 2020 and March 2021. The calls targeted four specific government services in every U.S. state: Unemployment Insurance (UI), Income Tax, Medicaid, and the Supplemental Nutrition Assistance Program (SNAP).

Key Insights and Ideas

  • The "State-Level" Effect: One of the most significant findings is that customer support quality is not random or siloed by agency. If a state is good at answering the phone for UI, it is likely good at answering for Medicaid. This suggests that the quality of support is driven by state-wide administrative culture or funding rather than individual agency management.
Correlation between Income Tax and Medicaid representative accessibility by state
  • The Hierarchy of Help: There is a clear "pecking order" in which services are easiest to access. Income tax assistance was the most accessible, followed by Medicaid, then SNAP. Unemployment Insurance (UI) was consistently the most difficult service to reach a human for.
  • No Digital Trade-off: A common defense for poor phone support is that resources are being diverted to "better" digital channels. The researchers found no evidence for this. States with low phone connectivity did not have better websites or more frequent live chat features. In fact, the "bad" states were often bad across all platforms.

Key Results

  • Extreme Heterogeneity: The gap between the best and worst performers is staggering. In New Hampshire and Wisconsin, RAs reached a human over 80% of the time. In Georgia and New Jersey, they were successful less than 20% of the time.
  • UI Failures: Unemployment Insurance was particularly dire. In 16 states, the success rate for reaching a UI representative was 10% or lower (meaning 9 out of 10 RAs failed to reach anyone after 45 minutes of trying).
  • Quantitative Ranking: On average across all states, Income Tax help was the easiest to reach, while UI was the hardest.
Average fraction of phone calls leading to a live representative by government office

Limitations and Future Work

The authors acknowledge several limitations. The study only measures access (reaching a person), not the quality of the help provided or whether the representative actually solved the caller's problem. Furthermore, it doesn't account for how difficult the initial application process is before a user even feels the need to call for help.

Future research could expand this framework to measure the accuracy of the information provided by representatives. The authors suggest this "mystery shopping" framework could be a powerful tool for investigative reporters and watchdogs to hold state governments accountable.

Summary updated

Abstract

Many government services are provided at the state level such as unemployment insurance, Medicaid, and SNAP.Given the lack of competition, a natural worry is that customer support provided by states for these services is less than adequate.While there are many different measures of how a state can support beneficiaries, we focus on just one in this short and applied report: the ability to get a live representative on the phone to help with an application question.To do this, we take a "mystery shopping" approach and make 2,000 phone calls to state government offices.We find substantial heterogeneity in the availability of live phone representatives across states and types of service (UI, Medicaid, etc.).For example, live representatives in New Jersey and Georgia were reached less than 20% of the time while representatives in New Hampshire and Wisconsin were reached more than 80% of the time.We hope that this report provides a simple example for how academics, investigative reporters, and watch groups can help states be more accountable for their customer support systems.
BibTeX
@misc{dube2021note,
  author = {Oeindrila Dube and Sendhil Mullainathan and Devin G. Pope},
  title = {{A Note on the Level of Customer Support by State Governments: A Mystery-Shopping Approach}},
  year = {2021},
  doi = {10.3386/w29055},
}
Figure