Within-Household Selection and Dual-Frame Telephone Surveys: A Comparative Experiment of Eleven Different Selection Methods

Jennifer Marlar, Manas Chattopadhyay, Jeff Jones, Stephanie Marken, Frauke KreuterSurvey 2018 · DOI

Abstract

Numerous within-household selection methods have been tested in general population surveys since the advent of telephone interviewing. However, very few selection studies have been conducted with a dual-frame (landline and cell phone) sample. This study tested 11 different within-household selection methods with approximately 11,000 landline respondents, combined with a parallel cell phone sample of 1,000 respondents. The methods included one probability-based method, four quasi-probability methods, and six nonprobability methods. Evaluation criteria included response rates, accuracy, demographic representation, and substantive results. The probability method had the lowest response rate, while nonprobability at-home methods had the highest. Accuracy rates were lowest for quasi-probability birthday methods. Notably, demographic and substantive differences between selection methods largely vanished when combined with the cell phone sample and properly weighted.

Introduction

In the traditional era of telephone surveying, landlines were the primary means of reaching households. Because a landline serves a household rather than an individual, researchers required rigorous methods to randomly select one adult from the home to ensure a representative sample. While mathematically sound methods like the "Kish grid" exist, they are often perceived as intrusive, leading to higher refusal rates. In today's dual-frame environment, where cell phones (treated as personal devices) provide a significant portion of data, the necessity and impact of complex landline selection methods require re-evaluation. This paper investigates whether these selection methods still materially affect survey outcomes when integrated into a dual-frame design.

Methods

The experiment tested 11 selection methods categorized into three tiers:

  1. Probability-based: A variant of the Rizzo-Brick-Park (R-B-P) method, which uses a Kish grid for larger households.
  2. Quasi-probability: Various "Birthday" methods (Most recent, Last, Next).
  3. Non-probability: Demographic-targeted methods (e.g., "Youngest Male/Oldest Female") either for the whole household or those currently at home.

The study utilized a landline sample of 10,999 random-digit-dial (RDD) interviews and a parallel cell phone sample of 1,000 interviews. Benchmarks were derived from the 2013 Current Population Survey (CPS).

Results

Response Rates and the Intrusiveness Penalty

The choice of selection method significantly impacted participation. The probability-based R-B-P method, which requires a more detailed household roster upfront, yielded the lowest response rate at 9.9%. In contrast, non-probability "at-home" methods, which are less burdensome for the respondent, achieved the highest response rates, peaking at 12.6% for the "Youngest male/oldest female, at home" method.

Table 1: Response, Refusal, Cooperation, and Contact Rates by Selection Method

Selection Accuracy and the Birthday Paradox

Accuracy varied widely across methods. The "Multisequence Youngest Male/Youngest Female" in-household method proved the most accurate (>99%). Conversely, "Birthday" methods—often favored for their low intrusiveness—showed significant errors. The "Most Recent Birthday" method was the least accurate, with an error rate of approximately 28% in households with two or more adults. This suggests that respondents struggle with the mental calculation required for "most recent" compared to "next" or "last" birthdays.

Table 2: Selection Accuracy by Method for Households with Complete Rosters

Demographic Representation and the "Cell Phone Stabilizer"

While different selection methods produced varying demographic skews in the landline-only data, the inclusion of the cell phone frame acted as a powerful corrective. Landline samples typically skew older and more female; however, once combined with the cell phone sample and weighted, almost all significant demographic and substantive differences (such as presidential approval ratings) disappeared. One notable exception was the "Youngest Male/Oldest Female" (YMOF) method, which over-corrected for the male deficit in landlines, leading to an overrepresentation of men when combined with the already male-leaning cell phone frame.

Conclusion

As the survey industry shifts toward cell-only RDD, within-household selection for landlines may become less critical. This study demonstrates that while selection methods affect response rates and accuracy, their impact on final dual-frame results is minimal after weighting. For researchers, this suggests that simpler, less intrusive selection methods may be preferable to maximize response rates without sacrificing the integrity of the final dual-frame data. These findings remain particularly relevant for modes where household-level sampling is still standard, such as face-to-face or mail-in surveys.

Summary updated

Abstract

Numerous within-household selection methods have been tested in general population surveys since the advent of telephone interviewing. However, very few selection studies, if any, have been conducted with a dual frame (landline and cell phone) sample. Landline and cell phone frames are known to represent demographically different groups of respondents, and selection methods that may result in more representative demographics in a landline frame may actually skew the results when combined with the cell phone frame. This study tested 11 different within-household selection methods with approximately 11,000 landline respondents. A parallel cell phone sample was also collected with 1,000 respondents, and the frames were combined for analysis. The selection methods tested included one probability-based method, four quasi-probability methods and six nonprobability methods. The methods were evaluated on four criteria: response rates, accuracy, demographic representation and substantive results. The demographic representativeness of each method was examined for the landline frame only and for the dual (landline and cell phone) frame combination. The probability method had the lowest response rate, while the nonprobability at-home methods had the highest. Accuracy rates were lowest for the quasi-probability birthday methods. There were few demographic differences between selection methods, and no substantive differences, when combined with the cell phone sample.
BibTeX
@inproceedings{marlar2018within,
  author = {Jennifer Marlar and Manas Chattopadhyay and Jeff Jones and Stephanie Marken and Frauke Kreuter},
  title = {{Within-Household Selection and Dual-Frame Telephone Surveys: A Comparative Experiment of Eleven Different Selection Methods}},
  booktitle = {Survey},
  year = {2018},
  doi = {10.29115/sp-2018-0031},
}
Figure