Abstract
Numerous within-household selection methods have been tested in general population surveys since the advent of telephone interviewing. However, very few selection studies have been conducted with a dual-frame (landline and cell phone) sample. This study tested 11 different within-household selection methods with approximately 11,000 landline respondents, combined with a parallel cell phone sample of 1,000 respondents. The methods included one probability-based method, four quasi-probability methods, and six nonprobability methods. Evaluation criteria included response rates, accuracy, demographic representation, and substantive results. The probability method had the lowest response rate, while nonprobability at-home methods had the highest. Accuracy rates were lowest for quasi-probability birthday methods. Notably, demographic and substantive differences between selection methods largely vanished when combined with the cell phone sample and properly weighted.
Introduction
In the traditional era of telephone surveying, landlines were the primary means of reaching households. Because a landline serves a household rather than an individual, researchers required rigorous methods to randomly select one adult from the home to ensure a representative sample. While mathematically sound methods like the "Kish grid" exist, they are often perceived as intrusive, leading to higher refusal rates. In today's dual-frame environment, where cell phones (treated as personal devices) provide a significant portion of data, the necessity and impact of complex landline selection methods require re-evaluation. This paper investigates whether these selection methods still materially affect survey outcomes when integrated into a dual-frame design.
Methods
The experiment tested 11 selection methods categorized into three tiers:
- Probability-based: A variant of the Rizzo-Brick-Park (R-B-P) method, which uses a Kish grid for larger households.
- Quasi-probability: Various "Birthday" methods (Most recent, Last, Next).
- Non-probability: Demographic-targeted methods (e.g., "Youngest Male/Oldest Female") either for the whole household or those currently at home.
The study utilized a landline sample of 10,999 random-digit-dial (RDD) interviews and a parallel cell phone sample of 1,000 interviews. Benchmarks were derived from the 2013 Current Population Survey (CPS).
Results
Response Rates and the Intrusiveness Penalty
The choice of selection method significantly impacted participation. The probability-based R-B-P method, which requires a more detailed household roster upfront, yielded the lowest response rate at 9.9%. In contrast, non-probability "at-home" methods, which are less burdensome for the respondent, achieved the highest response rates, peaking at 12.6% for the "Youngest male/oldest female, at home" method.
Selection Accuracy and the Birthday Paradox
Accuracy varied widely across methods. The "Multisequence Youngest Male/Youngest Female" in-household method proved the most accurate (>99%). Conversely, "Birthday" methods—often favored for their low intrusiveness—showed significant errors. The "Most Recent Birthday" method was the least accurate, with an error rate of approximately 28% in households with two or more adults. This suggests that respondents struggle with the mental calculation required for "most recent" compared to "next" or "last" birthdays.
Demographic Representation and the "Cell Phone Stabilizer"
While different selection methods produced varying demographic skews in the landline-only data, the inclusion of the cell phone frame acted as a powerful corrective. Landline samples typically skew older and more female; however, once combined with the cell phone sample and weighted, almost all significant demographic and substantive differences (such as presidential approval ratings) disappeared. One notable exception was the "Youngest Male/Oldest Female" (YMOF) method, which over-corrected for the male deficit in landlines, leading to an overrepresentation of men when combined with the already male-leaning cell phone frame.
Conclusion
As the survey industry shifts toward cell-only RDD, within-household selection for landlines may become less critical. This study demonstrates that while selection methods affect response rates and accuracy, their impact on final dual-frame results is minimal after weighting. For researchers, this suggests that simpler, less intrusive selection methods may be preferable to maximize response rates without sacrificing the integrity of the final dual-frame data. These findings remain particularly relevant for modes where household-level sampling is still standard, such as face-to-face or mail-in surveys.
Summary updated