New Sources for Comparative Social Science: Historical Population Panel Data From East Asia

Hao Dong, Cameron Campbell, Satomi Kurosu, Wen-Shan Yang, James LeeDemography 2015 · DOI

TL;DR

This paper introduces five new historical population panel datasets from East Asia, offering unprecedented opportunities for comparative social science research. By reconstructing millions of individual life histories from historical registers, these datasets enable detailed analysis of demographic and social behaviors across diverse East Asian contexts, bridging a significant gap in global historical data.

What Problem Does This Paper Address?

Comparative social science, particularly historical demography, thrives on comparable data. While extensive micro-level historical datasets have been compiled for North America and Europe, similar resources for East Asia have been largely absent. This lack of data has hindered in-depth, cross-regional comparisons, limiting our understanding of universal versus context-specific social and demographic patterns. The authors aim to rectify this by introducing and detailing five new East Asian historical panel datasets that are designed for comparability.

Methods

The core methodological contribution lies in the reconstruction of historical population registers into longitudinal individual-level panel datasets. This involved:

  • Transcription and Digitization: Converting historical administrative records (population registers) into machine-readable formats.
  • Longitudinal Linkage: Developing sophisticated methods to link records of the same individual across different registers over time. This was particularly challenging due to variations in register organization, naming conventions, and the survival of records.
  • For some datasets (CMGPD-LN, CMGPD-SC), linkage was more straightforward due to consistent ordering.
  • For others (KMGPD-TS), a multi-pass approach involving software-assisted candidate linking followed by manual adjudication was necessary.
  • Specialized data entry programs and unique identifiers were used for datasets like CTHRD and NAC-SN.
  • Family Linkage: Utilizing recorded relationships to household heads to reconstruct family structures and identify intergenerational links, enabling pedigree reconstruction.
  • Data Harmonization (Implied): While not explicitly detailed as a separate step, the goal of creating comparable datasets implies efforts to standardize variables and formats where possible.

Datasets

The paper introduces five new East Asian historical panel datasets:

  • China Multi-Generational Panel Dataset-Liaoning (CMGPD-LN): 1749-1909
  • China Multi-Generational Panel Dataset-Shuangcheng (CMGPD-SC): 1866-1913
  • Japanese Ninbetsu-Aratame-Cho Population Register Database-Shimomoriya and Niita (NAC-SN): 1716-1870
  • Korea Multi-Generational Panel Dataset-Tansung (KMGPD-TS): 1678-1888
  • Colonial Taiwan Household Registration Database (CTHRD): 1906-1945

These datasets collectively contain over 3.7 million linked observations of 610,000 individuals. They are made available online or by application.

Key Insights and Ideas

  • The Power of Registers: Historical population registers, originally created for administrative purposes (taxation, population control), are incredibly rich sources for understanding past societies at the individual level.
  • Bridging the Data Divide: The creation of these East Asian datasets is a crucial step in democratizing historical demographic research, moving beyond a Eurocentric bias.
  • Longitudinality is Key: Individual life histories, not just snapshots, are essential for understanding the causes and consequences of demographic and social behaviors.
  • Context Matters: The datasets allow researchers to examine how individual behavior is shaped by community, household, and kinship contexts, offering a nuanced view of social organization.
  • Cultural Similarities: East Asian societies, despite regional differences, share common cultural underpinnings (e.g., Confucian ideology, emphasis on family/kinship) that make intra-regional comparisons particularly insightful.
  • Data Limitations Inform Research: Understanding the specific limitations of each dataset (e.g., missing registers, differential recording of sexes/ages) is vital for appropriate methodological application and interpretation.

Key Results

  • The five datasets provide detailed information on vital events (birth, death, marriage, migration), individual characteristics (sex, age, SES proxies), household context, and kinship relationships.
  • Longitudinal linkage rates are generally high (up to 90-95% for some datasets), enabling the reconstruction of life histories.
  • Multigenerational linkage is possible, with CMGPD-LN showing the highest proportion of individuals linked across multiple generations.
  • The datasets reveal patterns in mortality and marriage that are broadly consistent with each other and with Western data, though with some notable regional variations.
  • Significant data limitations exist, including:
  • Missing registers: Leading to unexplained disappearances of individuals (up to 52.54% in KMGPD-TS).
  • Differential recording: Children, especially daughters, are often underrepresented in some datasets.
  • Inconsistent age calculation: Traditional East Asian age-counting methods require careful conversion.
  • Variations in SES measures: Direct comparison of SES indicators across datasets is challenging.

Limitations and Future Work

  • Representativeness: Most datasets are not nationally representative, covering specific communities or regions.
  • Data Quality: Issues like missing registers, differential recording, and inconsistent age calculation require careful handling.
  • Comparability of SES: While the datasets provide information on social and economic status, direct comparability across populations is limited.
  • Future Work: The authors envision combining these datasets into a single large file for more robust statistical modeling and comparisons. They also highlight the potential for these datasets to contribute to a better understanding of human agency and behavior in general, and to inform comparative social science on a global scale. The ongoing development of the East Asian Population and Family History Project (EAP II) aims to foster further collaboration and detailed comparisons within East Asia and with other regions.

Summary updated

Abstract

Comparison and comparability lie at the heart of any comparative social science. Still, precise comparison is virtually impossible without using similar methods and similar data. In recent decades, social demographers, historians, and economic historians have compiled and made available a large number of micro-level data sets of historical populations for North America and Europe. Studies using these data have already made important contributions to many academic disciplines. In a similar spirit, we introduce five new micro-level historical panel data sets from East Asia, including the China Multi-Generational Panel Dataset-Liaoning (CMGPD-LN) 1749-1909, the China Multi-Generational Panel Dataset-Shuangcheng (CMGPD-SC) 1866-1913, the Japanese Ninbetsu-Aratame-Cho Population Register Database-Shimomoriya and Niita (NAC-SN) 1716-1870, the Korea Multi-Generational Panel Dataset-Tansung (KMGPD-TS) 1678-1888, and the Colonial Taiwan Household Registration Database (CTHRD) 1906-1945. These data sets in total contain more than 3.7 million linked observations of 610,000 individuals and are the first such Asian data to be made available online or by application. We discuss the key features and historical institutions that originally collected these data; the subsequent processes by which the data were reconstructed into individual-level panels; their particular data limitations and strengths; and their potential for comparative social scientific research.
BibTeX
@inproceedings{dong2015new,
  author = {Hao Dong and Cameron Campbell and Satomi Kurosu and Wen-Shan Yang and James Lee},
  title = {{New Sources for Comparative Social Science: Historical Population Panel Data From East Asia}},
  booktitle = {Demography},
  year = {2015},
  doi = {10.1007/s13524-015-0397-y},
}
Figure