TL;DR
This paper introduces "Open e-commerce 1.0," a novel dataset of over 1.8 million U.S. Amazon purchase histories from 5,027 users, linked to their demographic and lifestyle information. This resource democratizes access to rich consumer spending data, traditionally held by private companies, enabling broader research into consumer behavior and its societal impacts.
What Problem Does This Paper Address?
Traditionally, detailed consumer spending data has been collected through costly government surveys, which are experiencing declining response rates. Meanwhile, companies amass vast amounts of similar data through digital platforms, but this information remains largely inaccessible to researchers. This creates a gap in our ability to understand consumer behavior, economic trends, and societal dynamics. The authors aim to bridge this gap by democratizing access to such data, making it available for public research while respecting user privacy and consent.
Methods
The researchers employed a crowdsourcing approach to collect the data. Participants were recruited through online platforms and invited to export their Amazon order histories. A custom survey tool was developed to facilitate this process, ensuring that data only left the participant's machine with their explicit consent. The survey also collected demographic, lifestyle, and health information from participants. Data preprocessing involved cleaning, de-duplication, and anonymization to remove personally identifiable information. The dataset was then validated against public Amazon sales data and analyzed for seasonal trends in specific product categories.
Datasets
The primary dataset introduced in this paper is the "Open e-commerce 1.0" dataset, comprising:
- Over 1.8 million U.S. Amazon purchase histories from 5,027 consumers, spanning 2018-2022.
- Associated survey data including participant demographics (gender, age, income, education, race, ethnicity), lifestyle, and health information.
For validation, the authors used:
- Publicly available Amazon net sales data (North America segment).
- U.S. Census Bureau data for demographic comparisons.
- U.S. Bureau of Labor Statistics (BLS) data for footwear CPI and retail sales.
- World Health Organization (WHO) COVID-19 data.
Key Insights and Ideas
- Democratizing Data Access: The core idea is to make rich, granular consumer data accessible to the research community, moving beyond traditional, often less detailed, government surveys.
- Privacy-Preserving Crowdsourcing: The methodology highlights how to collect sensitive purchase data ethically, emphasizing informed consent and on-device processing to protect user privacy.
- Validation is Crucial: The paper strongly emphasizes the importance of validating crowdsourced data against external sources (like public sales data and government statistics) to ensure its reliability and representativeness.
- Seasonal and Event-Driven Trends: The data clearly reveal predictable seasonal purchasing patterns (e.g., footwear) and significant shifts in consumer behavior in response to major events like the COVID-19 pandemic.
- Demographic Correlations: The dataset allows for the exploration of how demographics and lifestyle factors correlate with purchasing behaviors, offering insights into consumer segmentation and targeted analysis.
Key Results
- The "Open e-commerce 1.0" dataset contains over 1.8 million purchase records from 5,027 U.S. Amazon users, spanning 2018-2022.
- The dataset's purchase expenditure data showed a strong correlation with public Amazon sales data (Pearson r = 0.978, p < 0.001).
- Analyses of specific product categories demonstrated expected seasonal trends, such as summer peaks for sandals and winter peaks for boots.
- Purchasing patterns, like face mask purchases, showed clear correlations with external events such as the COVID-19 pandemic.
- The sample demographics, while showing some biases (e.g., under-representation of older adults, over-representation of females), were found to be geographically correlated with U.S. population data (Pearson r = 0.977, p < 0.001).
Limitations and Future Work
- Dataset Size: The authors acknowledge that the dataset, while substantial, is relatively small compared to the total number of Amazon Prime users, which may limit the scope of certain analyses, particularly those requiring fine-grained geographic or migration data.
- Demographic Biases: The sample exhibits biases in age, gender, income, and education compared to U.S. census data. While stratification techniques can mitigate some of these, they remain a consideration.
- Gift Card Purchases: A significant portion of purchases are gift cards, which can skew certain analyses. The paper provides guidance on how to handle these.
- Future Work: The authors encourage the collection of complementary datasets to enhance the utility of this resource. They suggest exploring more sophisticated reweighting methods to improve representativeness, investigating the inference of sensitive attributes from purchase data, and using this data to inform official statistics like the Consumer Price Index (CPI). They also highlight the potential for product-level analysis by linking product codes to external product information.
Summary updated