How to choose an eCommerce data provider for investment research
Judge an eCommerce pricing and assortment dataset on ten things: how far back the history goes, whether the same retailers are collected on a fixed schedule, how gaps and restatements are handled, whether prices are measured like-for-like on the same products, how products are classified, whether brands are mapped to their owners and tickers with dates, what is observed at SKU level, how the data is delivered, what validation evidence exists, and how the data is collected. Ask every vendor for evidence on each, including a backtest against reported results.
Ten things to check
1. History
Ask: How far back does continuous history go for the retailers and brands I care about?
Red flag: History that starts when the vendor started selling it, or that was back-filled from a different source.
Data Purl: Weekly history back to 2013, collected the same way throughout.
2. A fixed panel
Ask: Are the same retailers collected on a fixed schedule, and is a retailer joining or leaving disclosed?
Red flag: Coverage that grows every quarter with no way to separate new retailers from real change.
Data Purl: 400+ retailers on a fixed schedule; panel changes are disclosed, for example on the brand headlines.
3. Gaps and restatements
Ask: What happens when a retailer is not collected one week, and when is history restated?
Red flag: Missing weeks silently dropped or interpolated, or history rewritten without notice.
Data Purl: 95.9% of scheduled retailer-weeks captured without a gap; gaps filled and flagged. Restatement rules are on the methodology page.
4. Like-for-like prices
Ask: Can I see price change on the same products a year apart, separately from changes in the range?
Red flag: Only average prices, which move when the assortment changes as well as when prices do.
Data Purl: Same-SKU growth alongside totals.
5. Classification
Ask: Is every product placed in one standardized category tree, and how is classification checked?
Red flag: Each retailer's own site menu used as the category, so the same product sits in different places.
Data Purl: One tree of about 4,000 categories; in Hardlines & Consumables, 91% of products are verified or set by audited rules.
6. Brand, owner and ticker mapping
Ask: Are brands mapped to their owners and listed tickers with start and end dates?
Red flag: A static brand-to-ticker table, which leaks future information into backtests after acquisitions.
Data Purl: Point-in-time brand-to-ticker mapping, including licences, across 1,900+ public companies.
7. SKU-level observation
Ask: Are availability and markdowns observed per size and variant, or per product?
Red flag: Product-level data only, which hides sizes selling out and partial markdowns.
Data Purl: More than 100 million SKUs tracked weekly, with in-stock and markdown status per SKU.
8. Delivery
Ask: Can the data arrive where my team works, and is there a self-serve way to explore it first?
Red flag: Spreadsheets on request, or an API with no history download.
Data Purl: Snowflake or S3 feeds plus a self-serve analytics portal.
9. Validation evidence
Ask: What evidence shows the data is complete and tracks reported results?
Red flag: Claims of accuracy with no published collection rates, quality screens or backtests.
Data Purl: Collection rates, quality screens and backtests on validation, and a retailer page set beside its record on what we capture.
10. Collection practices
Ask: Is only public product information collected, with no personal data, and has it been through compliance review?
Red flag: No written collection policy or due diligence questionnaire.
Data Purl: Public product information only, no personal data; see compliance.
Questions
What should I ask an eCommerce data vendor before buying?
Ask how far back continuous history goes, whether the retailer panel is fixed and disclosed, how gaps and restatements are handled, whether prices are measured on the same products a year apart, how products are classified, whether brands are mapped to owners and tickers with dates, whether data is observed at SKU level, how it is delivered, what validation evidence exists, and how it is collected.
Should I build my own retailer web scraper instead?
A scraper starts its history on the day it is built, and its sample changes whenever pages block or change layout. For research that needs year-on-year comparisons and backtests, a dataset with a fixed panel and years of history is usually the faster route.
How do I test a vendor's data before buying?
Run a backtest: compare the vendor's measures for companies you know against their reported results over several years, and check that the brand-to-ticker mapping is point-in-time so acquisitions do not leak into earlier periods.
See also Data Purl vs AI web scraping.
Put Data Purl through the checklist
We answer due diligence questionnaires and can run a backtest on the companies you follow.