2scraper
What the Etsy scraper collects
Scrapes Etsy search results, category listings, shop fronts and single listing pages into a shared row schema with Etsy's own columns appended. Listing mode paginates at about 62 rows a page; product mode adds description, images, material, GTIN and shipping origin; shop mode adds the seller's rating and location in the sidecar.
Search & category listings
Input: An Etsy /search?q=…, /c/… or /market/… URLOne row per listing tile: price, original price and computed discount, sponsored flag, shop id and shop rating, and whether the price is a 'from' minimum.
Listing pages
Input: An Etsy /listing/<id>/… URLOne row with the listing's own rating and review count, description, every image, material, GTIN, ships_from and free_shipping.
Shop fronts
Input: An Etsy /shop/<name> URLA seller's catalogue as listing rows, with the shop's name, location, rating and review count in the run's sidecar.
From zero to Etsy data in 3 steps
Install
Clone the repository and install one browser engine. Python 3.9+.
$ git clone https://github.com/2scraper/etsy-scraper.git && cd etsy-scraper $ pip install -r requirements.txt $ pip install -r requirements-playwright.txt && playwright install chromium $ cp .env.example .env $ python3 env_config.py
Run a scraper
Pick what you need and run it. Every flag is documented in the README.
$ python3 playwright_scraper.py --url "https://www.etsy.com/search?q=handmade+mug" --pages 3 --format both --out mugs$ python3 playwright_scraper.py --mode product --url "https://www.etsy.com/listing/519688604/handthrown-pottery-mug"$ python3 playwright_scraper.py --mode shop --url "https://www.etsy.com/shop/StonehousePotteryOH" --pages 3Get structured data
Rows land in JSON or CSV (
--format), with a.meta.jsonsidecar recording status, pages fetched and why the run stopped.
Etsy data fields
30 columns per row in a predictable schema. The most useful ones:
| Field | What it holds |
|---|---|
sku | Etsy listing id |
title | Listing title as the storefront shows it |
brand | The shop name — the seller is the brand on Etsy |
price | Displayed price as a number (a minimum when price_is_from is true) |
currency | Currency of the storefront that answered, read from the page |
original_price | Pre-discount price, where one is shown |
discount_pct | Discount computed from the two prices, never read from the badge |
is_ad | Sponsored placement, from the link's own locale-independent parameter |
shop_id | Seller's shop id |
shop_rating | The SHOP's star rating shown on the tile |
shop_review_count | The shop's review count |
price_is_from | True when the listing has variations and price is the lowest |
price_max | Highest variation price, where published |
rating | The LISTING's own rating (product mode only) |
ships_from | Shipping origin (product mode only) |
material | Listed material (product mode only) |
[
{
"source": "etsy.com",
"scraped_at": "2026-09-09T21:20:44.081810+00:00",
"url": "https://www.etsy.com/de/listing/1881219122/handgemachter-pilzbecher-wald",
"sku": "1881219122",
"title": "Handgemachter Pilzbecher • Wald-Inspirierte Tasse • Cottagecore Keramik • Größe S oder …",
"brand": "KuzuArte",
"price": 65.0,
"currency": "EUR",
"original_price": null,
"discount_pct": null,
"rating": null,
"review_count": null,
"in_stock": true,
"image_url": "https://i.etsystatic.com/17705718/r/il/73f165/7533232038/il_765x956.7533232038_inwr.jpg",
"category": "handmade mug",
"price_source": "dom",
"page": 1,
"position": 2,
"is_ad": true,
"shop_id": "17705718",
"shop_rating": 4.9,
"shop_review_count": 14,
"price_is_from": true,
"price_max": null,
"free_shipping": null,
"ships_from": null,
"material": null,
"gtin": null,
"description": null,
"images": null
},
{
"source": "etsy.com",
"scraped_at": "2026-09-09T21:20:44.076473+00:00",
"url": "https://www.etsy.com/de/listing/1765159813/ladybug-mug-coffee-mug-ceramic-mug-tea",
"sku": "1765159813",
"title": "Ladybug Ceramic Mug, Handmade Pottery Coffee Cup, Nature Lover Gift, Stoneware Tea Mug",
"brand": "DaNaKeramikShop",
"price": 54.9,
"currency": "EUR",
"original_price": null,
"discount_pct": null,
"rating": null,
"review_count": null,
"in_stock": true,
"image_url": "https://i.etsystatic.com/52168346/r/il/e6077a/8084684510/il_765x956.8084684510_ct1x.jpg",
"category": "handmade mug",
"price_source": "jsonld+dom",
"page": 1,
"position": 1,
"is_ad": false,
"shop_id": "52168346",
"shop_rating": 5.0,
"shop_review_count": 495,
"price_is_from": null,
"price_max": null,
"free_shipping": null,
"ships_from": null,
"material": null,
"gtin": null,
"description": null,
"images": null
}
]First 2 of 5 rows from a real run — sample_output.json. Long values shortened here.
Handles what breaks naive Etsy scrapers
Sponsored listings flagged in any language
Etsy labels ads in the page's own language. is_ad comes from the tile link's ls=a / ls=s parameter instead, verified against the text label with 280 agreements and zero disagreements.
Shop ratings are not listing ratings
The stars on a listing tile are the shop's. They land in shop_rating / shop_review_count, while the listing's own rating comes only from its detail page — three different numbers kept in three different columns.
DataDome states told apart
A banned address (t=bv), a solvable slider (t=fe) and an undecided interstitial are classified separately; the engines wait out the interstitial instead of reporting a block that was about to clear.
Cross-storefront diffs refused
The exit IP decides the storefront and currency. diff_runs.py compares the rows' currencies and refuses to diff a USD run against a EUR one, where every price would differ by the exchange rate.
Scraping infrastructure included
The same tested core runs every 2scraper repository, so the hard parts work the same way on every site.
CAPTCHA handling
Detects the challenge a page really serves and, with a 2Captcha key, solves it only when it blocks the data — a capped number of solves per page.
Proxy rotation
Single proxy or a pool from a file, rotated on blocks with a fresh browser per exit. Credentials stay out of argv and logs.
Fingerprints
Optional complete device identity from the 2Captcha Fingerprint API — user agent, client hints, locale and timezone that agree with each other.
Scraping Browser API
Point --cdp-endpoint at a managed Chromium with its own exit country — no browser infrastructure to run yourself.
Honest run metadata
Every run writes a .meta.json sidecar and distinct exit codes: complete, blocked, empty and partial are never confused, and a failed run never overwrites good data.
Diff two runs
diff_runs.py compares two complete Etsy snapshots by id — new, removed and changed rows for monitoring.
What teams do with Etsy data
Handmade market research
Pull search and category results to see price ranges, discounts and variation pricing for a product niche.
Competitor shop tracking
Scrape a seller's whole catalogue with shop mode and diff runs to catch new listings and price changes.
Ad visibility analysis
Measure how many results on a query are sponsored, and which shops buy placement, using is_ad.
Product detail enrichment
Fetch descriptions, materials, images, GTINs and shipping origin for listings you already track.
Free to run. Pay only for what the site requires.
Needs a residential exit or the Scraping Browser API
Etsy is behind DataDome and refused datacentre addresses outright when measured in Sep 2026; even a headless Chromium through a residential exit was refused. The path that worked was the 2Captcha Scraping Browser API (--cdp-endpoint), which served full pages with no captcha solve bought, used with Playwright or Puppeteer. Captcha solving for the DataDome slider is wired but opt-in (--solve-captcha always) and needs a proxy; fingerprints apply only to a local browser.
Optional 2Captcha services — captcha solving, the Scraping Browser API, residential proxies and fingerprints — work behind one key.
Good to know
- A Scraping Browser profile can be refused for its first requests and vary through the day; retry or switch profile.
- gtin is null on most rows because most handmade listings carry no barcode.
- Selenium cannot use an authenticated Scraping Browser endpoint, so on this site it has no working access path.
2scraper vs. hosted scraper APIs
| 2scraper | Typical hosted API | |
|---|---|---|
| Code | Open source, MIT — read and patch it | Closed |
| Cost | Free; optional services billed only when used | Subscription or per request / per result |
| Runs on | Your machine, server or CI | The vendor's infrastructure |
| Output | JSON/CSV files + run metadata | API responses |
| When the site changes | Fix it yourself or open an issue / PR | Wait for the vendor |
Etsy scraper FAQ
What formats does it write?
<out>.json and/or <out>.csv (--format both for both), plus a <out>.meta.json sidecar with the run's status and, in shop mode, the seller's own facts.
Can I run it from my own machine without a paid product?
The README says not to count on it: a local browser was refused from a hosting network, and headless Chromium was refused even from a residential one. The Scraping Browser API is the path that worked.
Which engine should I use?
Playwright. Puppeteer also reached Etsy through the Scraping Browser API. Selenium cannot authenticate a remote CDP endpoint or a proxy, so it refuses up front with that reason.
Why are rating and review_count null on listing rows?
Etsy publishes no per-listing rating on listing pages. The stars on a tile are the shop's and are in shop_rating; the listing's own rating comes from --mode product.
How does pagination work?
Etsy has no rel="next", so the scraper pages with ?page=N, cross-checked against the site's numbered links and bounded by its own initial_total_pages where published.
Is it legal?
It reads public search, category, shop and listing pages a visitor sees without signing in, and never touches an account or checkout. Follow Etsy's terms and the law where you operate.