2scraper
What the Pottery Barn scraper collects
Scrapes Pottery Barn (US) category listings and product pages. Listings come from the site's own listing API with no proxy, browser or key, one row per product with its price range; product pages give one row per purchasable SKU with its own price, markdown, stock state and options. JSON or CSV, with a sidecar that says whether the result is complete.
Category listing
Input: A category group id or category URLOne row per listing result: price and price_max, original price, max discount, lead SKU, price type and flags.
Product SKUs
Input: One or more product URLsOne row per purchasable SKU: price, markdown, availability, back-order date, options, colour and dimensions.
SKUs from a listing run
Input: The JSON output of a category runFetches each product page a listing run found and writes all of its SKUs.
Catalogue discovery
Input: An optional name filterLists category and product URLs from the site's sitemaps, with hubs, leaves and open-box items told apart.
From zero to Pottery Barn data in 3 steps
Install
Clone the repository and install one browser engine. Python 3.9+.
$ git clone https://github.com/2scraper/potterybarn-scraper $ cd potterybarn-scraper $ pip install -r requirements.txt $ pip install -r requirements-playwright.txt && playwright install chromium $ cp .env.example .env
Run a scraper
Pick what you need and run it. Every flag is documented in the README.
$ python3 api_scraper.py --group-id sofa --all-pages --out sofas$ python3 http_scraper.py --url https://www.potterybarn.com/products/delaney-marble-end-table/ --out skus$ python3 http_scraper.py --from-listing sofas.json --out sofa_skus$ python3 catalog_walk.py categories --leaf-only --grep sofaGet structured data
Rows land in JSON or CSV (
--format), with a.meta.jsonsidecar recording status, pages fetched and why the run stopped.
Pottery Barn data fields
22 columns per row in a predictable schema. The most useful ones:
| Field | What it holds |
|---|---|
sku | Product slug on listing rows; SKU id on SKU rows |
product_id | Product slug, the join key between listing and SKU rows |
title | Product or SKU title |
price | Lowest price on a listing row; the SKU's own price on a SKU row |
price_max | Top of the product's price range (listing rows) |
original_price | Price before markdown |
max_discount_pct | Largest markdown across the range (listing rows) |
discount_pct | The SKU's markdown (SKU rows) |
currency | Currency as the page states it |
leader_sku | The listing's lead SKU |
sub_group_id | Finish slice, when a listing result is a sub-group of a product |
availability | ON_HAND, BACK_ORDERED or NLA (SKU rows) |
back_ordered_date | Expected date for back-ordered SKUs |
options | The SKU's options, e.g. finish and quantity |
width_in | Width in inches (SKU rows) |
price_type | The site's price type, e.g. Discount |
[
{
"source": "potterybarn.com",
"scraped_at": "2026-09-24T14:12:50.768372+00:00",
"url": "https://www.potterybarn.com/products/york-slope-arm-deep-slipcovered-sofa-collection/",
"sku": "york-slope-arm-deep-slipcovered-sofa-collection",
"title": "York Slope Arm Deep Seat Slipcovered Sofa (60\"-108\")",
"image_url": "https://assets.pbimgs.com/pbimgs/rk/images/dp/wcm/202629/0076/york-slope-arm-deep-seat-…",
"price": 1679.0,
"currency": "USD",
"category": "sofa",
"price_source": "constructor-browse",
"product_id": "york-slope-arm-deep-slipcovered-sofa-collection",
"sub_group_id": null,
"price_max": 4199.0,
"original_price": 1999.0,
"original_price_max": 4199.0,
"max_discount_pct": 20.0,
"leader_sku": "865462",
"pip_type": "guided",
"price_type": "Discount",
"flags": [
"freePers",
"inHome"
],
"page": 1,
"row_index": 0
},
{
"source": "potterybarn.com",
"scraped_at": "2026-09-24T14:12:50.768442+00:00",
"url": "https://www.potterybarn.com/products/buchanan-upholstered-sofa-collection/",
"sku": "buchanan-upholstered-sofa-collection",
"title": "Buchanan Roll Arm Sofa (63\"-94\")",
"image_url": "https://assets.pbimgs.com/pbimgs/rk/images/dp/wcm/202632/0006/buchanan-roll-arm-sofa-63…",
"price": 999.0,
"currency": "USD",
"category": "sofa",
"price_source": "constructor-browse",
"product_id": "buchanan-upholstered-sofa-collection",
"sub_group_id": null,
"price_max": 2799.0,
"original_price": null,
"original_price_max": null,
"max_discount_pct": null,
"leader_sku": "12720827",
"pip_type": "guided",
"price_type": "Regular",
"flags": [
"freePers",
"inHome"
],
"page": 1,
"row_index": 1
}
]First 2 of 24 rows from a real run — sample_output.json. Long values shortened here.
Handles what breaks naive Pottery Barn scrapers
Compressed SKUs decoded
Configurable products like sofas list no SKUs in plain sight; they ship Brotli-compressed inside the page. A naive parser returns zero rows per sofa; this one decodes all of them and checks the count against the site's own skuCount.
Lost results recovered
The listing API's page order is not always stable, so a repeat means another product was skipped. When distinct rows fall short of the total, a second sweep in price order recovers the gap, or the run reports partial.
UK store never mistaken for US
A non-US exit is redirected to potterybarn.co.uk in GBP. That hop is classified geo_redirect and rotated to a fresh US exit instead of being parsed.
A 200 must be a product
A generic prerendered page with no product in it is not counted as a product page; the classifier requires the product details node.
Scraping infrastructure included
The same tested core runs every 2scraper repository, so the hard parts work the same way on every site.
CAPTCHA handling
Detects the challenge a page really serves and, with a 2Captcha key, solves it only when it blocks the data — a capped number of solves per page.
Proxy rotation
Single proxy or a pool from a file, rotated on blocks with a fresh browser per exit. Credentials stay out of argv and logs.
Fingerprints
Optional complete device identity from the 2Captcha Fingerprint API — user agent, client hints, locale and timezone that agree with each other.
Scraping Browser API
Point --cdp-endpoint at a managed Chromium with its own exit country — no browser infrastructure to run yourself.
Honest run metadata
Every run writes a .meta.json sidecar and distinct exit codes: complete, blocked, empty and partial are never confused, and a failed run never overwrites good data.
Diff two runs
diff_runs.py compares two complete Pottery Barn snapshots by id — new, removed and changed rows for monitoring.
What teams do with Pottery Barn data
Furniture price tracking
Track price ranges and markdowns across whole categories daily with no proxy, diffing runs by sku.
Per-SKU stock monitoring
Follow ON_HAND, BACK_ORDERED and NLA states and back-order dates for each finish and size of a product.
Assortment and variant analysis
Map options, colours and dimensions across thousands of SKUs to see how a collection is configured and priced.
Catalogue coverage
Walk the category and product sitemaps to build a complete target list, open-box items optional.
Free to run. Pay only for what the site requires.
Free to run · paid add-ons optional
Category listings need nothing: no proxy, browser or key, from any address. Product pages need a US residential exit, fetched over plain curl, which the README measured working where local browsers were refused. The Scraping Browser API is the only browser path that reaches product pages, served 14 of 14 sessions on 2026-10-01. Captcha solving is wired in but has never had anything to solve here; fingerprints are not needed over CDP.
Optional 2Captcha services — captcha solving, the Scraping Browser API, residential proxies and fingerprints — work behind one key.
Good to know
- US store only; non-US exits are redirected to the UK store.
- No rating or review columns: the site publishes none where this scraper reads.
- A local browser is refused on product pages; use curl or the Scraping Browser API.
2scraper vs. hosted scraper APIs
| 2scraper | Typical hosted API | |
|---|---|---|
| Code | Open source, MIT — read and patch it | Closed |
| Cost | Free; optional services billed only when used | Subscription or per request / per result |
| Runs on | Your machine, server or CI | The vendor's infrastructure |
| Output | JSON/CSV files + run metadata | API responses |
| When the site changes | Fix it yourself or open an issue / PR | Wait for the vendor |
Pottery Barn scraper FAQ
Do I need a proxy?
Not for category listings. For product pages yes: a US residential exit set as POTTERYBARN_PROXY in .env.
Why two kinds of row?
A listing result is a product with a price range; a product page lists each SKU with its own price. One sofa has 4,843 SKUs, so the mode picks the grain. Join the two on product_id, never on sku.
Where does the listing data come from?
From Constructor.io's browse API, the same call the storefront's grid makes with its public key. That host is not covered by potterybarn.com's robots.txt; the README explains this so you can decide.
How does pagination work?
24 results a page via page=<n>; a run stops when a page adds no new product id, and the site's total is checked afterwards, never used to stop.
Are ratings included?
No. Neither the product page nor the listing API carries rating data, so there is no rating column.
Is scraping Pottery Barn legal?
It reads public catalogue data as an anonymous visitor, respects robots.txt on www.potterybarn.com, and touches no cart or account. Follow the site's terms and the law.