2scraper
What the Kohl's scraper collects
Scrapes kohls.com category grids, search results and product pages into JSON and CSV: the lowest payable price and the top of the range, regular prices, Kohl's own sale and clearance labels, ratings, stock, brand and size-by-colour SKU availability. Each listing page's 120 products are read from the site's server-rendered data, and a .meta.json sidecar records the listing's own product count and page total.
Category & brand listings
Input: A /catalog/…jsp?CN=… category or brand URLOne row per product: price and price range, regular price and discount, sale/clearance label, rating, review count, stock and the site's own sort order.
Search results
Input: A /search.jsp?search=… URLThe same row shape as a category grid, for a keyword search, with pages numbered as the server served them.
Product pages
Input: One /product/prd-NNN/… URLAdds brand, the deepest category crumb, and how many size × colour SKUs exist and how many are in stock.
From zero to Kohl's data in 3 steps
Install
Clone the repository and install one browser engine. Python 3.9+.
$ git clone https://github.com/2scraper/kohls-scraper.git $ cd kohls-scraper $ python3 -m venv venv $ ./venv/bin/pip install -r requirements.txt -r requirements-playwright.txt $ cp .env.example .env $ ./venv/bin/python env_config.py
Run a scraper
Pick what you need and run it. Every flag is documented in the README.
$ python3 playwright_scraper.py --url "https://www.kohls.com/catalog/cuisinart.jsp?CN=Brand:Cuisinart" --pages 3 --out cuisinart$ python3 playwright_scraper.py --mode listing --url "https://www.kohls.com/search.jsp?search=coffee+maker" --pages 2$ python3 playwright_scraper.py --mode product --url "https://www.kohls.com/product/prd-7580242/juniors-disneys-minnie-mouse-winter-icons-lux-plush-fleece-pajama-pants.jsp"Get structured data
Rows land in JSON or CSV (
--format), with a.meta.jsonsidecar recording status, pages fetched and why the run stopped.
Kohl's data fields
24 columns per row in a predictable schema. The most useful ones:
| Field | What it holds |
|---|---|
sku | Kohl's webID, the number in /product/prd-NNN/ |
title | Product title |
brand | Brand, filled in product mode (the listing payload has none) |
price | Lowest price payable now: sale if running, else regular |
price_max | Top of the price range; null for a single price |
currency | From the page's own JSON-LD; never a compiled-in default |
original_price | The regular (Reg./Orig.) price, only when a sale runs below it |
discount_pct | Computed from the low ends of price and original_price |
price_label | Kohl's own label: sale, clearance, sale+clearance, … |
rating | Average rating; null when the product has no reviews |
review_count | Number of reviews; null with no reviews |
in_stock | Ship/pickup availability on listings; any SKU in stock in product mode |
category | Category slug from the URL, or the deepest breadcrumb |
sort | The ordering the site reports, e.g. Featured or Price Low-High |
skus_total | Product mode: number of size × colour SKUs |
skus_in_stock | Product mode: how many of those are in stock |
[
{
"source": "kohls.com",
"scraped_at": "2026-09-25T13:48:35.547243+00:00",
"url": "https://www.kohls.com/product/prd-7580242/juniors-disneys-minnie-mouse-winter-icons-lux…",
"sku": "7580242",
"title": "Juniors' Disney's Minnie Mouse Winter Icons Lux Plush Fleece Pajama Pants",
"brand": null,
"price": 2.0,
"currency": "USD",
"original_price": 20.0,
"discount_pct": 90.0,
"rating": 4.1,
"review_count": 9,
"in_stock": true,
"image_url": "https://media.kohlsimg.com/is/image/kohls/7580242?wid=180&hei=180&op_sharpen=1",
"category": "womens-clothing",
"price_source": "catalog",
"page": 1,
"position": 1,
"price_max": null,
"original_price_max": null,
"price_label": "clearance",
"sort": "Price Low-High",
"skus_total": null,
"skus_in_stock": null
},
{
"source": "kohls.com",
"scraped_at": "2026-09-25T13:48:35.547993+00:00",
"url": "https://www.kohls.com/product/prd-6850910/juniors-so-seamless-cheeky-bikini-panty-so75-…",
"sku": "6850910",
"title": "Juniors' SO® Seamless Cheeky Bikini Panty SO75-028",
"brand": null,
"price": 5.5,
"currency": "USD",
"original_price": 11.0,
"discount_pct": 50.0,
"rating": 4.2,
"review_count": 46,
"in_stock": true,
"image_url": "https://media.kohlsimg.com/is/image/kohls/6850910_Rainbow_Hearts?wid=180&hei=180&op_sha…",
"category": "womens-clothing",
"price_source": "catalog",
"page": 1,
"position": 29,
"price_max": 7.99,
"original_price_max": null,
"price_label": "sale+clearance",
"sort": "Price Low-High",
"skus_total": null,
"skus_in_stock": null
}
]First 2 of 8 rows from a real run — sample_output.json. Long values shortened here.
Handles what breaks naive Kohl's scrapers
Every page verified against the server
Pagination is a zero-based WS= offset, planned against the site's totalPages. A page whose server says it is a different page ends the run as end_of_listing instead of being recorded under the wrong number.
Sponsored and collection tiles dropped
0 to 12 sponsored placements a page, varying between fetches, plus outfit collection entries are removed, and position counts only the rows emitted, so runs line up.
JSON-LD decoy ignored for rows
Each listing page's ItemList holds 15 items against 120 products. The scraper reads products from the page's own server-rendered data and uses the JSON-LD for the currency only.
Drifting order detected
The default Featured order moved mid-run (22 repeats in one run), so the README recommends a sorted URL such as &S=4 for monitoring, and diff_runs.py refuses to compare runs under different orderings.
Scraping infrastructure included
The same tested core runs every 2scraper repository, so the hard parts work the same way on every site.
CAPTCHA handling
Detects the challenge a page really serves and, with a 2Captcha key, solves it only when it blocks the data — a capped number of solves per page.
Proxy rotation
Single proxy or a pool from a file, rotated on blocks with a fresh browser per exit. Credentials stay out of argv and logs.
Fingerprints
Optional complete device identity from the 2Captcha Fingerprint API — user agent, client hints, locale and timezone that agree with each other.
Scraping Browser API
Point --cdp-endpoint at a managed Chromium with its own exit country — no browser infrastructure to run yourself.
Honest run metadata
Every run writes a .meta.json sidecar and distinct exit codes: complete, blocked, empty and partial are never confused, and a failed run never overwrites good data.
Diff two runs
diff_runs.py compares two complete Kohl's snapshots by id — new, removed and changed rows for monitoring.
What teams do with Kohl's data
Price and markdown monitoring
Track price, price ranges, regular prices and sale/clearance labels for a brand or category, run after run, with diff_runs.py reporting what changed.
Brand assortment tracking
Pull every product of a brand listing (e.g. Cuisinart) and compare counts and prices against the site's own productCount.
Size and colour availability
Use product mode to see how many size × colour SKUs a product has and how many are in stock.
Ratings research
Collect ratings and review counts across a category to compare products and brands.
Free to run. Pay only for what the site requires.
Needs a residential exit or the Scraping Browser API
On 2026-09-25 kohls.com's Akamai refused every local browser and HTTP client tried from US residential addresses, and served the Scraping Browser API with a US exit. So the product this site needs is the Scraping Browser API (--cdp-endpoint), or the 2Captcha Scraper API routed through it. Akamai's refusal page carried no captcha widget, so captcha solving, proxies and fingerprints were not what got pages served; a residential proxy with a local browser remains supported and worth a try, since Akamai's scoring moves.
Optional 2Captcha services — captcha solving, the Scraping Browser API, residential proxies and fingerprints — work behind one key.
Good to know
- in_stock was true on 863 of 863 measured listing rows, so a false on listings is unobserved.
- Keep the ?CN= filter on a category URL; it is what selects the category.
- Akamai's behavioural challenge is detected and reported as blocked; this repo does not implement a solver for it.
2scraper vs. hosted scraper APIs
| 2scraper | Typical hosted API | |
|---|---|---|
| Code | Open source, MIT — read and patch it | Closed |
| Cost | Free; optional services billed only when used | Subscription or per request / per result |
| Runs on | Your machine, server or CI | The vendor's infrastructure |
| Output | JSON/CSV files + run metadata | API responses |
| When the site changes | Fix it yourself or open an issue / PR | Wait for the vendor |
Kohl's scraper FAQ
What does it extract?
Category grids, search results and product pages: prices and ranges, regular prices, discounts, sale and clearance labels, ratings, stock, brand and per-SKU availability, as JSON and CSV.
Do I need a proxy or the Scraping Browser API?
In the README's 2026-09-25 measurement, yes: every local browser and HTTP client was refused and the Scraping Browser API with a US exit was served. Selenium cannot use it, so use Playwright or pyppeteer.
How does pagination work?
Kohl's uses a WS= offset of 120 products per page. The scraper plans against the site's own totalPages and verifies each page's currentPage.
Why is brand empty on listing rows?
The listing payload carries no brand (0 of 863 measured rows). Run --mode product to fill it.
Is a complete run the whole category?
It is everything you asked for, not everything the site has: the sidecar records total_results and pages_available, so a 3-page run of a 1,317-page category reads as a 0.2% sample.
Is scraping Kohl's legal?
It reads public product data. Respect kohls.com's terms of service and robots.txt, rate-limit your requests, and follow the law where you operate.