2scraper
What the Woolworths scraper collects
Reads woolworths.com.au search results, category nodes and the specials catalogue, and writes one row per product with 41 columns: price, unit price, real discounts, stock, merchandising hierarchy, health star rating, ingredients and allergens. The data comes from the site's own API, called from inside a loaded page, to JSON or CSV.
Search results
Input: A woolworths.com.au search URLOne row per product matching the term, with sponsored rows flagged and duplicates removed.
Category browse
Input: A /shop/browse/ category or child-node URLOne row per product in the category, paged until no new organic product appears.
Specials
Input: The specials category URLThe current specials catalogue, with original price, savings and computed discount.
From zero to Woolworths data in 3 steps
Install
Clone the repository and install one browser engine. Python 3.9+.
$ git clone https://github.com/2scraper/woolworths-scraper.git && cd woolworths-scraper $ pip install -r requirements.txt -r requirements-playwright.txt $ playwright install chromium $ python3 playwright_scraper.py --url "https://www.woolworths.com.au/shop/browse/bakery" --pages 3 --format both --out bakery
Run a scraper
Pick what you need and run it. Every flag is documented in the README.
$ python3 playwright_scraper.py --url "https://www.woolworths.com.au/shop/search/products?searchTerm=milk" --pages 3$ python3 playwright_scraper.py --url "https://www.woolworths.com.au/shop/browse/bakery" --pages 3 --format both --out bakery$ python3 playwright_scraper.py --url "https://www.woolworths.com.au/shop/browse/specials" --pages 3Get structured data
Rows land in JSON or CSV (
--format), with a.meta.jsonsidecar recording status, pages fetched and why the run stopped.
Woolworths data fields
41 columns per row in a predictable schema. The most useful ones:
| Field | What it holds |
|---|---|
sku | Woolworths stockcode |
title | Product name with pack size |
brand | Brand |
barcode | Product barcode |
price | Shelf price as a number |
currency | AUD |
original_price | Was-price, only when the product is really discounted |
discount_pct | Discount computed from price and original_price |
cup_price | Unit price as a number |
cup_string | Unit price as the site prints it, e.g. $2.35 / 1L |
package_size | Pack size |
is_on_special | On special flag |
is_sponsored | True for promoted ad rows the site injects |
is_in_stock | Stock flag |
category | Woolworths' own merchandising category |
health_star_rating | Australian Health Star Rating, 0.5-5 |
[
{
"source": "www.woolworths.com.au",
"scraped_at": "2026-09-16T17:42:03.474962+00:00",
"url": "https://www.woolworths.com.au/shop/productdetails/855750/cadbury-raspberry-mini-rolls",
"sku": "855750",
"title": "Cadbury Raspberry Mini Rolls 125g 5 pack",
"brand": "Cadbury",
"description": "Cadbury Raspberry Mini Rolls 125g 5 pack",
"variety": "Rolls",
"barcode": "5000221600077",
"price": 6.9,
"currency": "AUD",
"original_price": null,
"savings_amount": null,
"discount_pct": null,
"price_source": "api",
"cup_price": 1.38,
"cup_measure": "1EA",
"cup_string": "$1.38 / 1EA",
"package_size": "125g 5 pack",
"unit": "Each",
"is_on_special": false,
"is_half_price": false,
"is_sponsored": true,
"ad_status": "Promoted",
"offer_id": null,
"is_available": true,
"is_in_stock": true,
"is_purchasable": true,
"supply_limit": 18,
"department": "PROPRIETARY BAKERY",
"category": "PROPRIETARY BAKERY",
"subcategory": "RESALE CAKE",
"health_star_rating": null,
"dietary_claims": "Vegetarian",
"allergy_statement": null,
"ingredients": null,
"storage_instructions": null,
"image_url": "https://cdn1.woolworths.media/content/wowproductimages/large/855750.jpg",
"data_source": "api-category",
"page": 1
},
{
"source": "www.woolworths.com.au",
"scraped_at": "2026-09-16T17:42:03.475041+00:00",
"url": "https://www.woolworths.com.au/shop/productdetails/277728/woolworths-white-sandwich-brea…",
"sku": "277728",
"title": "Woolworths White Sandwich Bread Loaf 650g",
"brand": "Woolworths",
"description": "Woolworths White Sandwich Bread Loaf 650g",
"variety": "Loaf",
"barcode": "9339687336265",
"price": 2.5,
"currency": "AUD",
"original_price": null,
"savings_amount": null,
"discount_pct": null,
"price_source": "api+dom",
"cup_price": 0.38,
"cup_measure": "100G",
"cup_string": "$0.38 / 100G",
"package_size": "650g",
"unit": "Each",
"is_on_special": false,
"is_half_price": false,
"is_sponsored": false,
"ad_status": null,
"offer_id": null,
"is_available": true,
"is_in_stock": true,
"is_purchasable": true,
"supply_limit": 36,
"department": "PROPRIETARY BAKERY",
"category": "PROPRIETARY BAKERY",
"subcategory": "RESALE BREAD",
"health_star_rating": 3.5,
"dietary_claims": "Low Fat,Low Sugar,Source of Fibre,Vegetarian",
"allergy_statement": "Dairy Free,Egg Free,Fish Free,Lactose Free",
"ingredients": "Wheat Flour, Water, Yeast, Vinegar, Iodised Salt, Fermented Wheat Flour, Canola Oil, So…",
"storage_instructions": null,
"image_url": "https://cdn1.woolworths.media/content/wowproductimages/large/277728.jpg",
"data_source": "api-category",
"page": 1
}
]First 2 of 12 rows from a real run — sample_output.json. Long values shortened here.
Handles what breaks naive Woolworths scrapers
No product data in the HTML — so it asks the API
Woolworths renders its catalogue into shadow DOM from JSON. The scraper navigates once so Akamai issues a session, then calls the site's own API for each page from inside that page, with the browser's cookies and TLS.
Sponsored rows flagged and stopped on
Promoted products appear on every page and keep coming past the last real page. is_sponsored marks them, and the run ends when a page adds no new organic product, not when a page comes back empty.
Real discounts only
The site's WasPrice equals Price on most rows. original_price stays null unless it is genuinely higher, so the file is not filled with 0% discounts.
Category ids resolved, not guessed
A category id is opaque and not derivable from its slug, and sending the slug returns 200 with zero products. The scraper resolves it against the site's own tree first.
Scraping infrastructure included
The same tested core runs every 2scraper repository, so the hard parts work the same way on every site.
CAPTCHA handling
Detects the challenge a page really serves and, with a 2Captcha key, solves it only when it blocks the data — a capped number of solves per page.
Proxy rotation
Single proxy or a pool from a file, rotated on blocks with a fresh browser per exit. Credentials stay out of argv and logs.
Fingerprints
Optional complete device identity from the 2Captcha Fingerprint API — user agent, client hints, locale and timezone that agree with each other.
Scraping Browser API
Point --cdp-endpoint at a managed Chromium with its own exit country — no browser infrastructure to run yourself.
Honest run metadata
Every run writes a .meta.json sidecar and distinct exit codes: complete, blocked, empty and partial are never confused, and a failed run never overwrites good data.
Diff two runs
diff_runs.py compares two complete Woolworths snapshots by id — new, removed and changed rows for monitoring.
What teams do with Woolworths data
Grocery price monitoring
Track shelf and unit prices for a basket of products across Australia's largest supermarket.
Specials and half-price tracking
Pull the specials catalogue weekly and record savings, original prices and half-price flags.
Nutrition and allergen datasets
Collect health star ratings, dietary claims, allergy statements and ingredients by category.
Retail media analysis
Measure which brands buy promoted placements in which searches using is_sponsored and ad_status.
Free to run. Pay only for what the site requires.
Free to run · paid add-ons optional
From a residential address you need nothing: no account, no key, no proxy. From a datacentre, a visible (headful) browser was served 4 of 4 URLs where headless was refused 4 of 4, so headful is the default and the Dockerfile runs under xvfb. If your address is refused anyway, a residential proxy or the Scraping Browser API is what helps — an Australian exit is not required. No captcha has been observed on this site, so a 2Captcha key buys the proxy and the Scraping Browser here, not a solve.
Optional 2Captcha services — captcha solving, the Scraping Browser API, residential proxies and fingerprints — work behind one key.
Good to know
- A single product page is refused: every listing row already carries the full product object.
- --concurrency works on the Playwright engine only, and is refused with --cdp-endpoint.
- Selenium cannot authenticate a proxy or a credentialled CDP endpoint.
2scraper vs. hosted scraper APIs
| 2scraper | Typical hosted API | |
|---|---|---|
| Code | Open source, MIT — read and patch it | Closed |
| Cost | Free; optional services billed only when used | Subscription or per request / per result |
| Runs on | Your machine, server or CI | The vendor's infrastructure |
| Output | JSON/CSV files + run metadata | API responses |
| When the site changes | Fix it yourself or open an issue / PR | Wait for the vendor |
Woolworths scraper FAQ
Do I need an Australian proxy?
No. A US exit was served the full catalogue. A residential address is what matters.
Why does --headless fail from my server?
Akamai refused headless Chromium on every URL tried from a datacentre and served a visible browser on every one. On a machine with no display, use xvfb-run rather than headless.
Why does the product count change between runs?
The number of promoted rows the site injects varies, and they are deduplicated by sku — three pages of one category gave 117 rows on one run and 111 on another.
Why is there no rating column?
The API's rating object was empty on every one of 670 rows checked, and a column that is always null is worse than a missing one.
Does it cover Woolworths New Zealand or South Africa?
No. woolworths.co.nz and woolworths.co.za are different companies on different platforms, and both are refused by name.
Is scraping Woolworths legal?
It reads public product listings an anonymous shopper sees, with no login and no personal data. Woolworths' terms, robots.txt and your request rate are your responsibility.