2scraper
What the Home Depot scraper collects
Scrapes Home Depot category listings (12 products a page) and product pages into one row schema: price, store-level stock, brand, model, ratings, badges and the site's own category hierarchy. Output is JSON or CSV with a sidecar that says whether the run is complete, and a catalog walker lists every leaf category from the site's sitemaps.
Category listings
Input: A leaf category URL (/b/…/N-…)One row per item with price, store stock, local-delivery stock, brand, model, rating, badges and category hierarchy.
Product pages
Input: A product URL (/p/…/<itemId>)Adds gtin13, colour, dimensions and the full description to the listing columns.
Category discovery
Input: An optional name filterLists leaf category URLs from the taxonomy sitemaps the site publishes, to use as scrape targets without crawling.
From zero to Home Depot data in 3 steps
Install
Clone the repository and install one browser engine. Python 3.9+.
$ git clone https://github.com/2scraper/homedepot-scraper $ cd homedepot-scraper $ pip install -r requirements.txt $ pip install -r requirements-playwright.txt && playwright install chromium $ cp .env.example .env
Run a scraper
Pick what you need and run it. Every flag is documented in the README.
$ python3 http_scraper.py --url https://www.homedepot.com/b/Tools-Power-Tools-Saws-Miter-Saws/N-5yc1vZc2d7 --pages 3 --out miter$ python3 playwright_scraper.py --mode product --url https://www.homedepot.com/p/.../321488310 --out saw$ python3 catalog_walk.py categories --grep 'Saws' --limit 20Get structured data
Rows land in JSON or CSV (
--format), with a.meta.jsonsidecar recording status, pages fetched and why the run stopped.
Home Depot data fields
35 columns per row in a predictable schema. The most useful ones:
| Field | What it holds |
|---|---|
sku | Home Depot item id (the number at the end of a /p/ URL) |
title | Product title |
brand | Brand name |
model_number | Manufacturer model number |
price | Price at the assigned store |
original_price | Was-price, where discounted |
discount_pct | Discount computed from the two prices |
availability | Stock status, e.g. InStock (listing rows) |
store_id | The store the price and stock belong to |
store_quantity | Units on the shelf for in-store pickup |
local_delivery_quantity | Units available for local delivery from that store |
rating | Average customer rating |
review_count | Number of reviews |
category_hierarchy | The site's own category path |
store_sku | Home Depot's store SKU, distinct from the item id |
gtin13 | Barcode (product pages) |
[
{
"source": "homedepot.com",
"scraped_at": "2026-09-22T11:56:20.360165+00:00",
"url": "https://www.homedepot.com/p/DEWALT-15-Amp-Corded-12-in-Double-Bevel-Sliding-Compound-Mi…",
"sku": "321488310",
"title": "15 Amp Corded 12 in. Double Bevel Sliding Compound Miter Saw with XPS technology, Blade…",
"image_url": "https://images.thdstatic.com/productImages/7a5da777-efd1-4bdb-958c-f1da097c431f/svn/dew…",
"price": 649.0,
"currency": "USD",
"category": "Tools-Power-Tools-Saws-Miter-Saws",
"price_source": "apollo-searchnav",
"original_price": null,
"discount_pct": null,
"brand": "DEWALT",
"model_number": "DWS780",
"parent_id": "341241630",
"store_sku": "1008223977",
"is_super_sku": true,
"product_type": "MERCHANDISE",
"gtin13": null,
"color": null,
"availability": "InStock",
"store_id": "121",
"store_name": "Cumberland",
"store_postal_code": "30339",
"store_quantity": 3,
"local_delivery_quantity": 23,
"rating": 4.6837,
"review_count": 1451,
"category_hierarchy": [
"Tools",
"Power Tools",
"Saws",
"… 1 more"
],
"badges": [
"Top Rated"
],
"unit_of_measure": "each",
"unit_price": null,
"returnable": "90-Day",
"page": 1,
"row_index": 0
},
{
"source": "homedepot.com",
"scraped_at": "2026-09-22T11:56:20.360423+00:00",
"url": "https://www.homedepot.com/p/DEWALT-15-Amp-Corded-12-in-Compound-Double-Bevel-Miter-Saw-…",
"sku": "308351915",
"title": "15 Amp Corded 12 in. Compound Double Bevel Miter Saw",
"image_url": "https://images.thdstatic.com/productImages/2deb1e5c-47b3-4ddb-af8d-8f5b54a6e7bd/svn/dew…",
"price": 349.0,
"currency": "USD",
"category": "Tools-Power-Tools-Saws-Miter-Saws",
"price_source": "apollo-searchnav",
"original_price": 429.0,
"discount_pct": 18.65,
"brand": "DEWALT",
"model_number": "DWS716",
"parent_id": "308351915",
"store_sku": "1004154267",
"is_super_sku": false,
"product_type": "MERCHANDISE",
"gtin13": null,
"color": null,
"availability": "InStock",
"store_id": "121",
"store_name": "Cumberland",
"store_postal_code": "30339",
"store_quantity": 6,
"local_delivery_quantity": 11,
"rating": 4.6375,
"review_count": 582,
"category_hierarchy": [
"Tools",
"Power Tools",
"Saws",
"… 1 more"
],
"badges": null,
"unit_of_measure": "each",
"unit_price": null,
"returnable": "90-Day",
"page": 1,
"row_index": 1
}
]First 2 of 35 rows from a real run — sample_output.json. Long values shortened here.
Handles what breaks naive Home Depot scrapers
Every price tied to its store
Home Depot prices and stocks per location; every row carries store_id, store_name and store_postal_code, because a price without its store is not a fact.
Carousels kept out of the grid
Hubs and the home page carry 22-24 recommendation products. Rows are scoped to the listing's own product list, so a hub reports zero products instead of 24 wrong ones.
Item id, not the misleading sku
The JSON-LD sku is the store SKU; the scraper keys rows on the item id from the URL and keeps the store SKU in its own column.
Blocked vs. plumbing told apart
Exit 3 means the site refused; exit 5 means a timeout, dropped CDP socket, locked profile or dead proxy credential, so you are not sent to fix the wrong thing.
Scraping infrastructure included
The same tested core runs every 2scraper repository, so the hard parts work the same way on every site.
CAPTCHA handling
Detects the challenge a page really serves and, with a 2Captcha key, solves it only when it blocks the data — a capped number of solves per page.
Proxy rotation
Single proxy or a pool from a file, rotated on blocks with a fresh browser per exit. Credentials stay out of argv and logs.
Fingerprints
Optional complete device identity from the 2Captcha Fingerprint API — user agent, client hints, locale and timezone that agree with each other.
Scraping Browser API
Point --cdp-endpoint at a managed Chromium with its own exit country — no browser infrastructure to run yourself.
Honest run metadata
Every run writes a .meta.json sidecar and distinct exit codes: complete, blocked, empty and partial are never confused, and a failed run never overwrites good data.
Diff two runs
diff_runs.py compares two complete Home Depot snapshots by id — new, removed and changed rows for monitoring.
What teams do with Home Depot data
Price monitoring
Track prices and markdowns on tools, appliances and building supplies at a specific store, diffing runs by sku.
Local stock tracking
Watch store_quantity and local_delivery_quantity to see availability by location.
Assortment analysis
Walk thousands of leaf categories from the sitemaps to map brands, models and badges across a department.
Product data enrichment
Pull gtin13, model numbers and ratings from product pages to match Home Depot items against your own catalogue.
Free to run. Pay only for what the site requires.
Needs a residential exit or the Scraping Browser API
The HTTP path needs a US residential exit: from non-US addresses every URL, robots.txt included, answered 403 (measured Sep 2026). Exits are scored individually, so a residential proxy pool beats a single address. A local Chromium was refused outright, so the Scraping Browser API with country-us is the only browser transport measured to work. Captcha solving is armed but has never had a challenge to solve on listing or product pages.
Optional 2Captcha services — captcha solving, the Scraping Browser API, residential proxies and fingerprints — work behind one key.
Good to know
- availability is null on product pages: stock is loaded after page load there; listing rows carry it.
- Selenium and pyppeteer have not been run live against homedepot.com.
- Prices and stock reflect the store your exit is assigned to.
2scraper vs. hosted scraper APIs
| 2scraper | Typical hosted API | |
|---|---|---|
| Code | Open source, MIT — read and patch it | Closed |
| Cost | Free; optional services billed only when used | Subscription or per request / per result |
| Runs on | Your machine, server or CI | The vendor's infrastructure |
| Output | JSON/CSV files + run metadata | API responses |
| When the site changes | Fix it yourself or open an issue / PR | Wait for the vendor |
Home Depot scraper FAQ
Do I need a proxy?
Yes, a US residential exit, set as HOMEDEPOT_PROXY in .env. With it, the free http_scraper.py returns the full grid without any browser.
Why not just use a local browser?
In 10 paired trials a headless Playwright Chromium was refused 10 times through the same exits that served curl. Over the Scraping Browser API, Playwright cleared Akamai's interstitial every time.
How does pagination work?
Home Depot uses an offset, ?Nao=, 12 products a page. A run stops when a page adds no new item id rather than trusting totalProducts, which is live inventory and moves.
Why does a category return zero products?
It is probably a hub, not a leaf. Hubs have no product grid; use catalog_walk.py to find leaf categories.
What output formats are there?
JSON or CSV with the same field order, plus <out>.meta.json and a latest_attempt sidecar written on every run, so a blocked run cannot pass for yesterday's success.
Is scraping Home Depot legal?
The scraper reads public /b/ and /p/ pages that robots.txt allows, with no login or cart. It is public data; follow the site's terms and the law.