2scraper
What the Lidl US scraper collects
Takes a search term, a category path or a lidl.com URL and returns a flat list of grocery products: title, brand, price, unit price and pack size. It reads each tile's own embedded product data, walks Lidl's batched, virtualized grid, and writes JSON or CSV with a run sidecar.
Product search
Input: A search termOne row per product in the results grid, with price, unit price and pack size, up to --max-results.
Category listing
Input: A category path copied from lidl.com's navigationOne row per product in that category's grid, walked through every View More Products batch.
Any lidl.com listing URL
Input: A full lidl.com search or category URLThe same product rows, for an address you already have.
From zero to Lidl US data in 3 steps
Install
Clone the repository and install one browser engine. Python 3.9+.
$ git clone https://github.com/2scraper/lidl-scraper.git && cd lidl-scraper $ pip install -r requirements-playwright.txt && playwright install chromium $ cp .env.example .env $ python3 playwright_scraper.py --query "whole milk" --format json --out lidl_results.json
Run a scraper
Pick what you need and run it. Every flag is documented in the README.
$ python3 playwright_scraper.py --query "whole milk" --format json --out lidl_results.json$ python3 playwright_scraper.py --category food-wine/s10068374$ python3 playwright_scraper.py --url "https://www.lidl.com/q/search?q=milk"Get structured data
Rows land in JSON or CSV (
--format), with a.meta.jsonsidecar recording status, pages fetched and why the run stopped.
Lidl US data fields
20 columns per row in a predictable schema. The most useful ones:
| Field | What it holds |
|---|---|
sku | Lidl's own product id |
title | Product name |
brand | Brand printed on the item, when the site gives one |
category | The aisle or category Lidl assigns |
price | Shelf price as a number |
currency | ISO currency code (USD) |
unit_price | Price per unit, as the tile prints it |
unit_size | Pack size, e.g. 128 fl.oz. |
price_source | Which path produced the row: embedded_json or dom |
product_url | Product detail page URL |
original_price | Pre-deal price, when one is captured |
is_weekly_deal | Weekly-deal flag, when one is captured |
zip_code | The --zip you asked for, recorded on the row |
scraped_at | UTC timestamp of the run |
[
{
"sku": "lidl-11675058",
"source": "lidl.com",
"category": "Food",
"title": "whole milk",
"brand": null,
"price": 3.3,
"currency": "USD",
"price_source": "dom",
"product_url": "https://www.lidl.com/p/whole-milk/p11675058",
"image_url": null,
"scraped_at": "2026-09-20T13:13:11Z",
"unit_price": 0.03,
"unit_size": "128 fl.oz.",
"original_price": null,
"discount_pct": null,
"is_weekly_deal": null,
"deal_valid_from": null,
"deal_valid_until": null,
"store_id": null,
"zip_code": null
},
{
"sku": "lidl-11242918",
"source": "lidl.com",
"category": "Food",
"title": "Lactaid® whole milk",
"brand": "LACTAID®",
"price": 6.38,
"currency": "USD",
"price_source": "dom",
"product_url": "https://www.lidl.com/p/lactaid-whole-milk/p11242918",
"image_url": null,
"scraped_at": "2026-09-20T13:13:11Z",
"unit_price": 0.07,
"unit_size": "96 fl.oz.",
"original_price": null,
"discount_pct": null,
"is_weekly_deal": null,
"deal_valid_from": null,
"deal_valid_until": null,
"store_id": null,
"zip_code": null
}
]First 2 of 2 rows from a real run — sample_output.json. Long values shortened here.
Handles what breaks naive Lidl US scrapers
Reads the tile's own data
Lidl's search page carries no product JSON-LD. Each tile does carry a data-gridbox-impression attribute with clean id, name, brand, category and price, and that is the primary extraction path.
Batched, virtualized grid walked fully
The grid recycles tiles as you scroll. Each engine steps through it a viewport at a time, clicks View More Products only once a batch has settled, and dedupes by sku.
Completeness checked by arithmetic
The page prints its own "N Products" counter. A run that captured fewer than min(N, --max-results) rows is reported partial (exit 6), never a plausible-looking complete result.
Unverified fields left null, not guessed
Image and weekly-deal columns stay null until Lidl's real markup for them has been captured, so a column never holds an invented value.
Scraping infrastructure included
The same tested core runs every 2scraper repository, so the hard parts work the same way on every site.
CAPTCHA handling
Detects the challenge a page really serves and, with a 2Captcha key, solves it only when it blocks the data — a capped number of solves per page.
Proxy rotation
Single proxy or a pool from a file, rotated on blocks with a fresh browser per exit. Credentials stay out of argv and logs.
Fingerprints
Optional complete device identity from the 2Captcha Fingerprint API — user agent, client hints, locale and timezone that agree with each other.
Scraping Browser API
Point --cdp-endpoint at a managed Chromium with its own exit country — no browser infrastructure to run yourself.
Honest run metadata
Every run writes a .meta.json sidecar and distinct exit codes: complete, blocked, empty and partial are never confused, and a failed run never overwrites good data.
Diff two runs
diff_runs.py compares two complete Lidl US snapshots by id — new, removed and changed rows for monitoring.
What teams do with Lidl US data
Grocery price tracking
Follow shelf and unit prices for a basket of search terms over time and diff runs by sku.
Unit-price comparison
Compare price per ounce or per count across pack sizes and brands in one category.
Private-label research
See how Lidl's own-brand items are priced against branded ones in the same aisle.
Assortment monitoring
Record which products a category lists and catch additions and removals between runs.
Free to run. Pay only for what the site requires.
Runs with no key, proxy or account
The default is an ordinary local headless Chromium with no proxy, no key and no account, and no bot challenge of any kind was met on lidl.com in the September 2026 captures and live runs. Proxies, the Scraping Browser API and fingerprints are opt-in for volume, a specific exit country or a consistent device identity; the README notes those were not live-measured on this site.
Optional 2Captcha services — captcha solving, the Scraping Browser API, residential proxies and fingerprints — work behind one key.
Good to know
- Images and weekly-deal fields are not yet live-verified and are left null.
- Store/zip selection is not confirmed to change what a search returns.
- Selenium and pyppeteer engines have not been live-verified like Playwright has.
2scraper vs. hosted scraper APIs
| 2scraper | Typical hosted API | |
|---|---|---|
| Code | Open source, MIT — read and patch it | Closed |
| Cost | Free; optional services billed only when used | Subscription or per request / per result |
| Runs on | Your machine, server or CI | The vendor's infrastructure |
| Output | JSON/CSV files + run metadata | API responses |
| When the site changes | Fix it yourself or open an issue / PR | Wait for the vendor |
Lidl US scraper FAQ
Do I need a proxy or a 2Captcha key?
No. A local headless Chromium with no proxy or key returned complete runs. The paid options are there for volume or a specific exit country.
Does it read prices for my store?
Not yet confirmed. A plain search returns generic prices with no store set; --zip and --store-id are recorded on every row, but no engine is confirmed to switch region yet.
Why is there no --pages flag?
Lidl's results are one batched, scroll-based page. --max-results, --max-scrolls and --stall-rounds control how far the scraper walks it.
What output formats are there?
JSON or CSV with a 20-column Product schema, plus a <out>.meta.json sidecar with status and price coverage. A failed or empty run writes nothing, so it never overwrites a good one.
Which Lidl sites are covered?
lidl.com, Lidl US.
Is scraping lidl.com legal?
It reads public product listings. Follow Lidl's terms and the law in your jurisdiction.