2scraper
What the LG scraper collects
Give it any lg.com category URL on the /ru or /ua platform — televisions, refrigerators, monitors, audio — and it reads the whole product grid, one row per model with 27 columns. The recommended engine calls the catalogue API the page itself describes; three browser engines and 2Captcha's Scraper API read the rendered grid as a fallback.
Category via catalogue API
Input: An lg.com /ru or /ua category URLPOSTs the parameters read off the category page to LG's own catalogue endpoint and pages through the whole category, one row per model.
Category via rendered grid
Input: The same category URLOpens the page in a browser and reads each card's microdata and data-model attributes; verified to return the same products in the same order as the API path.
From zero to LG data in 3 steps
Install
Clone the repository and install one browser engine. Python 3.9+.
$ git clone https://github.com/2scraper/lg-scraper.git && cd lg-scraper $ pip install -r requirements.txt $ python3 catalog_client.py --url "https://www.lg.com/ru/televisions" --pages 10 --out tv_products
Run a scraper
Pick what you need and run it. Every flag is documented in the README.
$ python3 catalog_client.py --url "https://www.lg.com/ru/televisions" --pages 10 --out tv_products$ python3 playwright_scraper.py --url "https://www.lg.com/ru/televisions" --pages 3Get structured data
Rows land in JSON or CSV (
--format), with a.meta.jsonsidecar recording status, pages fetched and why the run stopped.
LG data fields
27 columns per row in a predictable schema. The most useful ones:
| Field | What it holds |
|---|---|
sku | LG's model code, e.g. OLED83W69LA |
model_id | LG's internal model id |
sales_model_code | The model code with its market suffix |
title | The site's own long product name |
url | Canonical product page |
image_url | Primary product image |
model_year | Model year |
screen_size | Screen size in inches |
sibling_sizes | Other sizes of the same model |
product_category | Category name as the API states it |
super_category | Top-level category slug |
status | Model status, e.g. ACTIVE |
where_to_buy_url | The "where to buy" link the catalogue offers instead of a checkout |
currency | Currency the page's microdata declares |
price_source | api or dom — which path produced the row |
[
{
"source": "lg.com",
"scraped_at": "2026-09-21T15:22:25.547492+00:00",
"url": "https://www.lg.com/ru/televisions/lg-oled83w69la",
"sku": "OLED83W69LA",
"title": "83-дюймовый телевизор Smart TV Wallpaper TV LG OLED evo AI W6 4K 2026 года",
"price": null,
"currency": "RUB",
"original_price": null,
"discount_pct": null,
"category": "televisions",
"price_source": "api",
"page": 1,
"position": 1,
"model_id": "MD07610675",
"sales_model_code": "OLED83W69LA.ARUG",
"model_year": 2026,
"screen_size": "83",
"sibling_sizes": "83\", 77\"",
"product_category": "Телевизоры",
"product_category_slug": "televisions",
"super_category": "tv-audio-video",
"rating": null,
"review_count": 0,
"status": "ACTIVE",
"energy_label": null,
"image_url": "https://www.lg.com/ru/images/televisions/md07610675/md07610675-350x350.jpg",
"where_to_buy_url": "https://www.lg.com/ru/televisions/lg-oled83w69la#pdp_where"
},
{
"source": "lg.com",
"scraped_at": "2026-09-21T15:22:25.547648+00:00",
"url": "https://www.lg.com/ru/televisions/lg-55nu800b6la",
"sku": "55NU800B6LA",
"title": "55-дюймовый телевизор Smart TV LG NANO 4K UHD AI NU80 2026",
"price": null,
"currency": "RUB",
"original_price": null,
"discount_pct": null,
"category": "televisions",
"price_source": "api",
"page": 1,
"position": 2,
"model_id": "MD09003547",
"sales_model_code": "55NU800B6LA.ARUQ",
"model_year": 2026,
"screen_size": "55",
"sibling_sizes": null,
"product_category": "Телевизоры",
"product_category_slug": "televisions",
"super_category": "tv-audio-video",
"rating": null,
"review_count": 0,
"status": "ACTIVE",
"energy_label": null,
"image_url": "https://www.lg.com/ru/images/televisions/md09003547/md09003547-350x350.jpg",
"where_to_buy_url": "https://www.lg.com/ru/televisions/lg-55nu800b6la#pdp_where"
}
]First 2 of 6 rows from a real run — sample_output.json. Long values shortened here.
Handles what breaks naive LG scrapers
Parameters read off the page, never hardcoded
categoryId differs per category and per locale. A hardcoded one would quietly scrape a different catalogue while reporting success, so every run reads the form the page itself serves.
Rails and template rows excluded
A 7-product recommendation rail repeats on every page and the site's own unfilled template row sits in the grid. Counting either would stop the run early as "no new products"; extraction is scoped to the real grid and placeholder values are discarded.
Published zeros are not prices
All nine API price fields were 0 or null across 36 products and the site's own switch says prices are off. price stays null rather than listing every LG TV as free — and a diff will show the day prices appear.
Samples are labelled as samples
--pages is a ceiling. A run that stops before the category ends is marked limited, and diff_runs.py will not treat it as the whole assortment.
Scraping infrastructure included
The same tested core runs every 2scraper repository, so the hard parts work the same way on every site.
CAPTCHA handling
Detects the challenge a page really serves and, with a 2Captcha key, solves it only when it blocks the data — a capped number of solves per page.
Proxy rotation
Single proxy or a pool from a file, rotated on blocks with a fresh browser per exit. Credentials stay out of argv and logs.
Fingerprints
Optional complete device identity from the 2Captcha Fingerprint API — user agent, client hints, locale and timezone that agree with each other.
Scraping Browser API
Point --cdp-endpoint at a managed Chromium with its own exit country — no browser infrastructure to run yourself.
Honest run metadata
Every run writes a .meta.json sidecar and distinct exit codes: complete, blocked, empty and partial are never confused, and a failed run never overwrites good data.
Diff two runs
diff_runs.py compares two complete LG snapshots by id — new, removed and changed rows for monitoring.
What teams do with LG data
Assortment tracking
Snapshot every model LG lists in a category and diff runs to see launches and removals.
Model and size mapping
Map model codes, sales codes, model years and sibling sizes for catalogue or comparison databases.
Product-data enrichment
Pull official titles, images and product-page links per model code for your own listings.
Category monitoring
Watch category totals and statuses across the /ru and /ua platforms on a schedule.
Free to run. Pay only for what the site requires.
Runs with no key, proxy or account
You need none of the paid products for this site: every number in the README was measured with no key, no proxy and no browser, because the catalogue API answers a plain HTTPS POST. Proxies matter only if your exit is in a place Akamai refuses; the Scraper API and the Scraping Browser API offer the same without managing an exit yourself. The site configures reCAPTCHA only on support forms, which this scraper never touches.
Optional 2Captcha services — captcha solving, the Scraping Browser API, residential proxies and fingerprints — work behind one key.
Good to know
- Only the /ru and /ua platforms are implemented; lg.com/uk and lg.com/us are refused.
- price, original_price and rating are null on everything measured, because the site publishes none.
- The pyppeteer engine carries a known-vulnerable urllib3; prefer Playwright with proxy credentials.
2scraper vs. hosted scraper APIs
| 2scraper | Typical hosted API | |
|---|---|---|
| Code | Open source, MIT — read and patch it | Closed |
| Cost | Free; optional services billed only when used | Subscription or per request / per result |
| Runs on | Your machine, server or CI | The vendor's infrastructure |
| Output | JSON/CSV files + run metadata | API responses |
| When the site changes | Fix it yourself or open an issue / PR | Wait for the vendor |
LG scraper FAQ
Which lg.com sites does it support?
The /ru and /ua platforms — the two entries the site's own hreflang set declares. Other locales use different markup or refused the client, and are refused with that reason rather than returning zero rows.
Why is price null?
On these platforms LG publishes zero in every price field and its own switch turns prices off. A published zero is not a price, so the column stays null.
Do I need a browser?
No. catalog_client.py, the recommended engine, needs only requirements.txt. The browser engines are a fallback over the rendered grid.
How does pagination work?
12 products per page from the API's own pagination block; --pages is a ceiling and the run stops at the category's last page.
What output does it write?
JSON or CSV plus a .meta.json sidecar with status (complete, limited or partial), which pages failed and the category's own totals. Writes are atomic, so a killed run leaves the previous files whole.
Is this allowed?
It reads public catalogue pages only — nothing behind a login and no form submissions. Public data; follow the site's terms and the law.