2scraper
What the Tokopedia scraper collects
Scrapes Tokopedia search grids, category listings and product pages with three interchangeable browser engines. Listing rows carry price, was-price, discount, rating, sold count and shop location read from the rendered grid; product pages add the real product id, exact sold count, condition and weight. Output is JSON or CSV with a run sidecar.
Category listings
Input: A category URL (/p/<cat>/<sub>)Products from the category grid with price, brand, shop and image, paged with ?page=N.
Search results
Input: A search URLThe infinitely scrolling search grid, scrolled until a round adds nothing new: price, was-price, rating, sold count and shop location.
Product pages
Input: A product URL (/{shop}/{slug})One product with the site's real product id, exact sold count, review count, condition, weight, stock and its category URL.
From zero to Tokopedia data in 3 steps
Install
Clone the repository and install one browser engine. Python 3.9+.
$ git clone https://github.com/2scraper/tokopedia-scraper.git && cd tokopedia-scraper $ pip install -r requirements.txt -r requirements-playwright.txt $ playwright install chromium $ cp .env.example .env
Run a scraper
Pick what you need and run it. Every flag is documented in the README.
$ python playwright_scraper.py --url "https://www.tokopedia.com/p/makanan-minuman/minuman/kopi-bubuk" --pages 3 --format both$ python playwright_scraper.py --url "https://www.tokopedia.com/search?st=product&q=kopi"$ python playwright_scraper.py --mode product --url "https://www.tokopedia.com/zayn-snack-448/kopi-hitam-bubuk-robusta-original-berat-1-kg-coffee-kualitas-premium-pahitnya-pas-cocok-untuk-kopi-susu-1731177319241910164"Get structured data
Rows land in JSON or CSV (
--format), with a.meta.jsonsidecar recording status, pages fetched and why the run stopped.
Tokopedia data fields
30 columns per row in a predictable schema. The most useful ones:
| Field | What it holds |
|---|---|
sku | The product's URL path, the join key between listing and product rows |
title | Product title |
brand | Name shown on the tile |
price | Current price as a number |
currency | IDR |
original_price | Was-price, where discounted |
discount_pct | Discount computed from the two prices |
rating | Product rating |
sold | Units sold |
sold_is_floor | True when sold is a tile's rounded floor ("100rb+"), false when exact |
shop_slug | Seller shop slug |
shop_location | Seller location |
image_url | Real product image, or null for a lazy-load placeholder |
product_id | Tokopedia's real product id (product mode) |
condition | New or used (product mode) |
weight_grams | Shipping weight (product mode) |
[
{
"source": "tokopedia.com",
"scraped_at": "2026-09-10T10:23:34.301907+00:00",
"url": "https://www.tokopedia.com/zayn-snack-448/kopi-hitam-bubuk-robusta-original-berat-1-kg-c…",
"sku": "/zayn-snack-448/kopi-hitam-bubuk-robusta-original-berat-1-kg-coffee-kualitas-premium-pa…",
"title": "Kopi Hitam Bubuk Robusta Original Berat 1 Kg - Coffee Kualitas Premium - Pahitnya pas c…",
"brand": "Zayn Snack",
"price": 7500.0,
"currency": "IDR",
"original_price": 120000.0,
"discount_pct": 93.75,
"rating": 4.7,
"review_count": null,
"in_stock": null,
"image_url": "https://p16-images-sign-sg.tokopedia-static.net/tos-alisg-i-aphluv4xwc-sg/ed86f8de7b064…",
"category": null,
"price_source": "dom",
"page": 1,
"position": 2,
"shop_slug": "zayn-snack-448",
"shop_location": "Kab. Magetan",
"sold": 100000,
"sold_is_floor": true,
"slug_id": "1731177319241910164",
"product_id": null,
"shop_id": null,
"condition": null,
"weight_grams": null,
"stock_max": null,
"listed_at": null,
"category_url": null
},
{
"source": "tokopedia.com",
"scraped_at": "2026-09-10T10:23:34.310048+00:00",
"url": "https://www.tokopedia.com/gudang-grosir-meymey/kopi-good-day-1-renceng-10-sachet-all-va…",
"sku": "/gudang-grosir-meymey/kopi-good-day-1-renceng-10-sachet-all-varian-rasa-original-chococ…",
"title": "Kopi Good Day 1 Renceng (10 Sachet) All Varian Rasa Original / Chococinno / Mocacinno /…",
"brand": "Gudang Grosir Meymey",
"price": 20499.0,
"currency": "IDR",
"original_price": null,
"discount_pct": null,
"rating": 4.9,
"review_count": null,
"in_stock": null,
"image_url": "https://p16-images-sign-sg.tokopedia-static.net/tos-alisg-i-aphluv4xwc-sg/04cf0e117af74…",
"category": null,
"price_source": "dom",
"page": 1,
"position": 17,
"shop_slug": "gudang-grosir-meymey",
"shop_location": "Kab. Sukoharjo",
"sold": 3000,
"sold_is_floor": true,
"slug_id": "1735564139188488006",
"product_id": null,
"shop_id": null,
"condition": null,
"weight_grams": null,
"stock_max": null,
"listed_at": null,
"category_url": null
}
]First 2 of 6 rows from a real run — sample_output.json. Long values shortened here.
Handles what breaks naive Tokopedia scrapers
Search pagination done right
?page=2 on a search URL silently empties the result set. Search is scrolled instead and ends on data, while category listings use ?page=N, so a one-page result is never reported as complete.
Sold counts that say what they are
A tile prints "100rb+ terjual" for a product that sold 207,785; sold_is_floor marks the floor so listing and product runs are not confused.
Placeholders are not images
Lazy-loaded tiles carry placeholder images; a real product image is recognised positively and everything else is null, rather than a column that is full and half wrong.
Hubs are not grids
A one-segment /p/ discovery hub carries ~39 carousel links; scoped to the grid it correctly returns zero products instead of 39 rows of filler.
Scraping infrastructure included
The same tested core runs every 2scraper repository, so the hard parts work the same way on every site.
CAPTCHA handling
Detects the challenge a page really serves and, with a 2Captcha key, solves it only when it blocks the data — a capped number of solves per page.
Proxy rotation
Single proxy or a pool from a file, rotated on blocks with a fresh browser per exit. Credentials stay out of argv and logs.
Fingerprints
Optional complete device identity from the 2Captcha Fingerprint API — user agent, client hints, locale and timezone that agree with each other.
Scraping Browser API
Point --cdp-endpoint at a managed Chromium with its own exit country — no browser infrastructure to run yourself.
Honest run metadata
Every run writes a .meta.json sidecar and distinct exit codes: complete, blocked, empty and partial are never confused, and a failed run never overwrites good data.
Diff two runs
diff_runs.py compares two complete Tokopedia snapshots by id — new, removed and changed rows for monitoring.
What teams do with Tokopedia data
Indonesian price monitoring
Track IDR prices and discounts for a category or search term, diffing runs by sku.
Best-seller discovery
Rank products by sold count and rating across a search grid, then fetch product pages for exact figures.
Seller landscape mapping
Map shops and their locations across a category to see who sells what and from where.
Discount analysis
Measure how deep markdowns run, using discounts computed from was-price and price rather than the site's badge.
Free to run. Pay only for what the site requires.
Needs a residential exit or the Scraping Browser API
Tokopedia ignores datacentre addresses entirely: the connection is reset with no status code, for curl and real Chromium alike (measured Sep 2026). Any residential exit clears it, and every measurement in the README was taken through the Scraping Browser API. Browser-plus-residential-proxy did not work in testing for reasons not yet understood, so the README does not recommend it. Captcha solving is wired in, but no challenge has been observed on this site.
Optional 2Captcha services — captcha solving, the Scraping Browser API, residential proxies and fingerprints — work behind one key.
Good to know
- No shop-front mode, no reviews, and no GraphQL API access are implemented.
- The site publishes no result total for search.
- image_url fills only for tiles the lazy loader reached.
2scraper vs. hosted scraper APIs
| 2scraper | Typical hosted API | |
|---|---|---|
| Code | Open source, MIT — read and patch it | Closed |
| Cost | Free; optional services billed only when used | Subscription or per request / per result |
| Runs on | Your machine, server or CI | The vendor's infrastructure |
| Output | JSON/CSV files + run metadata | API responses |
| When the site changes | Fix it yourself or open an issue / PR | Wait for the vendor |
Tokopedia scraper FAQ
What do I need to run it?
A residential exit. The tested path is the Scraping Browser API via --cdp-endpoint, set as TOKOPEDIA_CDP_ENDPOINT in .env. The exit country does not change the data.
Does the exit country change prices or language?
No. Indonesian and US exits got the same Indonesian-language site in IDR with zero price differences across 68 shared products.
Why does --pages behave differently on search?
Search has no per-page addresses, so --pages scrolls further instead, and --concurrency above 1 is refused for search URLs.
Why do some category rows lack rating and sold count?
Category tiles do not print them. Search grids and --mode product do.
Which formats are supported?
JSON, CSV or both, plus <out>.meta.json; a run that finds nothing does not overwrite earlier output unless you pass --allow-empty.
Is scraping Tokopedia legal?
It reads public listing and product pages with no login. It is public data; follow the site's terms, the law, and keep your request rate reasonable.