2scraper
What the Rakuten Ichiba scraper collects
Scrapes Rakuten Ichiba keyword searches, genre listings and single item pages into 42 columns: price in JPY, Rakuten points and point rate, shipping fee, delivery estimate, shop, genre, review score and count, and per-variant prices on item pages. Output is JSON or CSV with a sidecar that records the site's own result cap.
Keyword search
Input: A search.rakuten.co.jp/search/mall/{keyword}/ URL45 products per page with price, points, shipping, shop, genre, rating and review count; the site's own filters in the URL survive pagination.
Genre listing
Input: A genre URL by Rakuten's genre idEvery product in a Rakuten genre, paginated with ?p=N, with the same columns as a search.
Single item
Input: An item.rakuten.co.jp/{shop}/{code}/ URLOne item with per-variant prices, a variant count and a was-price only when Rakuten marks it verified.
Filtered search
Input: A search URL with price-band or other filtersNarrows a query with Rakuten's own filters — the way to reach more of a result set the site caps at 150 pages.
From zero to Rakuten Ichiba data in 3 steps
Install
Clone the repository and install one browser engine. Python 3.9+.
$ git clone https://github.com/2scraper/rakuten-scraper.git && cd rakuten-scraper $ python3 -m venv .venv && . .venv/bin/activate $ pip install -r requirements.txt -r requirements-playwright.txt $ playwright install chromium
Run a scraper
Pick what you need and run it. Every flag is documented in the README.
$ python3 playwright_scraper.py --url "https://search.rakuten.co.jp/search/mall/コーヒー/" --pages 3$ python3 playwright_scraper.py --url "https://www.rakuten.co.jp/category/100356/" --pages 5 --format csv$ python3 playwright_scraper.py --mode product --url "https://item.rakuten.co.jp/sawaicoffee-tea/solandluna/"$ python3 playwright_scraper.py --url "https://search.rakuten.co.jp/search/mall/コーヒー/?min=1000&max=3000" --pages 3Get structured data
Rows land in JSON or CSV (
--format), with a.meta.jsonsidecar recording status, pages fetched and why the run stopped.
Rakuten Ichiba data fields
42 columns per row in a predictable schema. The most useful ones:
| Field | What it holds |
|---|---|
sku | {shopCode}:{manageNumber} — Rakuten's own item code, the same on listing and item pages |
title | Item title as the merchant writes it |
price | Price as a number |
currency | JPY, as stated by the site's own page-1 JSON-LD |
points | Rakuten points earned on purchase |
point_rate | Point multiplier (e.g. 10 for a 10x campaign) |
shipping_fee | Shipping fee; 0 when free |
delivery_estimate | The site's delivery estimate text |
subscription_price | Subscription (定期購入) price where offered |
price_max | Highest variant price when variants differ |
rating | Review score; null on unreviewed items |
review_count | Number of reviews |
shop_name | Merchant selling the item |
brand | Brand from Rakuten's own brand tag |
category | Genre path as text |
genre_rank | The item's rank in its genre ranking, where published |
[
{
"source": "rakuten.co.jp",
"scraped_at": "2026-09-21T09:22:22.033307+00:00",
"url": "https://item.rakuten.co.jp/ajinomoto/r4901111371248/",
"sku": "ajinomoto:r4901111371248",
"title": "【9/24まで P10倍！】味の素AGF ブレンディ スティックブラック100本コーヒー スティックコーヒー インスタントコーヒー 簡単 大容量 まとめ買い AGF ギフト お中元",
"brand": "味の素AGF",
"price": 2447.0,
"currency": "JPY",
"original_price": null,
"discount_pct": null,
"rating": 4.76,
"review_count": 475,
"in_stock": true,
"image_url": "https://thumbnail.image.rakuten.co.jp/@0_mall/ajinomoto/cabinet/prd/48/r4901111371248_2…",
"category": "水・ソフトドリンク > コーヒー > インスタントコーヒー",
"price_source": "state",
"page": 1,
"position": 1,
"variant_id": "r4901111371248",
"item_number": "10000338",
"price_max": 6975.0,
"has_price_range": true,
"subscription_price": 2324.0,
"points": 220,
"point_rate": 10,
"shipping_fee": 0.0,
"free_shipping": true,
"delivery_estimate": "最短9/23お届け",
"shop_name": "味の素グループ公式ショップ",
"shop_code": "ajinomoto",
"shop_id": 383220,
"shop_url": "https://www.rakuten.co.jp/ajinomoto/",
"genre_id": 303017,
"genre_path": "/0/100316/100356/303017",
"genre_rank": 2,
"is_super_deal": false,
"is_39shop": true,
"is_official_shop": true,
"variant_count": null,
"original_price_label": null
},
{
"source": "rakuten.co.jp",
"scraped_at": "2026-09-21T09:22:22.033563+00:00",
"url": "https://item.rakuten.co.jp/sawaicoffee-tea/ac-sale-1960/",
"sku": "sawaicoffee-tea:ac-sale-1960",
"title": "＼先着300円OFFクーポン配布／ 送料無料 コーヒー コーヒー豆 コーヒー粉 珈琲豆 豆のまま \"グルメ大賞9度受賞の1番選ばれるコーヒーセット\" 澤井珈琲 1kg 2kg…",
"brand": "澤井珈琲",
"price": 2788.0,
"currency": "JPY",
"original_price": null,
"discount_pct": null,
"rating": 4.7,
"review_count": 50175,
"in_stock": true,
"image_url": "https://thumbnail.image.rakuten.co.jp/@0_mall/sawaicoffee-tea/cabinet/thum/ff26/ac-sale…",
"category": "水・ソフトドリンク > コーヒー > コーヒー豆",
"price_source": "state",
"page": 1,
"position": 2,
"variant_id": "ac-sale-1960-80-as-m",
"item_number": "10001136",
"price_max": 8888.0,
"has_price_range": true,
"subscription_price": 2509.0,
"points": 50,
"point_rate": 2,
"shipping_fee": 0.0,
"free_shipping": true,
"delivery_estimate": "最短9/23お届け",
"shop_name": "澤井珈琲Beans&Leaf",
"shop_code": "sawaicoffee-tea",
"shop_id": 198667,
"shop_url": "https://www.rakuten.co.jp/sawaicoffee-tea/",
"genre_id": 201402,
"genre_path": "/0/100316/100356/201402",
"genre_rank": 1,
"is_super_deal": false,
"is_39shop": true,
"is_official_shop": false,
"variant_count": null,
"original_price_label": null
}
]First 2 of 9 rows from a real run — sample_output.json. Long values shortened here.
Handles what breaks naive Rakuten Ichiba scrapers
Skips the decoy JSON-LD
A search page's only JSON-LD holds 10 SEO-carousel items while the page holds 45 products. The scraper reads the page's own __INITIAL_STATE__ payload instead, using JSON-LD only for the currency.
Detects page-1 repeats past the end
Rakuten serves page 1 again for out-of-range ?p= values with HTTP 200. Every page is checked against the offset the server says it used, so a run never fills up with duplicates.
Sponsored slots filtered
Paid placements injected into the result list are dropped and position counts emitted rows, so the same listing gives the same positions in every engine.
Points tracked, throttles waited out
A 10x point campaign is a discount that never touches price, so diff_runs.py tracks points too. A 503 throttle gets a wait at the same exit rather than being reported as a block.
Scraping infrastructure included
The same tested core runs every 2scraper repository, so the hard parts work the same way on every site.
CAPTCHA handling
Detects the challenge a page really serves and, with a 2Captcha key, solves it only when it blocks the data — a capped number of solves per page.
Proxy rotation
Single proxy or a pool from a file, rotated on blocks with a fresh browser per exit. Credentials stay out of argv and logs.
Fingerprints
Optional complete device identity from the 2Captcha Fingerprint API — user agent, client hints, locale and timezone that agree with each other.
Scraping Browser API
Point --cdp-endpoint at a managed Chromium with its own exit country — no browser infrastructure to run yourself.
Honest run metadata
Every run writes a .meta.json sidecar and distinct exit codes: complete, blocked, empty and partial are never confused, and a failed run never overwrites good data.
Diff two runs
diff_runs.py compares two complete Rakuten Ichiba snapshots by id — new, removed and changed rows for monitoring.
What teams do with Rakuten Ichiba data
Japanese e-commerce price monitoring
Track prices, points and shipping for a product set across the many shops that sell the same article.
Effective-price analysis
Combine price, point_rate and shipping_fee to see the real cost of an item through a Super Sale or point campaign.
Merchant and assortment research
Map which shops sell in a genre, how many are official stores, and how they rank and are reviewed.
Market entry and category sizing
Use the site's own result totals and genre listings to gauge category depth and price bands in Japan.
Free to run. Pay only for what the site requires.
Runs with no key, proxy or account
Listings and item pages need no key, proxy or account: measured 2026-09-21 from a datacentre address, a 3-page run returned 135 rows, all priced, and a daily canary runs from a bare GitHub runner. What Rakuten refuses is a client whose claimed identity contradicts itself, so do not add a bare user agent. The paid products buy specific things: residential proxies for volume (avoiding the 503 throttle), the Scraping Browser API for a chosen exit country without local browsers, a complete fingerprint for a consistent identity, and the Scraper API for one HTTP request per page.
Optional 2Captcha services — captcha solving, the Scraping Browser API, residential proxies and fingerprints — work behind one key.
Good to know
- Capped by the site at 150 pages of 45 per query; slice large queries with filters.
- ranking.rakuten.co.jp and merchant storefront pages are not implemented.
- in_stock is true on every listing row because search excludes sold-out items.
2scraper vs. hosted scraper APIs
| 2scraper | Typical hosted API | |
|---|---|---|
| Code | Open source, MIT — read and patch it | Closed |
| Cost | Free; optional services billed only when used | Subscription or per request / per result |
| Runs on | Your machine, server or CI | The vendor's infrastructure |
| Output | JSON/CSV files + run metadata | API responses |
| When the site changes | Fix it yourself or open an issue / PR | Wait for the vendor |
Rakuten Ichiba scraper FAQ
Do I need a proxy for Rakuten?
No. Listings and item pages were served to a plain datacentre address in the README's tests. Proxies help with volume if you hit the 503 throttle; raising --delay is the first fix.
How many results can I get per query?
Rakuten serves at most 6,750 results per query — 150 pages of 45 — however many it matched. The sidecar records the site's total and the cap; narrow with filters to reach more.
Why is original_price empty on listings?
The listing payload publishes no was-price. Item pages carry one only when Rakuten marks it verified, and product mode reads it from there.
Does --locale en translate products?
No. ?lang=en changes the navigation chrome; product names, shop names and tags are merchant-authored Japanese and come back unchanged.
Which URLs are supported?
Keyword searches, genre listings and item pages. Merchant storefront pages are refused with the reason, and ranking pages are not implemented.
Is scraping Rakuten legal?
It reads public pages, uses a delay between pages and never logs in or touches checkout. Follow the site's terms and the law where you operate.