2scraper
What the Weibo scraper collects
Scrapes weibo.com in three modes: the public hot feed, one account's posts walked by cursor, and one post's comments. Truncated long posts are re-fetched in full, and engagement counts are read as integers rather than the site's display text. All modes write the same 36 columns to JSON or CSV, through a browserless client or Playwright, Selenium or Puppeteer.
Hot feed
Input: Nothing: the public recommended feedPosts from Weibo's public hot timeline with author, text, engagement counts, pictures and topics. The feed re-rolls, so --pages N means N fetches of it.
Account posts
Input: A weibo.com profile URLOne account's posts walked by the cursor the site hands back, with full text recovered for long posts.
Post comments
Input: A weibo.com post URLOne post's comments, followed by max_id continuation, with comment and reply ids.
From zero to Weibo data in 3 steps
Install
Clone the repository and install one browser engine. Python 3.9+.
$ git clone https://github.com/2scraper/weibo-scraper.git && cd weibo-scraper $ pip install -r requirements.txt $ python weibo_api.py --mode hot --pages 3 --format both --out weibo_posts
Run a scraper
Pick what you need and run it. Every flag is documented in the README.
$ python weibo_api.py --mode hot --pages 3 --format both --out weibo_posts$ python playwright_scraper.py --mode user --url https://weibo.com/u/2803301701 --pages 5$ python playwright_scraper.py --mode post --url https://weibo.com/2803301701/RiPCAfklUGet structured data
Rows land in JSON or CSV (
--format), with a.meta.jsonsidecar recording status, pages fetched and why the run stopped.
Weibo data fields
36 columns per row in a predictable schema. The most useful ones:
| Field | What it holds |
|---|---|
sku | The post's mblogid, the base-62 id in its URL |
mid | Numeric post id, needed to fetch comments |
title | The post body (posts have no title) |
text_source | Which read produced the text, e.g. longtext |
text_truncated | True only when the full text could not be fetched |
created_at | Post time |
author_id | Author's user id |
author_name | Author's screen name |
author_verified | Whether the author is verified |
reposts_count | Reposts, as an integer |
comments_count | Comments, as an integer |
attitudes_count | Likes, as an integer |
region | Posting province, prefix removed; null when the site does not say |
pic_urls | Picture URLs |
topics | Hashtag topics |
repost_of_sku | Original post's id when the row is a repost |
[
{
"source": "weibo.com",
"scraped_at": "2026-09-21T08:26:23.336567+00:00",
"url": "https://weibo.com/900000001/RiZDX9H6U",
"sku": "RiZDX9H6U",
"title": "示例正文内容。示例正文内容。示例正文内容。示例正文内容。示例正文内容。示例正文内容。示例正文内容。示例正文内容。示例正文内容。示例正文内容。示例正文内容。示例正文内容。示例正…",
"text_source": "longtext",
"text_truncated": false,
"text_length_claimed": 1776,
"created_at": "2026-09-20T17:00:19+08:00",
"created_at_raw": "Sun Sep 20 17:00:19 +0800 2026",
"author_id": "900000001",
"author_name": "示例用户1",
"author_url": "https://weibo.com/u/900000001",
"author_verified": true,
"author_verified_type": 7,
"author_followers": null,
"reposts_count": 3858,
"comments_count": 3137,
"attitudes_count": 34799,
"region": "重庆",
"client": null,
"pic_count": 2,
"pic_urls": [
"https://wx1.example/1_0.jpg",
"https://wx1.example/1_1.jpg"
],
"video_url": null,
"topics": [
"Fly回归重庆狼队"
],
"links": [],
"is_long_text": true,
"is_ad": false,
"repost_of_sku": null,
"repost_of_author": null,
"mid": "5345268652310672",
"parent_sku": null,
"comment_id": null,
"reply_to_comment_id": null,
"page": 1,
"position": 1
},
{
"source": "weibo.com",
"scraped_at": "2026-09-21T08:26:23.336615+00:00",
"url": "https://weibo.com/900000002/Rj2jIdDhS",
"sku": "Rj2jIdDhS",
"title": "示例正文内容。示例正文内容。",
"text_source": "inline",
"text_truncated": false,
"text_length_claimed": 71,
"created_at": "2026-09-20T23:48:42+08:00",
"created_at_raw": "Sun Sep 20 23:48:42 +0800 2026",
"author_id": "900000002",
"author_name": "示例用户2",
"author_url": "https://weibo.com/u/900000002",
"author_verified": true,
"author_verified_type": 0,
"author_followers": null,
"reposts_count": 14427,
"comments_count": 10024,
"attitudes_count": 49360,
"region": "中国澳门",
"client": null,
"pic_count": 6,
"pic_urls": [
"https://wx1.example/2_0.jpg",
"https://wx1.example/2_1.jpg",
"https://wx1.example/2_2.jpg",
"… 3 more"
],
"video_url": null,
"topics": [
"2026湾区升明月",
"2026湾区升明月大湾区电影音乐晚会"
],
"links": [],
"is_long_text": false,
"is_ad": false,
"repost_of_sku": null,
"repost_of_author": null,
"mid": "5345371423249288",
"parent_sku": null,
"comment_id": null,
"reply_to_comment_id": null,
"page": 1,
"position": 2
}
]First 2 of 5 rows from a real run — sample_output.json. Long values shortened here.
Handles what breaks naive Weibo scrapers
Long posts recovered in full
Posts flagged isLongText arrive cut at about 150 characters with nothing saying so. The scraper fetches the long text for each one; on a live hot feed, two posts went from 149 and 146 characters to 1,034 and 1,235.
Real counts, not the display rule
Every post carries "100万+" as the site's rule for printing large numbers. The scraper reads the integer counts beside it, never that string.
API reads that Weibo accepts
Navigating straight to Weibo's API answers 403. The engines land on weibo.com once and read every payload as an XHR from that page, the way the site's own front end does.
A re-rolling feed is labelled as one
The hot feed has no page 2. The sidecar records feed_not_addressable and feed_rerolled, and account runs record the post count the account claims beside the rows read.
Scraping infrastructure included
The same tested core runs every 2scraper repository, so the hard parts work the same way on every site.
CAPTCHA handling
Detects the challenge a page really serves and, with a 2Captcha key, solves it only when it blocks the data — a capped number of solves per page.
Proxy rotation
Single proxy or a pool from a file, rotated on blocks with a fresh browser per exit. Credentials stay out of argv and logs.
Fingerprints
Optional complete device identity from the 2Captcha Fingerprint API — user agent, client hints, locale and timezone that agree with each other.
Scraping Browser API
Point --cdp-endpoint at a managed Chromium with its own exit country — no browser infrastructure to run yourself.
Honest run metadata
Every run writes a .meta.json sidecar and distinct exit codes: complete, blocked, empty and partial are never confused, and a failed run never overwrites good data.
Diff two runs
diff_runs.py compares two complete Weibo snapshots by id — new, removed and changed rows for monitoring.
What teams do with Weibo data
Chinese social listening
Track what trends on Weibo's public hot feed, with topics, regions and engagement counts.
Brand and KOL monitoring
Collect an account's posts with reposts, comments and likes to follow its reach over time.
Comment sentiment analysis
Pull a post's comment thread for sentiment or topic modelling.
Media and research archives
Keep full-text copies of public posts, including the long posts other tools truncate.
Free to run. Pay only for what the site requires.
Runs with no key, proxy or account
No account, API key or proxy, and no Chinese exit IP: the three modes were measured from a datacentre address in Finland with no geographic block and no rate-limit refusals (Sep 2026). Search and some timeline routes sit behind a login wall, and this repo does not implement account login. The Scraping Browser API buys a residential exit and no local browser, and proxies buy volume from many addresses; no captcha was met on the routes this repo reads.
Optional 2Captcha services — captcha solving, the Scraping Browser API, residential proxies and fingerprints — work behind one key.
Good to know
- Search, mymblog and m.weibo.cn routes need a logged-in account, which this repo does not implement.
- An account run ends where Weibo's cursor ends, often far short of the account's claimed post count.
- The Scraper API path was measured getting 403 on Weibo's API URLs; use the HTTP client or a browser engine.
2scraper vs. hosted scraper APIs
| 2scraper | Typical hosted API | |
|---|---|---|
| Code | Open source, MIT — read and patch it | Closed |
| Cost | Free; optional services billed only when used | Subscription or per request / per result |
| Runs on | Your machine, server or CI | The vendor's infrastructure |
| Output | JSON/CSV files + run metadata | API responses |
| When the site changes | Fix it yourself or open an issue / PR | Wait for the vendor |
Weibo scraper FAQ
Do I need a Chinese proxy?
No. Weibo served these routes to a Finnish datacentre address. The routes that refuse do so because they need a logged-in account, which no exit address changes.
Do I need a browser?
No. weibo_api.py runs with two libraries and no Chromium. Use a browser engine when you want --cdp-endpoint or a fingerprint.
Can it search Weibo?
Not yet. s.weibo.com redirects to the login page, and this repo does not implement account login.
Why does --pages 3 on the hot feed not give pages 1 to 3?
The hot feed has no page 2; it re-rolls on every request. --pages 3 means three fetches of a moving feed, and --concurrency is refused in that mode.
What output formats are there?
JSON or CSV (or both) with 36 columns, plus a .meta.json sidecar. Writes are atomic, CSV neutralises formula-like cells, and a run that finds nothing writes nothing.
Is scraping Weibo legal?
It reads content Weibo serves publicly to an anonymous visitor and never logs in or posts. Read Weibo's terms and your local law before running it at volume.