2scraper
What the Medium scraper collects
Reads a Medium tag feed, a tag's day archive (up to 128 stories per day), an author's profile or a single story, and writes one row per story: title, subtitle, author, publication, dates, claps, responses, reading time, word count, paywall flag, tags and — for a single story — the full body text. Output is JSON and CSV with a run-metadata sidecar.
Tag feed
Input: A /tag/{slug} URLThe stories a tag feed ships in one fetch (53 on /tag/python, Sep 2026), with title, author, claps and tags.
Tag day archive
Input: A /tag/{slug}/archive/{yyyy}/{mm}/{dd} URLUp to 128 stories per day with reading time, word count and language; walk several days with --pages, in parallel with --concurrency.
Author page
Input: A /@{username} URLOne row per story on the writer's profile with claps, responses and reading time.
Single story
Input: A /p/{id} URL or any story URLOne row for the story including its full text in content and the character count.
From zero to Medium data in 3 steps
Install
Clone the repository and install one browser engine. Python 3.9+.
$ git clone https://github.com/2scraper/medium-scraper.git && cd medium-scraper $ python3 -m venv .venv && . .venv/bin/activate $ pip install -r requirements.txt -r requirements-playwright.txt $ playwright install chromium
Run a scraper
Pick what you need and run it. Every flag is documented in the README.
$ python3 playwright_scraper.py --url "https://medium.com/tag/python"$ python3 playwright_scraper.py --url "https://medium.com/tag/python/archive/2026/09/10" --pages 5 --concurrency 3 --delay 20$ python3 playwright_scraper.py --url "https://medium.com/@quincylarson"$ python3 playwright_scraper.py --url "https://medium.com/p/d373fe2c96b7" --format jsonGet structured data
Rows land in JSON or CSV (
--format), with a.meta.jsonsidecar recording status, pages fetched and why the run stopped.
Medium data fields
28 columns per row in a predictable schema. The most useful ones:
| Field | What it holds |
|---|---|
sku | Medium's own 12-hex post id — survives renames and moves |
title | Story title |
subtitle | Story subtitle |
author | Author display name |
author_username | Author handle |
publication | Publication the story appears in, if any |
published_at | First publication time |
claps | Clap count |
responses | Number of responses |
reading_time_min | Estimated reading time in minutes |
word_count | Word count |
is_paywalled | Member-only story flag |
language | Story language |
tags | List of story tags |
content | Full story text (single-story mode) |
data_source | Which view built the row: obvinit, apollo, jsonld or dom |
[
{
"source": "medium.com",
"scraped_at": "2026-09-16T11:38:11.223347+00:00",
"url": "https://medium.com/@dailypythonpro/i-tried-python-3-15s-lazy-imports-here-s-what-happen…",
"sku": "bc8b23e21c66",
"title": "I Tried Python 3.15’s Lazy Imports. Here’s What Happened",
"subtitle": "What I learned about startup time, deferred work, and the traps you shouldn’t ignore.",
"canonical_url": null,
"author": "Daily Python",
"author_username": "dailypythonpro",
"author_url": "https://medium.com/@dailypythonpro",
"publication": null,
"publication_url": null,
"published_at": "2026-09-12T19:37:42.037000+00:00",
"updated_at": "2026-09-12T19:37:42.037000+00:00",
"claps": 73,
"responses": 1,
"reading_time_min": 13.93,
"word_count": 3333,
"is_paywalled": true,
"is_series": false,
"language": "en",
"tags": [
"python",
"python-programming",
"python3",
"… 2 more"
],
"preview_image_url": "https://miro.medium.com/v2/resize:fit:1200/1*mj0tV8VSIYZ0ZnykVXGVlA.png",
"content": null,
"content_chars": null,
"data_source": "obvinit",
"page": 1,
"position": 1
},
{
"source": "medium.com",
"scraped_at": "2026-09-16T11:38:11.223371+00:00",
"url": "https://medium.com/@mayursurani/building-a-production-grade-end-to-end-data-engineering…",
"sku": "a268c4aefacc",
"title": "Building a Production-Grade End-to-End Data Engineering Project: Agentic Databases, Dat…",
"subtitle": "End to End Production grade databrick + DBT + Agentic Database + Airflow Project",
"canonical_url": null,
"author": "Mayurkumar Surani",
"author_username": "mayursurani",
"author_url": "https://medium.com/@mayursurani",
"publication": null,
"publication_url": null,
"published_at": "2026-09-12T08:47:46.149000+00:00",
"updated_at": "2026-09-12T08:47:46.149000+00:00",
"claps": 21,
"responses": 0,
"reading_time_min": 12.95,
"word_count": 3212,
"is_paywalled": true,
"is_series": false,
"language": "en",
"tags": [
"databricks",
"airflow",
"agentic-ai",
"… 2 more"
],
"preview_image_url": "https://miro.medium.com/v2/resize:fit:1200/1*dCzIMpQo_yqAUINUHfhugg.png",
"content": null,
"content_chars": null,
"data_source": "obvinit",
"page": 1,
"position": 2
}
]First 2 of 6 rows from a real run — sample_output.json. Long values shortened here.
Handles what breaks naive Medium scrapers
Gets past the HeadlessChrome block
Medium's Cloudflare hard-blocks the HeadlessChrome token in the User-Agent, not headless browsing. These engines build their UA from the browser's own version without that token (11 of 11 blocked vs 0 of 11, measured Sep 2026).
Two renderers, both read
Tag feeds and author pages ship Apollo state; day archives ship a legacy payload with 84 fields per story. data_source records which view built each row, so a null reading time on a tag feed is the site, not a failed read.
Managed challenge retried in a fresh context
Cloudflare's transient 'Just a moment' page cleared on the next attempt in a new browser context in 9 of 9 cases, so --retries opens a fresh context and no solve is ever paid for.
Traps refused with the reason
Publication home pages ship post stubs with no data, ?page=2 on a tag feed is ignored, and an empty archive day redirects to its month — each is detected rather than written as plausible rows.
Scraping infrastructure included
The same tested core runs every 2scraper repository, so the hard parts work the same way on every site.
CAPTCHA handling
Detects the challenge a page really serves and, with a 2Captcha key, solves it only when it blocks the data — a capped number of solves per page.
Proxy rotation
Single proxy or a pool from a file, rotated on blocks with a fresh browser per exit. Credentials stay out of argv and logs.
Fingerprints
Optional complete device identity from the 2Captcha Fingerprint API — user agent, client hints, locale and timezone that agree with each other.
Scraping Browser API
Point --cdp-endpoint at a managed Chromium with its own exit country — no browser infrastructure to run yourself.
Honest run metadata
Every run writes a .meta.json sidecar and distinct exit codes: complete, blocked, empty and partial are never confused, and a failed run never overwrites good data.
Diff two runs
diff_runs.py compares two complete Medium snapshots by id — new, removed and changed rows for monitoring.
What teams do with Medium data
Topic and trend research
Walk a tag's day archives to collect every story published on a topic, with claps and responses as engagement signals.
Author and competitor tracking
Follow writers' output, reading times and engagement over time and diff runs by post id.
Content datasets
Collect full story text with tags and language for NLP corpora or content analysis.
Paywall and publication analysis
Measure the share of member-only stories per tag or publication using is_paywalled and publication.
Free to run. Pay only for what the site requires.
Runs with no key, proxy or account
No key and no proxy needed: every number in the README was measured with Playwright's bundled Chromium from an ordinary datacentre address. What 2Captcha's products add here is not getting in: the Scraper API returns identical rows with no browser at $0.0005 a request, and the Scraping Browser API let an author page scroll to 55 rows against 10 locally (Sep 2026). For long archive walks the limit is rate — --delay first, rotating proxies to scale.
Optional 2Captcha services — captcha solving, the Scraping Browser API, residential proxies and fingerprints — work behind one key.
Good to know
- --pages and --concurrency only apply to day archives; tag feeds and author pages are one fetch.
- Concurrency is implemented in the Playwright engine only.
- towardsdatascience.com moved off Medium and is refused.
2scraper vs. hosted scraper APIs
| 2scraper | Typical hosted API | |
|---|---|---|
| Code | Open source, MIT — read and patch it | Closed |
| Cost | Free; optional services billed only when used | Subscription or per request / per result |
| Runs on | Your machine, server or CI | The vendor's infrastructure |
| Output | JSON/CSV files + run metadata | API responses |
| When the site changes | Fix it yourself or open an issue / PR | Wait for the vendor |
Medium scraper FAQ
Do I need a proxy or a 2Captcha key?
No. All README measurements were taken with no key and no proxy. Proxies help only when a long archive walk hits Medium's rate limit.
How do I get many stories?
Use --mode archive on a tag's day archive: up to 128 stories per day, walkable over several days with --pages and --concurrency.
Why are reading time and word count null on some rows?
A tag feed's payload does not contain them. Archive, author and single-story rows carry them; data_source shows which view built the row.
Can it scrape a publication's home page?
No — it ships post ids with no data. The scraper refuses it and suggests archive mode on a tag the publication uses, or post mode on story URLs.
Does it get the full article text?
Yes, in single-story mode: content holds the body and content_chars its length.
Is scraping Medium legal?
It reads public pages. Follow Medium's terms and the law where you operate, and respect authors' rights over their text.