2scraper
What the Hacker News scraper collects
Reads news.ycombinator.com pages: the front page and every other story list, one item's entire comment tree, or a user profile. Each mode writes JSON and CSV plus a sidecar with the run's status, and keeps what the official API does not: the rank and page a story held when you looked.
Story lists
Input: A feed (front page, newest, ask, show, jobs, best, active, one day's front), a user's submissions, a domain, or a list URLOne row per story: title, link, author, points, comment count, timestamp, domain, kind and the rank the site printed.
Comment trees
Input: An item id (a story, or a single comment)One row per comment in the whole tree: author, timestamp, depth, parent id, status (ok, flagged, dead, deleted), plain text and full link addresses.
User profiles
Input: A usernameOne row per profile: creation date, karma, the about text and its links.
From zero to Hacker News data in 3 steps
Install
Clone the repository and install one browser engine. Python 3.9+.
$ git clone https://github.com/2scraper/hackernews-scraper && cd hackernews-scraper $ python3 -m venv .venv $ .venv/bin/pip install -r requirements.txt -r requirements-playwright.txt $ .venv/bin/playwright install chromium $ .venv/bin/python playwright_scraper.py --pages 3 --out front
Run a scraper
Pick what you need and run it. Every flag is documented in the README.
$ python playwright_scraper.py --url "https://news.ycombinator.com/newest" --pages 3$ python playwright_scraper.py --mode item --item 49901736$ python playwright_scraper.py --mode user --user dangGet structured data
Rows land in JSON or CSV (
--format), with a.meta.jsonsidecar recording status, pages fetched and why the run stopped.
Hacker News data fields
15 columns per row in a predictable schema. The most useful ones:
| Field | What it holds |
|---|---|
sku | The HN item id, as a string |
title | Story title |
url | Where the story points (its own item page for a text post) |
hn_url | The discussion page on news.ycombinator.com |
author | Submitter's username (null on a job posting) |
points | Score at the moment of the run |
comments | Comment count; 0 on a story showing "discuss" |
posted_at | The site's own timestamp, ISO 8601 UTC — never "9 hours ago" |
domain | The domain the site shows beside the title |
kind | story, ask, show, job or text |
rank | The rank the site printed, continuing across pages |
page | Which list page the row came from |
position | Position on that page; page + position is unique across a run |
parent_id | Comment mode: the item this comment replies to |
depth | Comment mode: the site's indent, 0 for top level |
karma | Profile mode: the user's karma |
[
{
"source": "news.ycombinator.com",
"scraped_at": "2026-09-30T08:43:27.752616+00:00",
"url": "https://github.com/ninjahawk/livenerf",
"sku": "49901736",
"title": "Livenerf: Has Opus 5.5 been nerfed yet?",
"author": "user_a",
"points": 536,
"comments": 230,
"posted_at": "2026-09-29T22:36:14Z",
"domain": "github.com/ninjahawk",
"kind": "story",
"rank": 1,
"hn_url": "https://news.ycombinator.com/item?id=49901736",
"page": 1,
"position": 1
},
{
"source": "news.ycombinator.com",
"scraped_at": "2026-09-30T08:43:27.752889+00:00",
"url": "https://exyr.org/2026/solving-factorio-quality/",
"sku": "49887343",
"title": "Solving Factorio Quality",
"author": "user_b",
"points": 29,
"comments": 4,
"posted_at": "2026-09-29T02:27:54Z",
"domain": "exyr.org",
"kind": "story",
"rank": 2,
"hn_url": "https://news.ycombinator.com/item?id=49887343",
"page": 1,
"position": 2
}
]First 2 of 5 rows from a real run — sample_output.json. Long values shortened here.
Handles what breaks naive Hacker News scrapers
Whole threads in one request
An item page holds every comment in one response, so a 1,104-comment thread came back in 12 seconds as a flat table whose story_id and parent_id rebuild the tree.
Cursor lists walked correctly
/newest, /jobs, submissions and domain lists paginate by a ?next= cursor, not ?p=N. The scraper compares page 1's More link with what ?p=2 would build and only parallelises lists that really are addressable.
Typos refused, not scraped
The site answers a malformed day with the latest one and ?p=0 with page 1, both HTTP 200. --day and p are validated first, so a typo never becomes a healthy-looking run of the wrong thing.
"Nothing there" is an answer
No such item, No such user and a future day come back as plain text with HTTP 200; the scraper reports them as zero rows (exit 4), not as a block.
Scraping infrastructure included
The same tested core runs every 2scraper repository, so the hard parts work the same way on every site.
Proxy rotation
Single proxy or a pool from a file, rotated on blocks with a fresh browser per exit. Credentials stay out of argv and logs.
Fingerprints
Optional complete device identity from the 2Captcha Fingerprint API — user agent, client hints, locale and timezone that agree with each other.
Scraping Browser API
Point --cdp-endpoint at a managed Chromium with its own exit country — no browser infrastructure to run yourself.
Honest run metadata
Every run writes a .meta.json sidecar and distinct exit codes: complete, blocked, empty and partial are never confused, and a failed run never overwrites good data.
Diff two runs
diff_runs.py compares two complete Hacker News snapshots by id — new, removed and changed rows for monitoring.
What teams do with Hacker News data
Front-page tracking
Record which stories held which rank, with points and comment counts, at the moment you looked — data the official API does not keep.
Discussion analysis
Pull an entire comment tree, with depth, status and full link addresses, for sentiment, topic or link analysis.
Launch monitoring
Follow Show HN, Ask HN or everything submitted from your domain, and diff two runs to see what is new.
Hiring signals
Read the jobs feed on a schedule to follow which YC companies are posting roles.
Free to run. Pay only for what the site requires.
Runs with no key, proxy or account
You need no key, no proxy and no account: every engine, and plain curl, was served from a datacentre VPS with no challenge on any page this scraper reads, and a canary runs from a GitHub runner with no secrets. This repo does not implement captcha solving, because there has been nothing to solve. The paid products buy more addresses than one if you run a lot, a browser you do not host (Scraping Browser API or the Scraper API), and a consistent device identity with --fingerprint.
Optional 2Captcha services — captcha solving, the Scraping Browser API, residential proxies and fingerprints — work behind one key.
Good to know
- author, points and comments are null on job postings, and rank is null on domain (/from) lists — the site prints none.
- Flagged comments carry the site's [flagged] placeholder, not the author's text.
- It never logs in, so every page is what a logged-out reader sees.
2scraper vs. hosted scraper APIs
| 2scraper | Typical hosted API | |
|---|---|---|
| Code | Open source, MIT — read and patch it | Closed |
| Cost | Free; optional services billed only when used | Subscription or per request / per result |
| Runs on | Your machine, server or CI | The vendor's infrastructure |
| Output | JSON/CSV files + run metadata | API responses |
| When the site changes | Fix it yourself or open an issue / PR | Wait for the vendor |
Hacker News scraper FAQ
Why does it wait 30 seconds between pages?
Hacker News's robots.txt asks for a Crawl-delay of 30 for every agent, and --delay defaults to 30 to honour it. You can lower it if you decide to.
Why not just use the official API?
Use the Firebase API or Algolia search when they answer your question. This scraper reads the pages, which carry what those do not: the rank and page a story held when you looked, and what the site chose to show or hide.
How deep does the front page go?
26 pages of 30 when measured: ?p=26 (ranks 751-780) is the last, and the run ends there as complete.
Do --pages and --concurrency work everywhere?
--concurrency works only for ?p=N lists such as the front page, show and best; cursor lists are walked link to link. --pages means nothing for an item, because one page holds every comment.
What formats does it write?
JSON and CSV plus a <out>.meta.json sidecar. Writes are atomic, a run that finds nothing writes nothing, and CSV cells that would be read as a spreadsheet formula are escaped in the CSV only.
Is it legal to scrape Hacker News?
It reads public pages only and never logs in. Follow the site's terms and the law in your jurisdiction.