2scraper
What the YouTube scraper collects
Scrapes YouTube comment threads and replies, video metadata and video search, read from the same endpoint YouTube's own front end calls. One row per comment, in the order the site sent them, with likes, reply counts, pinned and creator-hearted flags and the sort it came from; output is JSON or CSV with a run sidecar.
Comments and replies
Input: A video URL or video idOne row per comment, top or newest first, with likes, reply count, author flags, pinned and creator-hearted; --replies adds reply threads.
Video metadata
Input: A video URL or video idOne row per video with the exact like count, exact upload date, duration, category and keywords.
Video search
Input: A search queryVideos matching a query, to feed into --mode comments.
From zero to YouTube data in 3 steps
Install
Clone the repository and install one browser engine. Python 3.9+.
$ git clone https://github.com/2scraper/youtube-scraper.git $ cd youtube-scraper $ python3 -m venv .venv && ./.venv/bin/pip install -r requirements.txt $ ./.venv/bin/python youtube_scraper.py --url "https://www.youtube.com/watch?v=dQw4w9WgXcQ" --pages 3
Run a scraper
Pick what you need and run it. Every flag is documented in the README.
$ python3 youtube_scraper.py --url dQw4w9WgXcQ --pages 5 --sort newest --replies$ python3 youtube_scraper.py --url dQw4w9WgXcQ --mode video$ python3 youtube_scraper.py --url "web scraping tutorial" --mode search --pages 3Get structured data
Rows land in JSON or CSV (
--format), with a.meta.jsonsidecar recording status, pages fetched and why the run stopped.
YouTube data fields
32 columns per row in a predictable schema. The most useful ones:
| Field | What it holds |
|---|---|
sku | Comment id (a reply's id contains its parent's) |
video_id | The video the comment belongs to |
title | The video's title, repeated on every row |
text | Comment text |
author_name | Commenter's handle |
author_is_verified | Whether the commenter is verified |
like_count | Likes as a number (approximate: the site publishes "318K") |
like_count_text | The like count exactly as the site wrote it |
reply_count | Number of replies |
published_time_text | Relative time as the site wrote it, e.g. "1 year ago" |
published_at_approx | Latest instant the comment could have been written |
published_at_precision | How precise the derived date is (e.g. year) |
parent_id | Parent comment id for replies |
is_pinned | Whether the comment is pinned |
creator_hearted | Whether the creator hearted it |
sort | Which ordering the row came from: top or newest |
[
{
"source": "youtube.com",
"scraped_at": "2026-09-21T08:43:43Z",
"url": "https://www.youtube.com/watch?v=dQw4w9WgXcQ&lc=FIXTURE_CMT_1",
"sku": "FIXTURE_CMT_1",
"title": "Rick Astley - Never Gonna Give You Up (Official Video) (4K Remaster)",
"video_id": "dQw4w9WgXcQ",
"text": "fix turetex: tn otwri tten by ap",
"author_name": "@fixture_user_1",
"author_channel_id": "UCFFFFFFFFFFFFFFFFFFFF01",
"author_channel_url": "https://www.youtube.com/@fixture_user_1",
"author_avatar_url": "https://yt3.ggpht.com/FIXTURE_AVATAR=s88-c-k-c0x00ffffff-no-rj",
"author_is_verified": true,
"author_is_creator": false,
"author_is_artist": false,
"like_count": 318000,
"like_count_text": "318K",
"reply_count": 962,
"reply_count_text": "962",
"published_time_text": "1 year ago",
"published_at_approx": "2025-09-21T02:54:31Z",
"published_at_precision": "year",
"edited": false,
"reply_level": 0,
"parent_id": null,
"is_pinned": true,
"pinned_by": "Pinned by @RickAstleyYT",
"creator_hearted": true,
"mentions": null,
"sort": "top",
"data_source": "innertube.entity",
"page": 1,
"position": 1
},
{
"source": "youtube.com",
"scraped_at": "2026-09-21T08:43:43Z",
"url": "https://www.youtube.com/watch?v=dQw4w9WgXcQ&lc=FIXTURE_CMT_2",
"sku": "FIXTURE_CMT_2",
"title": "Rick Astley - Never Gonna Give You Up (Official Video) (4K Remaster)",
"video_id": "dQw4w9WgXcQ",
"text": "f ixtu retext no twritt enbyap erson fixture te xtn ot writt e nbya pe rsonfi xtu rete …",
"author_name": "@fixture_user_2",
"author_channel_id": "UCFFFFFFFFFFFFFFFFFFFF02",
"author_channel_url": "https://www.youtube.com/@fixture_user_2",
"author_avatar_url": "https://yt3.ggpht.com/FIXTURE_AVATAR=s88-c-k-c0x00ffffff-no-rj",
"author_is_verified": false,
"author_is_creator": false,
"author_is_artist": false,
"like_count": 101000,
"like_count_text": "101K",
"reply_count": 546,
"reply_count_text": "546",
"published_time_text": "6 years ago (edited)",
"published_at_approx": "2020-09-20T21:48:31Z",
"published_at_precision": "year",
"edited": true,
"reply_level": 0,
"parent_id": null,
"is_pinned": false,
"pinned_by": null,
"creator_hearted": false,
"mentions": null,
"sort": "top",
"data_source": "innertube.entity",
"page": 1,
"position": 7
}
]First 2 of 9 rows from a real run — sample_output.json. Long values shortened here.
Handles what breaks naive YouTube scrapers
No browser by default
Comments come from the POST endpoint YouTube's own front end calls, which answered plain HTTPS from a datacentre address. A browser starts only if the site refuses.
Approximate numbers labelled as such
YouTube publishes comment likes as "318K" and times as "1 year ago". Each row keeps the site's text beside the derived number and states the date's precision.
Turned-off comments are not a block
Disabled comments, deleted and private videos all answer HTTP 200 with a healthy-looking payload; they exit 4 (no comments) rather than 3, so you are not sent to buy a proxy.
Complete is not exhaustive
Every run logs what share of the video's comments it collected, and a missed reply thread or failed metadata call makes the run partial, with the losses named in the sidecar.
Scraping infrastructure included
The same tested core runs every 2scraper repository, so the hard parts work the same way on every site.
CAPTCHA handling
Detects the challenge a page really serves and, with a 2Captcha key, solves it only when it blocks the data — a capped number of solves per page.
Proxy rotation
Single proxy or a pool from a file, rotated on blocks with a fresh browser per exit. Credentials stay out of argv and logs.
Fingerprints
Optional complete device identity from the 2Captcha Fingerprint API — user agent, client hints, locale and timezone that agree with each other.
Scraping Browser API
Point --cdp-endpoint at a managed Chromium with its own exit country — no browser infrastructure to run yourself.
Honest run metadata
Every run writes a .meta.json sidecar and distinct exit codes: complete, blocked, empty and partial are never confused, and a failed run never overwrites good data.
Diff two runs
diff_runs.py compares two complete YouTube snapshots by id — new, removed and changed rows for monitoring.
What teams do with YouTube data
Audience sentiment analysis
Collect comment threads and replies on a video to analyse reactions, questions and themes.
Creator engagement tracking
Find pinned and creator-hearted comments and see how a channel engages with its viewers.
Topic and competitor research
Search a query, then pull metadata and comments for the videos it returns.
Video performance monitoring
Record exact like counts, upload dates, duration and keywords for a list of videos over time.
Free to run. Pay only for what the site requires.
Runs with no key, proxy or account
No key, proxy, cookies or account: 60 consecutive pages, 1,200 comments, zero refusals from a bare datacentre address (measured Sep 2026). The paid integrations are readiness rather than routine here — they buy volume from many addresses, a specific exit country, and somewhere to run a browser that is not your machine. The README also points to the official YouTube Data API if you need exact comment like counts and timestamps.
Optional 2Captcha services — captcha solving, the Scraping Browser API, residential proxies and fingerprints — work behind one key.
Good to know
- Comment like counts are rounded and comment timestamps are relative, because that is all the site publishes.
- A comments run is sequential: each page is addressed by a token from the previous one.
- A non-English --locale leaves the like-count and timestamp text columns null on purpose.
2scraper vs. hosted scraper APIs
| 2scraper | Typical hosted API | |
|---|---|---|
| Code | Open source, MIT — read and patch it | Closed |
| Cost | Free; optional services billed only when used | Subscription or per request / per result |
| Runs on | Your machine, server or CI | The vendor's infrastructure |
| Output | JSON/CSV files + run metadata | API responses |
| When the site changes | Fix it yourself or open an issue / PR | Wait for the vendor |
YouTube scraper FAQ
Do I need a YouTube API key or account?
No. The scraper reads the endpoint the web front end uses, with no key, quota or login.
Why are comment like counts and dates approximate?
YouTube's web client publishes only "318K" and "1 year ago" for comments. The scraper keeps both the text and a derived value with its precision; video rows carry exact figures.
What is a page?
A continuation: 20 top-level comments, or 10 replies. --pages 3 gives 60 comments; --replies costs one extra request per thread that has replies.
Are top and newest the same comments in a different order?
No, they are two different samples. Each row records its sort, and diff_runs.py refuses to compare runs with different orderings.
Which formats are supported?
JSON or CSV, one row per comment, plus <out>.meta.json; a run that finds nothing does not overwrite earlier output unless you pass --allow-empty.
Is scraping YouTube comments legal?
It reads publicly visible pages without circumventing authentication. Comments are people's own words: collect thoughtfully, follow YouTube's terms and the law.