2scraper
What the Goodreads scraper collects
Scrapes goodreads.com listings (Listopia lists, an author's books, searches, genre shelves, series), full book pages with contributors, rating distribution, genres, awards and edition details, and a book's community reviews walked as deep as you ask. Output is JSON and CSV with a .meta.json sidecar that records the site's own count beside the rows read.
Lists, authors, search, shelves & series
Input: A Listopia list, author list, search, genre shelf or series URLOne row per book: title, author, average rating and count, publication year, editions, and a list's score and votes. 100 per list page, 30 per author page, 20 per search page.
Search
Input: A /search?q= URLSearch results as book rows. Run headful (xvfb-run -a on a server): a headless browser is refused on /search.
Full book pages
Input: A book URL, or a listing URL to open every book on itEvery contributor with role, the rating distribution, review counts per language, genres, series, awards, characters, places, ISBN, ASIN, pages, publisher, dates and the blurb.
Community reviews
Input: A book URL, with optional --sort, --rating and --review-languageOne row per review: stars, text, date, likes, comments, the reviewer's shelf and tags, and the reviewer, read through the site's own GraphQL API.
From zero to Goodreads data in 3 steps
Install
Clone the repository and install one browser engine. Python 3.9+.
$ git clone https://github.com/2scraper/goodreads-scraper.git && cd goodreads-scraper $ python3 -m venv .venv && . .venv/bin/activate $ pip install -r requirements.txt -r requirements-playwright.txt $ playwright install chromium
Run a scraper
Pick what you need and run it. Every flag is documented in the README.
$ python3 playwright_scraper.py --url https://www.goodreads.com/author/list/656983.J_R_R_Tolkien --pages 30$ python3 playwright_scraper.py --url "https://www.goodreads.com/search?q=dune" --pages 5 --headful$ python3 playwright_scraper.py --mode book --url https://www.goodreads.com/list/show/1.Best_Books_Ever --max-books 20 --concurrency 2$ python3 playwright_scraper.py --mode reviews --url https://www.goodreads.com/book/show/5907.The_Hobbit --sort newest --pages 10Get structured data
Rows land in JSON or CSV (
--format), with a.meta.jsonsidecar recording status, pages fetched and why the run stopped.
Goodreads data fields
23 columns per row in a predictable schema. The most useful ones:
| Field | What it holds |
|---|---|
sku | The edition's id, the number in /book/show/5907 |
title | Book title as listed, including series |
author | Author name |
author_id | Goodreads author id |
avg_rating | Average rating of the work; null when unrated |
ratings_count | Number of ratings of the work |
published_year | Publication year, where the listing shows it |
editions_count | Number of editions |
work_id | The work all editions share; ratings and reviews belong to it |
cover_url | Cover image URL |
list_score | A Listopia list's score for the book |
list_votes | Votes the book received on the list |
series_position | Position within a series |
listing_kind | Which kind of listing the row came from: list, author, search, shelf, series |
page | Listing page the row was on |
position | Rank within the run |
[
{
"source": "goodreads.com",
"scraped_at": "2026-09-27T08:07:58.849961+00:00",
"url": "https://www.goodreads.com/book/show/2767052-the-hunger-games",
"sku": "2767052",
"title": "The Hunger Games (The Hunger Games, #1)",
"author": "Suzanne Collins",
"author_id": "153394",
"author_url": "https://www.goodreads.com/author/show/153394.Suzanne_Collins",
"avg_rating": 4.36,
"ratings_count": 10292712,
"published_year": null,
"editions_count": null,
"work_id": null,
"cover_url": "https://i.gr-assets.com/images/S/compressed.photo.goodreads.com/books/1586722975i/27670…",
"list_score": 4547035,
"list_votes": 46205,
"shelved_count": null,
"series_position": null,
"listing_kind": "list",
"page": 1,
"position": 1,
"mode": "list",
"data_source": "microdata"
},
{
"source": "goodreads.com",
"scraped_at": "2026-09-27T08:07:58.850368+00:00",
"url": "https://www.goodreads.com/book/show/1885.Pride_and_Prejudice",
"sku": "1885",
"title": "Pride and Prejudice",
"author": "Jane Austen",
"author_id": "1265",
"author_url": "https://www.goodreads.com/author/show/1265.Jane_Austen",
"avg_rating": 4.3,
"ratings_count": 5017334,
"published_year": null,
"editions_count": null,
"work_id": null,
"cover_url": "https://i.gr-assets.com/images/S/compressed.photo.goodreads.com/books/1320399351i/1885.…",
"list_score": 3085040,
"list_votes": 31599,
"shelved_count": null,
"series_position": null,
"listing_kind": "list",
"page": 1,
"position": 2,
"mode": "list",
"data_source": "microdata"
}
]First 2 of 30 rows from a real run — sample_output.json. Long values shortened here.
Handles what breaks naive Goodreads scrapers
A page past the end is detected
Goodreads answers ?page=800 of a 100-page list with HTTP 200 and the last page, and a shelf answers page 2 with page 1. Every page is checked for which page it actually served, and a mismatch ends the listing cleanly instead of re-collecting it.
AWS WAF challenge waited out, not called a block
The WAF's free challenge action is passed by a real browser in 1.0-2.4 s. The engines wait for it instead of reporting a block, so a normal run pays nothing.
Author lists that repeat and hide books
Sorted by popularity, Tolkien's 791-place list served 720 distinct books and 71 twice. Repeats are dropped, counted in the sidecar, and the README shows how two orderings together recover all 791.
Review filters checked before sending
The review API answers a language with no reviews, or a page size of 200, with zero rows rather than an error. --review-language is checked against the languages the book page lists, and oversized pages are refused, before anything is sent.
Scraping infrastructure included
The same tested core runs every 2scraper repository, so the hard parts work the same way on every site.
CAPTCHA handling
Detects the challenge a page really serves and, with a 2Captcha key, solves it only when it blocks the data — a capped number of solves per page.
Proxy rotation
Single proxy or a pool from a file, rotated on blocks with a fresh browser per exit. Credentials stay out of argv and logs.
Fingerprints
Optional complete device identity from the 2Captcha Fingerprint API — user agent, client hints, locale and timezone that agree with each other.
Scraping Browser API
Point --cdp-endpoint at a managed Chromium with its own exit country — no browser infrastructure to run yourself.
Honest run metadata
Every run writes a .meta.json sidecar and distinct exit codes: complete, blocked, empty and partial are never confused, and a failed run never overwrites good data.
Diff two runs
diff_runs.py compares two complete Goodreads snapshots by id — new, removed and changed rows for monitoring.
What teams do with Goodreads data
Review and sentiment datasets
Walk every community review of a book with stars, text, date, likes and language, filtered by rating or language, for sentiment or reception analysis.
Reading-list and catalogue research
Pull Listopia lists, genre shelves and series with ratings, counts, scores and votes to study what readers rank and shelve.
Author bibliographies
Collect every book an author is credited on, with editions, publication years and ratings, then open each one's full page.
Book metadata enrichment
Fill ISBN, ASIN, page count, publisher, genres, awards and rating distributions for a list of titles from their book pages.
Free to run. Pay only for what the site requires.
Runs with no key, proxy or account
All three modes run with no key, no proxy and no account — measured on 2026-09-27 from a datacentre VPS, and by a daily canary that runs with no secrets. Searches need a headful browser locally. 2Captcha proxies add volume from more than one address, the Scraping Browser API gives a remote browser with a chosen exit country and serves /search on a server with no display, and the captcha path covers AWS WAF's CAPTCHA via AmazonTask should a session be shown it.
Optional 2Captcha services — captcha solving, the Scraping Browser API, residential proxies and fingerprints — work behind one key.
Good to know
- This repo does not implement --mode reviews through the Scraper API; that path reads lists and single book pages.
- Searches are refused to a headless local browser; run them --headful or through the Scraping Browser API.
- Selenium cannot authenticate a proxy or a remote CDP endpoint.
2scraper vs. hosted scraper APIs
| 2scraper | Typical hosted API | |
|---|---|---|
| Code | Open source, MIT — read and patch it | Closed |
| Cost | Free; optional services billed only when used | Subscription or per request / per result |
| Runs on | Your machine, server or CI | The vendor's infrastructure |
| Output | JSON/CSV files + run metadata | API responses |
| When the site changes | Fix it yourself or open an issue / PR | Wait for the vendor |
Goodreads scraper FAQ
Doesn't Goodreads have an API?
Goodreads retired its public API in December 2020. This repo reads what its pages show every visitor and what its own front end fetches.
Do I need a proxy or a 2Captcha key?
No, not for any of the three modes. Paid products help with volume, a specific exit country, or running searches on a server without a virtual display.
How many reviews can it get for one book?
As many pages as you ask for, 30 reviews per page by default (1-100). A live walk read 11,000 reviews in 110 pages of 100 with no ceiling and no duplicate.
Why does a listing say complete but not have every book?
Goodreads states far more than it serves: the Best Books Ever list states 79,607 books and serves 100 pages of 100. The sidecar records total_results, reachable_max and capped_by_site.
Why do two editions have the same rating?
Ratings, counts, the distribution and reviews belong to the work (work_id), not the edition (sku), so all editions of one book share them.
Is it legal to scrape Goodreads?
It reads public data an anonymous visitor sees and nothing behind a login. Reviews are people's writing under their names; follow the site's terms and the law where you operate.