2scraper
What the Perplexity scraper collects
Reads public perplexity.ai articles — classic Pages (/page/...) and Discover articles — from a URL, a file of URLs or the Discover feed. Each row comes from the site's own /rest/article/ API: title, author, summary, publish and update times, view, like and fork counts, every cited source, and section and word counts, as JSON or CSV.
Discover feed
Input: A Discover topic (top)Reads the feed 20 articles per page up to --max-results, then fetches each article's full record.
Single article
Input: A classic Page or Discover article URLOne row with title, author, counters, summary, timestamps and every cited source.
URL batch
Input: A text file, one article URL per lineOne row per article; dead URLs are reported as not found, never as blocks.
No local browser
Input: TWOCAPTCHA_KEY and a profile in .envThe feed and every article API call go through 2Captcha's Scraper API, routed through your Scraping Browser profile.
From zero to Perplexity data in 3 steps
Install
Clone the repository and install one browser engine. Python 3.9+.
$ git clone https://github.com/2scraper/perplexity-scraper.git && cd perplexity-scraper $ python3 -m venv .venv && source .venv/bin/activate $ pip install -r requirements-playwright.txt $ playwright install chromium $ cp .env.example .env $ python3 playwright_scraper.py --discover top --max-results 20
Run a scraper
Pick what you need and run it. Every flag is documented in the README.
$ python3 playwright_scraper.py --discover top --max-results 40 --format csv --out discover.csv$ python3 playwright_scraper.py --url "https://www.perplexity.ai/page/ai-generated-images-tools-prom-efxu3L04SpufSVPD532HQg"$ python3 playwright_scraper.py --urls-file articles.txt$ python3 playwright_scraper.py --scraper-api --discover top --max-results 20Get structured data
Rows land in JSON or CSV (
--format), with a.meta.jsonsidecar recording status, pages fetched and why the run stopped.
Perplexity data fields
24 columns per row in a predictable schema. The most useful ones:
| Field | What it holds |
|---|---|
sku | perplexity- plus the article's stable 22-character id |
title | Article title |
author | The article's author handle |
category | The Discover topic; null for a classic Page |
product_url | The article's canonical URL |
summary | Discover summary; null for a classic Page |
view_count | The site's view counter |
like_count | The site's like counter |
fork_count | The site's fork counter |
sources_json | Every distinct cited source as JSON {url, title} |
source_count | Number of distinct cited sources |
section_count | Number of sections in the article |
word_count | Words across the whole article |
read_time_minutes | Read time as the site reports it |
published_at | Publish time |
updated_at | Last update time |
[
{
"sku": "perplexity-VzUTuvQVSIqru3QGvPihlg",
"source": "perplexity.ai",
"category": null,
"title": "Perplexity AI Pages Guide: Generate & Share Content Seamlessly",
"brand": null,
"price": null,
"currency": null,
"price_source": null,
"product_url": "https://www.perplexity.ai/page/perplexity-ai-pages-guide-gene-VzUTuvQVSIqru3QGvPihlg",
"image_url": "https://pplx-res.cloudinary.com/image/fetch/s--mc_0liAQ--/t_limit/https://framerusercon…",
"scraped_at": "2026-09-30T08:34:56Z",
"author": "perplexity_guy",
"view_count": 2280,
"like_count": 0,
"fork_count": 30,
"source_count": 38,
"sources_json": "[{\"url\": \"https://www.youtube.com/watch?v=EH3886nGmOk\", \"title\": \"Perplexity Pages - Th…",
"section_count": 14,
"word_count": 1758,
"slug": "perplexity-ai-pages-guide-gene-VzUTuvQVSIqru3QGvPihlg",
"summary": null,
"read_time_minutes": 8,
"published_at": "2024-06-01T22:51:39.145100+00:00",
"updated_at": "2024-07-08T16:35:32.009233"
},
{
"sku": "perplexity-heYaECNnQuaM0AZ0QSWjaw",
"source": "perplexity.ai",
"category": "top",
"title": "OpenAI unveils Dots, an always-on AI agent, at DevDay 2026",
"brand": null,
"price": null,
"currency": null,
"price_source": null,
"product_url": "https://www.perplexity.ai/discover/top/openai-unveils-dots-an-always-heYaECNnQuaM0AZ0QS…",
"image_url": "https://pplx-res.cloudinary.com/image/fetch/s--ujP6_Ewb--/t_limit/https://static.simonw…",
"scraped_at": "2026-09-30T08:35:03Z",
"author": "pagesandbits",
"view_count": 0,
"like_count": 0,
"fork_count": 0,
"source_count": 6,
"sources_json": "[{\"url\": \"https://openai.com/index/introducing-dots/\", \"title\": \"Introducing dots\"}, {\"…",
"section_count": 4,
"word_count": 380,
"slug": "openai-unveils-dots-an-always-heYaECNnQuaM0AZ0QSWjaw",
"summary": "Powered by GPT-6 Astra, each Dot gets its own cloud computer and can connect to over 4,…",
"read_time_minutes": 3,
"published_at": "2026-09-29T17:14:10.246772+00:00",
"updated_at": "2026-09-29T17:15:35.292639"
}
]First 2 of 2 rows from a real run — sample_output.json. Long values shortened here.
Handles what breaks naive Perplexity scrapers
The article API, not the markup
The HTML's meta tags are the same on every page, so rows come from /rest/article/{id}, called with fetch() from inside the article page where the browser already holds the site's cookies.
Throttling handled, not hidden
A refused API call is retried after 3, 5, 8 and 15 seconds, and articles are paced 2 s apart — at 0.5 s the API throttled after 13. A run that still gets throttled says rate_limited, never complete.
Not found is not blocked
An API 400 means the article does not exist: no row, no retry, and not counted as a block. Only a challenge that never cleared or a persistent refusal is reported as blocked.
Every cited source, deduplicated
Sources are collected across all sections, deduplicated, and stored as a flat JSON string so a CSV row stays one row.
Scraping infrastructure included
The same tested core runs every 2scraper repository, so the hard parts work the same way on every site.
CAPTCHA handling
Detects the challenge a page really serves and, with a 2Captcha key, solves it only when it blocks the data — a capped number of solves per page.
Proxy rotation
Single proxy or a pool from a file, rotated on blocks with a fresh browser per exit. Credentials stay out of argv and logs.
Fingerprints
Optional complete device identity from the 2Captcha Fingerprint API — user agent, client hints, locale and timezone that agree with each other.
Scraping Browser API
Point --cdp-endpoint at a managed Chromium with its own exit country — no browser infrastructure to run yourself.
Honest run metadata
Every run writes a .meta.json sidecar and distinct exit codes: complete, blocked, empty and partial are never confused, and a failed run never overwrites good data.
Diff two runs
diff_runs.py compares two complete Perplexity snapshots by id — new, removed and changed rows for monitoring.
What teams do with Perplexity data
Citation analysis
See which domains Perplexity's articles cite and how often, from sources_json across many articles.
Content and trend monitoring
Snapshot the top Discover feed daily and diff runs to see which stories enter and leave it.
Engagement tracking
Follow view, like and fork counters on specific Pages over time.
Research corpora
Collect article titles, summaries, word counts and sources for media or AI-search research.
Free to run. Pay only for what the site requires.
Needs a residential exit or the Scraping Browser API
Cloudflare kept every local browser tried on its challenge page, so the recommended setup is a 2Captcha Scraping Browser API profile (US) in .env — it got through on every live run of 2026-09-30 and 2026-10-01 with no paid captcha solves. Alternatively the 2Captcha Scraper API runs it with no browser at all, routed through the same profile. This repo does not implement a local captcha solver, so a local run without the Browser API is reported as blocked.
Optional 2Captcha services — captcha solving, the Scraping Browser API, residential proxies and fingerprints — work behind one key.
Good to know
- Local browsers were blocked by Cloudflare in every test; use the Scraping Browser API or the Scraper API.
- One Scraping Browser connection per profile; a second run started right away can time out.
- Only the top Discover topic has items for an anonymous visitor.
2scraper vs. hosted scraper APIs
| 2scraper | Typical hosted API | |
|---|---|---|
| Code | Open source, MIT — read and patch it | Closed |
| Cost | Free; optional services billed only when used | Subscription or per request / per result |
| Runs on | Your machine, server or CI | The vendor's infrastructure |
| Output | JSON/CSV files + run metadata | API responses |
| When the site changes | Fix it yourself or open an issue / PR | Wait for the vendor |
Perplexity scraper FAQ
Which Perplexity content can it read?
Published articles: classic Pages (/page/...) and Discover articles (/discover/{topic}/...). Live answers, accounts and anything behind a login are out of scope.
Do I need a paid product?
In practice yes: local browsers were held on Cloudflare's challenge in every test, while a Scraping Browser API profile got through. Put it in PERPLEXITY_CDP_ENDPOINT in .env.
Which Discover topics work?
Only top has items for an anonymous visitor; other topics return an empty feed (exit 4).
Why do some counters read 0?
The site's view, like and fork counters can read 0 for about a day after an article is published.
What output does it produce?
JSON or CSV, one row per article, plus a .meta.json sidecar with status, failed URLs and whether --max-results capped the run. A run that collects nothing never overwrites your last good file.
Is it legal to scrape Perplexity?
It reads public articles only and skips paths robots.txt disallows. Public data; follow the site's terms and the law.