2scraper
What the Wellfound scraper collects
Scrapes wellfound.com (formerly AngelList Talent) role landing pages, the /jobs discovery feed and individual job pages into one row per listing with 38 columns. Rows carry the company's size, badges and tagline, parsed salary and equity ranges, remote locations and the full job description, as JSON or CSV.
Role landing page
Input: A role slug (optionally --remote or --location), or a role URLThe richest rows — description, job type, badges, salary and equity — paginated with ?page=N against the listing's own page count.
Jobs discovery feed
Input: NothingOne page of the /jobs feed; thinner rows with no description, job type or badges.
Single job
Input: A wellfound.com/jobs/{id}-{slug} URLOne row from the job's schema.org JobPosting, adding salary period, benefits and industry.
From zero to Wellfound data in 3 steps
Install
Clone the repository and install one browser engine. Python 3.9+.
$ git clone https://github.com/2scraper/wellfound-scraper.git && cd wellfound-scraper $ pip install -r requirements.txt -r requirements-playwright.txt $ playwright install chromium $ cp .env.example .env $ python3 playwright_scraper.py --url "https://wellfound.com/role/r/software-engineer" --pages 3
Run a scraper
Pick what you need and run it. Every flag is documented in the README.
$ python3 playwright_scraper.py --role software-engineer --remote --pages 3$ python3 playwright_scraper.py --mode jobs$ python3 playwright_scraper.py --mode job --url "https://wellfound.com/jobs/4697947-senior-software-engineer"Get structured data
Rows land in JSON or CSV (
--format), with a.meta.jsonsidecar recording status, pages fetched and why the run stopped.
Wellfound data fields
38 columns per row in a predictable schema. The most useful ones:
| Field | What it holds |
|---|---|
sku | Wellfound's job id |
title | Job title |
company_name | Hiring company |
company_size | Headcount band, e.g. 11-50 |
company_badges | Badges such as Actively Hiring or YC Funded |
job_type | full-time, contract, … |
years_experience_min | Minimum years of experience |
posted_at | When the job was posted |
remote | Whether the job is remote |
remote_locations | Where remote candidates may be based |
compensation | The pay and equity string as rendered |
salary_min | Parsed lower salary bound |
salary_max | Parsed upper salary bound |
salary_currency | Salary currency, e.g. USD or INR |
equity_min | Lower equity bound, in percent |
has_equity | true, false ("No equity") or null when unstated |
[
{
"source": "wellfound.com",
"scraped_at": "2026-09-17T15:44:38.519921+00:00",
"url": "https://wellfound.com/jobs/4697947-senior-software-engineer",
"sku": "4697947",
"title": "Senior Software Engineer",
"company_name": "Confident LIMS",
"company_slug": "confidentlims",
"company_id": "829771",
"company_url": "https://wellfound.com/company/confidentlims",
"company_logo_url": "https://photos.wellfound.com/startups/i/829771-0deed706d7c625c16e51b403c4b1ad20-medium_…",
"company_size": "11-50",
"company_tagline": "The easiest way to test",
"company_badges": [
"Actively Hiring",
"Top 1% of responders",
"Responds within a day",
"… 4 more"
],
"job_slug": "senior-software-engineer",
"job_type": "full-time",
"primary_role": "Software Engineer",
"years_experience_min": 5,
"years_experience_max": null,
"posted_at": "2026-09-10T17:59:14+00:00",
"locations": [
"San Francisco Bay Area"
],
"remote": true,
"remote_kind": "REMOTE",
"wfh_flexible": false,
"remote_locations": [
"Canada",
"South America",
"United States",
"… 1 more"
],
"compensation": "$100k – $180k • 0.05% – 0.25%",
"salary_min": 100000.0,
"salary_max": 180000.0,
"salary_currency": "USD",
"salary_period": null,
"equity_min": 0.05,
"equity_max": 0.25,
"has_equity": true,
"description": "Confident powers analytical testing labs — the labs that test samples on behalf of busi…",
"benefits": null,
"industry": null,
"data_source": "apollo",
"page": 1,
"position": 1
},
{
"source": "wellfound.com",
"scraped_at": "2026-09-17T15:44:38.519921+00:00",
"url": "https://wellfound.com/jobs/4579450-software-engineer",
"sku": "4579450",
"title": "Software Engineer",
"company_name": "Oklo",
"company_slug": "oklo",
"company_id": "5268250",
"company_url": "https://wellfound.com/company/oklo",
"company_logo_url": "https://photos.wellfound.com/startups/i/5268250-a0d534809c2c6fa47e19e6c82e50f089-medium…",
"company_size": "201-500",
"company_tagline": "Making reactors people want",
"company_badges": [
"Actively Hiring",
"B2B",
"Growth Stage",
"… 3 more"
],
"job_slug": "software-engineer",
"job_type": "full-time",
"primary_role": "Software Engineer",
"years_experience_min": null,
"years_experience_max": null,
"posted_at": "2026-08-12T16:29:14+00:00",
"locations": [
"Santa Clara"
],
"remote": true,
"remote_kind": null,
"wfh_flexible": null,
"remote_locations": [
"United States"
],
"compensation": "$110k – $200k",
"salary_min": 110000.0,
"salary_max": 200000.0,
"salary_currency": "USD",
"salary_period": null,
"equity_min": null,
"equity_max": null,
"has_equity": null,
"description": "Join us in pioneering the next generation of nuclear reactors! You'll leverage your sof…",
"benefits": null,
"industry": null,
"data_source": "apollo",
"page": 1,
"position": 2
}
]First 2 of 10 rows from a real run — sample_output.json. Long values shortened here.
Handles what breaks naive Wellfound scrapers
Salary and equity parsed correctly
"$100k – $180k • 0.05% – 0.25%" is split into pay and equity, either side may be absent, and L is read as the Indian lakh — ₹30L is 3,000,000, not 30.
Out-of-range pages detected
Asking for page 48 of a 47-page listing silently returns page 1 again. The run plans against the listing's own page count and checks which page the server says it served, stopping on a mismatch.
Pages counted in companies, not jobs
A landing page is twenty companies carrying 32 to 45 jobs. The sidecar records total jobs, total companies and pages available instead of a misleading per-page figure.
Challenge markers that do not lie
The obvious Cloudflare marker sits on 14 of 16 good pages because the site loads it itself. Refusals are detected by the challenge's own bootstrap vocabulary, present on every refusal and on no served page.
Scraping infrastructure included
The same tested core runs every 2scraper repository, so the hard parts work the same way on every site.
CAPTCHA handling
Detects the challenge a page really serves and, with a 2Captcha key, solves it only when it blocks the data — a capped number of solves per page.
Proxy rotation
Single proxy or a pool from a file, rotated on blocks with a fresh browser per exit. Credentials stay out of argv and logs.
Fingerprints
Optional complete device identity from the 2Captcha Fingerprint API — user agent, client hints, locale and timezone that agree with each other.
Scraping Browser API
Point --cdp-endpoint at a managed Chromium with its own exit country — no browser infrastructure to run yourself.
Honest run metadata
Every run writes a .meta.json sidecar and distinct exit codes: complete, blocked, empty and partial are never confused, and a failed run never overwrites good data.
Diff two runs
diff_runs.py compares two complete Wellfound snapshots by id — new, removed and changed rows for monitoring.
What teams do with Wellfound data
Startup salary benchmarking
Compare salary and equity ranges by role, location and company size across hundreds of listings.
Recruiting and sourcing
Track which startups are actively hiring for a role, with company badges, size and remote policy.
Job-market research
Follow posting volume and remote availability for a role over time, with diffs between complete runs.
Job-board aggregation
Pull full descriptions, compensation and locations into your own job feed or alerting tool.
Free to run. Pay only for what the site requires.
Needs a residential exit or the Scraping Browser API
You need a residential exit; nothing else is required. Measured 2026-09-17, a datacentre address was refused on every route including robots.txt, while a 2Captcha residential US exit got HTTP 200 on every route this scraper reads — it is one WELLFOUND_PROXY line in .env. The Scraping Browser API is an alternative if you have no browser infrastructure; captcha solving and fingerprints are wired in but a residential exit removed the challenge on these routes.
Optional 2Captcha services — captcha solving, the Scraping Browser API, residential proxies and fingerprints — work behind one key.
Good to know
- Company pages (/company/{slug}) are not implemented; they are gated even from residential exits.
- The /jobs feed is a single page and carries no descriptions.
- salary_period is filled only from job pages; years_experience_max is almost always null.
2scraper vs. hosted scraper APIs
| 2scraper | Typical hosted API | |
|---|---|---|
| Code | Open source, MIT — read and patch it | Closed |
| Cost | Free; optional services billed only when used | Subscription or per request / per result |
| Runs on | Your machine, server or CI | The vendor's infrastructure |
| Output | JSON/CSV files + run metadata | API responses |
| When the site changes | Fix it yourself or open an issue / PR | Wait for the vendor |
Wellfound scraper FAQ
Do I need a proxy?
Yes — a residential exit. From a datacentre address every route, including robots.txt, was refused by Cloudflare in testing; from a residential exit every route this scraper reads answered.
What is the difference between the three modes?
Role landing pages give the richest rows and paginate; the /jobs feed is one page with thinner rows; a job page adds salary period, benefits and industry from its JobPosting.
Can it scrape company pages?
No. /company/{slug} answered 403 even from a residential exit and a real browser, so there is no company mode.
Why is has_equity sometimes null?
false means the listing printed "No equity"; null means it said nothing. Most listings say nothing.
What output does it produce?
JSON or CSV with a .meta.json sidecar recording status, stop reason, failed pages and the listing's own totals. A run that finds nothing never overwrites your last good file.
Is scraping Wellfound legal?
It reads only routes robots.txt allows and public job listings. Public data; follow the site's terms and the law, and keep your request rate civil.