2scraper
What the Better Business Bureau scraper collects
Scrapes business listings and business profiles from bbb.org: name, address, phone numbers, BBB letter grade, accreditation, categories and coordinates, and on a profile the accreditation date, years in business and complaint counts. Listings are read from the JSON endpoint BBB's own front end uses; profiles are read from the rendered page. Output is JSON or CSV with a run sidecar.
Keyword search
Input: A search term and a locationOne row per business location matching the query, with grade, accreditation, address, phones and coordinates. Defaults to A-Z ordering rather than BBB's accredited-first Best Match.
Category listing
Input: A bbb.org category URLEvery business BBB lists in a category, read from the same JSON endpoint as search, with the same columns.
Business profile
Input: A bbb.org business profile URLOne fully populated row: accreditation date, years in business, entity type, review and complaint counts and rating reasons. Shares the listing sku, so a profile run joins a listing run.
From zero to Better Business Bureau data in 3 steps
Install
Clone the repository and install one browser engine. Python 3.9+.
$ git clone https://github.com/2scraper/bbb-scraper.git $ cd bbb-scraper $ pip install -r requirements.txt -r requirements-playwright.txt $ playwright install chromium
Run a scraper
Pick what you need and run it. Every flag is documented in the README.
$ python playwright_scraper.py --text restaurants --location "New York, NY" --pages 3$ python playwright_scraper.py --mode category --url "https://www.bbb.org/us/category/restaurants" --pages 3$ python playwright_scraper.py --mode profile --url "https://www.bbb.org/us/ny/bronx/profile/cleaning-services/proclean-maintenance-systems-inc-0121-134716"Get structured data
Rows land in JSON or CSV (
--format), with a.meta.jsonsidecar recording status, pages fetched and why the run stopped.
Better Business Bureau data fields
53 columns per row in a predictable schema. The most useful ones:
| Field | What it holds |
|---|---|
title | Business name |
sku | Business at a location: {bbbId}_{businessId}_{addressId} |
rating_grade | BBB letter grade (A+ to F); null when BBB has not graded the business |
rating_score | The same rating as BBB's numeric score; null when ungraded |
is_accredited | Whether the business is BBB Accredited |
category | Primary BBB category |
address | Street address |
city | City |
state | State or province |
phone | List of phone numbers |
latitude | Latitude of the location |
longitude | Longitude of the location |
accredited_since | Accreditation date (profile mode) |
years_in_business | Years in business (profile mode) |
complaints_total | Total complaints (profile mode) |
sort | Ordering the run used; it decides which businesses are in the file |
[
{
"source": "bbb.org",
"scraped_at": "2026-09-16T12:10:03.339781+00:00",
"url": "https://www.bbb.org/us/ny/new-york-/profile/restaurants/1849-bar-and-grill-0121-100402",
"sku": "0121_100402_93156",
"title": "'1849' Bar and Grill",
"business_id": "100402",
"address_id": "93156",
"bbb_id": "0121",
"bbb_office": "BBB Serving Metropolitan New York",
"rating_grade": null,
"rating_score": null,
"is_accredited": false,
"category": "Restaurants",
"category_id": "50544-000",
"categories": [
"Restaurants"
],
"address": "183 Bleecker Street",
"city": "New York",
"state": "NY",
"postal_code": "10012-1406",
"country": "USA",
"latitude": 40.7290153503418,
"longitude": -74.0008773803711,
"service_areas": null,
"has_service_area": false,
"phone": [
"(212) 505-3200"
],
"logo_url": null,
"local_profile_url": null,
"leave_review_url": "https://www.bbb.org/us/ny/new-york-/profile/restaurants/1849-bar-and-grill-0121-100402/…",
"request_quote_url": null,
"is_charity": false,
"charity_seal": false,
"accredited_charity": false,
"out_of_business": null,
"website": null,
"description": null,
"fax": null,
"accredited_since": null,
"bbb_file_opened": null,
"business_started": null,
"years_in_business": null
},
{
"source": "bbb.org",
"scraped_at": "2026-09-16T12:10:03.339804+00:00",
"url": "https://www.bbb.org/us/ny/hampton-bays/profile/restaurants/1-north-steakhouse-0121-184362",
"sku": "0121_184362_223774",
"title": "1 North Steakhouse",
"business_id": "184362",
"address_id": "223774",
"bbb_id": "0121",
"bbb_office": "BBB Serving Metropolitan New York",
"rating_grade": "A+",
"rating_score": 100.0,
"is_accredited": false,
"category": "Restaurants",
"category_id": "50544-000",
"categories": [
"Restaurants"
],
"address": "322 West Montauk Highway",
"city": "Hampton Bays",
"state": "NY",
"postal_code": "11946-3525",
"country": "USA",
"latitude": 40.87,
"longitude": -72.52,
"service_areas": null,
"has_service_area": false,
"phone": [
"(631) 594-3419"
],
"logo_url": null,
"local_profile_url": null,
"leave_review_url": "https://www.bbb.org/us/ny/hampton-bays/profile/restaurants/1-north-steakhouse-0121-1843…",
"request_quote_url": null,
"is_charity": false,
"charity_seal": false,
"accredited_charity": false,
"out_of_business": null,
"website": null,
"description": null,
"fax": null,
"accredited_since": null,
"bbb_file_opened": null,
"business_started": null,
"years_in_business": null
}
]First 2 of 10 rows from a real run — sample_output.json. Long values shortened here.
Handles what breaks naive Better Business Bureau scrapers
Reads BBB's own JSON, not its gated HTML
Search and category pages are built from a JSON endpoint that is not behind Cloudflare, so listings come back from a datacenter address that gets HTTP 403 on every HTML page (measured 2026-09-16).
Sidesteps the accredited-first default
BBB's Best Match returned 15 of 15 accredited businesses and no restaurants for a restaurant search. The scraper defaults to --sort a-z, records the ordering as a column, and refuses to diff runs taken under different sorts.
Complete is told apart from exhaustive
BBB caps every query at 15 pages of 15. The sidecar records total_results, pages_available and capped_by_site, and the scraper never asks for page 16, which BBB answers with HTTP 500.
Ungraded is null, not zero
BBB returns a 0.0 score for businesses it has not graded. Both rating columns go null so an unrated business does not drag every average down.
Scraping infrastructure included
The same tested core runs every 2scraper repository, so the hard parts work the same way on every site.
CAPTCHA handling
Detects the challenge a page really serves and, with a 2Captcha key, solves it only when it blocks the data — a capped number of solves per page.
Proxy rotation
Single proxy or a pool from a file, rotated on blocks with a fresh browser per exit. Credentials stay out of argv and logs.
Fingerprints
Optional complete device identity from the 2Captcha Fingerprint API — user agent, client hints, locale and timezone that agree with each other.
Scraping Browser API
Point --cdp-endpoint at a managed Chromium with its own exit country — no browser infrastructure to run yourself.
Honest run metadata
Every run writes a .meta.json sidecar and distinct exit codes: complete, blocked, empty and partial are never confused, and a failed run never overwrites good data.
Diff two runs
diff_runs.py compares two complete Better Business Bureau snapshots by id — new, removed and changed rows for monitoring.
What teams do with Better Business Bureau data
Lead generation
Build lists of local businesses by category and location with addresses, phone numbers, websites and coordinates.
Vendor due diligence
Check a supplier's BBB grade, accreditation date, years in business and complaint counts before signing.
Reputation monitoring
Track grade changes and complaint counts for your own business or competitors with scheduled profile runs and run diffs.
Market mapping
Slice a category by state to see how many businesses BBB lists and what share are accredited.
Free to run. Pay only for what the site requires.
Free to run · paid add-ons optional
Search and category listings need no account, key or proxy: the README measured a complete run from a datacenter VPS that gets 403 on every HTML page. Business profiles are Cloudflare-gated and need a residential exit or the Scraping Browser API, which is where every live profile measurement was taken. On that path the Scraping Browser's built-in captcha auto-solve clears the Managed Challenge; this repo does not implement Turnstile solving through the local solver.
Optional 2Captcha services — captcha solving, the Scraping Browser API, residential proxies and fingerprints — work behind one key.
Good to know
- BBB caps every query at 225 results (15 pages of 15).
- Profile mode needs a residential exit or the Scraping Browser API; Selenium cannot authenticate either.
- position describes one fetch: two locations of one business can swap order between runs.
2scraper vs. hosted scraper APIs
| 2scraper | Typical hosted API | |
|---|---|---|
| Code | Open source, MIT — read and patch it | Closed |
| Cost | Free; optional services billed only when used | Subscription or per request / per result |
| Runs on | Your machine, server or CI | The vendor's infrastructure |
| Output | JSON/CSV files + run metadata | API responses |
| When the site changes | Fix it yourself or open an issue / PR | Wait for the vendor |
Better Business Bureau scraper FAQ
Do I need a proxy or a 2Captcha key?
Not for search or category listings, which read an ungated JSON endpoint. Profile mode needs a residential proxy or a Scraping Browser API session, because profile pages are behind Cloudflare.
Why doesn't it use BBB's default sort?
BBB's Best Match returned only accredited businesses, often outside the searched category. The scraper defaults to A-Z, which is also the only ordering stable between runs; pass --sort best-match to see what a visitor sees.
How many results can I get per query?
BBB serves at most 15 pages of 15 per query, whatever the total. Narrow the query, for example with --state NY, and each slice gets its own 15 pages.
Why does one business appear twice?
Each row is a business at a location. One business with two addresses yields two real rows with the same businessId and different sku values; dedupe on sku.
What output formats are there?
JSON and CSV with the same columns, plus a .meta.json sidecar recording status, sort, totals and whether the site capped the result set. A run that finds nothing writes nothing.
Is scraping BBB legal?
It reads public pages a visitor sees and deliberately skips the named officers BBB lists. Follow the site's terms and the law in your jurisdiction.