2scraper
What the Skyscanner scraper collects
Runs a skyscanner.com flight search and captures the web-unified-search response the page itself polls, turning it into one flat row per itinerary: price, currency, airline, departure and arrival times, duration and stops, with return-leg columns for round trips. Single searches or a CSV of routes through batch_scraper.py, written as JSON or CSV.
One-way search
Input: Origin, destination and a departure dateEvery itinerary the search returns, one row each, with price, airline, times, duration and stops.
Round trip with filters
Input: Route, both dates, passengers, cabin, stops and sortOutbound and return legs in separate columns of one row, filtered by cabin class and stops and kept in the site's own order for --sort.
Search URL
Input: A full skyscanner.com search URLScrapes exactly that search, bypassing the URL builder.
Batch of routes
Input: A CSV file of routesRuns each route in your local Chrome session, merges and dedupes all itineraries into one file, with per-route results in the sidecar.
From zero to Skyscanner data in 3 steps
Install
Clone the repository and install one browser engine. Python 3.9+.
$ git clone https://github.com/2scraper/skyscanner-scraper.git && cd skyscanner-scraper $ pip install -r requirements-playwright.txt && playwright install chromium $ cp .env.example .env $ python3 playwright_scraper.py --origin LHR --destination JFK --depart-date 2026-11-15 --wait-for-human 120
Run a scraper
Pick what you need and run it. Every flag is documented in the README.
$ python3 playwright_scraper.py --origin LHR --destination JFK --depart-date 2026-10-15 --format json --out sky_results.json$ python3 playwright_scraper.py --origin LHR --destination JFK --depart-date 2026-10-15 --return-date 2026-10-22 --adults 2 --currency EUR --cabin-class business --stops direct --sort cheapest$ python3 playwright_scraper.py --url "https://www.skyscanner.com/transport/flights/lhr/jfk/261015/"$ python3 batch_scraper.py --routes-file routes.example.csv --out skyscanner_batch.jsonGet structured data
Rows land in JSON or CSV (
--format), with a.meta.jsonsidecar recording status, pages fetched and why the run stopped.
Skyscanner data fields
27 columns per row in a predictable schema. The most useful ones:
| Field | What it holds |
|---|---|
sku | Deterministic itinerary fingerprint, stable across days so price changes diff correctly |
title | Route and airline, e.g. LHR → JFK · jetBlue |
brand | The outbound leg's marketing airline |
price | Itinerary price as a number |
currency | Currency the data states, never the requested one |
product_url | Skyscanner's deep link for the itinerary |
origin | Origin airport or city code |
destination | Destination airport or city code |
depart_date | Outbound date |
return_date | Return date; null on one-way |
cabin_class | Cabin class searched |
stops | Stops on the outbound leg, e.g. Direct |
departure_time | Outbound departure time |
arrival_time | Outbound arrival time |
duration | Outbound flight duration |
return_airline | The return leg's airline; null on one-way |
[
{
"sku": "LHRJFK-20261115-5f7259c39207",
"source": "skyscanner.com",
"category": "flights",
"title": "LHR → JFK · jetBlue",
"brand": "jetBlue",
"price": 322.29,
"currency": "GBP",
"price_source": "search_api",
"product_url": "https://www.skyscanner.com/transport_deeplink/4.0/UK/en-GB/GBP/gtaf/1/13554.12712.2026-…",
"image_url": null,
"scraped_at": "2026-09-29T09:03:46Z",
"origin": "LHR",
"destination": "JFK",
"depart_date": "2026-11-15",
"return_date": null,
"adults": 1,
"cabin_class": "economy",
"stops": "Direct",
"departure_time": "2026-11-15T07:45:00",
"arrival_time": "2026-11-15T11:00:00",
"duration": "8h 15m",
"sort": "best",
"return_departure_time": null,
"return_arrival_time": null,
"return_duration": null,
"return_stops": null,
"return_airline": null
},
{
"sku": "LHRJFK-20261115-48cf924019df",
"source": "skyscanner.com",
"category": "flights",
"title": "LHR → JFK · TAP Air Portugal",
"brand": "TAP Air Portugal",
"price": 419.97,
"currency": "GBP",
"price_source": "search_api",
"product_url": "https://www.skyscanner.com/transport_deeplink/4.0/UK/en-GB/GBP/eduk/1/13554.12712.2026-…",
"image_url": null,
"scraped_at": "2026-09-29T09:03:46Z",
"origin": "LHR",
"destination": "JFK",
"depart_date": "2026-11-15",
"return_date": null,
"adults": 1,
"cabin_class": "economy",
"stops": "2 stops",
"departure_time": "2026-11-15T06:00:00",
"arrival_time": "2026-11-15T17:29:00",
"duration": "16h 29m",
"sort": "best",
"return_departure_time": null,
"return_arrival_time": null,
"return_duration": null,
"return_stops": null,
"return_airline": null
}
]First 2 of 3 rows from a real run — sample_output.json. Long values shortened here.
Handles what breaks naive Skyscanner scrapers
The search API, not the markup
A search page's HTML carries no flights. All three engines capture the web-unified-search XHR the page polls, and keep polling while the site says the search is incomplete.
Stable itinerary ids
Flights have no SKU, so sku is a fingerprint of route, dates, airline, times and cabin — everything except price. The same itinerary on two days gets the same id and diff_runs.py reports a real price change.
Incomplete searches are marked partial
A run that stops while the search API still reports incomplete is partial, and the sidecar records how many itineraries the site offered against how many were taken.
A person clears the challenge once
--wait-for-human waits for a person to complete PerimeterX's "Press & Hold" in a visible browser, then re-opens the search and carries on in that session; batch runs reuse it for later routes.
Scraping infrastructure included
The same tested core runs every 2scraper repository, so the hard parts work the same way on every site.
CAPTCHA handling
Detects the challenge a page really serves and, with a 2Captcha key, solves it only when it blocks the data — a capped number of solves per page.
Proxy rotation
Single proxy or a pool from a file, rotated on blocks with a fresh browser per exit. Credentials stay out of argv and logs.
Fingerprints
Optional complete device identity from the 2Captcha Fingerprint API — user agent, client hints, locale and timezone that agree with each other.
Scraping Browser API
Point --cdp-endpoint at a managed Chromium with its own exit country — no browser infrastructure to run yourself.
Honest run metadata
Every run writes a .meta.json sidecar and distinct exit codes: complete, blocked, empty and partial are never confused, and a failed run never overwrites good data.
Diff two runs
diff_runs.py compares two complete Skyscanner snapshots by id — new, removed and changed rows for monitoring.
What teams do with Skyscanner data
Fare monitoring
Re-run the same route daily; stable itinerary ids make price drops show up as changes, not as new rows.
Route comparison
Batch many origin/destination/date combinations into one deduped file to compare prices and durations.
Airline and schedule analysis
See which airlines serve a route, with direct versus connecting options and departure times.
Travel-deal research
Pull cheapest or best-sorted results for business or economy cabins to feed alerts or content.
Free to run. Pay only for what the site requires.
No key needed · one manual “Press & Hold” per profile
No key, proxy or account is required — the default is an ordinary local browser. What returned real results live (measured 2026-09-29) was your own local Chrome with --wait-for-human: a person held PerimeterX's "Press & Hold" once, and later runs in that profile returned full result sets. Proxies, fingerprints and the Scraping Browser API are opt-in for volume, a specific exit country or a consistent device identity; in testing so far, fully automated runs were challenged. This repo does not implement a PerimeterX solve.
Optional 2Captcha services — captcha solving, the Scraping Browser API, residential proxies and fingerprints — work behind one key.
Good to know
- Fully automated runs were challenged by PerimeterX in every test so far; the verified path needs a person to clear it once.
- Only the search-API path is verified live; the embedded-JSON and DOM fallbacks are unverified, and --scraper-api gets HTML only.
- No --pages, --category or --concurrency: there is one scroll-based results page per search.
2scraper vs. hosted scraper APIs
| 2scraper | Typical hosted API | |
|---|---|---|
| Code | Open source, MIT — read and patch it | Closed |
| Cost | Free; optional services billed only when used | Subscription or per request / per result |
| Runs on | Your machine, server or CI | The vendor's infrastructure |
| Output | JSON/CSV files + run metadata | API responses |
| When the site changes | Fix it yourself or open an issue / PR | Wait for the vendor |
Skyscanner scraper FAQ
Where does the flight data come from?
From the web-unified-search response the search page polls; the initial HTML carries no flights. Rows from it are marked price_source search_api.
Do I need a proxy or a 2Captcha key?
No. The path verified live is your own local Chrome with a person clearing the PerimeterX challenge once via --wait-for-human; paid products are optional extras.
Can it run many routes at once?
Yes — batch_scraper.py takes a CSV of routes, runs them in one local Chrome session and merges the itineraries into one deduped file.
How does pagination work?
Skyscanner lazy-loads on scroll, so the engines scroll and re-parse until --max-results is reached or --stall-rounds scrolls add nothing, capped by --max-scrolls.
What does a round-trip row look like?
One row per itinerary: outbound columns (departure_time, stops, brand…) plus return_departure_time, return_arrival_time, return_duration, return_stops and return_airline.
Is scraping Skyscanner legal?
It reads search results a visitor sees and never follows deep links to booking partners. Public data; follow the site's terms and the law.