2scraper
What the Transfermarkt scraper collects
Scrapes transfermarkt.com's player market-value ranking, a club's full squad for a season, the latest-transfers ledger and individual player profiles. Player rows carry market value in EUR, position, age, nationalities, club and — on a profile — birth date, height, agent, contract and caps; transfer rows have their own schema. Output is JSON or CSV with a run-metadata sidecar.
Market-value ranking
Input: Nothing, or a ranking URL on any supported hostThe most valuable players, one row each with rank, market value in EUR, position, age, nationalities and club.
Club squad
Input: Transfermarkt's numeric club id and a seasonEvery player in a club's squad for that season with shirt number, position, age, nationality and market value (null where none is published).
Latest transfers
Input: Nothing — the site's latest-transfers ledgerOne transfer row per move with the clubs involved, fee and fee type; independently addressable pages, so it can run with --concurrency.
Player profile
Input: A player id or a profile URLOne full profile: full name, birth date and place, height, agent, league, joined date, contract expiry, caps and international goals.
From zero to Transfermarkt data in 3 steps
Install
Clone the repository and install one browser engine. Python 3.9+.
$ git clone https://github.com/2scraper/transfermarkt-scraper && cd transfermarkt-scraper $ python3 -m venv .venv && . .venv/bin/activate $ pip install -r requirements.txt -r requirements-playwright.txt $ playwright install chromium
Run a scraper
Pick what you need and run it. Every flag is documented in the README.
$ python3 playwright_scraper.py --mode market-values --pages 2$ python3 playwright_scraper.py --mode club-squad --club-id 281 --season 2025$ python3 playwright_scraper.py --mode transfers --pages 5 --concurrency 4$ python3 playwright_scraper.py --mode player --player-id 418560Get structured data
Rows land in JSON or CSV (
--format), with a.meta.jsonsidecar recording status, pages fetched and why the run stopped.
Transfermarkt data fields
32 columns per row in a predictable schema. The most useful ones:
| Field | What it holds |
|---|---|
sku | Transfermarkt player id (or a transfer's own id in transfers mode) |
title | Player name |
price | Market value in EUR; null when none is published, never 0 |
currency | EUR on every supported host |
rank | Position in the market-value ranking |
position | Playing position, normalised to English |
age | Player age |
nationalities | Every nationality shown (nationality holds the first) |
club | Current club |
club_id | Transfermarkt club id |
shirt_number | Squad number (club-squad mode) |
birth_date | ISO birth date (player profile) |
height_m | Height in metres (player profile) |
agent | Player's agent (player profile) |
contract_until | Contract expiry date (player profile) |
caps | International caps (player profile) |
[
{
"source": "transfermarkt.com",
"scraped_at": "2026-09-10T08:21:28.159063+00:00",
"url": "https://www.transfermarkt.com/x/profil/spieler/937958",
"sku": "937958",
"title": "Lamine Yamal",
"image_url": "https://img.a.transfermarkt.technology/portrait/small/937958-1773173768.jpg?lm=4711",
"price": 220000000.0,
"currency": "EUR",
"price_source": "listing",
"category": "market-values",
"rank": 1,
"page": 1,
"row_index": 0,
"position": "Right Winger",
"shirt_number": null,
"age": 19,
"nationality": "Spain",
"nationalities": [
"Spain"
],
"club": "FC Barcelona",
"club_id": "131",
"full_name": null,
"birth_date": null,
"birth_place": null,
"height_m": null,
"agent": null,
"league": null,
"league_id": null,
"joined_date": null,
"contract_until": null,
"market_value_last_update": null,
"caps": null,
"international_goals": null
},
{
"source": "transfermarkt.com",
"scraped_at": "2026-09-10T08:21:28.159519+00:00",
"url": "https://www.transfermarkt.com/x/profil/spieler/418560",
"sku": "418560",
"title": "Erling Haaland",
"image_url": "https://img.a.transfermarkt.technology/portrait/small/418560-1788115885.jpg?lm=4711",
"price": 220000000.0,
"currency": "EUR",
"price_source": "listing",
"category": "market-values",
"rank": 1,
"page": 1,
"row_index": 1,
"position": "Centre-Forward",
"shirt_number": null,
"age": 26,
"nationality": "Norway",
"nationalities": [
"Norway"
],
"club": "Manchester City",
"club_id": "281",
"full_name": null,
"birth_date": null,
"birth_place": null,
"height_m": null,
"agent": null,
"league": null,
"league_id": null,
"joined_date": null,
"contract_until": null,
"market_value_last_update": null,
"caps": null,
"international_goals": null
}
]First 2 of 26 rows from a real run — sample_output.json. Long values shortened here.
Handles what breaks naive Transfermarkt scrapers
Five locales normalised into one vocabulary
German, Russian, Korean and Japanese hosts are parsed alongside English: money formats, day-first and year-first dates and translated positions all come out as one English, ISO-dated schema. Unknown strings pass through with a warning, never guessed.
Null is not zero
Youth and fringe players with no published market value report price null — 7 of 43 on one measured squad — so a squad total that skips nulls stays correct.
The rare broken pagination shape handled
Market-value pages occasionally ship a doubled next-link that re-serves page 1. Page 1's link is checked against the constructed URL, and only then does the run chain the site's own links inside one session.
AWS WAF challenge waited for, not misreported
Transfermarkt sits behind AWS WAF on CloudFront. A bundled probe tells you whether your address is challenged, and the AWS WAF captcha solve path is implemented as the fallback.
Scraping infrastructure included
The same tested core runs every 2scraper repository, so the hard parts work the same way on every site.
CAPTCHA handling
Detects the challenge a page really serves and, with a 2Captcha key, solves it only when it blocks the data — a capped number of solves per page.
Proxy rotation
Single proxy or a pool from a file, rotated on blocks with a fresh browser per exit. Credentials stay out of argv and logs.
Fingerprints
Optional complete device identity from the 2Captcha Fingerprint API — user agent, client hints, locale and timezone that agree with each other.
Scraping Browser API
Point --cdp-endpoint at a managed Chromium with its own exit country — no browser infrastructure to run yourself.
Honest run metadata
Every run writes a .meta.json sidecar and distinct exit codes: complete, blocked, empty and partial are never confused, and a failed run never overwrites good data.
Diff two runs
diff_runs.py compares two complete Transfermarkt snapshots by id — new, removed and changed rows for monitoring.
What teams do with Transfermarkt data
Player valuation tracking
Snapshot the market-value ranking over time and diff runs to see whose value rose or fell.
Squad value and contract analysis
Pull a club's squad with values and contract expiry dates to model squad cost and upcoming renewals.
Transfer market monitoring
Follow the latest-transfers ledger with fees and fee types for scouting, journalism or betting research.
Player database enrichment
Fill in birth dates, heights, agents, caps and nationalities from player profiles for your own football dataset.
Free to run. Pay only for what the site requires.
Free to run · paid add-ons optional
Whether you need anything depends on your address: Transfermarkt is fronted by AWS WAF, and challenge rates varied from 8 of 10 requests on a datacentre address to 0 of 10 from a home connection on the same day (2026-09-16). Run tools/waf_probe.sh to find out for yours; if it is clean, a local engine needs no paid product. The README's documented default is the Scraping Browser API, which brings its own exit and profile; captcha solving (AWS WAF task) is the fallback, proxies spread volume over many exits, and fingerprints give a consistent identity.
Optional 2Captcha services — captcha solving, the Scraping Browser API, residential proxies and fingerprints — work behind one key.
Good to know
- The AWS WAF solve path is implemented but has not yet met a live challenge.
- Locale coverage is uneven; the transfers listing's fee-type text is matched in English only.
- Four page kinds only — not every category Transfermarkt publishes.
2scraper vs. hosted scraper APIs
| 2scraper | Typical hosted API | |
|---|---|---|
| Code | Open source, MIT — read and patch it | Closed |
| Cost | Free; optional services billed only when used | Subscription or per request / per result |
| Runs on | Your machine, server or CI | The vendor's infrastructure |
| Output | JSON/CSV files + run metadata | API responses |
| When the site changes | Fix it yourself or open an issue / PR | Wait for the vendor |
Transfermarkt scraper FAQ
Do I need a proxy or the Scraping Browser API?
Only if AWS WAF challenges your address. Run bash tools/waf_probe.sh 10 market-values; if it comes back clean, a local engine works on its own.
Which Transfermarkt sites are supported?
www.transfermarkt.com, .de, .world, .co.kr and .jp, each checked live. Other locale hosts are refused with the reason. Pass a full --url to use a locale.
What currency are market values in?
EUR on every supported host — Transfermarkt does not convert, only the formatting differs.
Can I run pages in parallel?
--concurrency is available for --mode transfers, whose pages are independently addressable, and is refused with --cdp-endpoint because a Scraping Browser profile allows one live connection.
Why are transfers in a different schema?
A transfer has fees and two clubs; a player has a position and a market value. Separate schemas avoid columns that are always null, while both share the same family prefix.
Is scraping Transfermarkt legal?
It reads publicly rendered pages and nothing behind an account. Comply with Transfermarkt's terms, robots.txt and the data protection law that applies to you.