2scraper
What the Screener.in scraper collects
Scrapes screener.in result tables — any stock screen (/screens/{id}/{slug}/) and any sector or industry listing (/market/…) — into JSON and CSV, one row per company in the table's own order. Rows carry price in INR, market cap, P/E, dividend yield, quarterly sales and profit with their changes, ROCE, and every other column the page shows under its own name and unit. A .meta.json sidecar records the site's own result count and whether the run was exhaustive.
Sector & industry listings
Input: A /market/… sector or industry URLOne row per company in the sector, sorted by market cap as the site shows it, with the listing's heading as category.
Stock screens
Input: A /screens/{id}/{slug}/ URL — any public screenThe screen's result table: the standard metrics plus every custom column the screen adds, kept under its own name and unit in extra_metrics.
Plain HTTP, no browser
Input: Any screen or sector URLThe same rows from the served HTML with only requirements.txt installed: 1.2 s and 47 MB RSS against Playwright's 2.9 s and 150 MB, measured 2026-09-28.
From zero to Screener.in data in 3 steps
Install
Clone the repository and install one browser engine. Python 3.9+.
$ git clone https://github.com/2scraper/screener-scraper && cd screener-scraper $ python3 -m venv venv && ./venv/bin/pip install -r requirements.txt -r requirements-playwright.txt $ ./venv/bin/playwright install chromium
Run a scraper
Pick what you need and run it. Every flag is documented in the README.
$ python3 playwright_scraper.py --url "https://www.screener.in/market/IN08/IN0801/IN080101/" --pages 1 --out it_software$ python3 playwright_scraper.py --url "https://www.screener.in/screens/59/magic-formula/" --pages 1$ python3 http_scraper.py --url "https://www.screener.in/market/IN08/IN0801/IN080101/" --pages 1Get structured data
Rows land in JSON or CSV (
--format), with a.meta.jsonsidecar recording status, pages fetched and why the run stopped.
Screener.in data fields
22 columns per row in a predictable schema. The most useful ones:
| Field | What it holds |
|---|---|
sku | The site's own company id |
title | Company name as the table links it |
ticker | What the company URL carries: letters (TCS) or a six-digit code |
price | Current market price |
currency | INR, read from the price column's own unit |
rank | The site's own S.No., global across pages |
consolidated | Whether the row links consolidated or standalone figures |
market_cap_cr | Market capitalisation, Rs. crore |
pe | Price to earnings |
dividend_yield_pct | Dividend yield, % |
net_profit_qtr_cr | Latest quarter net profit, Rs. crore |
qtr_profit_var_pct | Quarterly profit change, % |
sales_qtr_cr | Latest quarter sales, Rs. crore |
qtr_sales_var_pct | Quarterly sales change, % |
roce_pct | Return on capital employed, % |
extra_metrics | Every other column, keyed by the site's own name and unit |
[
{
"source": "screener.in",
"scraped_at": "2026-09-25T09:16:57.869422+00:00",
"url": "https://www.screener.in/company/SPARC/",
"sku": "3244",
"title": "SPARC",
"price": 210.0,
"currency": "INR",
"category": "Magic Formula",
"page": 1,
"position": 1,
"rank": 1,
"ticker": "SPARC",
"consolidated": false,
"market_cap_cr": 6814.96,
"pe": 4.34,
"dividend_yield_pct": 0.0,
"net_profit_qtr_cr": -20.98,
"qtr_profit_var_pct": 59.68,
"sales_qtr_cr": 39.92,
"qtr_sales_var_pct": 314.11,
"roce_pct": 164.99,
"extra_metrics": {
"Return on invested capital (%)": 134.34,
"Earnings yield (%)": 22.73,
"Book value (Rs.)": 41.09
}
},
{
"source": "screener.in",
"scraped_at": "2026-09-25T09:16:57.869917+00:00",
"url": "https://www.screener.in/company/532015/",
"sku": "1203",
"title": "Gravity (India)",
"price": 14.46,
"currency": "INR",
"category": "Magic Formula",
"page": 1,
"position": 2,
"rank": 2,
"ticker": "532015",
"consolidated": false,
"market_cap_cr": 112.82,
"pe": 6.03,
"dividend_yield_pct": 0.0,
"net_profit_qtr_cr": 6.01,
"qtr_profit_var_pct": 8485.71,
"sales_qtr_cr": 60.0,
"qtr_sales_var_pct": 19254.84,
"roce_pct": 158.68,
"extra_metrics": {
"Return on invested capital (%)": 88.05,
"Earnings yield (%)": 22.28,
"Book value (Rs.)": 1.91
}
}
]First 2 of 5 rows from a real run — sample_output.json. Long values shortened here.
Handles what breaks naive Screener.in scrapers
Columns matched by name and unit, not position
A screen that adds a column moves every column after it. Each header is identified by its metric name and unit, and a header in a different unit goes to extra_metrics rather than into a typed column that promises the wrong unit.
Out-of-range pages detected
?page=8 of a 7-page listing returns page 7's rows with HTTP 200. The scraper reads which page the site says it served and plans against "Showing page 1 of 7", so it never duplicates the last page.
Lost pages proven by rank
The site numbers rows globally across pages, so the sidecar's rank_gaps lists any site rank missing from the merged rows — proof of a lost page rather than a guess.
The registration wall is not an empty screen
When screener.in answers with its /register/ page, the run reports it as blocked (exit 3) with advice, and never parses it as a screen with no results.
Scraping infrastructure included
The same tested core runs every 2scraper repository, so the hard parts work the same way on every site.
CAPTCHA handling
Detects the challenge a page really serves and, with a 2Captcha key, solves it only when it blocks the data — a capped number of solves per page.
Proxy rotation
Single proxy or a pool from a file, rotated on blocks with a fresh browser per exit. Credentials stay out of argv and logs.
Fingerprints
Optional complete device identity from the 2Captcha Fingerprint API — user agent, client hints, locale and timezone that agree with each other.
Scraping Browser API
Point --cdp-endpoint at a managed Chromium with its own exit country — no browser infrastructure to run yourself.
Honest run metadata
Every run writes a .meta.json sidecar and distinct exit codes: complete, blocked, empty and partial are never confused, and a failed run never overwrites good data.
Diff two runs
diff_runs.py compares two complete Screener.in snapshots by id — new, removed and changed rows for monitoring.
What teams do with Screener.in data
Screen result tracking
Run a public stock screen such as Magic Formula on a schedule and diff the companies entering and leaving it by sku.
Sector fundamentals snapshots
Pull every company in an industry with market cap, P/E, ROCE and quarterly sales and profit for comparison.
Quant research inputs
Feed typed fundamentals and a screen's custom metrics (ROIC, earnings yield, book value) into a model or spreadsheet.
Lightweight scheduled jobs
Use the plain-HTTP engine on a small server: no browser, about 1 second per page.
Free to run. Pay only for what the site requires.
Runs with no key, proxy or account
From an ordinary connection you need no key, no proxy and no account: on 2026-09-25 plain curl from a residential connection was served sector and screen pages in full, and the daily canary passes from a bare GitHub runner. 2Captcha proxies add volume and an Indian exit; the Scraping Browser API gives a remote browser with a chosen country. No captcha is configured on any page captured; the solver is wired in and bounded to one solve per page in case one appears.
Optional 2Captcha services — captcha solving, the Scraping Browser API, residential proxies and fingerprints — work behind one key.
Good to know
- Pages after the first are disallowed by screener.in's robots.txt; every engine warns when a run would request them.
- Prices and ranks move during Indian market hours, so compare runs by sku.
- --concurrency and --fingerprint are implemented in the Playwright engine only; the plain-HTTP engine has no captcha solving.
2scraper vs. hosted scraper APIs
| 2scraper | Typical hosted API | |
|---|---|---|
| Code | Open source, MIT — read and patch it | Closed |
| Cost | Free; optional services billed only when used | Subscription or per request / per result |
| Runs on | Your machine, server or CI | The vendor's infrastructure |
| Output | JSON/CSV files + run metadata | API responses |
| When the site changes | Fix it yourself or open an issue / PR | Wait for the vendor |
Screener.in scraper FAQ
Which pages does it scrape?
Any stock screen (/screens/{id}/{slug}/) and any sector or industry listing (/market/…). Nothing is hardcoded: the URL you give decides what a run covers.
Do I need a proxy or a 2Captcha key?
No, not from an ordinary connection. Paid products buy volume, an Indian exit or no local browser.
Can I get every page of a listing?
The scraper can plan every page from the site's own page count, but pages after the first are ?page= URLs, which screener.in's robots.txt disallows for all user agents. Read the robots.txt and the site's terms before running multi-page jobs.
Can I get 50 rows per page?
No. ?limit=10 and ?limit=50 both returned 25 rows to an anonymous visitor, so there is deliberately no --limit flag.
What formats does it write?
JSON and CSV, plus a .meta.json sidecar with status, total_results, pages_available, coverage (exhaustive, tail or window) and a SHA-256 of each data file.
Is scraping screener.in legal?
It reads publicly available tables. Respect robots.txt and the site's terms, rate-limit with --delay, and follow the law where you operate; the data itself is supplied to screener.in by a third party.