NetWeir: A Web Scraper Where Rust Does the Work and You Write Python

Overview

I needed data for Sentrello research, so I scraped for it. The open-source tools got me decent results. Decent, though, isn’t what I was after, and I thought I could build something more powerful.

NetWeir is what I built instead: a web scraper with a Rust core and a Python API. Fetching, parsing, crawl scheduling and extraction all run in Rust, off Python’s GIL. You write the spider in Python, the way you would anyway.

The problem

Most sites don’t block a scraper by reading its code. They block it by the first few hundred bytes of the connection: which TLS ciphers and extensions arrive, what the HTTP/2 settings frame says, which headers come first. A Python HTTP library gives itself away before it has asked for anything.

Then there’s the long crawl. Somewhere around page 600 the site serves a challenge page as a 403, or the process dies, or a redesign renames the one class your price selector depended on. Each of those ends the same way: an empty column you don’t notice until later.

What I built

NetWeir sends what Chrome sends. Not something close: the same ClientHello, the same HTTP/2 settings and priorities, the same headers in the same order. Firefox and Safari profiles are there too, each with its own habits copied. You don’t have to take that on trust, either. Chrome was recorded making four navigations, the test suite has NetWeir make the same four, and it fails if a single request differs; and the public checker at tls.peet.ws reports the same JA4 fingerprint for Chrome and for NetWeir.

If you’ve used Scrapy, the API won’t surprise you:

import netweir

page = netweir.get("https://books.toscrape.com/")
page.css(".price_color::text").getall()  # ["£51.77", "£53.74", ...]

CSS with ::text and ::attr(), all of XPath 1.0, and Beautiful Soup’s find_all family all work, so your selectors come with you unchanged. Describe a spider declaratively and Rust does every page without running any Python at all; against a local 1,050-page shop, that took about two thirds of the time of the same spider written as a callback.

When the site pushes back

Three things go wrong on a long crawl, and NetWeir handles each one without being asked.

A block page from Cloudflare, Akamai, DataDome, HUMAN, Kasada, Imperva or AWS WAF is recognised as a block, not mistaken for a page. The request gets a fresh session, the site gets slowed down, and if it still won’t budge, a real Chrome can take that one request and hand the cookies it earns back to the fast HTTP client.

A crawl that dies, whether from a crash, a reboot or a kill -9, picks up where it stopped. No page fetched twice, no item written twice. The tests kill crawls at random moments to prove it.

A selector that stops matching after a redesign finds the element most like the one it remembers, and tells you it had to. You get a warning instead of an empty column.

Where it is now

Days old, and honest about it. Version 0.1.0 is on PyPI with wheels for Linux, macOS and Windows, Python 3.10 and up. The README lists what works today and what doesn’t yet; next on that list is Firefox in the browser driver, once there’s a Firefox that doesn’t announce it’s automated.

Try it

pip install netweir

Pages that need a real browser want Chrome too, and netweir install chrome fetches the version NetWeir is tuned for. The code, the benchmarks and the design notes are all on GitHub.