Apparently, I Don't Know How to Build Small Projects

Now that I’ve officially jumped into open source, I’ve discovered a problem.

I have too many ideas floating around in my head.

Seriously. I’ve spent most of my career building things behind closed doors, and now that I’m putting projects out in public, I feel like I’m bursting at the seams. There’s so much I want to build, so much I want to share, and so much I want to contribute to the open-source community.

Apparently, I don’t have an off switch.

I recently released Sentrello, a self-hosted Unified Business Management Platform I’ve been building solo in my spare time for the past three and a half years. It’s a massive undertaking, and it’s nowhere near finished. Sentrello alone could keep a team of developers busy for years.

Naturally, I have two more massive projects in the works.

Apparently, one enormous project wasn’t enough.

Meet NetWeir

Project number two

The first of those projects just landed on GitHub: NetWeir.

NetWeir started with a fairly straightforward problem. I was doing research for Sentrello and needed to scrape websites for data. I started experimenting with a few existing open-source projects and got decent results, but felt I could build something a bit more powerful.

As Ricky Bobby famously put it, “If you ain’t first, you’re last.”

Is that a sound engineering principle? Absolutely not.

Has it stopped me before? Also absolutely not.

I’ve been building my own web scrapers for longer than some people reading this have been alive. Web scraping isn’t new to me, but the demands on modern scrapers have changed. Speed matters. Reliability matters. And websites have become considerably better at recognizing automated traffic.

I wanted something that could handle all three without making me compromise on the things that matter.

I couldn’t find a crawler that met my expectations, so I started researching, planning, and testing.

A few hours later, NetWeir was born.

There may also have been a few beers involved. I’m not going to pretend every great engineering decision in history happened in a brightly lit conference room with a whiteboard and a dozen people arguing over architecture diagrams.

Sometimes, you just need to sit down, think, experiment, and see what happens.

Fast, Resilient, and Built for Scale

Rust underneath, Python on top

NetWeir is an early-stage project with an ambitious goal: build an ultra-fast, self-healing web scraper that can extract clean data at scale.

The architecture puts the heavy lifting in a high-concurrency Rust core: fetching, HTML parsing, crawl scheduling and extraction all run in Rust, outside Python’s GIL. Python is where you write the scraper itself, with an API that feels familiar if you’ve used Scrapy or Beautiful Soup.

The idea is to combine the strengths of both languages rather than force one to do everything.

NetWeir is being designed around three primary goals:

• Speed. High concurrency and a Rust-powered core designed to process large workloads efficiently.
• Resilience. Self-healing behavior intended to help the scraper recover when websites change or individual operations fail.
• Reliable data extraction. A fast parser with CSS, XPath, and Beautiful Soup-style queries, built to turn messy web content into clean, usable data.

The project also aims to handle anti-bot defenses, a persistent challenge for modern web scraping. That is a particularly difficult part of the problem, and there’s still plenty of work ahead before I can claim NetWeir has delivered on every ambition.

These are the goals, not a declaration that the entire problem has already been solved.

There’s a difference between having a promising architecture and having a mature, battle-tested product. I’m very aware of that distinction.

It’s Early. Very Early.

Days, not years

Unlike Sentrello, which has been in development for three and a half years, NetWeir’s first commit landed only a few days ago.

It’s already on PyPI, though. Version 0.1.0 went out the same week, so if you want to kick the tyres, pip install netweir is all it takes.

It’s still in its early stages.

The initial benchmarks are encouraging, which is a good sign. But benchmarks are just one piece of the puzzle. Real-world workloads, changing website structures, failure recovery, and sustained performance will all have their say as development continues.

I’m not going to pretend a few promising numbers mean the project is finished. They don’t.

What they do mean is that there’s enough potential here to make me want to keep pushing.

And now that the project is public, I can build it in the open.

That’s the part I’m particularly excited about.

One Project Down. One More to Go.

Building it in the open

NetWeir is the first of two additional major projects I’ve been working on alongside Sentrello. I’m not ready to reveal everything about the other one just yet, but there’s more coming.

For now, I want to focus on getting NetWeir moving in the right direction, improving the implementation, testing the architecture, and turning the early promise into something people can actually put to work.

I’m also looking forward to seeing what happens when other developers get involved. Open source isn’t just about publishing code and walking away. It’s about making the work visible, learning from people with different perspectives, and giving others an opportunity to help shape what a project becomes.

After spending so much of my career working on private enterprise systems, that’s a refreshing change.

And a slightly terrifying one.

There’s nowhere to hide a questionable implementation when the repository is public. On the other hand, there’s no better motivation to keep improving your work.

If you’re interested in high-performance web scraping, Rust, Python, or just watching an ambitious project grow from its earliest stages, take a look at NetWeir on GitHub.

It’s early days, and there’s plenty left to build.

Which, as it turns out, is exactly how I like it.

I should probably learn to relax at some point.

Maybe after the third project.