Personal
sitesift
A URL classification pipeline that splits deterministic evidence collection from LLM judgment, producing structured and reproducible records of what a site is, what it is about, and how confident the verdict was.
Given a list of URLs, sitesift produces one structured record per URL: what kind of site it is, what it is about, what language it is in, technical metadata, quality flags and, for each decision, how confident it was and which method produced it.
- Input
- A list of URLs
- Output
- Validated records with per-decision method and confidence
- Split
- Deterministic evidence, then LLM judgment
- Role
- Author
- Repository
- VincenzoImp/sitesift
The problem
Pulling text and metadata out of a page is solved: sitesift uses trafilatura for it. The gap
is turning that evidence into a judgment: this is a news site, it is about football, it is
parked, this one is a login wall. Judgments are what downstream work needs, and they are what
nothing gives you cheaply, reproducibly, and at a scale where every extra model call is a line
item.
What I built
The pipeline is split in two, and the split is the design.
The deterministic layer – normalise, fetch, extract – never calls a model. It deduplicates by
registrable domain, respects robots, guards against SSRF, rate-limits, extracts with
trafilatura, reads JSON-LD, and detects language with a language identifier rather than by
asking a model to guess. Its job is to produce every canonical fact about a page.
The judgment layer is where the model decides. It reads the whole evidence bundle and returns
the site type and topic hierarchy, starting with a cheap model and escalating to a stronger one
only when the cheap one is not confident. Every record carries which method produced it:
llm_small, llm_large, or blocked.
The one decision the deterministic layer is allowed to make is to skip a page that has no content at all: dead, parked, a soft 404, not HTML. Those never reach a model, so no call is spent on a page with nothing to read.
Hard parts
Deciding where the model is allowed to be. Letting a model do the extraction too would be simpler to write and would destroy every property that matters here: a run could not be resumed after a crash, tests could not run offline, reclassifying under a new prompt would mean re-fetching the whole corpus, and prompt caching would have nothing stable to cache. Keeping acquisition deterministic is what makes the expensive layer replaceable.
Escalation as a cost control, not a quality knob. Sending everything to the strong model is the accurate option and the unaffordable one; sending everything to the cheap model is affordable and wrong on the hard cases. Routing on the cheap model’s own confidence means the bill scales with corpus difficulty rather than corpus size, and it means the confidence signal has to be trustworthy, which is why it is recorded per record rather than assumed.
Saying what the output is not. The records are indicative flags. One page is fetched per URL, JavaScript is not rendered, and nothing here is certified brand-safety. Those limits are stated in the repository because a structured, confident-looking record invites being used as if it were authoritative.