Personal
sitesift
A URL classification pipeline that splits deterministic evidence collection from LLM judgment, producing structured and reproducible records of what a site is, what it is about, and how confident the verdict was.
Given a list of URLs, sitesift produces one structured record per URL: what kind of site it is, what it is about, what language it is in, technical metadata, quality flags and, for each decision, how confident it was and which method produced it.
- Input
- A list of URLs
- Output
- Validated records with per-decision method and confidence
- Split
- Deterministic evidence, then LLM judgment
- Role
- Author
- Repository
- VincenzoImp/sitesift
The problem
Pulling text and metadata out of a page is solved: sitesift uses trafilatura for it. The gap
is turning that evidence into a judgment: this is a news site, it is about football, it is
parked, this one is a login wall. Judgments are what downstream work needs, and they are what
nothing gives you cheaply, reproducibly, and at a scale where every extra model call is a line
item.
What I built
The pipeline is split in two, and the split is the design.
The deterministic layer – normalise, fetch, extract – never calls a model. It deduplicates by
registrable domain, respects robots, guards against SSRF, rate-limits, extracts with
trafilatura, reads JSON-LD, and detects language with a language identifier rather than by
asking a model to guess. Its job is to produce every canonical fact about a page.
The judgment layer is where the model decides. It reads the whole evidence bundle and returns
the site type and topic hierarchy, starting with a cheap model and escalating to a stronger one
only when the cheap one is not confident. Every record carries which method produced it:
llm_small, llm_large, or blocked.
The one decision the deterministic layer is allowed to make is to skip a page that has no content at all: dead, parked, a soft 404, not HTML. Those never reach a model, so no call is spent on a page with nothing to read.
Hard parts
Keeping evidence reusable. Separating acquisition from classification lets a changed prompt reuse the collected evidence. Fetching, normalization and parsing can be tested independently of provider calls, and a failed classification does not require collecting the page again.
Using confidence to control escalation. The smaller model’s reported confidence decides which records receive a second pass. This limits stronger-model calls to a subset of the corpus, but depends on the usefulness of that confidence signal. Recording it makes the routing decision auditable; it does not establish that the signal is calibrated or that the second answer is right.
Saying what the output is not. The records are indicative flags. One page is fetched per URL, JavaScript is not rendered, and nothing here is certified brand-safety. Those limits are stated in the repository because a structured, confident-looking record invites being used as if it were authoritative.