A failed crawl is not an empty result

A failed request and an empty response are different facts. If a measurement pipeline stores both as “zero records”, every analysis built on it can turn a collection failure into a finding about the world.

That distinction shaped the infrastructure behind two studies: one followed links from more than 498 million Telegram messages into the open web; the other evaluated tracking-pixel detection on 44,710 real emails. The subjects differ, but both depend on knowing not only what was observed, but what could not be observed and why.

Three states, not one

For any source and collection window, there are at least three outcomes:

1. Observed. The source answered and the collector stored what it returned.
2. Absent. The source answered successfully and the requested record was not there.
3. Unavailable. The source timed out, rejected the request, changed shape, or returned something the collector could not interpret.

Only the second supports an absence claim. The third supports a claim about collection coverage. Treating them as the same value is convenient at ingestion time and expensive at review time, when somebody asks whether a missing record means “not present” or “not seen”.

Retries do not remove this problem. They change how often the unavailable state occurs. A source can refuse every retry, answer only part of a request, or return a successful response whose body is an interstitial rather than the requested content. The pipeline still needs a durable account of what happened.

Raw observations and derived facts have different lifetimes

The collector stores the response and its collection context before asking what the response means. Normalised records, classifications, resolved destinations, and aggregate counts are separate derived facts that retain a link to that observation.

This costs storage and makes queries less direct. It also makes correction possible. A parser, normalisation rule, or classifier can be changed and rerun over existing observations. If the normalised row overwrites the raw one, the same correction requires collecting the source again, assuming the source still exists and will answer in the same way.

That assumption is especially weak on the open web. Pages disappear, redirects change, domains expire, and platforms alter what an unauthenticated client can retrieve. Re-collection is not a neutral replay; it is a new measurement taken under different conditions.

Checkpoints protect progress, not validity

At this scale, interruption is ordinary. Collection is split into resumable units, and a committed checkpoint records which unit completed. That prevents a worker crash from restarting a run over hundreds of millions of records.

A checkpoint proves less than it appears to. It can prove that the worker finished processing a unit; it cannot prove that the source returned everything expected inside that unit. Completeness needs its own evidence: boundary checks, response counts, explicit failure records, and coverage summaries that remain visible to the analysis.

The useful operational question is therefore not only “where do I resume?” but “which gaps will a resume preserve?” A pipeline that advances its checkpoint after a partial response is reliable in the narrow sense that it keeps running, and unreliable in the sense that matters to a paper.

Provenance is part of the result

The USD 71 million figure in the Telegram study is the end of a chain: message, extracted URL, resolved destination, classified service, attributed monetisation mechanism, aggregate. Each edge has to remain traversable in the opposite direction.

That does not make the result automatically correct. It makes it inspectable. A reviewer can move from the aggregate to the records that contributed to it; a changed classifier can identify which outputs it affects; an unavailable source remains unavailable rather than becoming a zero.

The general rule is simple: preserve observations, derivations, and collection failures as different data. Storage is cheaper than recollection, and uncertainty recorded at ingestion is more useful than certainty reconstructed after publication.