BigBrotr

BigBrotr combines active monitoring of Nostr relays with an archive of the signed events they serve. I independently build and operate the OpenSats-funded project, preserving both the content and the evidence of where and when it was observed.

Role

Independent development and operation

Funding

OpenSats grant

Implementation

Go, append-only journal, ClickHouse, PostgreSQL

Operations

Self-hosted Linux, Proxmox, ZFS, Prometheus and Grafana

BigBrotr/bigbrotrRelay discovery and monitoring, verified event archiving, provenance analytics and shared query access.Go

As of 7 September 2026, the public default branch still contains the Python implementation. The Go storage, recovery and serving paths described below are current development work; they are not presented as a release available from that default branch.

The problem

An archive and a network monitor answer different questions. The archive records what a relay served. The monitor records whether the relay was reachable, how it responded and which protocol capabilities it exposed. A failed request is not evidence that a relay has no events, and receiving the same event from another relay is not a redundant measurement.

These distinctions matter when sources are independently operated. Relays can return short pages, disappear during collection or deliver events that have already been archived elsewhere. The system has to preserve observations across retries without mistaking a partial collection for a completed one.

What I built

The Go pipeline separates acquisition, durable storage, analytical indexing, monitoring and serving. PostgreSQL owns relay and service coordination state. An append-only journal holds the event and provenance records from which ClickHouse indexes can be rebuilt. Badger holds disposable deduplication state, not the only copy of an observation.

That division sets the recovery contract. Losing an analytical index must not require asking unreliable external sources to serve the archive again. Losing PostgreSQL is a different failure: its control state needs its own backup, not just journal replay.

The project I operate monitors more than 5,000 relays, as of September 2026. This is the operated-project scope, not a benchmark of the Go implementation or a guarantee that every relay is reachable or revisited within a fixed interval.

Discovery feeds candidates to protocol validation. The monitor records reachability, response latency and capability observations with bounded concurrency and adaptive backoff. Stored measurements do not require public broadcasting: signed NIP-66 and NIP-85 publication is separately configurable and opt-in.

The HTTP API, Nostr data service and read-only relay use one bounded read core. Filter rules, limits and access to derived data are implemented once rather than independently in each interface. Analytics readers select completed windows; derivations reject truncated input instead of allowing a partial result to look finished.

Hard parts

Defining what is durable

The ingestion boundary is a sealed, flushed journal segment. The synchronizer does not advance its durable checkpoint or mark an event as stored before that boundary. A crash can therefore cause a retry, but progress is not recorded ahead of the source records it claims to represent.

Replay is at least once. Preserving provenance consequently requires more than detecting the same event ID: a later delivery must not overwrite an earlier observation at the same relay. The indexing logic preserves the earliest observation across replays. The extra record is part of the measurement, even when its event body is already known.

I verify recovery of event content, provenance and tags after ClickHouse loss in an automated database-rebuild test. That checks the recovery mechanism; it is not a production-scale recovery-time guarantee.

Separating scheduling from indexing

A shared service tick made replay compete with the scheduling of other work. The indexer uses a resident drain instead: it continues replay outside the supervisor’s per-cycle budget while retaining a single ordered writer.

This is a separation of responsibilities, not an assumption that more writers are faster. Ordering still governs replay and compaction. Compaction runs on the idle branch of that worker, so it does not race the replay path over journal state. Removing the scheduling dependency does not remove CPU, disk or database contention.

Compressing the authoritative copy safely

Compression changes the recovery source itself. The compactor writes and flushes a compressed temporary file, renames it into place, verifies decompression and the segment checksum, and only then retires the raw segment. An interruption must leave a readable copy rather than a registry entry pointing to half-written data.

Zstandard compression reduces the representation without dropping content or provenance. The implemented format uses sequential compressed frames; it is not a claim of indexed random access into compressed event bodies. I do not attach a whole-system storage-saving percentage to a component compression experiment.

Collecting without inventing completeness

A short relay response does not close a collection window by itself. The synchronizer requires an additional page with no unseen events and separately revisits late arrivals through a delayed safety tail. This costs additional reads but avoids treating a relay’s page limit as the end of its history.

The same discipline applies to operations. I own deployment, storage and backups on a self-hosted Linux server using Proxmox and ZFS, with application and host metrics and alerts in Prometheus and Grafana. The service’s recovery rules and the host’s backup strategy cover different failure modes; neither replaces the other.