An index must preserve the specification's distinctions

A specification can be implemented faithfully in the application layer and violated by one derived database column. The code still runs, the index still answers quickly, and the caller receives the wrong rows.

The line

A Nostr event carries tags: arrays whose first element is the tag name and whose remaining elements are its values. NIP-01 lets clients filter on them: give me events with this e tag, or that p tag.

To make that fast, an implementation extracts the tag values into an indexed column. And there the representation choice appears: do you store the value, or do you store the value with its tag key attached?

Store the bare value and the index cannot distinguish an e tag whose value is X from a p tag whose value is X. A filter on one matches the other. That is a merged pull request to fiatjaf/eventstore, the collection of storage backends maintained by the protocol’s author: prefix tag values with their tag key, for correct NIP-01 filtering.

It is a small diff. It is also the kind of bug that does not crash, does not appear in a log, and does not fail a test that nobody wrote. It just quietly returns the wrong rows to whoever asked.

The same line, in my own system

BigBrotr archives events from thousands of relays, and it has to answer the same filter questions over 250 million of them. Its events table carries a tagvalues column, computed at insert time and indexed with GIN precisely so those queries are cheap.

Which means I made this decision twice: once as a fix in someone else’s storage backend, and once as a schema choice in my own. The protocol does not decide it for you. It defines what a filter means and leaves every implementation to find a representation that preserves that meaning under indexing, and there is more than one way to get it wrong.

The schema bug and the specification use different languages for the same invariant. The specification says filters distinguish tag keys; the index must therefore preserve the tag key in its representation. Once the invariant is stated that way, both the implementation and its tests have something precise to enforce.

This is the useful boundary between specification and storage. A specification defines the distinctions an implementation must preserve. A storage design decides which of those distinctions remain representable after denormalisation, indexing, and compression. Performance work is correct only while that mapping remains intact.