The flag vocabularies, and the scan substrate they share

Two things, one idea. The idea is that the store's flag field is a substrate —
classify at write time, retrieve cheaply by class later — and every use of it is a
vocabulary: named bits, a writer, a consumer. The mechanism is built once; the
vocabularies attach as they go.

DESIGN-flag-vocabularies.md is the running inventory: the space×flags split (space
is the hard partition, flags the soft classification), the eight vocabularies with
the evidence already in the tree that each wants the same scan (events, sealing,
lineage, lifecycle, capability, association, audit, tiering), the v4 layout
decision (a 16-bit class mask in the index entry, never inside the value), and the
CUBE_OP_FLAG_SCAN op with the userspace scan as its oracle. Events are the first
consumer because the kernel already captures the boot record; the flag is the only
thing missing to make it retrievable by class.

cube-store::flagged is that oracle now: scan_by_flag / count_by_flag over flagged
records, Any and All modes, order-preserving, a zero mask matching nothing, and an
unknown bit matched by mask rather than interpreted — so a vocabulary can attach
later without a format change. Pinned by tests that build a Wi-Fi deauth, an OOM, a
throttle and a boot and scan them three ways.

This carries the EventFlags field from the previous commit into its first use, and
closes the "unused visited" warning the range-seek left in non-debug builds.

DESIGN-flag-vocabularies.md

@@ -0,0 +1,78 @@
+# The flag vocabularies, and the scan substrate they share
+
+This is a running inventory, not a one-time design. The store's flag field is a **substrate** — a
+way to *classify at write time and retrieve cheaply by class later* — and every use of it is a
+**vocabulary**: a set of named bits with its own meaning, its own writer, and its own consumer. The
+point of this document is to keep the mechanism and the vocabularies separate, so that a new use
+case adds bits and a writer, never a second format change.
+
+## 1. The mechanism, stated once
+
+A record already travels as `(space, key, value)`. The two axes that make the whole scheme work are
+already there:
+
+* **space** is the hard partition — *who / where*. Root `0x00`, boot `0xFC`, keystore `0xFD`, portal
+  `0xFE`, edge `0xFF`, and the named spaces. Crossing a space is structurally impossible from a
+  query; you must name it.
+* **flags** are the soft classification — *what kind / what state*. A record carries a mask of bits;
+  a reader tests bits. Classification is done once, at write time, by the writer that had the most
+  context it will ever have (the kernel, for kernel-captured events); every later reader is spared
+  re-deriving it.
+
+The substrate is therefore exactly one thing: **the record carries a class mask, and a
+`scan_by_flag` op returns the records whose mask matches, in `O(log n + matches)`** — the same
+binary-search-then-forward-read shape `CUBE_OP_RANGE` already has. The mask lives in the *index*
+entry, not inside the value, which is the entire difference between a wordscan and another
+`O(records)` walk.
+
+## 2. The vocabularies, as they emerge
+
+Each row is a vocabulary: the question it answers, the bits it will own, and the evidence already in
+the tree that it wants the same scan. This table is the "as we go" part — it grows as a use case
+proves itself, and the column that matters is *the hint that's already there*, because a vocabulary
+without a live need is a bit that will never be read.
+
+| vocabulary | the question it answers | the hint already in the tree |
+|---|---|---|
+| **events** | what kind of event did the kernel just capture | the boot record (build #52); the `autonomic-os` Tier-0 loop |
+| **sealing** | sealed vs plaintext | `HeaderFlags::ENCRYPTED`; the "0 of 69,638 records sealed" finding |
+| **lineage** | kernel wrote this vs a client wrote it | the write-authority handover; the boot record's self-authored value |
+| **lifecycle** | current vs superseded vs tombstone; fold-eligible | the WAL overlay; deletes; `checkpoint_every` |
+| **capability** | in destroy-scope / reachable by this principal | keystore `0xFD`; `PERM_ROOT`/`USER`; the kill switch |
+| **association** | edge vs vertex | `ASSOC_EDGE`; portals |
+| **audit** | part of a transaction / rolled back / under retention | (none yet) |
+| **tiering** | hot / cold / archive | (none yet — the per-call-open and sparse-index cost notes point at it) |
+
+The first consumer is **events**, because it is the one with a gate already half-built (the kernel
+captures the boot record today; the flag is the only thing missing to make it retrievable by class).
+
+## 3. The layout decision
+
+The class mask is **one small per-record field in the index**, not a per-vocabulary field and not a
+single field argued over by every vocabulary. The reason is the split in §1: **space is already the
+partition; flags are the subclassification inside it.** A vocabulary that needs room takes its own
+space — events, being cross-cutting, get a reserved space (`0xFB`) so a scan can mean either "every
+event anywhere" or "events of this class in this space".
+
+Concretely, `v4` extends the index entry from `key | value_off | value_len` to
+`key | flags(2) | value_off(8) | value_len(8)`. `flags` is the raw 16-bit mask; the *meaning* of its
+bits is a vocabulary's business, never the format's. The writer sets it at write time (the kernel's
+`append` gains a `flags` argument; the boot record passes `EventFlags::BOOT`); a reader never
+interprets a bit it does not know, and an unknown set bit survives a round-trip untouched — the same
+out-of-band discipline `HeaderFlags` already keeps.
+
+## 4. The op
+
+`CUBE_OP_FLAG_SCAN` (its own `size`-versioned block, like `cube_range_args`): a space (or the
+reserved "any" space), a 16-bit `mask`, a `mode` (any / all), a cursor, and a buffer. Counts can be
+derived, but a `count` answer over a walk the kernel already does is the honest O(1) version the
+notes name: maintain per-class counters in the index and answer a count without walking. The gate
+diffs the kernel's scan against the userspace one over the same flagged records, exactly as
+`verify-enum` diffs the walk — the userspace scan is the oracle, the kernel op is what must match it.
+
+## 5. What is deliberately out of scope, for now
+
+A model in the correction path. The substrate is deterministic and kernel-authored: the kernel
+stamps the flag, a reader tests bits. A model's job is over the *residue* a scan surfaces — the
+events that share a class but still surprise — and it proposes; a deterministic layer executes with
+preconditions. That layer is the subject of a later document, not of the substrate.

crates/cube-store/src/flagged.rs

@@ -0,0 +1,121 @@
+//! The scan half of the flag substrate: classify at write, retrieve cheaply by class later.
+//!
+//! This is the userspace oracle. The kernel's `CUBE_OP_FLAG_SCAN` walks a sorted index whose
+//! entries carry a class mask; this module is the same semantics over the records a userspace store
+//! holds, so a gate can diff the two and know a mismatch is the kernel's, not the model's.
+//!
+//! Nothing here interprets a bit. A mask is a raw `u16`; the *meaning* of its bits is a vocabulary's
+//! business ([`EventFlags`] today, sealing / lineage / lifecycle later). A reader that does not know
+//! a bit still matches it by mask, which is what lets vocabularies attach without a format change.
+
+use cube_core::EventFlags;
+
+/// Whether a scan matches a mask when the record has *any* of its bits, or *all* of them.
+///
+/// `Any` answers "every error" (`ERROR`), "every Wi-Fi event" (`WIFI`); `All` answers "every Wi-Fi
+/// error" (`WIFI | ERROR`). One field, three questions — the point of a set over a category.
+#[derive(Copy, Clone, Debug, PartialEq, Eq)]
+pub enum FlagMode {
+    /// The record's mask shares at least one bit with the scan mask.
+    Any,
+    /// The record's mask contains every bit of the scan mask.
+    All,
+}
+
+/// The records whose class mask matches `mask` under `mode`, in the order they were given.
+///
+/// Kept order-preserving rather than sorted on purpose: the kernel op returns records in the
+/// index's key order, and the gate compares against this in the same order, so neither side is
+/// asked to agree on a sort it did not promise.
+pub fn scan_by_flag<'a, T>(records: &'a [(EventFlags, T)], mask: u16, mode: FlagMode) -> Vec<&'a T> {
+    records
+        .iter()
+        .filter(|(flags, _)| matches(*flags, mask, mode))
+        .map(|(_, payload)| payload)
+        .collect()
+}
+
+/// How many records match `mask` under `mode` — the count the kernel can answer in O(1) with
+/// per-class counters rather than by walking.
+pub fn count_by_flag<T>(records: &[(EventFlags, T)], mask: u16, mode: FlagMode) -> usize {
+    records
+        .iter()
+        .filter(|(flags, _)| matches(*flags, mask, mode))
+        .count()
+}
+
+/// The one predicate both directions of the scan share. A mask of zero matches nothing — asking
+/// "what is this" with no class named is asking no question, and a scan that returned everything
+/// on an empty mask would be a walk wearing a scan's hat.
+fn matches(flags: EventFlags, mask: u16, mode: FlagMode) -> bool {
+    if mask == 0 {
+        return false;
+    }
+    match mode {
+        FlagMode::Any => flags.matches_any(mask),
+        FlagMode::All => flags.matches_all(mask),
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    /// The classify-then-scan loop end to end, with the events a kernel would actually stamp: a
+    /// Wi-Fi deauthentication (WIFI|ERROR), an OOM (MEM|ERROR), a thermal throttle (CPU|POWER), and
+    /// the boot (BOOT). The scan is then asked the three questions the vocabulary exists for.
+    #[test]
+    fn the_scan_answers_any_all_and_count_from_the_same_bits() {
+        let deauth = EventFlags::from_bits(EventFlags::WIFI | EventFlags::ERROR);
+        let oom = EventFlags::from_bits(EventFlags::MEM | EventFlags::ERROR);
+        let throttle = EventFlags::from_bits(EventFlags::CPU | EventFlags::POWER);
+        let boot = EventFlags::from_bits(EventFlags::BOOT);
+        let events = [
+            (deauth, "deauth"),
+            (oom, "oom"),
+            (throttle, "throttle"),
+            (boot, "boot"),
+        ];
+
+        // Any error: the two failures.
+        assert_eq!(
+            scan_by_flag(&events, EventFlags::ERROR, FlagMode::Any),
+            vec![&"deauth", &"oom"]
+        );
+        assert_eq!(count_by_flag(&events, EventFlags::ERROR, FlagMode::Any), 2);
+
+        // Any Wi-Fi: the deauth alone.
+        assert_eq!(
+            scan_by_flag(&events, EventFlags::WIFI, FlagMode::Any),
+            vec![&"deauth"]
+        );
+
+        // Wi-Fi *and* error — the set — the deauth alone, not the OOM.
+        assert_eq!(
+            scan_by_flag(&events, EventFlags::WIFI | EventFlags::ERROR, FlagMode::All),
+            vec![&"deauth"]
+        );
+
+        // Order is preserved: a scan for ERROR returns them in the order they were written.
+        let errors = scan_by_flag(&events, EventFlags::ERROR, FlagMode::Any);
+        assert_eq!(errors, vec![&"deauth", &"oom"], "the kernel walks the index in key order");
+    }
+
+    /// An empty mask is no question, so it matches nothing — never everything.
+    #[test]
+    fn an_empty_mask_matches_nothing() {
+        let events = [(EventFlags::from_bits(EventFlags::BOOT), "boot")];
+        assert_eq!(scan_by_flag(&events, 0, FlagMode::Any).len(), 0);
+        assert_eq!(scan_by_flag(&events, 0, FlagMode::All).len(), 0);
+    }
+
+    /// A reader that does not know a bit still matches it by mask — an unknown set bit survives and
+    /// is returned, not dropped or misread, which is what lets a vocabulary attach later.
+    #[test]
+    fn an_unknown_bit_is_matched_by_mask_not_interpreted() {
+        let unknown = 1 << 15; // reserved, no named meaning
+        let events = [(EventFlags::from_bits(unknown), "mystery")];
+        assert_eq!(scan_by_flag(&events, unknown, FlagMode::Any), vec![&"mystery"]);
+        assert_eq!(scan_by_flag(&events, EventFlags::BOOT, FlagMode::Any).len(), 0);
+    }
+}

crates/cube-store/src/lib.rs

@@ -62,6 +62,7 @@
 
 #[cfg(feature = "debug")]
 pub mod debug;
+pub mod flagged;
 pub mod wal;
 
 use cube_core::{Coord, Curve, Morton, Point, SpaceId};
@@ -314,6 +315,10 @@ impl<C: Curve> Store for MemStore<C> {
     fn range(&self, space: &SpaceId, region: &Region) -> Result<Vec<(Coord, Vec<u8>)>, StoreError> {
         // The seek, when the curve has earned one; see `span_scan`.
         let (out, visited) = span_scan::<C>(&self.map, space, region);
+        // `visited` is the honest read footprint and only the `debug` counters record it; without
+        // the feature it is still the seek's cost, just unobserved.
+        #[cfg(not(feature = "debug"))]
+        let _ = visited;
 
         #[cfg(feature = "debug")]
         {