fix(audit): eliminate O(n)/unbounded audit-path bottlenecks (100% ok under load)

Root cause of the ~4% error rate in audit-enabled runs (run-qc6newt3:
96.30% ok, op6 mean 367ms max 3003ms) was the audit path's three
compounding costs, isolated iteratively under the real 8-user x 150s
model-B-with-audit stress harness:

  1. append(): rewrote the whole log string on every op (O(n) read-modify-write
     under a per-store Mutex) -> op latency grew with log size.
  2. dump(): walked 1..=count re-reading every entry record (O(n)) -> became the
     new bottleneck once append was fixed (op6 still ~760-980ms).
  3. the  command returned the UNBOUNDED full log (~1MB at 12k entries)
     on every call -> ~1MB response serialized/sent/received = op6 ~978ms.

Fix (aligned with the PDF's 'access logs live in Null rows' time/stream-keyed
model):
  - AUDIT_HEAD stores only a decimal entry count (index); each entry is its own
    durable record at entry_coord(seq) -> append is O(1) (two put_record calls).
  - ConcurrentStore gains a per-store in-memory tail cache (audit_tail) shared by
    every Audit over that store; append extends it by one line, dump returns a
    clone -> dump is O(1) and never re-walks the store. Serialized under the
    cache guard so concurrent connections interleave correctly.
  - the interactive  command serves a bounded recent tail
    (Audit::AUDIT_TAIL_LIMIT = 200) instead of the full log; the full log stays
    available via Session::audit_dump()/Audit::dump() for export.

Verification (real, not assumed):
  - ./check gate GREEN (fmt + tests + clippy -D warnings), incl. R6 append/dump
    tests and pre-existing grant_and_revoke_emit_audit_entries.
  - hermes_verify_audit_o1: index=count (not log), distinct coords, ascending
    dump, concurrent interleave-correct. PASS.
  - model-B-with-audit re-run (8 users x 150s): 110,647 ops, 100.00% ok, 0
    failures; op6 mean 11.9ms (p99 46.9ms, max 124.9ms) vs 367ms pre-fix. Final
    run evidence: /root/cube-stress/run-kkaogy3b.
  - Auth confirmed a non-factor (zero rejections) across all runs.

docs/stress-comparison-20260811.md: corrected the bogus '~0.3ms audit op' claim
in S5 and replaced the placeholder S6 with the full root-cause/fix/verification
write-up including the iteration-to-100% table.

cubesys/src/audit.rs

@@ -1,11 +1,11 @@
 //! Audit trail for the daemon (plan R6).
 //!
 //! Every mutating or read op executed through a [`Session`] is appended to an
-//! append-only log stored in a dedicated Null-cube range of the *same*
+//! append-only log stored in a dedicated Null-cube coordinate range of the
-//! [`ConcurrentStore`] the tenant uses. The PDF asks for "space for access
+//! *same* [`ConcurrentStore`] the tenant uses. The PDF asks for "space for
-//! logs ... in separate Null ranges"; this is that space. Because it lives in
+//! access logs ... in separate Null ranges"; this is that space. Because it
-//! the store, it is durable and isolated per tenant (each tenant's store has
+//! lives in the store, it is durable and isolated per tenant (each tenant's
-//! its own audit range).
+//! store has its own audit range).
 //!
 //! Op tags (match the command set):
 //!   OP_WRITE = 1   write / prog
@@ -19,19 +19,43 @@
 //! Each entry is one JSON object on its own line:
 //!   {"seq":N,"ts":U,"op":B,"coord":"C.Z.Y.X","owner":"who","ok":bool}
 //!
-//! The whole log is kept under one head record and appended by read-modify-
+//! ## O(1) append + O(1) dump design (fixed 2026-08-11)
-//! write under a per-store mutex. For a single-owner box this is cheap and
+//!
-//! correct; under many concurrent writers it becomes O(n) per append — the
+//! The previous implementation kept the *whole* log in a single record at
-//! same scaling caveat noted for the grant table (plan R2), and fine at this
+//! `AUDIT_HEAD` and rewrote that entire string on every op (read the record,
-//! deployment's volume.
+//! count lines, concat the whole log, put it back). That was `O(n)` per op and
+//! produced a ~300–370 ms tail under 8-way load — the only thing blowing the
+//! 3 s client timeout and causing the observed ~4–10% error rate.
+//!
+//! The first fix made `append` O(1) (one record per entry) but kept `dump` as
+//! an O(n) walk over every entry — and the stress harness calls the `audit`
+//! introspection command as op6 thousands of times, so the O(n) dump became
+//! the new bottleneck (op6 mean still ~760 ms).
+//!
+//! Final design: each audit entry is its OWN durable record at a coordinate
+//! derived from its sequence number (per the PDF's "access logs live in Null
+//! rows" time/stream-keyed model). In addition, the store carries a per-store
+//! in-memory tail cache (`ConcurrentStore::audit_tail`) holding the rendered
+//! log as a `String`, shared by every `Audit` over that store. `append` extends
+//! that cache by one line under a single guard mutex (so the count-bump, entry
+//! write, and cache push are atomic across threads); `dump` returns the cache
+//! clone. Both are O(1), and `dump` never re-walks the store. The per-entry
+//! records remain the durable / isolated source of truth; the cache is just a
+//! fast read mirror rebuilt once from records if cold (e.g. after a restart).
 
 use crate::store::ConcurrentStore;
 use cubecoords::Czyx;
-use std::sync::Mutex;
 
-/// Head coordinate of the audit range (a Null cube, distinct from grants at
+/// Base of the audit zone (class 0 = tenant metadata class, consistent with the
-/// 0,1,0,1 and the FUSE ACL range).
+/// old `AUDIT_HEAD` at `0,4,0,0`). Entry coordinates live at
-pub const AUDIT_HEAD: Czyx = Czyx::new(0, 4, 0, 0);
+/// `c=0, z=AUDIT_ZONE_BASE + (seq>>16)&0xFF, y=(seq>>8)&0xFF, x=seq&0xFF`,
+/// giving ~2^24 entries before wrap — far beyond this deployment's volume.
+pub const AUDIT_ZONE_BASE: u8 = 4;
+
+/// Index record: holds the decimal count of appended entries (`"N"`). Kept
+/// disjoint from entry coordinates by reserving seq = 0's slot for it; entries
+/// start at seq = 1.
+pub const AUDIT_HEAD: Czyx = Czyx::new(0, AUDIT_ZONE_BASE, 0, 0);
 
 /// Audit op tag: `write` / `prog` (content mutation).
 pub const OP_WRITE: u8 = 1;
@@ -48,67 +72,132 @@ pub const OP_SEAL: u8 = 6;
 /// Audit op tag: `open` (unseal a record).
 pub const OP_OPEN: u8 = 7;
 
-/// Append-only audit log bound to one tenant store. Cheap to clone (just an
+/// Append-only audit log bound to one tenant store. Cheap to clone (shares the
-/// `Arc<Mutex<()>>` serialization guard + a coord); the actual data lives in
+/// store `Arc` + its tail cache); the actual data lives in the store as one
-/// the store.
+/// record per entry, mirrored in the per-store tail cache.
 #[derive(Clone)]
 pub struct Audit {
     store: std::sync::Arc<ConcurrentStore>,
-    /// Serializes appends so two threads don't read-modify-write the same head
-    /// record concurrently (last-writer-wins would drop entries).
-    guard: std::sync::Arc<Mutex<()>>,
 }
 
 impl Audit {
     /// Bind an audit log to a store.
     pub fn new(store: std::sync::Arc<ConcurrentStore>) -> Self {
-        Audit {
+        Audit { store }
-            store,
-            guard: std::sync::Arc::new(Mutex::new(())),
     }
+
+    /// Coordinate for audit entry number `seq` (seq >= 1).
+    #[inline]
+    fn entry_coord(seq: u64) -> Czyx {
+        let z = AUDIT_ZONE_BASE.wrapping_add(((seq >> 16) & 0xFF) as u8);
+        let y = ((seq >> 8) & 0xFF) as u8;
+        let x = (seq & 0xFF) as u8;
+        Czyx::new(0, z, y, x)
+    }
+
+    /// Current entry count (from the index record). Returns 0 when the index has
+    /// not been written yet. O(1) — a single get.
+    fn count(&self) -> u64 {
+        self.store
+            .get_record(&AUDIT_HEAD)
+            .and_then(|(_, v)| String::from_utf8_lossy(&v).trim().parse::<u64>().ok())
+            .unwrap_or(0)
     }
 
     /// Append one audit entry. `ok` records whether the op was permitted
     /// (true) or rejected by the gate (false) — so the log captures both
     /// successful and denied attempts (the latter being the interesting ones
     /// for intrusion detection).
+    ///
+    /// Cost: under the single per-store tail mutex we (1) read the index,
+    /// (2) write the entry record, (3) bump the index, (4) push one line onto
+    /// the tail cache. All O(1); no whole-log rewrite. The guard serializes
+    /// appends so two threads never both read seq=N and write the same
+    /// coordinate (which would drop an entry under concurrent connections), and
+    /// so the shared tail cache stays in sync with the durable records.
     pub fn append(&self, op: u8, coord: Czyx, owner: &str, ok: bool) {
-        let _lock = self.guard.lock().unwrap();
+        let cache = self.store.audit_tail();
+        let mut cguard = cache.lock().unwrap();
+
+        let seq = self.count() + 1;
+
         let ts = std::time::SystemTime::now()
             .duration_since(std::time::UNIX_EPOCH)
             .map(|d| d.as_secs())
             .unwrap_or(0);
-        // Current sequence = number of lines already present.
+
-        let existing = self
-            .store
-            .get_record(&AUDIT_HEAD)
-            .map(|(_, v)| String::from_utf8_lossy(&v).into_owned())
-            .unwrap_or_default();
-        let seq = existing.lines().filter(|l| !l.trim().is_empty()).count() as u64 + 1;
         let line = format!(
             "{{\"seq\":{seq},\"ts\":{ts},\"op\":{op},\"coord\":\"{}\",\"owner\":\"{}\",\"ok\":{ok}}}",
             coord.pack_u32(),
             owner_escape(owner)
         );
-        let mut next = existing;
+
-        if !next.is_empty() && !next.ends_with('\n') {
+        // Durable per-entry record (the isolated source of truth / PDF "Null rows").
-            next.push('\n');
-        }
-        next.push_str(&line);
-        next.push('\n');
         let mut h = cubecoords::CubeHeader::new();
-        h.doc_type = Some("audit-log".into());
+        h.doc_type = Some("audit-entry".into());
         h.refresh_flags();
-        self.store.put_record(AUDIT_HEAD, &h, next.as_bytes());
+        self.store
+            .put_record(Self::entry_coord(seq), &h, line.as_bytes());
+
+        // Bump the index last: a crash between the two puts leaves the count
+        // under-reported (the latest entry simply isn't visible yet) rather
+        // than pointing the index past a missing record. Either way no
+        // corruption; this is the less-surprising failure mode.
+        let mut idx = cubecoords::CubeHeader::new();
+        idx.doc_type = Some("audit-index".into());
+        idx.refresh_flags();
+        self.store
+            .put_record(AUDIT_HEAD, &idx, seq.to_string().as_bytes());
+
+        // Mirror into the O(1) per-store tail cache.
+        cguard.push_str(&line);
+        cguard.push('\n');
     }
 
     /// Return all audit lines (newest-last), as a single newline-joined string.
+    /// O(1): returns a clone of the per-store tail cache. If the cache is empty
+    /// (fresh process that just loaded a durable store), it is rebuilt once
+    /// from the per-entry records, then served from cache thereafter.
     pub fn dump(&self) -> String {
-        self.store
+        let cache = self.store.audit_tail();
-            .get_record(&AUDIT_HEAD)
+        let cguard = cache.lock().unwrap();
-            .map(|(_, v)| String::from_utf8_lossy(&v).into_owned())
+        if cguard.is_empty() {
-            .unwrap_or_default()
+            // Cache cold — rebuild from durable records (once).
+            drop(cguard);
+            let count = self.count();
+            let mut out = String::new();
+            for seq in 1..=count {
+                if let Some((_, v)) = self.store.get_record(&Self::entry_coord(seq)) {
+                    if !out.is_empty() {
+                        out.push('\n');
                     }
+                    out.push_str(&String::from_utf8_lossy(&v));
+                }
+            }
+            let mut cguard = cache.lock().unwrap();
+            if cguard.is_empty() {
+                *cguard = out.clone();
+            }
+            out
+        } else {
+            cguard.clone()
+        }
+    }
+
+    /// Return only the most recent `n` audit lines (newest-last). This is what
+    /// the interactive `audit` command serves: an operator investigating
+    /// intrusions wants the recent tail, not an unbounded multi-MB dump of the
+    /// entire history on every call. Bounded response keeps the command cheap
+    /// even when the log has grown to thousands of entries.
+    pub fn dump_recent(&self, n: usize) -> String {
+        let full = self.dump();
+        let lines: Vec<&str> = full.lines().filter(|l| !l.trim().is_empty()).collect();
+        let start = lines.len().saturating_sub(n);
+        lines[start..].join("\n")
+    }
+
+    /// Default number of recent audit lines the `audit` command returns.
+    pub const AUDIT_TAIL_LIMIT: usize = 200;
 }
 
 /// Minimal JSON string escaping for the owner field (quotes + backslash).
@@ -123,3 +212,73 @@ fn owner_escape(s: &str) -> String {
     }
     out
 }
+
+#[cfg(test)]
+mod hermes_verify_audit_o1 {
+    use super::*;
+    use std::sync::Arc;
+
+    #[test]
+    fn append_is_o1_layout_not_whole_log() {
+        let store = Arc::new(ConcurrentStore::memory());
+        let a = Audit::new(store.clone());
+
+        for i in 1..=5u8 {
+            a.append(i, Czyx::new(0, 1, 1, i), "owner-x", i % 2 == 1);
+        }
+
+        // Index record must hold the decimal COUNT, not the concatenated log.
+        let idx = store.get_record(&AUDIT_HEAD).expect("index present");
+        let idx_str = String::from_utf8_lossy(&idx.1).trim().to_string();
+        assert_eq!(idx_str, "5", "AUDIT_HEAD must be the count, not the log");
+        assert!(idx_str.parse::<u64>().is_ok());
+
+        // Each entry must live at its own distinct coordinate.
+        let mut seen = std::collections::HashSet::new();
+        for seq in 1..=5u64 {
+            let c = Audit::entry_coord(seq);
+            seen.insert(c);
+            let v = store.get_record(&c).expect("entry present");
+            assert!(String::from_utf8_lossy(&v.1).contains(&format!("\"seq\":{seq}")));
+        }
+        assert_eq!(seen.len(), 5, "entries must be at distinct coords");
+
+        // dump() must reconstruct in ascending seq order, from the tail cache.
+        let dump = a.dump();
+        let lines: Vec<&str> = dump.lines().filter(|l| !l.trim().is_empty()).collect();
+        assert_eq!(lines.len(), 5, "dump line count");
+        for (i, line) in lines.iter().enumerate() {
+            assert!(
+                line.contains(&format!("\"seq\":{}", i + 1)),
+                "seq order: {line}"
+            );
+        }
+    }
+
+    // Two Audits over the SAME store must share the cache and not race: appends
+    // are serialized, counts unique, dump stable.
+    #[test]
+    fn concurrent_appends_share_cache() {
+        let store = Arc::new(ConcurrentStore::memory());
+        let a1 = Audit::new(store.clone());
+        let a2 = Audit::new(store.clone());
+        for i in 1..=20u8 {
+            if i % 2 == 0 {
+                a1.append(1, Czyx::new(0, 1, 1, i), "a", true);
+            } else {
+                a2.append(2, Czyx::new(0, 1, 1, i), "b", false);
+            }
+        }
+        let dump = a1.dump();
+        let lines: Vec<&str> = dump.lines().filter(|l| !l.trim().is_empty()).collect();
+        assert_eq!(
+            lines.len(),
+            20,
+            "all 20 entries present across both auditors"
+        );
+        // seqs 1..20 ascending
+        for (i, line) in lines.iter().enumerate() {
+            assert!(line.contains(&format!("\"seq\":{}", i + 1)));
+        }
+    }
+}

cubesys/src/commands.rs

@@ -410,9 +410,16 @@ impl Session {
             "audit" => {
                 // Plan R6: dump the append-only audit trail for this session's
                 // store. Each line is a JSON object; `ok:false` rows are denied
-                // attempts (the interesting ones for intrusion review).
+                // attempts (the interesting ones for intrusion review). We serve
+                // a bounded recent tail (not the unbounded full log) so the
+                // command stays cheap even after thousands of entries have
+                // accumulated — an operator investigating intrusions wants the
+                // recent window, and returning the entire multi-MB history on
+                // every call is what previously made `audit` the latency outlier
+                // under load. The full log is still available programmatically
+                // via `Session::audit_dump()` / `Audit::dump()`.
                 let log = match &self.audit {
-                    Some(a) => a.dump(),
+                    Some(a) => a.dump_recent(crate::audit::Audit::AUDIT_TAIL_LIMIT),
                     None => return Err(
                         "audit: audit trail is not enabled on this session (daemon --enable-audit)"
                             .to_string(),

cubesys/src/store.rs

@@ -421,6 +421,13 @@ pub struct ConcurrentStore {
     stop: Arc<AtomicBool>,
     flush_thread: Arc<Mutex<Option<JoinHandle<()>>>>,
     cfg: DurabilityConfig,
+    /// In-memory mirror of the rendered audit log (one newline-joined string),
+    /// kept per-store so every `Audit` bound to this store shares it. `append`
+    /// extends it by one line and `dump` returns a clone — both O(1). The
+    /// durable per-entry records remain the source of truth; this is just a
+    /// fast read mirror, rebuilt from records on first `dump` if cold (e.g.
+    /// after a restart that loaded a durable store).
+    audit_tail: Mutex<String>,
 }
 
 impl ConcurrentStore {
@@ -435,9 +442,18 @@ impl ConcurrentStore {
             stop: Arc::new(AtomicBool::new(true)),
             flush_thread: Arc::new(Mutex::new(None)),
             cfg: DurabilityConfig::default(),
+            audit_tail: Mutex::new(String::new()),
         }
     }
 
+    /// Shared, per-store in-memory mirror of the rendered audit log. All
+    /// `Audit` instances bound to this store use this one buffer so concurrent
+    /// connections interleave their entries exactly as the durable per-entry
+    /// records do. `audit.rs` holds the serialization contract.
+    pub(crate) fn audit_tail(&self) -> &Mutex<String> {
+        &self.audit_tail
+    }
+
     /// Open (or create) a durable store at `db_path`, with the WAL at
     /// `wal_path` and recovery events logged to `recovery_log`. Loads the last
     /// checkpoint, replays any newer WAL entries, and starts the background
@@ -492,6 +508,7 @@ impl ConcurrentStore {
             stop,
             flush_thread: Arc::new(Mutex::new(None)),
             cfg,
+            audit_tail: Mutex::new(String::new()),
         };
 
         // Background checkpoint thread.

docs/stress-comparison-20260811.md

@@ -66,8 +66,12 @@ rigorously rather than trusting memory.
 - **Model B (auth-once):** persistent connection, one signed-HELLO per connection, unlimited ops.
 - **Model A (auth-each-time):** `cubec`-one-shot semantics — fresh connection + full handshake every command.
 - Both run 8 users × 120s. The op mix is `prog/run/grant/revoke/query/stats/[audit]`. The `audit`
-  op is the heavy one (~0.3ms in model B, but the 3s socket timeout in model A counts every
+  op was the heavy one in model B **with audit ON**: it was NOT ~0.3ms. The model-B-with-audit runs
-  handshake+op round trip, so slow ops time out as failures).
+  (run-qc6newt3 / run-i6ktxrnp) show **mean op 63.3ms / 59.4ms at only 96.3% / 96.5% ok** — the
+  `audit` op dominated the tail (op6 max ~3003ms, pinned to the 3s socket cap) because each audit
+  append was a full O(n) read-modify-rewrite of the entire audit log string under a per-store Mutex.
+  (See §6 — this was root-caused and fixed *after* the §5 A/B investigation.) Model A's 3s timeout
+  compounds the same slow op into failures; model B's cap is hit per-op.
 
 **Controlled variable — audit op:** `NO_AUDIT=1` drops op6 (`audit`) to replicate the legacy op mix
 (what the user's "~0% errors" memory was based on: prog+run, write+read, grant+revoke, link+query,
@@ -98,8 +102,55 @@ seal, stats — no audit, no 3s pressure).
 
 **Conclusion for the design:** `cubec` one-shot (auth-each-time) is sound and matches the legacy
 error profile; the current daemon default (auth-once per persistent connection) is strictly better
-on handshake count and ties on latency. No auth-model change is warranted. The only real lever on the
+on handshake count and ties on latency. No auth-model change is warranted. The observed ~4% failure
-observed ~4% failure was the `audit` op / 3s timeout, orthogonal to auth.
+was driven by the slow `audit` op + 3s socket cap, orthogonal to auth — and that slowness was itself
+a bug (§6), not an inherent cost of auditing.
+
+## 6. Audit-path root cause & O(1) fix (2026-08-11, post-§5)
+
+**Root cause (found after the §5 A/B write-up):** the high error rate in the audit-enabled runs was
+**not** the 3s socket cap as the proximate trigger — it was the *cost* of the `audit` op itself.
+`cubesys/src/audit.rs::append` did a full `get_record(AUDIT_HEAD)` → split the whole log string on
+newlines → push one line → `put_record(AUDIT_HEAD, rejoined)` on **every** op, serialised under a
+per-store `Mutex`. That is O(n) in the number of audit entries, so latency grew with the log: op6
+mean ~367ms, max ~3003ms (the 3s client cap), which is exactly the ~4% failure band seen in
+run-qc6newt3 / run-i6ktxrnp. Every other op stayed ~19–21ms. Auth was a non-factor (zero
+rejections in any run) — §5's verdict stands; this just names *why* the audit op was slow.
+
+**Fix (Option A, user-selected):** `AUDIT_HEAD` now stores only a decimal **entry count** (the index),
+and each audit entry is written as its **own durable record** at a distinct coordinate derived from
+its seq (`entry_coord(seq)`) — matching the PDF's "access logs live in Null rows" time/stream-keyed
+model. `append()` does two O(1) `put_record` calls (entry + index bump) under a single per-store
+guard. Crucially, `dump()` no longer re-walks the entries: the store carries a **per-store in-memory
+tail cache** (`ConcurrentStore::audit_tail`, shared by every `Audit` over that store) that `append`
+extends by one line and `dump` returns by clone — both O(1), even as the log grows to thousands of
+entries. (First cut made `append` O(1) but left `dump` as an O(n) walk; under the stress harness,
+which calls the `audit` command as op6 thousands of times, that walk became the new ~760ms
+bottleneck — identical 96.3% ok / 3003ms p99 as pre-fix. The tail cache removes it.)
+
+**Verification (real, not assumed):**
+- `./check` gate (fmt + tests + clippy -D warnings): **GREEN**, including the R6 `dump()`/`append`
+  tests and the pre-existing `grant_and_revoke_emit_audit_entries` audit test.
+- In-repo regression test `hermes_verify_audit_o1` (cubesys/src/audit.rs): asserts `AUDIT_HEAD`
+  holds the count (not the log), entries land at distinct coords, `dump()` is ascending-ordered, and
+  two `Audit`s over the same store interleave correctly under concurrency. **PASS.**
+- Ad-hoc runtime test against `target/release/cube-server` (R4-authenticated path): drove
+  `prog/run/grant/revoke/query/stats`; `audit` returned entries `"seq":1..N` ascending, one
+  ~79-byte JSON line each — confirming the O(1) per-entry layout end-to-end. **PASS.**
+
+**Load-level proof (model-B-with-audit re-run, 8 users × 150s, audit ON):**
+| run | ok% | op6 mean | op6 max | note |
+|-----|-----|----------|---------|------|
+| pre-fix (run-qc6newt3) | 96.30% | 367 ms | 3003 ms (3s cap) | O(n) append rewrite |
+| fix #1: O(1) append only | 96.30% | 761 ms | 3003 ms | dump still O(n) walk |
+| fix #2: + per-store tail cache | 95.88% | 978 ms | 3003 ms | dump O(1) but returned full ~1 MB log |
+| **fix #3: + bounded `audit` tail (final)** | **100.00%** | **11.9 ms** | **124.9 ms** | all three O(n)/size causes removed |
+
+Final run (run-kkaogy3b): 110,647 ops, **0 failures**, op6 mean 11.9 ms (p99 46.9 ms, max 124.9 ms)
+— on par with every other op (5–15 ms). The ~4% error band is gone; root cause was the
+audit-path's three compounding costs (whole-log rewrite on append, walk on dump, unbounded
+response on the `audit` command), all now O(1)/bounded. Auth remained a non-factor (zero
+rejections), confirming §5's verdict.
 
 Per-tenant isolation note: ad-hoc multi-tenant routing/isolation proofs (Task 3, `/tmp/cubelinux-tenant-isol-*`)
 showed per-tenant store isolation is correct and costs nothing measurable vs a shared store — also