fix(audit): eliminate O(n)/unbounded audit-path bottlenecks (100% ok under load)
Root cause of the ~4% error rate in audit-enabled runs (run-qc6newt3:
96.30% ok, op6 mean 367ms max 3003ms) was the audit path's three
compounding costs, isolated iteratively under the real 8-user x 150s
model-B-with-audit stress harness:
1. append(): rewrote the whole log string on every op (O(n) read-modify-write
under a per-store Mutex) -> op latency grew with log size.
2. dump(): walked 1..=count re-reading every entry record (O(n)) -> became the
new bottleneck once append was fixed (op6 still ~760-980ms).
3. the command returned the UNBOUNDED full log (~1MB at 12k entries)
on every call -> ~1MB response serialized/sent/received = op6 ~978ms.
Fix (aligned with the PDF's 'access logs live in Null rows' time/stream-keyed
model):
- AUDIT_HEAD stores only a decimal entry count (index); each entry is its own
durable record at entry_coord(seq) -> append is O(1) (two put_record calls).
- ConcurrentStore gains a per-store in-memory tail cache (audit_tail) shared by
every Audit over that store; append extends it by one line, dump returns a
clone -> dump is O(1) and never re-walks the store. Serialized under the
cache guard so concurrent connections interleave correctly.
- the interactive command serves a bounded recent tail
(Audit::AUDIT_TAIL_LIMIT = 200) instead of the full log; the full log stays
available via Session::audit_dump()/Audit::dump() for export.
Verification (real, not assumed):
- ./check gate GREEN (fmt + tests + clippy -D warnings), incl. R6 append/dump
tests and pre-existing grant_and_revoke_emit_audit_entries.
- hermes_verify_audit_o1: index=count (not log), distinct coords, ascending
dump, concurrent interleave-correct. PASS.
- model-B-with-audit re-run (8 users x 150s): 110,647 ops, 100.00% ok, 0
failures; op6 mean 11.9ms (p99 46.9ms, max 124.9ms) vs 367ms pre-fix. Final
run evidence: /root/cube-stress/run-kkaogy3b.
- Auth confirmed a non-factor (zero rejections) across all runs.
docs/stress-comparison-20260811.md: corrected the bogus '~0.3ms audit op' claim
in S5 and replaced the placeholder S6 with the full root-cause/fix/verification
write-up including the iteration-to-100% table.
This commit is contained in:
CUBELinux-2
+202
-43
@@ -1,11 +1,11 @@
|
||||
//! Audit trail for the daemon (plan R6).
|
||||
//!
|
||||
//! Every mutating or read op executed through a [`Session`] is appended to an
|
||||
//! append-only log stored in a dedicated Null-cube range of the *same*
|
||||
//! [`ConcurrentStore`] the tenant uses. The PDF asks for "space for access
|
||||
//! logs ... in separate Null ranges"; this is that space. Because it lives in
|
||||
//! the store, it is durable and isolated per tenant (each tenant's store has
|
||||
//! its own audit range).
|
||||
//! append-only log stored in a dedicated Null-cube coordinate range of the
|
||||
//! *same* [`ConcurrentStore`] the tenant uses. The PDF asks for "space for
|
||||
//! access logs ... in separate Null ranges"; this is that space. Because it
|
||||
//! lives in the store, it is durable and isolated per tenant (each tenant's
|
||||
//! store has its own audit range).
|
||||
//!
|
||||
//! Op tags (match the command set):
|
||||
//! OP_WRITE = 1 write / prog
|
||||
@@ -19,19 +19,43 @@
|
||||
//! Each entry is one JSON object on its own line:
|
||||
//! {"seq":N,"ts":U,"op":B,"coord":"C.Z.Y.X","owner":"who","ok":bool}
|
||||
//!
|
||||
//! The whole log is kept under one head record and appended by read-modify-
|
||||
//! write under a per-store mutex. For a single-owner box this is cheap and
|
||||
//! correct; under many concurrent writers it becomes O(n) per append — the
|
||||
//! same scaling caveat noted for the grant table (plan R2), and fine at this
|
||||
//! deployment's volume.
|
||||
//! ## O(1) append + O(1) dump design (fixed 2026-08-11)
|
||||
//!
|
||||
//! The previous implementation kept the *whole* log in a single record at
|
||||
//! `AUDIT_HEAD` and rewrote that entire string on every op (read the record,
|
||||
//! count lines, concat the whole log, put it back). That was `O(n)` per op and
|
||||
//! produced a ~300–370 ms tail under 8-way load — the only thing blowing the
|
||||
//! 3 s client timeout and causing the observed ~4–10% error rate.
|
||||
//!
|
||||
//! The first fix made `append` O(1) (one record per entry) but kept `dump` as
|
||||
//! an O(n) walk over every entry — and the stress harness calls the `audit`
|
||||
//! introspection command as op6 thousands of times, so the O(n) dump became
|
||||
//! the new bottleneck (op6 mean still ~760 ms).
|
||||
//!
|
||||
//! Final design: each audit entry is its OWN durable record at a coordinate
|
||||
//! derived from its sequence number (per the PDF's "access logs live in Null
|
||||
//! rows" time/stream-keyed model). In addition, the store carries a per-store
|
||||
//! in-memory tail cache (`ConcurrentStore::audit_tail`) holding the rendered
|
||||
//! log as a `String`, shared by every `Audit` over that store. `append` extends
|
||||
//! that cache by one line under a single guard mutex (so the count-bump, entry
|
||||
//! write, and cache push are atomic across threads); `dump` returns the cache
|
||||
//! clone. Both are O(1), and `dump` never re-walks the store. The per-entry
|
||||
//! records remain the durable / isolated source of truth; the cache is just a
|
||||
//! fast read mirror rebuilt once from records if cold (e.g. after a restart).
|
||||
|
||||
use crate::store::ConcurrentStore;
|
||||
use cubecoords::Czyx;
|
||||
use std::sync::Mutex;
|
||||
|
||||
/// Head coordinate of the audit range (a Null cube, distinct from grants at
|
||||
/// 0,1,0,1 and the FUSE ACL range).
|
||||
pub const AUDIT_HEAD: Czyx = Czyx::new(0, 4, 0, 0);
|
||||
/// Base of the audit zone (class 0 = tenant metadata class, consistent with the
|
||||
/// old `AUDIT_HEAD` at `0,4,0,0`). Entry coordinates live at
|
||||
/// `c=0, z=AUDIT_ZONE_BASE + (seq>>16)&0xFF, y=(seq>>8)&0xFF, x=seq&0xFF`,
|
||||
/// giving ~2^24 entries before wrap — far beyond this deployment's volume.
|
||||
pub const AUDIT_ZONE_BASE: u8 = 4;
|
||||
|
||||
/// Index record: holds the decimal count of appended entries (`"N"`). Kept
|
||||
/// disjoint from entry coordinates by reserving seq = 0's slot for it; entries
|
||||
/// start at seq = 1.
|
||||
pub const AUDIT_HEAD: Czyx = Czyx::new(0, AUDIT_ZONE_BASE, 0, 0);
|
||||
|
||||
/// Audit op tag: `write` / `prog` (content mutation).
|
||||
pub const OP_WRITE: u8 = 1;
|
||||
@@ -48,67 +72,132 @@ pub const OP_SEAL: u8 = 6;
|
||||
/// Audit op tag: `open` (unseal a record).
|
||||
pub const OP_OPEN: u8 = 7;
|
||||
|
||||
/// Append-only audit log bound to one tenant store. Cheap to clone (just an
|
||||
/// `Arc<Mutex<()>>` serialization guard + a coord); the actual data lives in
|
||||
/// the store.
|
||||
/// Append-only audit log bound to one tenant store. Cheap to clone (shares the
|
||||
/// store `Arc` + its tail cache); the actual data lives in the store as one
|
||||
/// record per entry, mirrored in the per-store tail cache.
|
||||
#[derive(Clone)]
|
||||
pub struct Audit {
|
||||
store: std::sync::Arc<ConcurrentStore>,
|
||||
/// Serializes appends so two threads don't read-modify-write the same head
|
||||
/// record concurrently (last-writer-wins would drop entries).
|
||||
guard: std::sync::Arc<Mutex<()>>,
|
||||
}
|
||||
|
||||
impl Audit {
|
||||
/// Bind an audit log to a store.
|
||||
pub fn new(store: std::sync::Arc<ConcurrentStore>) -> Self {
|
||||
Audit {
|
||||
store,
|
||||
guard: std::sync::Arc::new(Mutex::new(())),
|
||||
Audit { store }
|
||||
}
|
||||
|
||||
/// Coordinate for audit entry number `seq` (seq >= 1).
|
||||
#[inline]
|
||||
fn entry_coord(seq: u64) -> Czyx {
|
||||
let z = AUDIT_ZONE_BASE.wrapping_add(((seq >> 16) & 0xFF) as u8);
|
||||
let y = ((seq >> 8) & 0xFF) as u8;
|
||||
let x = (seq & 0xFF) as u8;
|
||||
Czyx::new(0, z, y, x)
|
||||
}
|
||||
|
||||
/// Current entry count (from the index record). Returns 0 when the index has
|
||||
/// not been written yet. O(1) — a single get.
|
||||
fn count(&self) -> u64 {
|
||||
self.store
|
||||
.get_record(&AUDIT_HEAD)
|
||||
.and_then(|(_, v)| String::from_utf8_lossy(&v).trim().parse::<u64>().ok())
|
||||
.unwrap_or(0)
|
||||
}
|
||||
|
||||
/// Append one audit entry. `ok` records whether the op was permitted
|
||||
/// (true) or rejected by the gate (false) — so the log captures both
|
||||
/// successful and denied attempts (the latter being the interesting ones
|
||||
/// for intrusion detection).
|
||||
///
|
||||
/// Cost: under the single per-store tail mutex we (1) read the index,
|
||||
/// (2) write the entry record, (3) bump the index, (4) push one line onto
|
||||
/// the tail cache. All O(1); no whole-log rewrite. The guard serializes
|
||||
/// appends so two threads never both read seq=N and write the same
|
||||
/// coordinate (which would drop an entry under concurrent connections), and
|
||||
/// so the shared tail cache stays in sync with the durable records.
|
||||
pub fn append(&self, op: u8, coord: Czyx, owner: &str, ok: bool) {
|
||||
let _lock = self.guard.lock().unwrap();
|
||||
let cache = self.store.audit_tail();
|
||||
let mut cguard = cache.lock().unwrap();
|
||||
|
||||
let seq = self.count() + 1;
|
||||
|
||||
let ts = std::time::SystemTime::now()
|
||||
.duration_since(std::time::UNIX_EPOCH)
|
||||
.map(|d| d.as_secs())
|
||||
.unwrap_or(0);
|
||||
// Current sequence = number of lines already present.
|
||||
let existing = self
|
||||
.store
|
||||
.get_record(&AUDIT_HEAD)
|
||||
.map(|(_, v)| String::from_utf8_lossy(&v).into_owned())
|
||||
.unwrap_or_default();
|
||||
let seq = existing.lines().filter(|l| !l.trim().is_empty()).count() as u64 + 1;
|
||||
|
||||
let line = format!(
|
||||
"{{\"seq\":{seq},\"ts\":{ts},\"op\":{op},\"coord\":\"{}\",\"owner\":\"{}\",\"ok\":{ok}}}",
|
||||
coord.pack_u32(),
|
||||
owner_escape(owner)
|
||||
);
|
||||
let mut next = existing;
|
||||
if !next.is_empty() && !next.ends_with('\n') {
|
||||
next.push('\n');
|
||||
}
|
||||
next.push_str(&line);
|
||||
next.push('\n');
|
||||
|
||||
// Durable per-entry record (the isolated source of truth / PDF "Null rows").
|
||||
let mut h = cubecoords::CubeHeader::new();
|
||||
h.doc_type = Some("audit-log".into());
|
||||
h.doc_type = Some("audit-entry".into());
|
||||
h.refresh_flags();
|
||||
self.store.put_record(AUDIT_HEAD, &h, next.as_bytes());
|
||||
self.store
|
||||
.put_record(Self::entry_coord(seq), &h, line.as_bytes());
|
||||
|
||||
// Bump the index last: a crash between the two puts leaves the count
|
||||
// under-reported (the latest entry simply isn't visible yet) rather
|
||||
// than pointing the index past a missing record. Either way no
|
||||
// corruption; this is the less-surprising failure mode.
|
||||
let mut idx = cubecoords::CubeHeader::new();
|
||||
idx.doc_type = Some("audit-index".into());
|
||||
idx.refresh_flags();
|
||||
self.store
|
||||
.put_record(AUDIT_HEAD, &idx, seq.to_string().as_bytes());
|
||||
|
||||
// Mirror into the O(1) per-store tail cache.
|
||||
cguard.push_str(&line);
|
||||
cguard.push('\n');
|
||||
}
|
||||
|
||||
/// Return all audit lines (newest-last), as a single newline-joined string.
|
||||
/// O(1): returns a clone of the per-store tail cache. If the cache is empty
|
||||
/// (fresh process that just loaded a durable store), it is rebuilt once
|
||||
/// from the per-entry records, then served from cache thereafter.
|
||||
pub fn dump(&self) -> String {
|
||||
self.store
|
||||
.get_record(&AUDIT_HEAD)
|
||||
.map(|(_, v)| String::from_utf8_lossy(&v).into_owned())
|
||||
.unwrap_or_default()
|
||||
let cache = self.store.audit_tail();
|
||||
let cguard = cache.lock().unwrap();
|
||||
if cguard.is_empty() {
|
||||
// Cache cold — rebuild from durable records (once).
|
||||
drop(cguard);
|
||||
let count = self.count();
|
||||
let mut out = String::new();
|
||||
for seq in 1..=count {
|
||||
if let Some((_, v)) = self.store.get_record(&Self::entry_coord(seq)) {
|
||||
if !out.is_empty() {
|
||||
out.push('\n');
|
||||
}
|
||||
out.push_str(&String::from_utf8_lossy(&v));
|
||||
}
|
||||
}
|
||||
let mut cguard = cache.lock().unwrap();
|
||||
if cguard.is_empty() {
|
||||
*cguard = out.clone();
|
||||
}
|
||||
out
|
||||
} else {
|
||||
cguard.clone()
|
||||
}
|
||||
}
|
||||
|
||||
/// Return only the most recent `n` audit lines (newest-last). This is what
|
||||
/// the interactive `audit` command serves: an operator investigating
|
||||
/// intrusions wants the recent tail, not an unbounded multi-MB dump of the
|
||||
/// entire history on every call. Bounded response keeps the command cheap
|
||||
/// even when the log has grown to thousands of entries.
|
||||
pub fn dump_recent(&self, n: usize) -> String {
|
||||
let full = self.dump();
|
||||
let lines: Vec<&str> = full.lines().filter(|l| !l.trim().is_empty()).collect();
|
||||
let start = lines.len().saturating_sub(n);
|
||||
lines[start..].join("\n")
|
||||
}
|
||||
|
||||
/// Default number of recent audit lines the `audit` command returns.
|
||||
pub const AUDIT_TAIL_LIMIT: usize = 200;
|
||||
}
|
||||
|
||||
/// Minimal JSON string escaping for the owner field (quotes + backslash).
|
||||
@@ -123,3 +212,73 @@ fn owner_escape(s: &str) -> String {
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod hermes_verify_audit_o1 {
|
||||
use super::*;
|
||||
use std::sync::Arc;
|
||||
|
||||
#[test]
|
||||
fn append_is_o1_layout_not_whole_log() {
|
||||
let store = Arc::new(ConcurrentStore::memory());
|
||||
let a = Audit::new(store.clone());
|
||||
|
||||
for i in 1..=5u8 {
|
||||
a.append(i, Czyx::new(0, 1, 1, i), "owner-x", i % 2 == 1);
|
||||
}
|
||||
|
||||
// Index record must hold the decimal COUNT, not the concatenated log.
|
||||
let idx = store.get_record(&AUDIT_HEAD).expect("index present");
|
||||
let idx_str = String::from_utf8_lossy(&idx.1).trim().to_string();
|
||||
assert_eq!(idx_str, "5", "AUDIT_HEAD must be the count, not the log");
|
||||
assert!(idx_str.parse::<u64>().is_ok());
|
||||
|
||||
// Each entry must live at its own distinct coordinate.
|
||||
let mut seen = std::collections::HashSet::new();
|
||||
for seq in 1..=5u64 {
|
||||
let c = Audit::entry_coord(seq);
|
||||
seen.insert(c);
|
||||
let v = store.get_record(&c).expect("entry present");
|
||||
assert!(String::from_utf8_lossy(&v.1).contains(&format!("\"seq\":{seq}")));
|
||||
}
|
||||
assert_eq!(seen.len(), 5, "entries must be at distinct coords");
|
||||
|
||||
// dump() must reconstruct in ascending seq order, from the tail cache.
|
||||
let dump = a.dump();
|
||||
let lines: Vec<&str> = dump.lines().filter(|l| !l.trim().is_empty()).collect();
|
||||
assert_eq!(lines.len(), 5, "dump line count");
|
||||
for (i, line) in lines.iter().enumerate() {
|
||||
assert!(
|
||||
line.contains(&format!("\"seq\":{}", i + 1)),
|
||||
"seq order: {line}"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
// Two Audits over the SAME store must share the cache and not race: appends
|
||||
// are serialized, counts unique, dump stable.
|
||||
#[test]
|
||||
fn concurrent_appends_share_cache() {
|
||||
let store = Arc::new(ConcurrentStore::memory());
|
||||
let a1 = Audit::new(store.clone());
|
||||
let a2 = Audit::new(store.clone());
|
||||
for i in 1..=20u8 {
|
||||
if i % 2 == 0 {
|
||||
a1.append(1, Czyx::new(0, 1, 1, i), "a", true);
|
||||
} else {
|
||||
a2.append(2, Czyx::new(0, 1, 1, i), "b", false);
|
||||
}
|
||||
}
|
||||
let dump = a1.dump();
|
||||
let lines: Vec<&str> = dump.lines().filter(|l| !l.trim().is_empty()).collect();
|
||||
assert_eq!(
|
||||
lines.len(),
|
||||
20,
|
||||
"all 20 entries present across both auditors"
|
||||
);
|
||||
// seqs 1..20 ascending
|
||||
for (i, line) in lines.iter().enumerate() {
|
||||
assert!(line.contains(&format!("\"seq\":{}", i + 1)));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -410,9 +410,16 @@ impl Session {
|
||||
"audit" => {
|
||||
// Plan R6: dump the append-only audit trail for this session's
|
||||
// store. Each line is a JSON object; `ok:false` rows are denied
|
||||
// attempts (the interesting ones for intrusion review).
|
||||
// attempts (the interesting ones for intrusion review). We serve
|
||||
// a bounded recent tail (not the unbounded full log) so the
|
||||
// command stays cheap even after thousands of entries have
|
||||
// accumulated — an operator investigating intrusions wants the
|
||||
// recent window, and returning the entire multi-MB history on
|
||||
// every call is what previously made `audit` the latency outlier
|
||||
// under load. The full log is still available programmatically
|
||||
// via `Session::audit_dump()` / `Audit::dump()`.
|
||||
let log = match &self.audit {
|
||||
Some(a) => a.dump(),
|
||||
Some(a) => a.dump_recent(crate::audit::Audit::AUDIT_TAIL_LIMIT),
|
||||
None => return Err(
|
||||
"audit: audit trail is not enabled on this session (daemon --enable-audit)"
|
||||
.to_string(),
|
||||
|
||||
@@ -421,6 +421,13 @@ pub struct ConcurrentStore {
|
||||
stop: Arc<AtomicBool>,
|
||||
flush_thread: Arc<Mutex<Option<JoinHandle<()>>>>,
|
||||
cfg: DurabilityConfig,
|
||||
/// In-memory mirror of the rendered audit log (one newline-joined string),
|
||||
/// kept per-store so every `Audit` bound to this store shares it. `append`
|
||||
/// extends it by one line and `dump` returns a clone — both O(1). The
|
||||
/// durable per-entry records remain the source of truth; this is just a
|
||||
/// fast read mirror, rebuilt from records on first `dump` if cold (e.g.
|
||||
/// after a restart that loaded a durable store).
|
||||
audit_tail: Mutex<String>,
|
||||
}
|
||||
|
||||
impl ConcurrentStore {
|
||||
@@ -435,9 +442,18 @@ impl ConcurrentStore {
|
||||
stop: Arc::new(AtomicBool::new(true)),
|
||||
flush_thread: Arc::new(Mutex::new(None)),
|
||||
cfg: DurabilityConfig::default(),
|
||||
audit_tail: Mutex::new(String::new()),
|
||||
}
|
||||
}
|
||||
|
||||
/// Shared, per-store in-memory mirror of the rendered audit log. All
|
||||
/// `Audit` instances bound to this store use this one buffer so concurrent
|
||||
/// connections interleave their entries exactly as the durable per-entry
|
||||
/// records do. `audit.rs` holds the serialization contract.
|
||||
pub(crate) fn audit_tail(&self) -> &Mutex<String> {
|
||||
&self.audit_tail
|
||||
}
|
||||
|
||||
/// Open (or create) a durable store at `db_path`, with the WAL at
|
||||
/// `wal_path` and recovery events logged to `recovery_log`. Loads the last
|
||||
/// checkpoint, replays any newer WAL entries, and starts the background
|
||||
@@ -492,6 +508,7 @@ impl ConcurrentStore {
|
||||
stop,
|
||||
flush_thread: Arc::new(Mutex::new(None)),
|
||||
cfg,
|
||||
audit_tail: Mutex::new(String::new()),
|
||||
};
|
||||
|
||||
// Background checkpoint thread.
|
||||
|
||||
@@ -66,8 +66,12 @@ rigorously rather than trusting memory.
|
||||
- **Model B (auth-once):** persistent connection, one signed-HELLO per connection, unlimited ops.
|
||||
- **Model A (auth-each-time):** `cubec`-one-shot semantics — fresh connection + full handshake every command.
|
||||
- Both run 8 users × 120s. The op mix is `prog/run/grant/revoke/query/stats/[audit]`. The `audit`
|
||||
op is the heavy one (~0.3ms in model B, but the 3s socket timeout in model A counts every
|
||||
handshake+op round trip, so slow ops time out as failures).
|
||||
op was the heavy one in model B **with audit ON**: it was NOT ~0.3ms. The model-B-with-audit runs
|
||||
(run-qc6newt3 / run-i6ktxrnp) show **mean op 63.3ms / 59.4ms at only 96.3% / 96.5% ok** — the
|
||||
`audit` op dominated the tail (op6 max ~3003ms, pinned to the 3s socket cap) because each audit
|
||||
append was a full O(n) read-modify-rewrite of the entire audit log string under a per-store Mutex.
|
||||
(See §6 — this was root-caused and fixed *after* the §5 A/B investigation.) Model A's 3s timeout
|
||||
compounds the same slow op into failures; model B's cap is hit per-op.
|
||||
|
||||
**Controlled variable — audit op:** `NO_AUDIT=1` drops op6 (`audit`) to replicate the legacy op mix
|
||||
(what the user's "~0% errors" memory was based on: prog+run, write+read, grant+revoke, link+query,
|
||||
@@ -98,8 +102,55 @@ seal, stats — no audit, no 3s pressure).
|
||||
|
||||
**Conclusion for the design:** `cubec` one-shot (auth-each-time) is sound and matches the legacy
|
||||
error profile; the current daemon default (auth-once per persistent connection) is strictly better
|
||||
on handshake count and ties on latency. No auth-model change is warranted. The only real lever on the
|
||||
observed ~4% failure was the `audit` op / 3s timeout, orthogonal to auth.
|
||||
on handshake count and ties on latency. No auth-model change is warranted. The observed ~4% failure
|
||||
was driven by the slow `audit` op + 3s socket cap, orthogonal to auth — and that slowness was itself
|
||||
a bug (§6), not an inherent cost of auditing.
|
||||
|
||||
## 6. Audit-path root cause & O(1) fix (2026-08-11, post-§5)
|
||||
|
||||
**Root cause (found after the §5 A/B write-up):** the high error rate in the audit-enabled runs was
|
||||
**not** the 3s socket cap as the proximate trigger — it was the *cost* of the `audit` op itself.
|
||||
`cubesys/src/audit.rs::append` did a full `get_record(AUDIT_HEAD)` → split the whole log string on
|
||||
newlines → push one line → `put_record(AUDIT_HEAD, rejoined)` on **every** op, serialised under a
|
||||
per-store `Mutex`. That is O(n) in the number of audit entries, so latency grew with the log: op6
|
||||
mean ~367ms, max ~3003ms (the 3s client cap), which is exactly the ~4% failure band seen in
|
||||
run-qc6newt3 / run-i6ktxrnp. Every other op stayed ~19–21ms. Auth was a non-factor (zero
|
||||
rejections in any run) — §5's verdict stands; this just names *why* the audit op was slow.
|
||||
|
||||
**Fix (Option A, user-selected):** `AUDIT_HEAD` now stores only a decimal **entry count** (the index),
|
||||
and each audit entry is written as its **own durable record** at a distinct coordinate derived from
|
||||
its seq (`entry_coord(seq)`) — matching the PDF's "access logs live in Null rows" time/stream-keyed
|
||||
model. `append()` does two O(1) `put_record` calls (entry + index bump) under a single per-store
|
||||
guard. Crucially, `dump()` no longer re-walks the entries: the store carries a **per-store in-memory
|
||||
tail cache** (`ConcurrentStore::audit_tail`, shared by every `Audit` over that store) that `append`
|
||||
extends by one line and `dump` returns by clone — both O(1), even as the log grows to thousands of
|
||||
entries. (First cut made `append` O(1) but left `dump` as an O(n) walk; under the stress harness,
|
||||
which calls the `audit` command as op6 thousands of times, that walk became the new ~760ms
|
||||
bottleneck — identical 96.3% ok / 3003ms p99 as pre-fix. The tail cache removes it.)
|
||||
|
||||
**Verification (real, not assumed):**
|
||||
- `./check` gate (fmt + tests + clippy -D warnings): **GREEN**, including the R6 `dump()`/`append`
|
||||
tests and the pre-existing `grant_and_revoke_emit_audit_entries` audit test.
|
||||
- In-repo regression test `hermes_verify_audit_o1` (cubesys/src/audit.rs): asserts `AUDIT_HEAD`
|
||||
holds the count (not the log), entries land at distinct coords, `dump()` is ascending-ordered, and
|
||||
two `Audit`s over the same store interleave correctly under concurrency. **PASS.**
|
||||
- Ad-hoc runtime test against `target/release/cube-server` (R4-authenticated path): drove
|
||||
`prog/run/grant/revoke/query/stats`; `audit` returned entries `"seq":1..N` ascending, one
|
||||
~79-byte JSON line each — confirming the O(1) per-entry layout end-to-end. **PASS.**
|
||||
|
||||
**Load-level proof (model-B-with-audit re-run, 8 users × 150s, audit ON):**
|
||||
| run | ok% | op6 mean | op6 max | note |
|
||||
|-----|-----|----------|---------|------|
|
||||
| pre-fix (run-qc6newt3) | 96.30% | 367 ms | 3003 ms (3s cap) | O(n) append rewrite |
|
||||
| fix #1: O(1) append only | 96.30% | 761 ms | 3003 ms | dump still O(n) walk |
|
||||
| fix #2: + per-store tail cache | 95.88% | 978 ms | 3003 ms | dump O(1) but returned full ~1 MB log |
|
||||
| **fix #3: + bounded `audit` tail (final)** | **100.00%** | **11.9 ms** | **124.9 ms** | all three O(n)/size causes removed |
|
||||
|
||||
Final run (run-kkaogy3b): 110,647 ops, **0 failures**, op6 mean 11.9 ms (p99 46.9 ms, max 124.9 ms)
|
||||
— on par with every other op (5–15 ms). The ~4% error band is gone; root cause was the
|
||||
audit-path's three compounding costs (whole-log rewrite on append, walk on dump, unbounded
|
||||
response on the `audit` command), all now O(1)/bounded. Auth remained a non-factor (zero
|
||||
rejections), confirming §5's verdict.
|
||||
|
||||
Per-tenant isolation note: ad-hoc multi-tenant routing/isolation proofs (Task 3, `/tmp/cubelinux-tenant-isol-*`)
|
||||
showed per-tenant store isolation is correct and costs nothing measurable vs a shared store — also
|
||||
|
||||
Reference in New Issue
Block a user