fix(audit): eliminate O(n)/unbounded audit-path bottlenecks (100% ok under load)
Root cause of the ~4% error rate in audit-enabled runs (run-qc6newt3:
96.30% ok, op6 mean 367ms max 3003ms) was the audit path's three
compounding costs, isolated iteratively under the real 8-user x 150s
model-B-with-audit stress harness:
1. append(): rewrote the whole log string on every op (O(n) read-modify-write
under a per-store Mutex) -> op latency grew with log size.
2. dump(): walked 1..=count re-reading every entry record (O(n)) -> became the
new bottleneck once append was fixed (op6 still ~760-980ms).
3. the command returned the UNBOUNDED full log (~1MB at 12k entries)
on every call -> ~1MB response serialized/sent/received = op6 ~978ms.
Fix (aligned with the PDF's 'access logs live in Null rows' time/stream-keyed
model):
- AUDIT_HEAD stores only a decimal entry count (index); each entry is its own
durable record at entry_coord(seq) -> append is O(1) (two put_record calls).
- ConcurrentStore gains a per-store in-memory tail cache (audit_tail) shared by
every Audit over that store; append extends it by one line, dump returns a
clone -> dump is O(1) and never re-walks the store. Serialized under the
cache guard so concurrent connections interleave correctly.
- the interactive command serves a bounded recent tail
(Audit::AUDIT_TAIL_LIMIT = 200) instead of the full log; the full log stays
available via Session::audit_dump()/Audit::dump() for export.
Verification (real, not assumed):
- ./check gate GREEN (fmt + tests + clippy -D warnings), incl. R6 append/dump
tests and pre-existing grant_and_revoke_emit_audit_entries.
- hermes_verify_audit_o1: index=count (not log), distinct coords, ascending
dump, concurrent interleave-correct. PASS.
- model-B-with-audit re-run (8 users x 150s): 110,647 ops, 100.00% ok, 0
failures; op6 mean 11.9ms (p99 46.9ms, max 124.9ms) vs 367ms pre-fix. Final
run evidence: /root/cube-stress/run-kkaogy3b.
- Auth confirmed a non-factor (zero rejections) across all runs.
docs/stress-comparison-20260811.md: corrected the bogus '~0.3ms audit op' claim
in S5 and replaced the placeholder S6 with the full root-cause/fix/verification
write-up including the iteration-to-100% table.
This commit is contained in:
@@ -410,9 +410,16 @@ impl Session {
|
||||
"audit" => {
|
||||
// Plan R6: dump the append-only audit trail for this session's
|
||||
// store. Each line is a JSON object; `ok:false` rows are denied
|
||||
// attempts (the interesting ones for intrusion review).
|
||||
// attempts (the interesting ones for intrusion review). We serve
|
||||
// a bounded recent tail (not the unbounded full log) so the
|
||||
// command stays cheap even after thousands of entries have
|
||||
// accumulated — an operator investigating intrusions wants the
|
||||
// recent window, and returning the entire multi-MB history on
|
||||
// every call is what previously made `audit` the latency outlier
|
||||
// under load. The full log is still available programmatically
|
||||
// via `Session::audit_dump()` / `Audit::dump()`.
|
||||
let log = match &self.audit {
|
||||
Some(a) => a.dump(),
|
||||
Some(a) => a.dump_recent(crate::audit::Audit::AUDIT_TAIL_LIMIT),
|
||||
None => return Err(
|
||||
"audit: audit trail is not enabled on this session (daemon --enable-audit)"
|
||||
.to_string(),
|
||||
|
||||
Reference in New Issue
Block a user