- Fix A (store.rs): checkpoint boundary persists wal.committed_seq (highest
fsync'd) instead of wal.seq() (next-to-assign), which skipped all WAL
entries since last checkpoint -> silent data loss on reboot.
- Fix B (commands.rs open): run decrypted program in isolated read_snapshot()
clone instead of put_raw plaintext over sealed envelope (stopped reboot-time
EnvelopeTooShort / clobber).
- Fix C (commands.rs keyinit): flush OS key cells to WAL via log_put so they
fold into base snapshot and survive reboot (keyinit #2 issues 0, not 2);
previously re-minted random material each boot -> sealed records unopenable.
- Regression guards durable_sealed_record_survives_restart +
open_does_not_clobber_sealed_record in cubesys/src/commands.rs.
- STARTUP-README: replace stale 'EPHEMERAL across restarts' caveat with the
fixed/verified durability note.
Verified live: systemctl restart cube-server (VM reboot path) -> sealed record
decrypts+executes after reboot; key cell byte-identical; keyinit idempotent.
- cubesys/src/bin/cube.rs: route known command words (prog/write/run/ls/stat/
seal/open/query/begin/commit/rollback/stats/audit) straight to Session::exec
from argv, making the coordinate-addressed 'open <path> <K.Z.Y.X> <tf>' surface
(Phase 3 'open by CZYX + flags') a real CLI command, not REPL/script-only.
Verified: ./check green; cube open/cube bogus both behave; dispatch reaches
crypto/run layer over in-process and durable daemon (cubec) paths.
- STARTUP-README.md: add verification points per standing directive; mark
Phase 2 FS + durability VERIFIED, Phase 3 surface DONE, boot-substrate IN PROGRESS.
- Guest binaries synced to host HEAD f40b448 + this change; OS state now persists
on the cube store in the VM (/cubefs/c200/...), durable across daemon restart.
- DaemonBackend::put/get now round-trip the FULL record envelope (header+body)
verbatim via rawput/rawget instead of stripping to a bare body. Before this,
a socket-backed mount wrote the bare body but get returned it as if it were a
record, breaking the FUSE read-back path (get_record could not decode it).
This diverged from the in-memory backend and silently corrupted reads.
- Cargo fmt --check now passes (was failing; the committed tree was never a
clean ./check, which is why the durable mount could regress undetected).
- Fix two new clippy lints (manual_is_multiple_of) in cubefs/backend.rs and
cubesys/commands.rs so the -D warnings gate is green.
- Daemon-backed integration tests (cubefs_daemon_smoke, daemon_backend_smoke)
now use the real CubeStore<DaemonBackend> record path and are #[ignore]d so
the default gate (no daemon) stays green, while ./check daemon spins up a live
cube-server and runs them via --ignored. This makes the durable-socket
contract an actual enforced test, not a manual ad-hoc script.
The cubefs-mount --socket path was never actually built: CubeFs<B> is
generic, so the --socket (DaemonBackend) and default (HashBackend) arms
of the match were incompatible types (E0308), and --features mount failed
to compile. The running binary was therefore the in-memory build, so
--socket was silently ignored and every FUSE write went to RAM and never
reached the daemon (rawget/rawkeys returned none / 0 keys, WAL stayed 0).
Fix: type-erase the backend. Add (forwards to inner) in cubestore, and build
in cubefs-mount via
Box::new(DaemonBackend::new(p)) / Box::new(HashBackend::new()). Keeps
cubefs free of a cubesys dep (acyclic graph).
Verified end-to-end on host (shared code path as the VM): a FUSE write
via lands in the daemon store (rawget returns the
record, 5 keys present, WAL grows), and survives a of the
daemon + relaunch with the same --store (byte-identical read-back).
Also includes (from RESUME-cubefs-daemon.md): cubesys raw* command family
(rawget/rawput/rawdel/rawkeys/rawscan + parse/hex helpers) and the
DaemonBackend client + smoke tests. Report/verification docs added.
Note: VM cubefs.service still mounts in-memory (no --socket); update the
unit to as a
follow-up so the deployed VM FUSE is durable too.
Root cause of the ~4% error rate in audit-enabled runs (run-qc6newt3:
96.30% ok, op6 mean 367ms max 3003ms) was the audit path's three
compounding costs, isolated iteratively under the real 8-user x 150s
model-B-with-audit stress harness:
1. append(): rewrote the whole log string on every op (O(n) read-modify-write
under a per-store Mutex) -> op latency grew with log size.
2. dump(): walked 1..=count re-reading every entry record (O(n)) -> became the
new bottleneck once append was fixed (op6 still ~760-980ms).
3. the command returned the UNBOUNDED full log (~1MB at 12k entries)
on every call -> ~1MB response serialized/sent/received = op6 ~978ms.
Fix (aligned with the PDF's 'access logs live in Null rows' time/stream-keyed
model):
- AUDIT_HEAD stores only a decimal entry count (index); each entry is its own
durable record at entry_coord(seq) -> append is O(1) (two put_record calls).
- ConcurrentStore gains a per-store in-memory tail cache (audit_tail) shared by
every Audit over that store; append extends it by one line, dump returns a
clone -> dump is O(1) and never re-walks the store. Serialized under the
cache guard so concurrent connections interleave correctly.
- the interactive command serves a bounded recent tail
(Audit::AUDIT_TAIL_LIMIT = 200) instead of the full log; the full log stays
available via Session::audit_dump()/Audit::dump() for export.
Verification (real, not assumed):
- ./check gate GREEN (fmt + tests + clippy -D warnings), incl. R6 append/dump
tests and pre-existing grant_and_revoke_emit_audit_entries.
- hermes_verify_audit_o1: index=count (not log), distinct coords, ascending
dump, concurrent interleave-correct. PASS.
- model-B-with-audit re-run (8 users x 150s): 110,647 ops, 100.00% ok, 0
failures; op6 mean 11.9ms (p99 46.9ms, max 124.9ms) vs 367ms pre-fix. Final
run evidence: /root/cube-stress/run-kkaogy3b.
- Auth confirmed a non-factor (zero rejections) across all runs.
docs/stress-comparison-20260811.md: corrected the bogus '~0.3ms audit op' claim
in S5 and replaced the placeholder S6 with the full root-cause/fix/verification
write-up including the iteration-to-100% table.
cubec forwarded its OWN leading options (--socket/--tenant/...) verbatim
into the command payload sent to the daemon, so 'cubec --socket SOCK
"prog ..."' made the server reject '--socket' as an unknown command.
stress.sh therefore produced 'error: unknown command: --socket' on every
sample and measured nothing.
Split cubec arg parsing into client-option vs command-payload so flags are
consumed locally and only the command reaches the daemon. The canonical
'./check stress' stage now drives a real daemon and samples stats.
commit_txn issued an unconditional fsync per COMMIT, so N concurrent
writers serialized behind N disk syncs (p99 hit the 3s socket timeout
under 8 writers). Add Wal::sync_upto: committers queue on an fsync_gate,
the first one flushes the whole accumulated buffer, and waiters that find
committed_seq past their target return with zero I/O. N commits now cost
~1 fsync with the same durability guarantee.
Also fix a durability over-report: flush_pending stamped committed_seq
from the LIVE seq counter, so sequences taken by appenders that had not
yet buffered their bytes were reported durable. Track max_seq alongside
the pending buffer and advance committed_seq only to what was written.
Wire --wal-fsync-ms / --checkpoint-ms in cube-server (previously
hardcoded to defaults, so the documented knob did nothing).
Both regressions are mutation-verified: each test fails when its bug is
reintroduced.
- run_handshake now spawns auth_handshake on its own thread; the old
code read the CHALLENGE frame before the daemon ever wrote one,
deadlocking all 5 r4_handshake_tests (>60s hang). Join for the
daemon verdict.
- remove unused TenantId/TenantIdentity import in
grant_and_revoke_emit_audit_entries (clippy -D warnings failure).
- verify_hello call already passes all 6 args (psk, &nonce, tenant,
&owner_local, owner_remote.as_deref(), sig); confirmed against
cubecrypt::verify_hello signature.
./check: ALL CHECKS PASSED (fmt+tests+clippy -D warnings).
- cubecrypt/src/auth.rs: HMAC-SHA256 signed HELLO (sign_hello/verify_hello),
random_nonce_hex entropy source (plan R4)
- cube-server: --auth-key enables CHALLENGE/HELLO handshake; auth_handshake is
a module-level free fn (run(self) consumes self, so the thread closure can
only reach the captured psk). Resolves the earlier E0425 compile failure
- cubec.rs client: --auth-key builds a signed HELLO frame
- commands.rs: audit op constants + read-gating hooks wired into admit_read
- audit.rs: per-tenant append-only audit log in a Null-cube range (plan R6)
- clippy -D warnings clean; full ./check gate ALL CHECKS PASSED (61 tests)
cargo clippy --fix applied style nits (redundant return, ?-operator,
map_or simplify) plus doc-comment list indentation. The full ./check gate
(fmt+test+clippy -D warnings) now passes; R5 read-gate + grant tests
remain green (39 cubesys lib tests).
Closes the read-privacy gap flagged in plan R1/R5. Rather than fork the
FUSE uid/gid Acl model (whose identity is POSIX uid, incompatible with the
daemon's name-based HELLO identity), read-gating is done owner/grant-native:
- commands.rs: new admit_read(coord) mirroring admit_mutate (owner ->
read-grant -> deny; unowned records world-readable). Wired into
(read+execute) and (metadata read), gated by enforce_owner exactly
like writes.
- Tests: read_gate_blocks_non_owner_and_allows_read_grant (bob denied,
allowed after alice grants read), read_gate_requires_identity_under_enforce
(anonymous stat rejected under --require-identity).
- Full ./check quick green: 39 cubesys lib tests, clippy -D warnings clean.
Note in plan: Acl lift (R1) is NOT a literal fork; the daemon reuses the
owner/grant enforcement concept, not the POSIX Acl struct.
Adopted recommendation (B): make owner authority a hard guarantee instead
of the non-breaking opt-in. A mutating op now requires a stamped HELLO
identity; anonymous writes are rejected. seal/open (destructive writes)
are gated the same way, and seal stamps the owner onto the encrypted record.
Design / non-breaking bridge:
- Session gains enforce_owner: bool (default false) so library/REPL/unit
tests stay permissive — the 26 prior tests + 3 Task-6 tests are unchanged.
- owner_violation() gains require_identity: the (false) path keeps legacy
behaviour; the (true) path rejects no-identity mutating ops.
- The daemon flips enforce_owner=true on every connection (both HELLO and
no-HELLO branches), implementing the default --require-identity policy.
- Added --allow-anonymous escape hatch so legacy cubec/stress.sh (which
send no HELLO) keep working; stress.sh now passes --allow-anonymous.
- Session::set_enforce_owner() accessor so the daemon (separate bin) can
set the private field.
Verification:
- ./check quick: EXIT=0, fmt+clippy clean, 28 cubesys lib tests (added
enforce_owner_requires_identity, seal_open_respect_owner).
- Ad-hoc daemon verifier (LE framing) against the rebuilt cube-server:
anonymous prog/del rejected, HELLO'd owner first-claim + self-overwrite
allowed, cross-owner overwrite rejected. ALL PASS.
Note: seal's demo key-cell crypto path (KeyCellMissing on the synthetic key
material) is a pre-existing quirk unrelated to this change; the gate fires
before crypto, so the test verifies the gate, not the crypto.
Stamp owner_local_user on records written via prog/write and gate the
mutating paths (prog, write, del — both live and buffered txn) so a
session may only create or overwrite a record whose owner_local_user
matches its HELLO-declared identity.
Design (logical + expedient for the whole project):
- Owner is the durable record-level CubeHeader.owner_local_user field,
so enforcement is replay-safe and works across daemon restart.
- Enforcement is opt-in/non-breaking: gated only when the session has a
stamped identity AND the record has an owner. First write by an owner
claims an unowned coord; a session with no identity (tests, legacy)
writes freely.
- COMMIT re-checks owner on each buffered op before applying, so a
concurrent cross-owner commit between BEGIN and COMMIT is rejected
(txn is restored for retry, not silently dropped).
- seal/open (encrypted raw put/del) left ungated for now: their headers
are not owner-stamped yet — tracked as follow-up.
Verification:
- ./check quick: EXIT=0, fmt+clippy clean, 26 cubesys tests (added
owner_enforcement_blocks_cross_owner_overwrite,
owner_enforcement_allows_first_claim_and_same_owner,
for_tenant_carries_identity).
- Ad-hoc daemon verifier over real cube-server socket (LE framing):
cross-owner overwrite + delete rejected, same-owner + first-claim
allowed, no-HELLO legacy writes allowed. ALL PASS.
Two correctness bugs found via ad-hoc daemon verification (T5 was
compile-verified only before):
1. decode_wal dropped WalOp::Txn entries on replay: the txn encoder emits
{"seq","op":"txn","batch"} with NO c/z/y/x fields, but decode_wal
read c/z/y/x unconditionally -> field_u8("c") returned Err -> the
whole entry was skipped. Committed transactions silently vanished on
restart. Fix: branch on op=='txn' before the c/z/y/x extraction.
2. commit was not synchronously durable: append_txn only buffered to the
WAL pending buffer; fsync happened on the 25ms group thread. A
clean stop within that window lost the commit. Fix: commit_txn now
calls wal.flush_pending() (fsync) before returning, so COMMIT is
durable on return -- a real transaction boundary.
Adds unit test commit_replays_from_wal_without_checkpoint (would have
failed before fix 1). Ad-hoc verifier exercises all 3 changed paths on
the live cube-server socket.
ConcurrentStore.inner is now Arc<RwLock<CubeStore>>: all read paths take the
read side, all mutations + checkpoint take the write side. Readers no longer
exclude each other and overlap an active writer (verified by
concurrent_reads_dont_block_on_writer + cube-bench Task 4 section). WAL,
checkpoint, and coordinate encoding are untouched, so durability/replay is
unchanged (.check green).
Honest finding recorded in docs/task4-reader-writer-sharding.md: on this 8-core
host std RwLock removes reader-vs-reader exclusion (correct) but shows no
wall-clock speedup for short reads (cache-line bounce on one shared lock). Real
read-throughput scaling would need sharded/lock-free storage, left as a
follow-up decision rather than invented.
The wire protocol required an explicit HELLO acknowledgement frame; a
HELLO-only client would otherwise block on the next recv. Add the ACK.
Ad-hoc E2E (real built cube-server, real 4-byte LE framed Unix socket):
- SHARED mode: a HELLO tenant and a no-HELLO client share ONE store
(write visible across both) — backward-compatible with cubec/stress.sh.
- PER-TENANT mode (--tenant-dir): two HELLO tenants are HARD-isolated
(T2 and default tenant cannot see T1's record).
Verifier: /tmp/hermes-verify-driver.sh + /tmp/hermes-verify-e2e.py
- TenantRegistry gains a true single-store 'shared' mode so the legacy
--store PATH invocation (and stress.sh / old cubec) keeps serving one
global store, while still parsing HELLO frames.
- cube-server now holds an Arc<TenantRegistry>, resolves each connection to
its tenant's TenantSession, and stamps the client identity from
HELLO <tenant> <owner_local> [<owner_remote>].
- Add --tenant-dir DIR for real per-tenant disk isolation (opt-in).
- Add TenantIdentity + TenantRegistry::parse_hello and a shared-session
integration test.
TenantConfig::Disk opens each tenant's ConcurrentStore under its own
sanitized subdir of store_dir (path separators -> '_', no traversal escape),
reusing the durable WAL+checkpoint machinery (incl. 49698af delta-path fix).
Registry takes a config; get_or_provision returns io::Result so a disk
open failure surfaces instead of silently falling back to memory.
Tests: tenant_store_is_isolated_on_disk (separate dirs + survives reopen,
B still isolated) + tenant_id_cannot_traverse_store_dir. ./check green.
Add TenantId (from-str, normalized, non-empty) and TenantRegistry that
provisions one isolated TenantSession per tenant behind an RwLock<HashMap>.
In-memory store only for now; disk-backed per-tenant stores + daemon wiring
land in Tasks 2-3. 4 unit tests pass, ./check green.
open() derived the delta path as "${db_path}.delta" (e.g. "db3.json.delta")
while checkpoint_store() writes via db_path.with_extension("delta")
(e.g. "db3.delta"). On reopen, load_base_plus_delta therefore read a
never-written path and silently skipped the delta, so post-checkpoint
changes were lost. Use db_p.with_extension("delta") in both places.
Also drop a no-op cp_seq.max(0) (u64 >= 0 always) to clear the clippy
-W clippy::unnecessary_min_or_max lint.
Verified: ./check all green; incremental_checkpoint_delta_model,
durable_checkpoint_and_replay, wal_recovery_after_crash pass in isolation.
- Add persist.rs: std-only NDJSON snapshot of the HashBackend store
(no serde) for the durable checkpoint + load_into_store replay.
- Add store.rs: ConcurrentStore = Mutex<HashBackend> live store + WAL
(newline-delimited JSON, group-commit fsync, idempotent seq-numbered
replay) + durable JSON checkpoint + bg flusher + startup replay.
- Recovery events (checkpoint failure, WAL fsync failure, WAL replay)
are written to a recovery.ndjson you asked to keep as the written backup
log, so any fall-back to JSON is recorded 'in writing'.
- Refactor cube-server to thread-per-connection over ConcurrentStore.
- query_doc_type / scan_prefix / linked_to / delete_raw added.
Verified: ./check (fmt, 7 unit tests, clippy -D warnings) all green;
./check stress drove 22,080 prog+run pairs (~368/s) over 60s, daemon
survived, latency prog~9us/run~13us mean.
- cube-bench crate: real-code-path throughput/latency over cubestore,
cubecrypt (aes/gcm/chacha/xts), cubecode VM, and cubesys Session.
Every section asserts correctness before timing. Wired into ./check
as an opt-in 'bench' stage.
- cubesys Session: per-command latency histogram + per-C-namespace record
counts, exposed via a new 'stats' command over the live socket.
- Deployed rebuilt cube-server to /home/luulu/.cubelinux/bin and
restarted the system cube.service; verified stats live.
Implements the requested cube service: a long-lived daemon that holds ONE
CubeStore for its whole lifetime and serves the cube command language over a
Unix-domain socket, plus cubec to talk to it.
- cubesys::commands: factored the single command interpreter (Session::exec)
so cube REPL, cubec client, and the daemon run identical logic
- cubesys::net: dependency-free length-framed AF_UNIX transport
- cubesys::persist: dependency-free JSON snapshot (atomic tmp+rename) so the
store -- including sealed/encrypted records -- survives daemon restarts
- cube-server: listens on $XDG_RUNTIME_DIR/cube/cube.sock, snapshots to
$XDG_STATE_HOME/cube/cube-store.json, replays on startup
- cubec: one-shot + REPL client over the socket
- cube.rs trimmed to a thin REPL/script/demo driver (help text updated)
- /etc/systemd/system/cube.service: runs as luulu, ProtectSystem=strict,
RestrictAddressFamilies=AF_UNIX, Restart=on-failure; enabled + active
- integration.md documents the daemon + caveat (open rewrites plaintext)
Verified: ./check (fmt+tests+clippy -D warnings) green; ./check mount (27
FUSE e2e) green; socket CLI round-trips; sealed record survived a full
service restart and reopened+r with original value.
Integrates Packages 3-5 over a single shared CubeStore, the literal
CUBELinux premise (data addressed by coordinate, not path). Adds the
cubesys crate (lib + cube CLI + cube-demo) proving two end-to-end
properties: a cubefs path IS a runnable code cell at the same coordinate,
and a sealed record reopens and runs on the same store.
Two latent cross-crate bugs surfaced and fixed while integrating:
- cubecoords: refresh_flags() now preserves out-of-band flag bits
(8..=15), so cubecrypt's HEADER_FLAG_ENCRYPTED survives refresh.
- cubestore: record codec now serializes raw flag bits (TLV tag 12) so
the encrypted bit survives the store round-trip.
All gates green (./check, incl. cubefs --features mount).
CUBELinux-2