cubelinux-2/.hermes/plans/2026-08-11_041500-concurrent-multitenant-db.md

Concurrent / Multi-Tenant CUBELinux-2 DB — Implementation Plan

For Hermes: Use subagent-driven-development skill to implement this plan task-by-task.

Goal: Upgrade the CUBELinux-2 store + daemon from "thread-per-connection sharing one global mutex (serialized commands, no tenant isolation)" to a genuinely concurrent, multi-tenant database: per-tenant isolated namespaces, reader/writer sharding for real concurrency, multi-command transactions (MVCC-style snapshot isolation), and per-client admission/auth — without regressing the durability gate (wal_recovery_after_crash, durable_checkpoint_and_replay, incremental_checkpoint_delta_model) that is green today.

Architecture (grounded in current code, not invented):

• Today: cube-server holds ONE Arc<Mutex<Session>>; each connection spawns a thread and runs s.exec(line) under the lock → concurrent readers but serialized commands. The store is ConcurrentStore = Mutex<CubeStore<HashBackend>> + a group-fsync NDJSON WAL (append-only, seq-monotonic) + scheduled checkpoint. The WAL is already append-only and multi-writer-friendly — that is our lever.
• The "not a real DB" gaps to close: (1) no tenant boundary in the write path — every client lands in the same keyspace; (2) one global lock → no real read/write concurrency under load; (3) no transactions — a client cannot do "read-modify-write" or multi-key atomic ops safely; (4) no per-client admission/auth — any socket connection writes to the shared store.
• We do NOT rewrite the WAL or the coordinate encoding. We layer: tenant routing (a tenant id prefixes the effective coordinate namespace / ACL), a sharded RwLock registry of per-tenant stores (readers run in parallel, writers serialized per-tenant), an optional MVCC snapshot per transaction, and a thin auth/admission check at connection accept time.

Tech Stack: Rust (existing workspace) · std::sync::{RwLock, Arc} (no new deps for the core) · existing ConcurrentStore/Wal/Session · optionally parking_lot if a faster rwlock is wanted (NOTE: add as a dependency only if std RwLock profiling shows writer starvation — YAGNI until proven). Keep ./check (fmt+test+clippy -D warnings) and the stress/mount/bench gates green throughout.

---

Current-state assumptions (verified this session)

• cubesys/src/store.rs: ConcurrentStore wraps Mutex<CubeStore<HashBackend>>; WAL is append-only NDJSON with monotonic seq; durability via wal_fsync_ms (25ms) + checkpoint_ms (2000ms); DELTA_COMPACT_BYTES=1MiB.
• cubesys/src/bin/cube-server.rs: Server { listener, session: Arc<Mutex<Session>> }; thread-per-connection; each command s.exec(line) under the session lock. cubesys has ZERO references to owner markings or ACL today.
• Owner markings ALREADY EXIST in the record header (this is the PDF model the user pointed at): cubecoords::CubeHeader carries owner_local_user: Option<String> and owner_remote_user: Option<String> (cubecoords/src/lib.rs:242-245), and HeaderFlags mirrors the PDF's title/type/date/size/permission flag layout (flags 1-4 explicit; flags 5-19 reserved for permissions + associations). So the owner metatag is already in every record — it just isn't enforced anywhere in the daemon path.
• ACL exists in cubefs only: cubefs::nullspace::Acl { uid, gid, mode } with a working permits(uid,gid,want) check, stored per-coordinate in null-space (NullSpace::get_acl/set_acl), but it is enforced ONLY at the FUSE mount layer — NOT in the cube-server daemon command path.
• The PDF's tenant axis already exists: REVIEW-cubelinux-pdf.md confirms "The C axis as environment/tenant selector, where the same logical coordinate resolves differently depending on active Null-cube state" — and the stress test already routes tenants via C=77. So tenant selection via the C axis is the PDF-blessed mechanism, not an invented one.
• "Permission grants" are reserved but unimplemented: the PDF reserves header flags 5-19 for "permissions and associations"; no delegated-grant model exists anywhere in the code. That is the new layer to build (see D5).
• Gate status now GREEN: fmt+test+clippy+stress(150s)+mount(27/27)+bench. Do not break these.

---

Design decisions (so the implementer doesn't guess)

D1. Tenant model + identity (uses the PDF's OWN model, not invented)

• A tenant = the PDF's C-axis environment selector. The connecting client declares its tenant via HELLO <tenant>; the daemon maps it to a C value (the stress harness already uses C=77 — that IS the tenant axis). Tenants physically isolated on disk under store_dir/<tenant>/ (own WAL + checkpoint), reusing the entire existing durable WAL machinery per tenant.
• Identity comes from the existing owner metatag, not a new token table. The PDF's CubeHeader already carries owner_local_user / owner_remote_user. HELLO therefore carries (tenant, owner_local_user, [owner_remote_user]) — i.e. the client asserts the owner identity that the record headers already model. This is the "owner markings in metatags" the user referenced; we now ENFORCE them (D4) instead of carrying them inertly.
• Cross-tenant access is impossible by construction (different Arc + different C axis). The "hard per-namespace partition" the user mandates for agents is enforced at the storage layer.

D2. Concurrency model

• Replace the single Arc<Mutex<Session>> with Arc<TenantRegistry> where TenantRegistry { tenants: RwLock<HashMap<TenantId, Arc<TenantSession>>>, ... }.
• TenantSession holds Arc<ConcurrentStore> (already internally mutexed) + telemetry. Because each ConcurrentStore already serializes its own writes, different tenants run fully in parallel; within a tenant, writes are serialized (correct) but reads via CubeStore getters can be made lock-free/copy-on-read if needed.
• This converts the current "all commands serial globally" into "serial per tenant, parallel across tenants" — real concurrency for the multi-tenant case, which is the actual ask.

D3. Transactions (MVCC-lite, YAGNI-scoped)

• Add BEGIN / COMMIT / ROLLBACK to the command language. A transaction snapshots the tenant store's in-RAM CubeStore at BEGIN (cheap Arc clone of the backend snapshot if we make HashBackend Clone; otherwise a per-transaction read-version counter).
• Writes inside a txn are buffered in the Session; on COMMIT they are applied under the store lock as one atomic batch and appended to the WAL as a single multi-op entry (new WalOp::Txn carrying [(op,coord,value),...]). On ROLLBACK the buffer is dropped — nothing hits the WAL.
• Isolation = snapshot at BEGIN; concurrent commits serialize per tenant (the store lock). This is "repeatable read" per tenant, not full serializable — explicitly enough for an agent memory store, and honest about the ceiling.

D4. Admission + owner-marking enforcement (the missing link)

• Today cubesys never checks owner/ACL. We add an enforcement layer in the daemon command path, reusing the fields that ALREADY exist:
  • Identity at connect: HELLO <tenant> <owner_local_user> [owner_remote_user]. The Session records the caller's (tenant, owner_local, owner_remote).
  • On every read/write of a record: the daemon reads the target record's CubeHeader (owner_local_user/owner_remote_user) and applies the SAME rule the FUSE layer already implements — Acl::permits(uid, gid, want) — but generalized to owner identity: a write/delete requires the caller's owner_local_user to match the record's owner_local_user (or a grant, see D5); a read requires the record to be readable by the caller (owner match, or group/other perms, or grant). This makes the header owner markings authoritative instead of decorative.
  • Reuse, don't duplicate: lift cubefs::nullspace::Acl + permits() into a shared crate (or cubesys) so both the FUSE mount and the daemon enforce identical rules. Today the ACL logic lives only in cubefs; the daemon must share it.
• This closes the "any socket connection writes to the shared store" gap using the PDF's own owner-centric model — no external token service.

D5. Permission grants (PDF flags 5-19 — the unbuilt layer)

• The PDF reserves header flags 5-19 for "permissions and associations" but nothing implements delegated grants. We add a grant as a first-class addressable null-space record (consistent with REVIEW's "metadata as reserved coordinate space, not a side table"):
  • Grant { granter: Owner, grantee: Owner, perms: rwx, scope: coord-prefix (Option<Czyx>) } stored at a null-space coord (e.g. GRANT_BUCKET).
  • Enforcement order on a protected op: (1) owner match → allow; (2) else any Grant whose grantee == caller and whose scope covers the coord and perms covers the op → allow; (3) else Acl::permits for other/group; (4) else deny.
  • Grants are themselves owner-marked (only the granter may revoke), and durable (live in the tenant's WAL like any other record).
• This is what turns "owner-only" into a real multi-tenant shared store: tenants can delegate read/write to other owners without flattening ownership.

---

Step-by-step plan (bite-sized; TDD; commit per task)

Task 1: Add TenantId type + TenantRegistry skeleton (no behavior change)

• Files: Create cubesys/src/tenant.rs; modify cubesys/src/lib.rs to pub mod tenant;.
• Step 1 (test): tenant.rs test tenant_id_roundtrips — TenantId::from_str("agent-a") == TenantId::from_str("agent-a"), distinct from "agent-b".
• Step 2: implement TenantId(String) + FromStr + Clone/Copy/Eq/Hash.
• Step 3: Twriteln TenantRegistry { tenants: RwLock<HashMap<TenantId, Arc<TenantSession>>> } with get_or_provision(&self, id) -> Arc<TenantSession> (provision returns a stub TenantSession with an in-memory ConcurrentStore for now).
• Step 4: cargo test -p cubesys tenant:: → PASS. Step 5: commit feat(cubesys): tenant id + registry skeleton.
• Verification: ./check still ALL CHECKS PASSED.

Task 2: Per-tenant durable stores on disk

• Files: Modify tenant.rs; modify ConcurrentStore::open call sites.
• Test: tenant_store_is_isolated_on_disk — two tenants get different WAL/checkpoint paths under tmp/<tenant>/; writing to A does not appear in B after reopen.
• Implement: TenantSession::open(tenant, store_dir, durability) builds its ConcurrentStore at store_dir/<tenant>/cube-store.json + .delta + .wal + .recovery.ndjson. Reuse existing ConcurrentStore::open (the delta-path fix in 49698af applies per tenant automatically).
• cargo test -p cubesys PASS; ./check green. Commit.

Task 3: Wire cube-server to multi-tenant registry + identity

• Files: Modify cubesys/src/bin/cube-server.rs (replace Arc<Mutex<Session>> with Arc<TenantRegistry>); add HELLO <tenant> <owner_local_user> [owner_remote_user] frame handling in cubesys/src/net.rs + Session.
• Test: integration test server_routes_by_tenant_and_owner — two cubec clients with different HELLO tenants see isolated stores; a write in tenant A is invisible to tenant B; a client asserting a different owner_local_user cannot overwrite another owner's record.
• Implement: accept thread does read_stream_frame → if first frame is HELLO <t> <owner> [remote], resolve registry.get_or_provision(t) and stamp Session.identity = (t, owner, remote), else reject. Each subsequent command runs against that tenant's Session with the stamped identity.
• ./check mount + ./check stress must stay green (stress harness already uses cubec; extend to 2 tenants / 2 owners). Commit.

Task 4: Reader/writer sharding (real concurrency)

• Files: tenant.rs, store.rs (expose read-only getters that don't take the full store lock if possible).
• Test: concurrent_reads_dont_block — N reader threads + 1 writer thread on one tenant; assert reader throughput is not serialized behind the writer (measure: readers complete while writer runs). Use a RwLock inside TenantSession separating "read snapshot" from "write apply".
• Implement: TenantSession { store: Arc<ConcurrentStore>, read_gate: RwLock<()> }; reads take read_gate.read(), writes take read_gate.write() then the store lock. This lets many reads run during a write.
• Benchmark before/after in docs/: expect reads to stop blocking. Commit.

Task 5: Transactions — BEGIN/COMMIT/ROLLBACK

• Files: cubesys/src/commands.rs (add opcodes), tenant.rs (TenantSession txn buffer), store.rs (WalOp::Txn).
• Test: txn_commit_atomic (all-or-nothing applied + single WAL entry), txn_rollback_drops (nothing in WAL, store unchanged), txn_isolation_snapshot (concurrent writer during txn does not affect txn's reads).
• Implement: BEGIN snapshots; commands buffer into Session.txn; COMMIT applies batch under store lock + appends one WalOp::Txn; ROLLBACK clears buffer. Replay already idempotent (put_raw), so a txn WAL entry replays as the same batch.
• Durability gate unchanged. ./check green. Commit.

Task 6: Owner-marking enforcement in the daemon path

• Files: cubesys/src/commands.rs (enforce on put/get/delete), cubefs/src/nullspace.rs (lift Acl/permits to a shared module — e.g. cubesys::acl or a new cubeacl crate — so daemon + FUSE share it), tenant.rs (Session.identity).
• Test: owner_cannot_overwrite_other (write to a record whose owner_local_user differs → denied), owner_can_write_own (passes), other_read_respects_mode (mode 0600 blocks non-owner read; 0644 allows).
• Implement: before applying a mutating op, read the record's CubeHeader; deny unless header.owner_local_user == caller.owner_local OR a matching grant exists (Task 6b). Reads check owner/other perms. Reuse the lifted permits().
• ./check stress + a denial test. Commit.

Task 6b: Permission grants (PDF flags 5-19) — DONE 2026-08-11

• Files: cubesys/src/grants.rs (NEW — Grant, Owner, Perm, GRANT_BUCKET at Czyx::new(0,1,0,1), grant/revoke/grant_allows/read_bucket/write_bucket); cubesys/src/commands.rs (GRANT/REVOKE opcodes, admit_mutate enforcement hook); cubesys/src/lib.rs (pub mod grants;).
• Enforcement model (as built): admit_mutate(coord, want) runs on every mutating op (prog/write/del/seal/open). Order: (1) owner match OR unowned (first-writer-claims) → allow; (2) any grant where grantee == caller, perms covers the op, and scope covers the coord (wildcard axis = 0) → allow; (3) else deny. No external KMS; grants live in the tenant WAL.
• Durability fix (key learning): the grant table is stored via put_record + a tagged CubeHeader{doc_type:"grant-table"}, NOT put_raw JSON — because dump_store (checkpoint) requires the [u32 header_len][hdr][body] framing. Storing raw bytes corrupted/ dropped the record on checkpoint (grant_survives_reopen initially failed). Now persists correctly.
• Identity gating: GRANT/REVOKE require a HELLO identity when the daemon runs --require-identity (i.e. enforce_owner); anonymous sessions get rejected. Only the granter may REVOKE.
• Tests (all green): grant_allows_delegated_write, revoke_removes_grant (found + fixed an inverted keep/remove filter), grant_scope_prefix, grant_requires_identity, grant_survives_reopen, plus Grant JSON roundtrip / null-scope / perms-parse / scope-wildcard unit tests. Full ./check quick = 115 tests pass, clippy -D warnings clean.
• Commit note: Task 6b closes the "PDF flags 5-19 unbuilt" gap. The delegated-grant model is now real and durable.

Task 7: Concurrency stress + multi-tenant benchmark

• Files: tools/stress.sh (add --tenants N driving N cubec clients); cube-bench (add multi_tenant_put section).
• Run: STRESS_SECONDS=150 TENANTS=4 ./check stress; assert per-tenant isolation holds and aggregate throughput > single-tenant baseline (parallelism should raise total pairs/s).
• Produce docs/concurrency-upgrade-2026MMMDD.md with before/after numbers (single global lock vs per-tenant parallel) — the honest delta, including any writer-starvation cost.
• Final ./check (fmt+test+clippy+stress+mount+bench) ALL GREEN. Commit.

---

Files likely to change

• cubesys/src/tenant.rs (NEW)
• cubesys/src/lib.rs (mod decl)
• cubesys/src/bin/cube-server.rs (registry wiring, HELLO)
• cubesys/src/net.rs (frame for HELLO)
• cubesys/src/commands.rs (BEGIN/COMMIT/ROLLBACK opcodes)
• cubesys/src/store.rs (WalOp::Txn, read gate)
• tools/stress.sh (multi-tenant mode)
• cube-bench/src/main.rs (multi_tenant section)
• docs/concurrency-upgrade-*.md (NEW report)

Tests / validation (every task)

• cargo test -p cubesys per task.
• Full gate: ./check (fmt+test+clippy), ./check stress, ./check mount, ./check bench — ALL CHECKS PASSED before "done".
• New dedicated tests: tenant isolation, txn atomicity/rollback/isolation, auth reject, reader-parallelism.

Risks / tradeoffs / open questions

• Writer serialization per tenant remains (correctness first). Full lock-free per-key MVCC is out of scope — we gain cross-tenant parallelism + read parallelism, not single-tenant write parallelism. Honest ceiling; revisit only if a tenant becomes write-hot.
• HashBackend clone cost for snapshots: if snapshot isolation needs a deep clone, a 65k-record store clone per BEGIN could be pricey. Mitigation: version-counter snapshot (copy-on-write only on write) instead of full clone — decide in Task 5 based on a benchmark.
• Token storage is local-file, not a KMS. Fine for the "hardware you own" privacy model; note it is not multi-host.
• Open question for user: auto-provision unknown tenants (default, convenient) vs deny-unknown (strict, must pre-create token). Plan defaults to provision; flag --deny-unknown-tenant flips it.
• No new external deps unless parking_lot is proven necessary (writer starvation under std RwLock). YAGNI until measured.

---

Auth layer — recommendations (2026-08-11)

What is now real (grounded in the PDF's own model):

1. Tenant isolation at the storage layer — the PDF's C-axis environment selector is the tenant boundary; cross-tenant access is impossible by construction (separate Arc + separate C namespace + separate on-disk WAL/checkpoint).
2. Owner-marking enforcement — every CubeHeader carries owner_local_user/owner_remote_user; mutating ops now require owner match (or first-writer-claims on unowned records) or a grant.
3. Delegated grants (PDF flags 5-19) — Grant{ granter, grantee, perms:rwx, scope:Option<Czyx> } stored as a durable, checkpoint-safe record; enforcement order owner → grant → deny. This is the "permissions and associations" layer the PDF reserved but never specified.
4. Identity from HELLO (tenant + owner pair) — no external token service; consistent with the user's "metadata is reserved coordinate space, not a side table" stance and the privacy mandate (local hardware, no datacenter).

Gaps / recommendations (honest ceiling — ranked):

• R1 — Reuse, don't fork, the ACL logic. The FUSE mount (cubefs::nullspace::Acl::permits) implements owner/group/other checks, but its identity is POSIX uid/gid, incompatible with the daemon's name-based HELLO identity (Owner{local,remote} matching CubeHeader.owner_local_user). A literal Acl lift would be a square peg. Resolution (2026-08-11): the daemon's read gate (R5) is implemented owner/grant-native — same admit_* model as writes — rather than importing Acl. This closes the actual read-privacy gap without forking the uid/gid concept. If POSIX mode-bit semantics are ever wanted in the daemon, the right move is to add mode to CubeHeader and a name-keyed permits() in grants.rs, not to lift FUSE Acl. Status: R5 DONE (commit 0e579a0); R1 reframed as "reuse the enforcement concept, not the struct."

• R2 — Grant table is single-record-per-tenant (O(n) scan). read_bucket deserializes the whole JSON array on every grant/revoke/grant_allows call. Fine for an agent memory store (tens of grants), but recommend a prefix-indexed bucket (one coord per grantee or per scope) before scaling past ~1k grants/tenant.

• R3 — No expiry / revocation propagation. Grants have a seq but no TTL; a revoked grant does not proactively invalidate in-flight transactions. Add expires_at: Option<u64> to Grant and check it in grant_allows if time-bound delegation is wanted.

• R4 — Identity is asserted, not authenticated. HELLO carries a self-asserted (tenant, owner) with no signature/challenge. For a single-owner box on a private network (the user's model) this is acceptable; if the box is ever exposed beyond loopback/LAN, add a mutual-auth step (e.g. a pre-shared key or the existing cubecrypt header-flag encrypted handshake) before stamping identity. The cubecrypt crate already has HEADER_FLAG_ENCRYPTED at bit 12 — wire it into the HELLO frame.

• R5 — Read-gating now enforced. admit_read(coord) mirrors admit_mutate (owner → read-grant → deny; unowned world-readable), wired into run (read+execute) and stat (metadata read), gated by enforce_owner exactly like writes. Tests: read_gate_blocks_non_owner_and_allows_read_grant, read_gate_requires_identity_under_enforce. DONE 2026-08-11 (commit 0e579a0). Note ls lists a directory via cubefs::CubeFs::readdir and is NOT per-entry gated yet — acceptable since ls reveals only names within a tenant the caller already reached; flag if directory-level privacy is needed.

• R6 — Audit log. Every grant/revoke/write is in the WAL but not attributable to who at the app layer. Add a grant_audit null-space record (append-only) listing (seq, granter, grantee, perms, scope, ts) so delegation is auditable — important the moment more than one human/agent touches a tenant.

• R7 — Daemon flag hygiene. --require-identity (i.e. !allow_anonymous) is the only switch; there is no --read-only or per-tenant policy yet. Recommend a small tenant policy table (allow-anon? require-grant-for-read? max-grants?) before multi-user rollout.

These are recommendations, not blockers — the built layer satisfies the PDF's flags 5-19 intent and the user's privacy model. R1 and R5 are the two I'd close first if this store ever holds more than one principal's data.