Commit Graph
73 Commits
Author SHA1 Message Date
admin 18f8eb9b4a cubesys/cubecli/cube-notes-mcp: note categories + project threads. notes use doc_type 'note:<category>' for categories and associate to a 'project' record (linked_records) for project threads; cube note add|list|search accept --project/--cat/--since/--until; new 'cube project list' + 'cube project show <coord>' (walk a project's thread). MCP server exposes note_write/list/search/show + project_list/project_show. 4 notes tests pass; workspace check green. 2026-09-08 05:29:06 -04:00
admin 15dd4c48a4 cube-notes-mcp: stdio MCP server exposing the cubesys::notes CLI (cube note add|list|search|show) as model-facing mcp__cubenotes__note_* tools for the DeepSeek Harness <-> CUBE session message path. 2026-09-08 04:02:21 -04:00
admin a601710d50 cubesys/cubecli: session note/message-save-path (notes) — WordFlags-tagged CZYX note log (doc_type=note, per-session owner + created_at day), store_note/list/search/show + note_command dispatcher, durable CubeStore+WAL+checkpoint; cube note add|list|search|show. New cubesys::notes module w/ 3 tests; workspace check green. 2026-09-08 04:01:06 -04:00
CUBELinux-2 cc896fc336 CUBELinux-2: WordFlags/tri-channel 16-bit flag field, tag-14 + scan_by_word_flag, cubetrace capture→replay→golden→lineage, CUBE VP-tree balling index (cubecode::ballindex), cubefs-visible WordFlags, and cube-mvw (sharded multi-writer MVCC store with per-shard WALs + group-commit + compaction, impl CubeBackend => CubeStore<MvwBackend> DB). Adds show-flags/scan-word-flags/trace-*/golden-capture CLI; ~ tests green. 2026-09-06 21:20:04 -04:00
CUBELinux-2andHermes Agent (upstage/solar-pro4:free) 3f007a1f38 bench(cube-bench): use saturating_sub for expected-value arithmetic
clippy -D warnings flagged implicit_saturating_sub on the c1/c2/c3
expected-count calculations. Replace manual - with saturating_sub so
the lint gate stays clean at any scale.

Co-Authored-By: Hermes Agent (upstage/solar-pro4:free)
2026-08-20 19:03:48 -04:00
CUBELinux-2andHermes Agent (upstage/solar-pro4:free) ae1a62a44e docs: reflect cubecli as standalone crate in README + integration docs
The  binary was extracted to  in a prior session; the
docs still referenced  as its home.

Co-Authored-By: Hermes Agent (upstage/solar-pro4:free)
2026-08-20 18:59:28 -04:00
CUBELinux-2andHermes Agent (upstage/solar-pro4:free) c090233056 bench(cube-bench): fix scan_prefix expected-value math for scale > 65536
coord_for spreads records across C=0,1,2,3 as i grows past 65536,
but the bench assumed everything beyond c=0 landed in c=1. At
scale=200k this asserted (got 65536, expected 134464). Fix: compute
per-bucket expected counts from the coord_for mapping and extend the
total-coverage assertion to include c=2 and c=3.

Verified: fmt clean, clippy -D clean, 100k + 200k both pass.

Co-Authored-By: Hermes Agent (upstage/solar-pro4:free)
2026-08-20 18:56:20 -04:00
CUBELinux-2andHermes Agent 7b0cf6fda5 chore(this session): land all 4 tasks — fixes, cube-bench run, cubecli extract, check gate passed
1. Fixes applied this session:
   - cubesys/src/lib.rs:136 — `store_code_cell` annotated with
     `#[allow(clippy::too_many_arguments)]` (8 args is inherent to the
     record-codec bridge; clippy -D warnings gate).
   - cubesys/src/store.rs:699 — `put_code_cell` annotated identically.
   - cubesys/src/commands.rs:819 — `cell.links().iter().copied().collect()`
     replaced with `cell.links().to_vec()`.
   - cubesys/src/commands.rs:991 — `while let Some(tok) = it.next()` loop
     rewritten as direct `for tok in it`.
   - cubesys/src/commands.rs:1187 — ls handler uses `parse_path` (not
     `path_to_czyx`) for directory ACL gate; derived prefix coordinate
     matches NullSpace acl_bucket convention.

2. cube-bench executed: ./check bench stage passes with correctness
   assertions on every measurement path (100k puts, 100k gets, crypto
   benchmarks, VM dispatch, session commands, concurrent RwLock stress).

3. cubecli extracted: standalone crate at cubecli/ with its own
   Cargo.toml + src/main.rs; cube binary unchanged (same CLI surface,
   demo, commands); cubesys/Cargo.toml no longer declares the `cube`
   bin; workspace members updated.

4. ./check gate: fmt + tests + clippy -D warnings all green;
   `./check bench` also passes.

Verification: ./check (full gate) + ./check bench both PASS.

Co-Authored-By: Hermes Agent
2026-08-20 18:08:19 -04:00
CUBELinux-2andHermes Agent ce8ac78a24 extract(cubecli): move cube CLI out of cubesys into standalone crate
The `cube` binary (REPL + script runner over one shared CubeStore) is
now its own crate at `cubecli/`, matching the spec's package outline
that lists cubecli as a separate crate from the composition layer.

What changed:
- New `cubecli/` crate with `Cargo.toml` (deps: cubecoords, cubestore,
  cubefs, cubecode, cubecrypt, cubesys) and `src/main.rs` (copy of
  the former `cubesys/src/bin/cube.rs`).
- `cubesys/Cargo.toml`: removed the `[[bin]]` entry for `cube`; keeps
  `cube-demo`, `cube-server`, and `cubec` as before.
- `Cargo.toml` (workspace): added `cubecli` to members.
- `cubesys/src/bin/cube.rs` removed (code now lives in cubecli).

The `cube` binary is unchanged: same CLI surface, same demo, same
commands. The `cube-demo` binary in cubesys still wraps
`cubesys::demo::run()` and is unaffected.

Verification: `./check` gate passes (fmt + tests + clippy -D warnings);
`cube --help` and `cube demo` both work.

Co-Authored-By: Hermes Agent
2026-08-20 18:05:56 -04:00
CUBELinux-2andHermes Agent fdfcd2c728 chore(cubesys): resolve 4 clippy -D warnings blocking the check gate
* commands.rs:819 — `cell.links().iter().copied().collect::<Vec<_>>()`
  replaced with `cell.links().to_vec()` (clippy: "calling to_vec() is
  both faster and more readable").

* commands.rs:991 — `while let Some(tok) = it.next()` loop over metatag
  tokens rewritten as a direct `for tok in it` (clippy: "this loop could
  be written as a for loop").

* store.rs:699 — `put_code_cell` (8 args) annotated with
  `#[allow(clippy::too_many_arguments)]` (argument count is inherent to
  the code-cell put API; splitting would add indirection without benefit).

* lib.rs:136 — `store_code_cell` (8 args) annotated identically; this is
  the shared implementation every front-end uses so the count cannot change
  without redesigning the record-codec bridge.

Verification: ./check gate passes (fmt + tests + clippy -D warnings).

Co-Authored-By: Hermes Agent
2026-08-20 17:49:38 -04:00
CUBELinux-2andHermes Agent 90e90ef926 fix(cubeai): add Default impl for CubeAi to satisfy clippy -D warnings
clippy flagged "you should consider adding a `Default` implementation for
`CubeAi`". Add the impl delegating to `new()`.

Also: impl block methods had been accidentally pulled out of `impl CubeAi`
into module scope during a prior edit; rewrite the impl block with all
methods (ingest_trace, classify, suggest, ingest_and_suggest, store)
inside it so it compiles.

Verification: all 5 cubeai tests pass, clippy clean.

Co-Authored-By: Hermes Agent
2026-08-20 17:41:02 -04:00
CUBELinux-2andHermes Agent ec25347af9 fix(cubedbt): add Default impl for CodeCache to satisfy clippy -D warnings
clippy flagged "you should consider adding a `Default` implementation for
`CodeCache`". Add the impl delegating to `new()`.

Also: `patch` method had been accidentally pulled out of `impl CodeCache`
into module scope during a prior edit; move it back inside the impl block
so it compiles.

Verification: all 4 cubedbt tests pass, clippy clean.

Co-Authored-By: Hermes Agent
2026-08-20 17:36:17 -04:00
CUBELinux-2andHermes Agent 6627505e70 fix(cubesys): ls on directory paths no longer rejected by path_to_czyx
The ls handler used path_to_czyx() (which enforces "path must name a
record") to derive the ACL gate coordinate. Directory prefixes like
/c001/z001/y001 are valid read targets but path_to_czyx rejects them as
"names a directory, not a record", so cube-bench's ls assertion panicked.

Fix: use cubefs::path::parse_path (accepts any well-formed prefix) and
derive the directory prefix coordinate (trailing axes zeroed) for the ACL
gate, matching what NullSpace acl_bucket uses for directory ACLs.

Also: clippy -D warnings — remove unnecessary `c as u8` cast in
hash_backend_keys_sorted_order test.

Verification: cube-bench runs to completion (ls section printed OK),
clippy clean.

Co-Authored-By: Hermes Agent
2026-08-20 17:31:54 -04:00
CUBELinux-2andHermes Agent ddca08ea73 fix(cubetrace): wrap Behavior::variant in behavior() helper in test call sites
Behavior is a newtype (Behavior(pub u16)), not an enum, so the 5 test
call sites that passed Behavior::PURE / Behavior::IO_HEAVY / etc. directly
to the ev() helper got type mismatch errors. Add a local behavior() wrapper
and use it at all 5 sites. Also clean up the unused CodeCell import and
extraneous parens that clippy flagged.

fix(cubesys): ls on directory paths no longer rejected by path_to_czyx

The ls handler used path_to_czyx() (which enforces "path must name a
record") to derive the ACL gate coordinate. Directory prefixes like
/c001/z001/y001 are valid read targets but path_to_czyx rejects them as
"names a directory, not a record", so cube-bench's ls assertion panicked.

Fix: use cubefs::path::parse_path (accepts any well-formed prefix) and
derive the directory prefix coordinate (trailing axes zeroed) for the ACL
gate, matching what NullSpace acl_bucket uses for directory ACLs.

Verification: cube-bench runs to completion (ls section printed OK), all
3 cubetrace tests pass, workspace compiles clean.

Co-Authored-By: Hermes Agent
2026-08-20 17:25:48 -04:00
CUBELinux-2andHermes Agent b97efa2a16 scrub: remove cube-store-raw + cube-fuse-proxy, the wrong-architecture duplicates
The workspace already has cubestore (CubeBackend trait + HashBackend
= HashMap<u32,Vec<u8>>), cubefs, and cubesys — the correct foundation
the PDF spec describes. cube-store-raw/ and cube-fuse-proxy/ were
scaffolded as redundant duplicates on the wrong architecture; they
broke `cargo check` (103 errors) and had no tests.

Also: add 7 direct HashBackend trait-level tests (PDF spec's
HashMap<u32,Vec<u8>> sketch) + clean up the _assert_backend_assoc
workaround + dead HashMap import in record_codec, both now unneeded
since HashBackend is exercised by real tests.

Verification: cargo test -p cubestore = 15/15 pass; cargo check
--workspace green (only pre-existing cubetrace warnings).

Co-Authored-By: Hermes Agent
2026-08-20 16:40:53 -04:00
CUBELinux-2 11e5ed7fca fix(metatag-dirs): exact-leaf + descendant path prefix semantics
scan_by_path_prefix now matches BOTH an exact leaf (p == dir) and any
descendant (p.starts_with(dir + '/')), so  returns
 (the leaf itself) as well as  (a child).
Previously a bare dir normalized to 'dir/' and rejected exact leaves.

Also fixes the unit test to assert the corrected semantics (exact leaf
answers its own parent query).

Falls out of wiring the OS migration scripts: their companion index
records (c200/z090 GNS band) carry path=/cubelinux/os/... metatags, and
the daemon  reconstructs the whole migrated tree from them.
2026-08-13 18:41:22 -04:00
CUBELinux-2 94681bbdc0 feat(metatag-dirs): nested directory hierarchy via path metatag + flags
Implements the source PDF's prescribed model for directory structure:
'treat the CZYX cube as the only persistent store, with system calls that
operate on CZYX records and Null-space flags rather than paths and inodes.'
Nested dirs are RECONSTRUCTED from a per-record `path` metatag, so the
filesystem does not need recursive inode trees.

- cubecoords: add CubeHeader.path (Option<String>) + HAS_PATH flag bit (1<<8);
  refresh_flags() sets it. Cubestore TLV codec now serializes/deserializes the
  path as tag 13 (persists across checkpoint/reopen).
- cubestore: scan_by_path (exact), scan_by_path_prefix (dir listing by path
  prefix; bare '/etc' normalized to '/etc/'), plus query_by_path/_prefix
  returning decoded (coord,header,body).
- cubesys: `put` verb gains path=<p>;
  new `query-path <dir>` verb reconstructs a directory listing from metatags.
- ConcurrentStore: query_by_path(_prefix) delegate to inner CubeStore.

Verified live in VM: migrated /etc/passwd, /etc/network/interfaces, /usr/bin/ls
at unrelated coordinates; 'query-path /etc' returns the two /etc files,
'/usr/bin/ls' correctly excluded. cubestore 8/8 tests pass.

Refs: OS-in-CUBE migration, query-layer prototype.
2026-08-13 18:26:58 -04:00
CUBELinux-2 2ff1dff02b feat(query): PDF Package-2 associative log lookup via flags/metatags
Implements the source PDF's 'put(C,Z,Y,X,bytes,flags)' + 'scan_by_flag'
associative-storage API so records are discoverable by WHAT they are
(event type / doc_type metatag) rather than WHERE they live (coordinate
path) — the 'log lookup by query' the OS layers want.

cubestore:
  - CubeStore::scan_by_flag(u16) / scan_by_type(&str): predicate scans
    over decoded CubeHeader (not prefix scans).
  - CubeStore::query_by_flag / query_by_type: same, but also return the
    decoded (label, header, body) so a query tool can present results.
  - get_record's raw/un-enveloped fallback now refresh_flags() so
    synthesized headers carry SIZE_BYTES and are visible to flag queries.
  - regression tests: scan_by_flag_finds_typed_records,
    scan_by_type_event_lookup.

cubesys (daemon):
  - new 'put' verb: enveloped write with optional doc_type=/title=
    metatags (the PDF's put API). rawput remains for bulk/append payloads.
  - ConcurrentStore::query_doc_type was already the backing predicate the
    'query <doc_type>' verb uses; it now matches these typed records.

Proven end-to-end in the VM: 'cubec query klog' returns the kernel-log
index record (doc_type=klog) published by cube-os-klog.sh, and
'cubec query fn' returns the 8 pre-existing typed program records.
2026-08-13 18:12:13 -04:00
CUBELinux-2 fe64b869e4 feat(trace+ai): add cubetrace package and wire DBT/AI/CUBEsys command + module edits (green-lit WIP)
Brings in the cubetrace crate (PDF Package 4, §547): wraps a DBI engine via an
FFI seam and streams trace events (basic blocks, syscalls) into cubestore,
tagging each with CZYX coordinates and header flags.

- Cargo.toml: register cubetrace workspace member
- cubeai/src/lib.rs, cubedbt/src/lib.rs: DBT/AI rule + trace integration edits
- cubecode/src/{cb,cell}.rs: code-cell bytecode/module plumbing for traces
- cubesys/src/commands.rs: trace/AI command surface expansion
- cubetrace/: new crate (builds; 2 non-fatal warnings)

All layers of the OS-in-CUBE migration pass; recorded per user green-light.
2026-08-13 17:31:40 -04:00
CUBELinux-2 ff2914a105 fix(cubestore): surface raw/un-enveloped records in get_record instead of returning None
get_record() previously returned None for any payload lacking a TLV envelope
(4-byte header-len | header | body). Records written through the raw path
(put_raw / the daemon's 'rawput' verb, used by every OS-layer service) carry
no envelope, so callers such as cubefs getattr/read and the 'stat' verb mapped
that None to size 0 / empty file: real bytes became SILENTLY INVISIBLE through
the filesystem while rawget still returned them.

This broke the OS-in-CUBE migration (2026-08-13): every rawput OS record listed
as a 0-byte file in /cubefs, which also surface-invalidated the 'the OS boots
from cube' resume path.

Now a payload that is not a well-formed envelope is surfaced as a raw body under
a synthesized header whose size_bytes reports the true length. Enveloped records
still decode their real header (no behavior change on the record path). Adds two
regression tests: get_record_surfaces_raw_unenveloped_payloads and
get_record_still_prefers_the_real_envelope (cargo test -p cubestore: 5/5).
2026-08-13 17:29:50 -04:00
CUBELinux-2 3121459138 feat(dbt+ai): build out cubedbt and cubeai packages (Package 4, PDF §549/§551)
cubedbt: DBT runtime that reads CZYX-stored translation rules and patches
them into a code cache to execute mimicked behavior. TranslationRule (CZYX
Variant record in c220 band) maps an op-class to a replacement bytecode
fragment; CodeCache patches an original CodeCell under eligible rules and
writes the result as a Variant (preserving behavior descriptors), and
DbRuntime.mimic() fetches -> patches -> runs in the real Vm, proving
're-run or modify behavior without the original binary'.

cubeai: models over cube-stored traces/graphs to classify blocks
(Computation/Branch/CallTrampoline/IoSection/Sequence from the op-class
histogram + behavior descriptors) and suggest new code sequences as
cubedbt TranslationRules (persisted into the c220 rule band, discoverable
by DbRuntime). End-to-end: trace -> classify -> suggest -> mimic.

Both crates are dependency-free and operate on the real CubeStore/CodeCell/
Vm/Behavior types, so they are exercisable today on HashBackend and slot
into ConcurrentStore later without an API change. ./check quick green:
cubedbt 4 tests, cubeai 5 tests, full workspace green.
2026-08-13 16:23:00 -04:00
CUBELinux-2 0074f3112c fix(os): cover full spec behavior-descriptor set (PDF §524-525)
cb.rs previously implemented only 3 of the 6 spec descriptors (pure/io/hot).
PDF §524-525 lists: pure function, I/O heavy, allocates memory, touches
network, hot path, security sensitive.

- Behavior now stores descriptor bits directly as dedicated header-flag bits
  in the spare 8..=15 range: PURE=9, IO_HEAVY=10, ALLOCATES=11, NETWORK=13,
  HOT_PATH=14, SECURITY_SENSITIVE=15. Bit 12 (ENCRYPTED) left clear; mask
  0xEE00. refresh_flags() preserves 8..=15 so round-trip is safe.
- K= CLI tokens gain alloc/net/sec (prog + kernel verbs).
- Add unit test for all-six round-trip + explicit security-sensitive bit.

Proven green via ./check quick.
2026-08-13 16:00:34 -04:00
CUBELinux-2 1539ecd1bb feat(os): OS operator kernels in CUBE + thin native effector (Steps 1 & 2)
- cubecode/src/cb.rs: Behavior descriptor bitflag (PURE/IO_HEAVY/HOT_PATH)
  encoded into HeaderFlags bits 13..15; C_OS_KERNEL=210 / C_OS_EFFECT=211
  coordinate bands; round-trip unit test.
- fix: HEADER_FLAG_BEHAVIOR mask was 0x7000 (bits 12-14) which grabbed the
  ENCRYPTED bit (12) and dropped HOT_PATH (bit 15). Corrected to 0xE000.
- store_code_cell/put_code_cell/header_for_code gain optional descriptor arg;
  all 12 prior call sites pass None (total change).
- CLI: prog K=<flag> tokens, kernel verb (links+descriptors), tick verb lays
  down the OS kernel call-graph (cfg->decide->summarize->tick), native-apply
  verb = thin effector (runs kernel in VM, reads computed result, emits effect).
- integration test os_kernels_live_in_cube_with_call_graph_and_descriptors.

Proven green via ./check quick (fmt+clippy -D warnings+tests).
2026-08-13 15:57:10 -04:00
CUBELinux-2 f6bd8bd01f cubesys: durability fixes A/B/C verified live on VM
- Fix A (store.rs): checkpoint boundary persists wal.committed_seq (highest
  fsync'd) instead of wal.seq() (next-to-assign), which skipped all WAL
  entries since last checkpoint -> silent data loss on reboot.
- Fix B (commands.rs open): run decrypted program in isolated read_snapshot()
  clone instead of put_raw plaintext over sealed envelope (stopped reboot-time
  EnvelopeTooShort / clobber).
- Fix C (commands.rs keyinit): flush OS key cells to WAL via log_put so they
  fold into base snapshot and survive reboot (keyinit #2 issues 0, not 2);
  previously re-minted random material each boot -> sealed records unopenable.
- Regression guards durable_sealed_record_survives_restart +
  open_does_not_clobber_sealed_record in cubesys/src/commands.rs.
- STARTUP-README: replace stale 'EPHEMERAL across restarts' caveat with the
  fixed/verified durability note.
Verified live: systemctl restart cube-server (VM reboot path) -> sealed record
decrypts+executes after reboot; key cell byte-identical; keyinit idempotent.
2026-08-13 12:31:34 -04:00
hermes ab40409316 cubefs: add .czyx.C.Z.Y.X FUSE magic-prefix for Phase-3 'open by CZYX' (same inode as canonical path) 2026-08-13 10:07:43 -04:00
hermes 868883ba1e docs: record OS operational-snapshot binding (real OS data on cube, survives power cycle) 2026-08-13 09:53:51 -04:00
hermes 0343ea6bcb docs: mark boot-substrate bind DONE (systemd units, survives power cycle); update open items 2026-08-13 09:43:44 -04:00
hermes b7467754f5 docs: mark RESUME NEXT STEPS #1/#2 done; VM durable stack verified 2026-08-13 09:32:54 -04:00
hermes f264527365 cubefs/phase3: expose 'open'/'seal' as first-class CLI commands; document startup verification points
- cubesys/src/bin/cube.rs: route known command words (prog/write/run/ls/stat/
  seal/open/query/begin/commit/rollback/stats/audit) straight to Session::exec
  from argv, making the coordinate-addressed 'open <path> <K.Z.Y.X> <tf>' surface
  (Phase 3 'open by CZYX + flags') a real CLI command, not REPL/script-only.
  Verified: ./check green; cube open/cube bogus both behave; dispatch reaches
  crypto/run layer over in-process and durable daemon (cubec) paths.
- STARTUP-README.md: add verification points per standing directive; mark
  Phase 2 FS + durability VERIFIED, Phase 3 surface DONE, boot-substrate IN PROGRESS.
- Guest binaries synced to host HEAD f40b448 + this change; OS state now persists
  on the cube store in the VM (/cubefs/c200/...), durable across daemon restart.
2026-08-13 09:32:45 -04:00
luulu f40b44823d cubefs: surface daemon put failures as FUSE EIO via put_checked
Keep CubeBackend::put returning () for source compat across cubesys/cubecrypt/
cube-bench; add put_checked() (default Ok(())) letting DaemonBackend report a
real socket error. FUSE create/write/truncate now call put_record_checked and
map the failure to FsError::WriteFailed -> errno 5 (EIO).

Verified: ./check (fmt+test+clippy) green; canonical ./check mount 57/57.
2026-08-13 07:50:05 -04:00
CUBELinux-2 314df8e42d docs: finalize RESUME — mount gate hardened (daemon-backed leg), VM deployment bugs fixed, snapshot taken 2026-08-13 07:08:14 -04:00
CUBELinux-2 da3aa3a07a test: ./check mount now also exercises the daemon-backed --socket FUSE path
The mount stage previously only tested the in-memory --seed mount, so the
daemon-backed write path (DaemonBackend) was never regression-guarded -- which
is exactly how a VM shipping an old cube-server (lacking the raw* commands)
shipped undetected. The mount stage now runs the full regression suite against
BOTH a --seed mount and a --socket mount (with its own fresh daemon), and adds
a durability-across-restart assertion proving the FUSE view is a real durable
store. run_fs_tests is now backend-agnostic (self-seeds its fixture).
2026-08-13 07:07:43 -04:00
CUBELinux-2 28e5bcc091 docs: finalize session status — gates green (check/daemon/mount), VM durable e2e + launcher verified 2026-08-13 06:35:01 -04:00
CUBELinux-2 0e3f18d106 docs: mark VM durable-socket e2e done (NEXT STEP 1) in RESUME doc
VM qcow2 was pre-fix; rebuilt cubefs-mount from fixed source inside the VM,
rewired cubefs.service to --socket /run/cube/cube.sock with
Requires=cube-server.service, and proved a FUSE write survives a daemon
restart via the durable store+WAL. ./check also green inside the VM.
2026-08-13 06:25:39 -04:00
CUBELinux-2 dd269e2b44 docs: record DaemonBackend envelope round-trip fix + gate status in RESUME doc 2026-08-13 05:58:40 -04:00
CUBELinux-2 0300def300 fix(cubefs): round-trip full record envelope through DaemonBackend + gate hardening
- DaemonBackend::put/get now round-trip the FULL record envelope (header+body)
  verbatim via rawput/rawget instead of stripping to a bare body. Before this,
  a socket-backed mount wrote the bare body but get returned it as if it were a
  record, breaking the FUSE read-back path (get_record could not decode it).
  This diverged from the in-memory backend and silently corrupted reads.
- Cargo fmt --check now passes (was failing; the committed tree was never a
  clean ./check, which is why the durable mount could regress undetected).
- Fix two new clippy lints (manual_is_multiple_of) in cubefs/backend.rs and
  cubesys/commands.rs so the -D warnings gate is green.
- Daemon-backed integration tests (cubefs_daemon_smoke, daemon_backend_smoke)
  now use the real CubeStore<DaemonBackend> record path and are #[ignore]d so
  the default gate (no daemon) stays green, while ./check daemon spins up a live
  cube-server and runs them via --ignored. This makes the durable-socket
  contract an actual enforced test, not a manual ad-hoc script.
2026-08-13 05:58:12 -04:00
CUBELinux-2 8f9e7e0025 fix(cubefs): make FUSE mount a durable view of cube-server daemon store
The cubefs-mount --socket path was never actually built: CubeFs<B> is
generic, so the --socket (DaemonBackend) and default (HashBackend) arms
of the match were incompatible types (E0308), and --features mount failed
to compile. The running binary was therefore the in-memory build, so
--socket was silently ignored and every FUSE write went to RAM and never
reached the daemon (rawget/rawkeys returned none / 0 keys, WAL stayed 0).

Fix: type-erase the backend. Add  (forwards to inner) in cubestore, and build
 in cubefs-mount via
Box::new(DaemonBackend::new(p)) / Box::new(HashBackend::new()). Keeps
cubefs free of a cubesys dep (acyclic graph).

Verified end-to-end on host (shared code path as the VM): a FUSE write
via  lands in the daemon store (rawget returns the
record, 5 keys present, WAL grows), and survives a  of the
daemon + relaunch with the same --store (byte-identical read-back).

Also includes (from RESUME-cubefs-daemon.md): cubesys raw* command family
(rawget/rawput/rawdel/rawkeys/rawscan + parse/hex helpers) and the
DaemonBackend client + smoke tests. Report/verification docs added.

Note: VM cubefs.service still mounts in-memory (no --socket); update the
unit to  as a
follow-up so the deployed VM FUSE is durable too.
2026-08-13 05:40:59 -04:00
CUBELinux-2 ad73f42e46 fix(audit): eliminate O(n)/unbounded audit-path bottlenecks (100% ok under load)
Root cause of the ~4% error rate in audit-enabled runs (run-qc6newt3:
96.30% ok, op6 mean 367ms max 3003ms) was the audit path's three
compounding costs, isolated iteratively under the real 8-user x 150s
model-B-with-audit stress harness:

  1. append(): rewrote the whole log string on every op (O(n) read-modify-write
     under a per-store Mutex) -> op latency grew with log size.
  2. dump(): walked 1..=count re-reading every entry record (O(n)) -> became the
     new bottleneck once append was fixed (op6 still ~760-980ms).
  3. the  command returned the UNBOUNDED full log (~1MB at 12k entries)
     on every call -> ~1MB response serialized/sent/received = op6 ~978ms.

Fix (aligned with the PDF's 'access logs live in Null rows' time/stream-keyed
model):
  - AUDIT_HEAD stores only a decimal entry count (index); each entry is its own
    durable record at entry_coord(seq) -> append is O(1) (two put_record calls).
  - ConcurrentStore gains a per-store in-memory tail cache (audit_tail) shared by
    every Audit over that store; append extends it by one line, dump returns a
    clone -> dump is O(1) and never re-walks the store. Serialized under the
    cache guard so concurrent connections interleave correctly.
  - the interactive  command serves a bounded recent tail
    (Audit::AUDIT_TAIL_LIMIT = 200) instead of the full log; the full log stays
    available via Session::audit_dump()/Audit::dump() for export.

Verification (real, not assumed):
  - ./check gate GREEN (fmt + tests + clippy -D warnings), incl. R6 append/dump
    tests and pre-existing grant_and_revoke_emit_audit_entries.
  - hermes_verify_audit_o1: index=count (not log), distinct coords, ascending
    dump, concurrent interleave-correct. PASS.
  - model-B-with-audit re-run (8 users x 150s): 110,647 ops, 100.00% ok, 0
    failures; op6 mean 11.9ms (p99 46.9ms, max 124.9ms) vs 367ms pre-fix. Final
    run evidence: /root/cube-stress/run-kkaogy3b.
  - Auth confirmed a non-factor (zero rejections) across all runs.

docs/stress-comparison-20260811.md: corrected the bogus '~0.3ms audit op' claim
in S5 and replaced the placeholder S6 with the full root-cause/fix/verification
write-up including the iteration-to-100% table.
2026-08-11 21:31:42 -04:00
CUBELinux-2 15ce36d488 docs: auth-model A/B comparison — error rate is op-mix, not auth model
Two harness drivers (model A per-command auth, model B persistent auth-once)
run across the real release cube-server to settle the "auth-each-time had ~0%
errors" memory. Conclusion: error rate is driven by the slow `audit` op /
3s socket cap, NOT the auth model — with audit removed, model A hits 94.81%
and model B 100%. Adds docs/stress-comparison-20260811.md §5 and the two
reusable harness scripts under tools/.
2026-08-11 19:09:44 -04:00
CUBELinux-2 4ab938fde8 tools/cubec: fix CLI so ./check stress works; separate client flags from command
cubec forwarded its OWN leading options (--socket/--tenant/...) verbatim
into the command payload sent to the daemon, so 'cubec --socket SOCK
"prog ..."' made the server reject '--socket' as an unknown command.
stress.sh therefore produced 'error: unknown command: --socket' on every
sample and measured nothing.

Split cubec arg parsing into client-option vs command-payload so flags are
consumed locally and only the command reaches the daemon. The canonical
'./check stress' stage now drives a real daemon and samples stats.
2026-08-11 18:33:28 -04:00
CUBELinux-2 ec84fbc725 cubesys: coalesce WAL fsync on commit; stop over-reporting durable seq
commit_txn issued an unconditional fsync per COMMIT, so N concurrent
writers serialized behind N disk syncs (p99 hit the 3s socket timeout
under 8 writers). Add Wal::sync_upto: committers queue on an fsync_gate,
the first one flushes the whole accumulated buffer, and waiters that find
committed_seq past their target return with zero I/O. N commits now cost
~1 fsync with the same durability guarantee.

Also fix a durability over-report: flush_pending stamped committed_seq
from the LIVE seq counter, so sequences taken by appenders that had not
yet buffered their bytes were reported durable. Track max_seq alongside
the pending buffer and advance committed_seq only to what was written.

Wire --wal-fsync-ms / --checkpoint-ms in cube-server (previously
hardcoded to defaults, so the documented knob did nothing).

Both regressions are mutation-verified: each test fails when its bug is
reintroduced.
2026-08-11 18:09:29 -04:00
CUBELinux-2 c87fef0514 fix(cubesys): unhang R4 handshake tests + drop unused import
- run_handshake now spawns auth_handshake on its own thread; the old
  code read the CHALLENGE frame before the daemon ever wrote one,
  deadlocking all 5 r4_handshake_tests (>60s hang). Join for the
  daemon verdict.
- remove unused TenantId/TenantIdentity import in
  grant_and_revoke_emit_audit_entries (clippy -D warnings failure).
- verify_hello call already passes all 6 args (psk, &nonce, tenant,
  &owner_local, owner_remote.as_deref(), sig); confirmed against
  cubecrypt::verify_hello signature.

./check: ALL CHECKS PASSED (fmt+tests+clippy -D warnings).
2026-08-11 16:38:38 -04:00
CUBELinux-2 94bddda3dd feat(cubesys/cubecrypt): R4 challenge-response auth + R5/R6 wiring
- cubecrypt/src/auth.rs: HMAC-SHA256 signed HELLO (sign_hello/verify_hello),
  random_nonce_hex entropy source (plan R4)
- cube-server: --auth-key enables CHALLENGE/HELLO handshake; auth_handshake is
  a module-level free fn (run(self) consumes self, so the thread closure can
  only reach the captured psk). Resolves the earlier E0425 compile failure
- cubec.rs client: --auth-key builds a signed HELLO frame
- commands.rs: audit op constants + read-gating hooks wired into admit_read
- audit.rs: per-tenant append-only audit log in a Null-cube range (plan R6)
- clippy -D warnings clean; full ./check gate ALL CHECKS PASSED (61 tests)
2026-08-11 15:59:10 -04:00
hermes 78cd5c0046 chore(cubesys): clear clippy -D warnings so full ./check gate is green
cargo clippy --fix applied style nits (redundant return, ?-operator,
map_or simplify) plus doc-comment list indentation. The full ./check gate
(fmt+test+clippy -D warnings) now passes; R5 read-gate + grant tests
remain green (39 cubesys lib tests).
2026-08-11 15:25:14 -04:00
hermes 8e31c15f69 docs: mark R5 done, reframe R1 in plan (owner/grant-native read gate) 2026-08-11 15:12:30 -04:00
hermes 0e579a02dd feat(cubesys): R5 — read-gating in daemon path (owner/grant-native)
Closes the read-privacy gap flagged in plan R1/R5. Rather than fork the
FUSE uid/gid Acl model (whose identity is POSIX uid, incompatible with the
daemon's name-based HELLO identity), read-gating is done owner/grant-native:

- commands.rs: new admit_read(coord) mirroring admit_mutate (owner ->
  read-grant -> deny; unowned records world-readable). Wired into
  (read+execute) and  (metadata read), gated by enforce_owner exactly
  like writes.
- Tests: read_gate_blocks_non_owner_and_allows_read_grant (bob denied,
  allowed after alice grants read), read_gate_requires_identity_under_enforce
  (anonymous stat rejected under --require-identity).
- Full ./check quick green: 39 cubesys lib tests, clippy -D warnings clean.

Note in plan: Acl lift (R1) is NOT a literal fork; the daemon reuses the
owner/grant enforcement concept, not the POSIX Acl struct.
2026-08-11 15:11:51 -04:00
hermes e77c650e9c feat(cubesys): Task 6b — permission grants (PDF flags 5-19)
Add a delegated-grant auth layer:
- cubesys/src/grants.rs: Grant/Owner/Perm model, GRANT_BUCKET at
  Czyx::new(0,1,0,1), grant/revoke/grant_allows, std-only JSON codec.
- Stored via put_record with a doc_type='grant-table' header so it
  survives checkpoint/restore (raw put_raw was dropped on dump_store).
- commands.rs: GRANT/REVOKE opcodes + admit_mutate() enforcement hook
  (owner -> grant -> deny). GRANT/REVOKE require HELLO identity under
  --require-identity.
- Enforce owner-match contract preserved (legacy/tests stay green).
- 11 new grant tests; full ./check quick = 115 pass, clippy -D clean.
2026-08-11 15:01:35 -04:00
CUBELinux-2 16cbf6c3cb feat(cubesys): Task 6 (B) — require HELLO identity, gate seal/open
Adopted recommendation (B): make owner authority a hard guarantee instead
of the non-breaking opt-in. A mutating op now requires a stamped HELLO
identity; anonymous writes are rejected. seal/open (destructive writes)
are gated the same way, and seal stamps the owner onto the encrypted record.

Design / non-breaking bridge:
- Session gains enforce_owner: bool (default false) so library/REPL/unit
  tests stay permissive — the 26 prior tests + 3 Task-6 tests are unchanged.
- owner_violation() gains require_identity: the (false) path keeps legacy
  behaviour; the (true) path rejects no-identity mutating ops.
- The daemon flips enforce_owner=true on every connection (both HELLO and
  no-HELLO branches), implementing the default --require-identity policy.
- Added --allow-anonymous escape hatch so legacy cubec/stress.sh (which
  send no HELLO) keep working; stress.sh now passes --allow-anonymous.
- Session::set_enforce_owner() accessor so the daemon (separate bin) can
  set the private field.

Verification:
- ./check quick: EXIT=0, fmt+clippy clean, 28 cubesys lib tests (added
  enforce_owner_requires_identity, seal_open_respect_owner).
- Ad-hoc daemon verifier (LE framing) against the rebuilt cube-server:
  anonymous prog/del rejected, HELLO'd owner first-claim + self-overwrite
  allowed, cross-owner overwrite rejected. ALL PASS.

Note: seal's demo key-cell crypto path (KeyCellMissing on the synthetic key
material) is a pre-existing quirk unrelated to this change; the gate fires
before crypto, so the test verifies the gate, not the crypto.
2026-08-11 14:12:42 -04:00
CUBELinux-2 abc1b56d24 feat(cubesys): Task 6 — owner enforcement on mutating commands
Stamp owner_local_user on records written via prog/write and gate the
mutating paths (prog, write, del — both live and buffered txn) so a
session may only create or overwrite a record whose owner_local_user
matches its HELLO-declared identity.

Design (logical + expedient for the whole project):
- Owner is the durable record-level CubeHeader.owner_local_user field,
  so enforcement is replay-safe and works across daemon restart.
- Enforcement is opt-in/non-breaking: gated only when the session has a
  stamped identity AND the record has an owner. First write by an owner
  claims an unowned coord; a session with no identity (tests, legacy)
  writes freely.
- COMMIT re-checks owner on each buffered op before applying, so a
  concurrent cross-owner commit between BEGIN and COMMIT is rejected
  (txn is restored for retry, not silently dropped).
- seal/open (encrypted raw put/del) left ungated for now: their headers
  are not owner-stamped yet — tracked as follow-up.

Verification:
- ./check quick: EXIT=0, fmt+clippy clean, 26 cubesys tests (added
  owner_enforcement_blocks_cross_owner_overwrite,
  owner_enforcement_allows_first_claim_and_same_owner,
  for_tenant_carries_identity).
- Ad-hoc daemon verifier over real cube-server socket (LE framing):
  cross-owner overwrite + delete rejected, same-owner + first-claim
  allowed, no-HELLO legacy writes allowed. ALL PASS.
2026-08-11 13:41:24 -04:00
CUBELinux-2 c36f64c78d fix(cubesys): make transaction commits durable + replayable
Two correctness bugs found via ad-hoc daemon verification (T5 was
compile-verified only before):

1. decode_wal dropped WalOp::Txn entries on replay: the txn encoder emits
   {"seq","op":"txn","batch"} with NO c/z/y/x fields, but decode_wal
   read c/z/y/x unconditionally -> field_u8("c") returned Err -> the
   whole entry was skipped. Committed transactions silently vanished on
   restart. Fix: branch on op=='txn' before the c/z/y/x extraction.

2. commit was not synchronously durable: append_txn only buffered to the
   WAL pending buffer; fsync happened on the 25ms group thread. A
   clean stop within that window lost the commit. Fix: commit_txn now
   calls wal.flush_pending() (fsync) before returning, so COMMIT is
   durable on return -- a real transaction boundary.

Adds unit test commit_replays_from_wal_without_checkpoint (would have
failed before fix 1). Ad-hoc verifier exercises all 3 changed paths on
the live cube-server socket.
2026-08-11 13:09:24 -04:00