Commit Graph
78 Commits
Author SHA1 Message Date
cubelinux 10cb4e4075 Point the notes MCP at this tree's new home
The repository moved from /home/CUBELinux/CUBELinux-2 to /home/CUBEdb, and the
MCP server hardcoded its own CLI's path. One line, no behaviour change; without it
the server would call a binary that no longer exists.
2026-09-18 17:57:24 -04:00
CUBELinux-2 b47a5df1b2 tools/cube-notes-agent: dedupe findings by ISSUE (not just coord)
_canonical_issue() strips FK/FK#/FA/FB/FLAG/FIX prefixes and normalizes separators so
duplicate representations of one root cause collapse (e.g. FK1-SELINUX_RESTORECON +
SELINUX-RESTORECON -> selinux_restorecon; FK3-WAL_CHECKPOINT +
CONCURRENT-STORE-CHECKPOINT -> checkpoint_before_exit via an alias map). Per issue only
the most-detailed note is reported. mark_covered now covers ALL uncovered finding coords
(including deduped-away duplicates) so nothing re-reports. Corresponding notes re-tagged:
issue logs -> finding, status notes -> impl (no uncategorized notes remain).
2026-09-08 07:46:08 -04:00
CUBELinux-2 d84f87e7bb tools: cube-notes-agent (headless notes<->model review agent) + integration README
Adds the headless, cron-driven note reviewer that uses the cubesys::notes
message-save-path as its memory: it reads only uncovered findings from CUBE, gets
the model to rank them, composes a grounded (verbatim) prioritized report, marks
them covered with a checkpoint note, and writes a review .txt. Documents how the
MCP cube-notes-mcp surface is wired into the harness bundle layer (inject:[tools])
and how prior run data saved in CUBE (action trace + finding/checkpoint notes) is
replayed and traced — surfacing 7 previously un-tagged logs (GIT-SHALLOW-CLONE,
POSTFIX-TLS-CERTS, SELINUX-RESTORECON, DSH-WEB-EADDRINUSE,
CONCURRENT-STORE-CHECKPOINT, MCP-BUNDLE-LAYER, CUBES-TWO-TREES).
2026-09-08 07:27:10 -04:00
CUBELinux-2 bd3e3c0512 cubesys/notes: note tag <coord> <cat> (in-place recategorize); close out the notes -> model CUBE integration + error-tracing from prior run data saved in CUBE
note tag
- New `cube note tag <coord> <cat>` subcommand: reclassify an existing note in place by
  setting its doc_type to `note:<cat>` and put_record under the same coord. Used to move
  mis-tagged notes into the right category (e.g. a finding accidentally logged as a debug
  or action). Puts are WAL+delta+checkpoint durable. Note: `delete_raw` is NOT durable here
  (the checkpoint/delta records Puts of surviving keys only — no tombstones — so a deleted
  key resurrects from the base snapshot on reload), so `tag` is the reliable in-place
  reclassify path. 5 notes tests pass.

Notes -> model CUBE integration (message-save-path)
- cubesys:🎶 WordFlags-tagged CZYX note log (doc_type=note:<category>); project threads
  via linked_records; project resume (<cubecoords> project resume <name> <text>) and context
  injection (<project context <name>>). Durable ConcurrentStore (WAL + delta + checkpoint);
  short-lived processes must checkpoint() before exit.
- CLI: cube note add|list|search|show; cube project list|show|resume|context.
- cube-notes-mcp (stdio MCP server) exposes mcp__cubenotes__note_*/project_* tools to the
  DeepSeek Harness; harness registers the mcp-client at the BUNDLE layer (inject: [tools])
  so the plugin actually mounts (profile-layer rows can't inject the tools service — the
  silent no-mount bug).

How we used it: trace errors from prior run data saved in CUBE
- The headless agent (cube-agent) treats CUBE as a searchable, replayable log: every action
  is written as a note (category=action) so a run can be replayed/fine-tuned; each review run
  reads ONLY uncovered findings (category=finding) and marks them covered with a checkpoint
  note (category=checkpoint) — true incremental, non-destructive review.
- The deterministic, grounded report tool reads finding notes verbatim (no model
  hallucination), assigns severity (HIGH/MEDIUM/LOW) + priority order, and writes a review
  .txt. It surfaced 7 previously un-categorized logs (GIT-SHALLOW-CLONE, POSTFIX-TLS-CERTS,
  SELINUX-RESTORECON, DSH-WEB-EADDRINUSE, CONCURRENT-STORE-CHECKPOINT, MCP-BUNDLE-LAYER,
  CUBES-TWO-TREES) that were logged but never tagged finding.
- Coverage is precise: the report marks exactly the findings it reviewed, so the next run
  reports only new ones. The notes store is also the harness GUI's CUBE surface
  (mcp-cubenotes note_list), so everything the agent writes is visible and reviewable.
2026-09-08 07:22:41 -04:00
admin 8c66478a42 cubesys/cube-notes-mcp: project resume + context. 'cube project resume <name> <text>' sets a resume marker (note category=resume, associated to the project); 'cube project context <name>' emits a compact 'where we are' summary (latest notes + resume marker) for context injection into a new session. MCP tools project_resume/project_context added. 5 notes tests pass. 2026-09-08 05:34:58 -04:00
admin 18f8eb9b4a cubesys/cubecli/cube-notes-mcp: note categories + project threads. notes use doc_type 'note:<category>' for categories and associate to a 'project' record (linked_records) for project threads; cube note add|list|search accept --project/--cat/--since/--until; new 'cube project list' + 'cube project show <coord>' (walk a project's thread). MCP server exposes note_write/list/search/show + project_list/project_show. 4 notes tests pass; workspace check green. 2026-09-08 05:29:06 -04:00
admin 15dd4c48a4 cube-notes-mcp: stdio MCP server exposing the cubesys::notes CLI (cube note add|list|search|show) as model-facing mcp__cubenotes__note_* tools for the DeepSeek Harness <-> CUBE session message path. 2026-09-08 04:02:21 -04:00
admin a601710d50 cubesys/cubecli: session note/message-save-path (notes) — WordFlags-tagged CZYX note log (doc_type=note, per-session owner + created_at day), store_note/list/search/show + note_command dispatcher, durable CubeStore+WAL+checkpoint; cube note add|list|search|show. New cubesys::notes module w/ 3 tests; workspace check green. 2026-09-08 04:01:06 -04:00
CUBELinux-2 cc896fc336 CUBELinux-2: WordFlags/tri-channel 16-bit flag field, tag-14 + scan_by_word_flag, cubetrace capture→replay→golden→lineage, CUBE VP-tree balling index (cubecode::ballindex), cubefs-visible WordFlags, and cube-mvw (sharded multi-writer MVCC store with per-shard WALs + group-commit + compaction, impl CubeBackend => CubeStore<MvwBackend> DB). Adds show-flags/scan-word-flags/trace-*/golden-capture CLI; ~ tests green. 2026-09-06 21:20:04 -04:00
CUBELinux-2andHermes Agent (upstage/solar-pro4:free) 3f007a1f38 bench(cube-bench): use saturating_sub for expected-value arithmetic
clippy -D warnings flagged implicit_saturating_sub on the c1/c2/c3
expected-count calculations. Replace manual - with saturating_sub so
the lint gate stays clean at any scale.

Co-Authored-By: Hermes Agent (upstage/solar-pro4:free)
2026-08-20 19:03:48 -04:00
CUBELinux-2andHermes Agent (upstage/solar-pro4:free) ae1a62a44e docs: reflect cubecli as standalone crate in README + integration docs
The  binary was extracted to  in a prior session; the
docs still referenced  as its home.

Co-Authored-By: Hermes Agent (upstage/solar-pro4:free)
2026-08-20 18:59:28 -04:00
CUBELinux-2andHermes Agent (upstage/solar-pro4:free) c090233056 bench(cube-bench): fix scan_prefix expected-value math for scale > 65536
coord_for spreads records across C=0,1,2,3 as i grows past 65536,
but the bench assumed everything beyond c=0 landed in c=1. At
scale=200k this asserted (got 65536, expected 134464). Fix: compute
per-bucket expected counts from the coord_for mapping and extend the
total-coverage assertion to include c=2 and c=3.

Verified: fmt clean, clippy -D clean, 100k + 200k both pass.

Co-Authored-By: Hermes Agent (upstage/solar-pro4:free)
2026-08-20 18:56:20 -04:00
CUBELinux-2andHermes Agent 7b0cf6fda5 chore(this session): land all 4 tasks — fixes, cube-bench run, cubecli extract, check gate passed
1. Fixes applied this session:
   - cubesys/src/lib.rs:136 — `store_code_cell` annotated with
     `#[allow(clippy::too_many_arguments)]` (8 args is inherent to the
     record-codec bridge; clippy -D warnings gate).
   - cubesys/src/store.rs:699 — `put_code_cell` annotated identically.
   - cubesys/src/commands.rs:819 — `cell.links().iter().copied().collect()`
     replaced with `cell.links().to_vec()`.
   - cubesys/src/commands.rs:991 — `while let Some(tok) = it.next()` loop
     rewritten as direct `for tok in it`.
   - cubesys/src/commands.rs:1187 — ls handler uses `parse_path` (not
     `path_to_czyx`) for directory ACL gate; derived prefix coordinate
     matches NullSpace acl_bucket convention.

2. cube-bench executed: ./check bench stage passes with correctness
   assertions on every measurement path (100k puts, 100k gets, crypto
   benchmarks, VM dispatch, session commands, concurrent RwLock stress).

3. cubecli extracted: standalone crate at cubecli/ with its own
   Cargo.toml + src/main.rs; cube binary unchanged (same CLI surface,
   demo, commands); cubesys/Cargo.toml no longer declares the `cube`
   bin; workspace members updated.

4. ./check gate: fmt + tests + clippy -D warnings all green;
   `./check bench` also passes.

Verification: ./check (full gate) + ./check bench both PASS.

Co-Authored-By: Hermes Agent
2026-08-20 18:08:19 -04:00
CUBELinux-2andHermes Agent ce8ac78a24 extract(cubecli): move cube CLI out of cubesys into standalone crate
The `cube` binary (REPL + script runner over one shared CubeStore) is
now its own crate at `cubecli/`, matching the spec's package outline
that lists cubecli as a separate crate from the composition layer.

What changed:
- New `cubecli/` crate with `Cargo.toml` (deps: cubecoords, cubestore,
  cubefs, cubecode, cubecrypt, cubesys) and `src/main.rs` (copy of
  the former `cubesys/src/bin/cube.rs`).
- `cubesys/Cargo.toml`: removed the `[[bin]]` entry for `cube`; keeps
  `cube-demo`, `cube-server`, and `cubec` as before.
- `Cargo.toml` (workspace): added `cubecli` to members.
- `cubesys/src/bin/cube.rs` removed (code now lives in cubecli).

The `cube` binary is unchanged: same CLI surface, same demo, same
commands. The `cube-demo` binary in cubesys still wraps
`cubesys::demo::run()` and is unaffected.

Verification: `./check` gate passes (fmt + tests + clippy -D warnings);
`cube --help` and `cube demo` both work.

Co-Authored-By: Hermes Agent
2026-08-20 18:05:56 -04:00
CUBELinux-2andHermes Agent fdfcd2c728 chore(cubesys): resolve 4 clippy -D warnings blocking the check gate
* commands.rs:819 — `cell.links().iter().copied().collect::<Vec<_>>()`
  replaced with `cell.links().to_vec()` (clippy: "calling to_vec() is
  both faster and more readable").

* commands.rs:991 — `while let Some(tok) = it.next()` loop over metatag
  tokens rewritten as a direct `for tok in it` (clippy: "this loop could
  be written as a for loop").

* store.rs:699 — `put_code_cell` (8 args) annotated with
  `#[allow(clippy::too_many_arguments)]` (argument count is inherent to
  the code-cell put API; splitting would add indirection without benefit).

* lib.rs:136 — `store_code_cell` (8 args) annotated identically; this is
  the shared implementation every front-end uses so the count cannot change
  without redesigning the record-codec bridge.

Verification: ./check gate passes (fmt + tests + clippy -D warnings).

Co-Authored-By: Hermes Agent
2026-08-20 17:49:38 -04:00
CUBELinux-2andHermes Agent 90e90ef926 fix(cubeai): add Default impl for CubeAi to satisfy clippy -D warnings
clippy flagged "you should consider adding a `Default` implementation for
`CubeAi`". Add the impl delegating to `new()`.

Also: impl block methods had been accidentally pulled out of `impl CubeAi`
into module scope during a prior edit; rewrite the impl block with all
methods (ingest_trace, classify, suggest, ingest_and_suggest, store)
inside it so it compiles.

Verification: all 5 cubeai tests pass, clippy clean.

Co-Authored-By: Hermes Agent
2026-08-20 17:41:02 -04:00
CUBELinux-2andHermes Agent ec25347af9 fix(cubedbt): add Default impl for CodeCache to satisfy clippy -D warnings
clippy flagged "you should consider adding a `Default` implementation for
`CodeCache`". Add the impl delegating to `new()`.

Also: `patch` method had been accidentally pulled out of `impl CodeCache`
into module scope during a prior edit; move it back inside the impl block
so it compiles.

Verification: all 4 cubedbt tests pass, clippy clean.

Co-Authored-By: Hermes Agent
2026-08-20 17:36:17 -04:00
CUBELinux-2andHermes Agent 6627505e70 fix(cubesys): ls on directory paths no longer rejected by path_to_czyx
The ls handler used path_to_czyx() (which enforces "path must name a
record") to derive the ACL gate coordinate. Directory prefixes like
/c001/z001/y001 are valid read targets but path_to_czyx rejects them as
"names a directory, not a record", so cube-bench's ls assertion panicked.

Fix: use cubefs::path::parse_path (accepts any well-formed prefix) and
derive the directory prefix coordinate (trailing axes zeroed) for the ACL
gate, matching what NullSpace acl_bucket uses for directory ACLs.

Also: clippy -D warnings — remove unnecessary `c as u8` cast in
hash_backend_keys_sorted_order test.

Verification: cube-bench runs to completion (ls section printed OK),
clippy clean.

Co-Authored-By: Hermes Agent
2026-08-20 17:31:54 -04:00
CUBELinux-2andHermes Agent ddca08ea73 fix(cubetrace): wrap Behavior::variant in behavior() helper in test call sites
Behavior is a newtype (Behavior(pub u16)), not an enum, so the 5 test
call sites that passed Behavior::PURE / Behavior::IO_HEAVY / etc. directly
to the ev() helper got type mismatch errors. Add a local behavior() wrapper
and use it at all 5 sites. Also clean up the unused CodeCell import and
extraneous parens that clippy flagged.

fix(cubesys): ls on directory paths no longer rejected by path_to_czyx

The ls handler used path_to_czyx() (which enforces "path must name a
record") to derive the ACL gate coordinate. Directory prefixes like
/c001/z001/y001 are valid read targets but path_to_czyx rejects them as
"names a directory, not a record", so cube-bench's ls assertion panicked.

Fix: use cubefs::path::parse_path (accepts any well-formed prefix) and
derive the directory prefix coordinate (trailing axes zeroed) for the ACL
gate, matching what NullSpace acl_bucket uses for directory ACLs.

Verification: cube-bench runs to completion (ls section printed OK), all
3 cubetrace tests pass, workspace compiles clean.

Co-Authored-By: Hermes Agent
2026-08-20 17:25:48 -04:00
CUBELinux-2andHermes Agent b97efa2a16 scrub: remove cube-store-raw + cube-fuse-proxy, the wrong-architecture duplicates
The workspace already has cubestore (CubeBackend trait + HashBackend
= HashMap<u32,Vec<u8>>), cubefs, and cubesys — the correct foundation
the PDF spec describes. cube-store-raw/ and cube-fuse-proxy/ were
scaffolded as redundant duplicates on the wrong architecture; they
broke `cargo check` (103 errors) and had no tests.

Also: add 7 direct HashBackend trait-level tests (PDF spec's
HashMap<u32,Vec<u8>> sketch) + clean up the _assert_backend_assoc
workaround + dead HashMap import in record_codec, both now unneeded
since HashBackend is exercised by real tests.

Verification: cargo test -p cubestore = 15/15 pass; cargo check
--workspace green (only pre-existing cubetrace warnings).

Co-Authored-By: Hermes Agent
2026-08-20 16:40:53 -04:00
CUBELinux-2 11e5ed7fca fix(metatag-dirs): exact-leaf + descendant path prefix semantics
scan_by_path_prefix now matches BOTH an exact leaf (p == dir) and any
descendant (p.starts_with(dir + '/')), so  returns
 (the leaf itself) as well as  (a child).
Previously a bare dir normalized to 'dir/' and rejected exact leaves.

Also fixes the unit test to assert the corrected semantics (exact leaf
answers its own parent query).

Falls out of wiring the OS migration scripts: their companion index
records (c200/z090 GNS band) carry path=/cubelinux/os/... metatags, and
the daemon  reconstructs the whole migrated tree from them.
2026-08-13 18:41:22 -04:00
CUBELinux-2 94681bbdc0 feat(metatag-dirs): nested directory hierarchy via path metatag + flags
Implements the source PDF's prescribed model for directory structure:
'treat the CZYX cube as the only persistent store, with system calls that
operate on CZYX records and Null-space flags rather than paths and inodes.'
Nested dirs are RECONSTRUCTED from a per-record `path` metatag, so the
filesystem does not need recursive inode trees.

- cubecoords: add CubeHeader.path (Option<String>) + HAS_PATH flag bit (1<<8);
  refresh_flags() sets it. Cubestore TLV codec now serializes/deserializes the
  path as tag 13 (persists across checkpoint/reopen).
- cubestore: scan_by_path (exact), scan_by_path_prefix (dir listing by path
  prefix; bare '/etc' normalized to '/etc/'), plus query_by_path/_prefix
  returning decoded (coord,header,body).
- cubesys: `put` verb gains path=<p>;
  new `query-path <dir>` verb reconstructs a directory listing from metatags.
- ConcurrentStore: query_by_path(_prefix) delegate to inner CubeStore.

Verified live in VM: migrated /etc/passwd, /etc/network/interfaces, /usr/bin/ls
at unrelated coordinates; 'query-path /etc' returns the two /etc files,
'/usr/bin/ls' correctly excluded. cubestore 8/8 tests pass.

Refs: OS-in-CUBE migration, query-layer prototype.
2026-08-13 18:26:58 -04:00
CUBELinux-2 2ff1dff02b feat(query): PDF Package-2 associative log lookup via flags/metatags
Implements the source PDF's 'put(C,Z,Y,X,bytes,flags)' + 'scan_by_flag'
associative-storage API so records are discoverable by WHAT they are
(event type / doc_type metatag) rather than WHERE they live (coordinate
path) — the 'log lookup by query' the OS layers want.

cubestore:
  - CubeStore::scan_by_flag(u16) / scan_by_type(&str): predicate scans
    over decoded CubeHeader (not prefix scans).
  - CubeStore::query_by_flag / query_by_type: same, but also return the
    decoded (label, header, body) so a query tool can present results.
  - get_record's raw/un-enveloped fallback now refresh_flags() so
    synthesized headers carry SIZE_BYTES and are visible to flag queries.
  - regression tests: scan_by_flag_finds_typed_records,
    scan_by_type_event_lookup.

cubesys (daemon):
  - new 'put' verb: enveloped write with optional doc_type=/title=
    metatags (the PDF's put API). rawput remains for bulk/append payloads.
  - ConcurrentStore::query_doc_type was already the backing predicate the
    'query <doc_type>' verb uses; it now matches these typed records.

Proven end-to-end in the VM: 'cubec query klog' returns the kernel-log
index record (doc_type=klog) published by cube-os-klog.sh, and
'cubec query fn' returns the 8 pre-existing typed program records.
2026-08-13 18:12:13 -04:00
CUBELinux-2 fe64b869e4 feat(trace+ai): add cubetrace package and wire DBT/AI/CUBEsys command + module edits (green-lit WIP)
Brings in the cubetrace crate (PDF Package 4, §547): wraps a DBI engine via an
FFI seam and streams trace events (basic blocks, syscalls) into cubestore,
tagging each with CZYX coordinates and header flags.

- Cargo.toml: register cubetrace workspace member
- cubeai/src/lib.rs, cubedbt/src/lib.rs: DBT/AI rule + trace integration edits
- cubecode/src/{cb,cell}.rs: code-cell bytecode/module plumbing for traces
- cubesys/src/commands.rs: trace/AI command surface expansion
- cubetrace/: new crate (builds; 2 non-fatal warnings)

All layers of the OS-in-CUBE migration pass; recorded per user green-light.
2026-08-13 17:31:40 -04:00
CUBELinux-2 ff2914a105 fix(cubestore): surface raw/un-enveloped records in get_record instead of returning None
get_record() previously returned None for any payload lacking a TLV envelope
(4-byte header-len | header | body). Records written through the raw path
(put_raw / the daemon's 'rawput' verb, used by every OS-layer service) carry
no envelope, so callers such as cubefs getattr/read and the 'stat' verb mapped
that None to size 0 / empty file: real bytes became SILENTLY INVISIBLE through
the filesystem while rawget still returned them.

This broke the OS-in-CUBE migration (2026-08-13): every rawput OS record listed
as a 0-byte file in /cubefs, which also surface-invalidated the 'the OS boots
from cube' resume path.

Now a payload that is not a well-formed envelope is surfaced as a raw body under
a synthesized header whose size_bytes reports the true length. Enveloped records
still decode their real header (no behavior change on the record path). Adds two
regression tests: get_record_surfaces_raw_unenveloped_payloads and
get_record_still_prefers_the_real_envelope (cargo test -p cubestore: 5/5).
2026-08-13 17:29:50 -04:00
CUBELinux-2 3121459138 feat(dbt+ai): build out cubedbt and cubeai packages (Package 4, PDF §549/§551)
cubedbt: DBT runtime that reads CZYX-stored translation rules and patches
them into a code cache to execute mimicked behavior. TranslationRule (CZYX
Variant record in c220 band) maps an op-class to a replacement bytecode
fragment; CodeCache patches an original CodeCell under eligible rules and
writes the result as a Variant (preserving behavior descriptors), and
DbRuntime.mimic() fetches -> patches -> runs in the real Vm, proving
're-run or modify behavior without the original binary'.

cubeai: models over cube-stored traces/graphs to classify blocks
(Computation/Branch/CallTrampoline/IoSection/Sequence from the op-class
histogram + behavior descriptors) and suggest new code sequences as
cubedbt TranslationRules (persisted into the c220 rule band, discoverable
by DbRuntime). End-to-end: trace -> classify -> suggest -> mimic.

Both crates are dependency-free and operate on the real CubeStore/CodeCell/
Vm/Behavior types, so they are exercisable today on HashBackend and slot
into ConcurrentStore later without an API change. ./check quick green:
cubedbt 4 tests, cubeai 5 tests, full workspace green.
2026-08-13 16:23:00 -04:00
CUBELinux-2 0074f3112c fix(os): cover full spec behavior-descriptor set (PDF §524-525)
cb.rs previously implemented only 3 of the 6 spec descriptors (pure/io/hot).
PDF §524-525 lists: pure function, I/O heavy, allocates memory, touches
network, hot path, security sensitive.

- Behavior now stores descriptor bits directly as dedicated header-flag bits
  in the spare 8..=15 range: PURE=9, IO_HEAVY=10, ALLOCATES=11, NETWORK=13,
  HOT_PATH=14, SECURITY_SENSITIVE=15. Bit 12 (ENCRYPTED) left clear; mask
  0xEE00. refresh_flags() preserves 8..=15 so round-trip is safe.
- K= CLI tokens gain alloc/net/sec (prog + kernel verbs).
- Add unit test for all-six round-trip + explicit security-sensitive bit.

Proven green via ./check quick.
2026-08-13 16:00:34 -04:00
CUBELinux-2 1539ecd1bb feat(os): OS operator kernels in CUBE + thin native effector (Steps 1 & 2)
- cubecode/src/cb.rs: Behavior descriptor bitflag (PURE/IO_HEAVY/HOT_PATH)
  encoded into HeaderFlags bits 13..15; C_OS_KERNEL=210 / C_OS_EFFECT=211
  coordinate bands; round-trip unit test.
- fix: HEADER_FLAG_BEHAVIOR mask was 0x7000 (bits 12-14) which grabbed the
  ENCRYPTED bit (12) and dropped HOT_PATH (bit 15). Corrected to 0xE000.
- store_code_cell/put_code_cell/header_for_code gain optional descriptor arg;
  all 12 prior call sites pass None (total change).
- CLI: prog K=<flag> tokens, kernel verb (links+descriptors), tick verb lays
  down the OS kernel call-graph (cfg->decide->summarize->tick), native-apply
  verb = thin effector (runs kernel in VM, reads computed result, emits effect).
- integration test os_kernels_live_in_cube_with_call_graph_and_descriptors.

Proven green via ./check quick (fmt+clippy -D warnings+tests).
2026-08-13 15:57:10 -04:00
CUBELinux-2 f6bd8bd01f cubesys: durability fixes A/B/C verified live on VM
- Fix A (store.rs): checkpoint boundary persists wal.committed_seq (highest
  fsync'd) instead of wal.seq() (next-to-assign), which skipped all WAL
  entries since last checkpoint -> silent data loss on reboot.
- Fix B (commands.rs open): run decrypted program in isolated read_snapshot()
  clone instead of put_raw plaintext over sealed envelope (stopped reboot-time
  EnvelopeTooShort / clobber).
- Fix C (commands.rs keyinit): flush OS key cells to WAL via log_put so they
  fold into base snapshot and survive reboot (keyinit #2 issues 0, not 2);
  previously re-minted random material each boot -> sealed records unopenable.
- Regression guards durable_sealed_record_survives_restart +
  open_does_not_clobber_sealed_record in cubesys/src/commands.rs.
- STARTUP-README: replace stale 'EPHEMERAL across restarts' caveat with the
  fixed/verified durability note.
Verified live: systemctl restart cube-server (VM reboot path) -> sealed record
decrypts+executes after reboot; key cell byte-identical; keyinit idempotent.
2026-08-13 12:31:34 -04:00
hermes ab40409316 cubefs: add .czyx.C.Z.Y.X FUSE magic-prefix for Phase-3 'open by CZYX' (same inode as canonical path) 2026-08-13 10:07:43 -04:00
hermes 868883ba1e docs: record OS operational-snapshot binding (real OS data on cube, survives power cycle) 2026-08-13 09:53:51 -04:00
hermes 0343ea6bcb docs: mark boot-substrate bind DONE (systemd units, survives power cycle); update open items 2026-08-13 09:43:44 -04:00
hermes b7467754f5 docs: mark RESUME NEXT STEPS #1/#2 done; VM durable stack verified 2026-08-13 09:32:54 -04:00
hermes f264527365 cubefs/phase3: expose 'open'/'seal' as first-class CLI commands; document startup verification points
- cubesys/src/bin/cube.rs: route known command words (prog/write/run/ls/stat/
  seal/open/query/begin/commit/rollback/stats/audit) straight to Session::exec
  from argv, making the coordinate-addressed 'open <path> <K.Z.Y.X> <tf>' surface
  (Phase 3 'open by CZYX + flags') a real CLI command, not REPL/script-only.
  Verified: ./check green; cube open/cube bogus both behave; dispatch reaches
  crypto/run layer over in-process and durable daemon (cubec) paths.
- STARTUP-README.md: add verification points per standing directive; mark
  Phase 2 FS + durability VERIFIED, Phase 3 surface DONE, boot-substrate IN PROGRESS.
- Guest binaries synced to host HEAD f40b448 + this change; OS state now persists
  on the cube store in the VM (/cubefs/c200/...), durable across daemon restart.
2026-08-13 09:32:45 -04:00
luulu f40b44823d cubefs: surface daemon put failures as FUSE EIO via put_checked
Keep CubeBackend::put returning () for source compat across cubesys/cubecrypt/
cube-bench; add put_checked() (default Ok(())) letting DaemonBackend report a
real socket error. FUSE create/write/truncate now call put_record_checked and
map the failure to FsError::WriteFailed -> errno 5 (EIO).

Verified: ./check (fmt+test+clippy) green; canonical ./check mount 57/57.
2026-08-13 07:50:05 -04:00
CUBELinux-2 314df8e42d docs: finalize RESUME — mount gate hardened (daemon-backed leg), VM deployment bugs fixed, snapshot taken 2026-08-13 07:08:14 -04:00
CUBELinux-2 da3aa3a07a test: ./check mount now also exercises the daemon-backed --socket FUSE path
The mount stage previously only tested the in-memory --seed mount, so the
daemon-backed write path (DaemonBackend) was never regression-guarded -- which
is exactly how a VM shipping an old cube-server (lacking the raw* commands)
shipped undetected. The mount stage now runs the full regression suite against
BOTH a --seed mount and a --socket mount (with its own fresh daemon), and adds
a durability-across-restart assertion proving the FUSE view is a real durable
store. run_fs_tests is now backend-agnostic (self-seeds its fixture).
2026-08-13 07:07:43 -04:00
CUBELinux-2 28e5bcc091 docs: finalize session status — gates green (check/daemon/mount), VM durable e2e + launcher verified 2026-08-13 06:35:01 -04:00
CUBELinux-2 0e3f18d106 docs: mark VM durable-socket e2e done (NEXT STEP 1) in RESUME doc
VM qcow2 was pre-fix; rebuilt cubefs-mount from fixed source inside the VM,
rewired cubefs.service to --socket /run/cube/cube.sock with
Requires=cube-server.service, and proved a FUSE write survives a daemon
restart via the durable store+WAL. ./check also green inside the VM.
2026-08-13 06:25:39 -04:00
CUBELinux-2 dd269e2b44 docs: record DaemonBackend envelope round-trip fix + gate status in RESUME doc 2026-08-13 05:58:40 -04:00
CUBELinux-2 0300def300 fix(cubefs): round-trip full record envelope through DaemonBackend + gate hardening
- DaemonBackend::put/get now round-trip the FULL record envelope (header+body)
  verbatim via rawput/rawget instead of stripping to a bare body. Before this,
  a socket-backed mount wrote the bare body but get returned it as if it were a
  record, breaking the FUSE read-back path (get_record could not decode it).
  This diverged from the in-memory backend and silently corrupted reads.
- Cargo fmt --check now passes (was failing; the committed tree was never a
  clean ./check, which is why the durable mount could regress undetected).
- Fix two new clippy lints (manual_is_multiple_of) in cubefs/backend.rs and
  cubesys/commands.rs so the -D warnings gate is green.
- Daemon-backed integration tests (cubefs_daemon_smoke, daemon_backend_smoke)
  now use the real CubeStore<DaemonBackend> record path and are #[ignore]d so
  the default gate (no daemon) stays green, while ./check daemon spins up a live
  cube-server and runs them via --ignored. This makes the durable-socket
  contract an actual enforced test, not a manual ad-hoc script.
2026-08-13 05:58:12 -04:00
CUBELinux-2 8f9e7e0025 fix(cubefs): make FUSE mount a durable view of cube-server daemon store
The cubefs-mount --socket path was never actually built: CubeFs<B> is
generic, so the --socket (DaemonBackend) and default (HashBackend) arms
of the match were incompatible types (E0308), and --features mount failed
to compile. The running binary was therefore the in-memory build, so
--socket was silently ignored and every FUSE write went to RAM and never
reached the daemon (rawget/rawkeys returned none / 0 keys, WAL stayed 0).

Fix: type-erase the backend. Add  (forwards to inner) in cubestore, and build
 in cubefs-mount via
Box::new(DaemonBackend::new(p)) / Box::new(HashBackend::new()). Keeps
cubefs free of a cubesys dep (acyclic graph).

Verified end-to-end on host (shared code path as the VM): a FUSE write
via  lands in the daemon store (rawget returns the
record, 5 keys present, WAL grows), and survives a  of the
daemon + relaunch with the same --store (byte-identical read-back).

Also includes (from RESUME-cubefs-daemon.md): cubesys raw* command family
(rawget/rawput/rawdel/rawkeys/rawscan + parse/hex helpers) and the
DaemonBackend client + smoke tests. Report/verification docs added.

Note: VM cubefs.service still mounts in-memory (no --socket); update the
unit to  as a
follow-up so the deployed VM FUSE is durable too.
2026-08-13 05:40:59 -04:00
CUBELinux-2 ad73f42e46 fix(audit): eliminate O(n)/unbounded audit-path bottlenecks (100% ok under load)
Root cause of the ~4% error rate in audit-enabled runs (run-qc6newt3:
96.30% ok, op6 mean 367ms max 3003ms) was the audit path's three
compounding costs, isolated iteratively under the real 8-user x 150s
model-B-with-audit stress harness:

  1. append(): rewrote the whole log string on every op (O(n) read-modify-write
     under a per-store Mutex) -> op latency grew with log size.
  2. dump(): walked 1..=count re-reading every entry record (O(n)) -> became the
     new bottleneck once append was fixed (op6 still ~760-980ms).
  3. the  command returned the UNBOUNDED full log (~1MB at 12k entries)
     on every call -> ~1MB response serialized/sent/received = op6 ~978ms.

Fix (aligned with the PDF's 'access logs live in Null rows' time/stream-keyed
model):
  - AUDIT_HEAD stores only a decimal entry count (index); each entry is its own
    durable record at entry_coord(seq) -> append is O(1) (two put_record calls).
  - ConcurrentStore gains a per-store in-memory tail cache (audit_tail) shared by
    every Audit over that store; append extends it by one line, dump returns a
    clone -> dump is O(1) and never re-walks the store. Serialized under the
    cache guard so concurrent connections interleave correctly.
  - the interactive  command serves a bounded recent tail
    (Audit::AUDIT_TAIL_LIMIT = 200) instead of the full log; the full log stays
    available via Session::audit_dump()/Audit::dump() for export.

Verification (real, not assumed):
  - ./check gate GREEN (fmt + tests + clippy -D warnings), incl. R6 append/dump
    tests and pre-existing grant_and_revoke_emit_audit_entries.
  - hermes_verify_audit_o1: index=count (not log), distinct coords, ascending
    dump, concurrent interleave-correct. PASS.
  - model-B-with-audit re-run (8 users x 150s): 110,647 ops, 100.00% ok, 0
    failures; op6 mean 11.9ms (p99 46.9ms, max 124.9ms) vs 367ms pre-fix. Final
    run evidence: /root/cube-stress/run-kkaogy3b.
  - Auth confirmed a non-factor (zero rejections) across all runs.

docs/stress-comparison-20260811.md: corrected the bogus '~0.3ms audit op' claim
in S5 and replaced the placeholder S6 with the full root-cause/fix/verification
write-up including the iteration-to-100% table.
2026-08-11 21:31:42 -04:00
CUBELinux-2 15ce36d488 docs: auth-model A/B comparison — error rate is op-mix, not auth model
Two harness drivers (model A per-command auth, model B persistent auth-once)
run across the real release cube-server to settle the "auth-each-time had ~0%
errors" memory. Conclusion: error rate is driven by the slow `audit` op /
3s socket cap, NOT the auth model — with audit removed, model A hits 94.81%
and model B 100%. Adds docs/stress-comparison-20260811.md §5 and the two
reusable harness scripts under tools/.
2026-08-11 19:09:44 -04:00
CUBELinux-2 4ab938fde8 tools/cubec: fix CLI so ./check stress works; separate client flags from command
cubec forwarded its OWN leading options (--socket/--tenant/...) verbatim
into the command payload sent to the daemon, so 'cubec --socket SOCK
"prog ..."' made the server reject '--socket' as an unknown command.
stress.sh therefore produced 'error: unknown command: --socket' on every
sample and measured nothing.

Split cubec arg parsing into client-option vs command-payload so flags are
consumed locally and only the command reaches the daemon. The canonical
'./check stress' stage now drives a real daemon and samples stats.
2026-08-11 18:33:28 -04:00
CUBELinux-2 ec84fbc725 cubesys: coalesce WAL fsync on commit; stop over-reporting durable seq
commit_txn issued an unconditional fsync per COMMIT, so N concurrent
writers serialized behind N disk syncs (p99 hit the 3s socket timeout
under 8 writers). Add Wal::sync_upto: committers queue on an fsync_gate,
the first one flushes the whole accumulated buffer, and waiters that find
committed_seq past their target return with zero I/O. N commits now cost
~1 fsync with the same durability guarantee.

Also fix a durability over-report: flush_pending stamped committed_seq
from the LIVE seq counter, so sequences taken by appenders that had not
yet buffered their bytes were reported durable. Track max_seq alongside
the pending buffer and advance committed_seq only to what was written.

Wire --wal-fsync-ms / --checkpoint-ms in cube-server (previously
hardcoded to defaults, so the documented knob did nothing).

Both regressions are mutation-verified: each test fails when its bug is
reintroduced.
2026-08-11 18:09:29 -04:00
CUBELinux-2 c87fef0514 fix(cubesys): unhang R4 handshake tests + drop unused import
- run_handshake now spawns auth_handshake on its own thread; the old
  code read the CHALLENGE frame before the daemon ever wrote one,
  deadlocking all 5 r4_handshake_tests (>60s hang). Join for the
  daemon verdict.
- remove unused TenantId/TenantIdentity import in
  grant_and_revoke_emit_audit_entries (clippy -D warnings failure).
- verify_hello call already passes all 6 args (psk, &nonce, tenant,
  &owner_local, owner_remote.as_deref(), sig); confirmed against
  cubecrypt::verify_hello signature.

./check: ALL CHECKS PASSED (fmt+tests+clippy -D warnings).
2026-08-11 16:38:38 -04:00
CUBELinux-2 94bddda3dd feat(cubesys/cubecrypt): R4 challenge-response auth + R5/R6 wiring
- cubecrypt/src/auth.rs: HMAC-SHA256 signed HELLO (sign_hello/verify_hello),
  random_nonce_hex entropy source (plan R4)
- cube-server: --auth-key enables CHALLENGE/HELLO handshake; auth_handshake is
  a module-level free fn (run(self) consumes self, so the thread closure can
  only reach the captured psk). Resolves the earlier E0425 compile failure
- cubec.rs client: --auth-key builds a signed HELLO frame
- commands.rs: audit op constants + read-gating hooks wired into admit_read
- audit.rs: per-tenant append-only audit log in a Null-cube range (plan R6)
- clippy -D warnings clean; full ./check gate ALL CHECKS PASSED (61 tests)
2026-08-11 15:59:10 -04:00
hermes 78cd5c0046 chore(cubesys): clear clippy -D warnings so full ./check gate is green
cargo clippy --fix applied style nits (redundant return, ?-operator,
map_or simplify) plus doc-comment list indentation. The full ./check gate
(fmt+test+clippy -D warnings) now passes; R5 read-gate + grant tests
remain green (39 cubesys lib tests).
2026-08-11 15:25:14 -04:00
hermes 8e31c15f69 docs: mark R5 done, reframe R1 in plan (owner/grant-native read gate) 2026-08-11 15:12:30 -04:00