Author SHA1 Message Date
cubelinux 10cb4e4075 Point the notes MCP at this tree's new home
The repository moved from /home/CUBELinux/CUBELinux-2 to /home/CUBEdb, and the
MCP server hardcoded its own CLI's path. One line, no behaviour change; without it
the server would call a binary that no longer exists.
2026-09-18 17:57:24 -04:00
CUBELinux-2 b47a5df1b2 tools/cube-notes-agent: dedupe findings by ISSUE (not just coord)
_canonical_issue() strips FK/FK#/FA/FB/FLAG/FIX prefixes and normalizes separators so
duplicate representations of one root cause collapse (e.g. FK1-SELINUX_RESTORECON +
SELINUX-RESTORECON -> selinux_restorecon; FK3-WAL_CHECKPOINT +
CONCURRENT-STORE-CHECKPOINT -> checkpoint_before_exit via an alias map). Per issue only
the most-detailed note is reported. mark_covered now covers ALL uncovered finding coords
(including deduped-away duplicates) so nothing re-reports. Corresponding notes re-tagged:
issue logs -> finding, status notes -> impl (no uncategorized notes remain).
2026-09-08 07:46:08 -04:00
CUBELinux-2 d84f87e7bb tools: cube-notes-agent (headless notes<->model review agent) + integration README
Adds the headless, cron-driven note reviewer that uses the cubesys::notes
message-save-path as its memory: it reads only uncovered findings from CUBE, gets
the model to rank them, composes a grounded (verbatim) prioritized report, marks
them covered with a checkpoint note, and writes a review .txt. Documents how the
MCP cube-notes-mcp surface is wired into the harness bundle layer (inject:[tools])
and how prior run data saved in CUBE (action trace + finding/checkpoint notes) is
replayed and traced — surfacing 7 previously un-tagged logs (GIT-SHALLOW-CLONE,
POSTFIX-TLS-CERTS, SELINUX-RESTORECON, DSH-WEB-EADDRINUSE,
CONCURRENT-STORE-CHECKPOINT, MCP-BUNDLE-LAYER, CUBES-TWO-TREES).
2026-09-08 07:27:10 -04:00
CUBELinux-2 bd3e3c0512 cubesys/notes: note tag <coord> <cat> (in-place recategorize); close out the notes -> model CUBE integration + error-tracing from prior run data saved in CUBE
note tag
- New `cube note tag <coord> <cat>` subcommand: reclassify an existing note in place by
  setting its doc_type to `note:<cat>` and put_record under the same coord. Used to move
  mis-tagged notes into the right category (e.g. a finding accidentally logged as a debug
  or action). Puts are WAL+delta+checkpoint durable. Note: `delete_raw` is NOT durable here
  (the checkpoint/delta records Puts of surviving keys only — no tombstones — so a deleted
  key resurrects from the base snapshot on reload), so `tag` is the reliable in-place
  reclassify path. 5 notes tests pass.

Notes -> model CUBE integration (message-save-path)
- cubesys:🎶 WordFlags-tagged CZYX note log (doc_type=note:<category>); project threads
  via linked_records; project resume (<cubecoords> project resume <name> <text>) and context
  injection (<project context <name>>). Durable ConcurrentStore (WAL + delta + checkpoint);
  short-lived processes must checkpoint() before exit.
- CLI: cube note add|list|search|show; cube project list|show|resume|context.
- cube-notes-mcp (stdio MCP server) exposes mcp__cubenotes__note_*/project_* tools to the
  DeepSeek Harness; harness registers the mcp-client at the BUNDLE layer (inject: [tools])
  so the plugin actually mounts (profile-layer rows can't inject the tools service — the
  silent no-mount bug).

How we used it: trace errors from prior run data saved in CUBE
- The headless agent (cube-agent) treats CUBE as a searchable, replayable log: every action
  is written as a note (category=action) so a run can be replayed/fine-tuned; each review run
  reads ONLY uncovered findings (category=finding) and marks them covered with a checkpoint
  note (category=checkpoint) — true incremental, non-destructive review.
- The deterministic, grounded report tool reads finding notes verbatim (no model
  hallucination), assigns severity (HIGH/MEDIUM/LOW) + priority order, and writes a review
  .txt. It surfaced 7 previously un-categorized logs (GIT-SHALLOW-CLONE, POSTFIX-TLS-CERTS,
  SELINUX-RESTORECON, DSH-WEB-EADDRINUSE, CONCURRENT-STORE-CHECKPOINT, MCP-BUNDLE-LAYER,
  CUBES-TWO-TREES) that were logged but never tagged finding.
- Coverage is precise: the report marks exactly the findings it reviewed, so the next run
  reports only new ones. The notes store is also the harness GUI's CUBE surface
  (mcp-cubenotes note_list), so everything the agent writes is visible and reviewable.
2026-09-08 07:22:41 -04:00
admin 8c66478a42 cubesys/cube-notes-mcp: project resume + context. 'cube project resume <name> <text>' sets a resume marker (note category=resume, associated to the project); 'cube project context <name>' emits a compact 'where we are' summary (latest notes + resume marker) for context injection into a new session. MCP tools project_resume/project_context added. 5 notes tests pass. 2026-09-08 05:34:58 -04:00
admin 18f8eb9b4a cubesys/cubecli/cube-notes-mcp: note categories + project threads. notes use doc_type 'note:<category>' for categories and associate to a 'project' record (linked_records) for project threads; cube note add|list|search accept --project/--cat/--since/--until; new 'cube project list' + 'cube project show <coord>' (walk a project's thread). MCP server exposes note_write/list/search/show + project_list/project_show. 4 notes tests pass; workspace check green. 2026-09-08 05:29:06 -04:00
admin 15dd4c48a4 cube-notes-mcp: stdio MCP server exposing the cubesys::notes CLI (cube note add|list|search|show) as model-facing mcp__cubenotes__note_* tools for the DeepSeek Harness <-> CUBE session message path. 2026-09-08 04:02:21 -04:00
admin a601710d50 cubesys/cubecli: session note/message-save-path (notes) — WordFlags-tagged CZYX note log (doc_type=note, per-session owner + created_at day), store_note/list/search/show + note_command dispatcher, durable CubeStore+WAL+checkpoint; cube note add|list|search|show. New cubesys::notes module w/ 3 tests; workspace check green. 2026-09-08 04:01:06 -04:00
CUBELinux-2 cc896fc336 CUBELinux-2: WordFlags/tri-channel 16-bit flag field, tag-14 + scan_by_word_flag, cubetrace capture→replay→golden→lineage, CUBE VP-tree balling index (cubecode::ballindex), cubefs-visible WordFlags, and cube-mvw (sharded multi-writer MVCC store with per-shard WALs + group-commit + compaction, impl CubeBackend => CubeStore<MvwBackend> DB). Adds show-flags/scan-word-flags/trace-*/golden-capture CLI; ~ tests green. 2026-09-06 21:20:04 -04:00
CUBELinux-2andHermes Agent (upstage/solar-pro4:free) 3f007a1f38 bench(cube-bench): use saturating_sub for expected-value arithmetic
clippy -D warnings flagged implicit_saturating_sub on the c1/c2/c3
expected-count calculations. Replace manual - with saturating_sub so
the lint gate stays clean at any scale.

Co-Authored-By: Hermes Agent (upstage/solar-pro4:free)
2026-08-20 19:03:48 -04:00
CUBELinux-2andHermes Agent (upstage/solar-pro4:free) ae1a62a44e docs: reflect cubecli as standalone crate in README + integration docs
The  binary was extracted to  in a prior session; the
docs still referenced  as its home.

Co-Authored-By: Hermes Agent (upstage/solar-pro4:free)
2026-08-20 18:59:28 -04:00
CUBELinux-2andHermes Agent (upstage/solar-pro4:free) c090233056 bench(cube-bench): fix scan_prefix expected-value math for scale > 65536
coord_for spreads records across C=0,1,2,3 as i grows past 65536,
but the bench assumed everything beyond c=0 landed in c=1. At
scale=200k this asserted (got 65536, expected 134464). Fix: compute
per-bucket expected counts from the coord_for mapping and extend the
total-coverage assertion to include c=2 and c=3.

Verified: fmt clean, clippy -D clean, 100k + 200k both pass.

Co-Authored-By: Hermes Agent (upstage/solar-pro4:free)
2026-08-20 18:56:20 -04:00
CUBELinux-2andHermes Agent 7b0cf6fda5 chore(this session): land all 4 tasks — fixes, cube-bench run, cubecli extract, check gate passed
1. Fixes applied this session:
   - cubesys/src/lib.rs:136 — `store_code_cell` annotated with
     `#[allow(clippy::too_many_arguments)]` (8 args is inherent to the
     record-codec bridge; clippy -D warnings gate).
   - cubesys/src/store.rs:699 — `put_code_cell` annotated identically.
   - cubesys/src/commands.rs:819 — `cell.links().iter().copied().collect()`
     replaced with `cell.links().to_vec()`.
   - cubesys/src/commands.rs:991 — `while let Some(tok) = it.next()` loop
     rewritten as direct `for tok in it`.
   - cubesys/src/commands.rs:1187 — ls handler uses `parse_path` (not
     `path_to_czyx`) for directory ACL gate; derived prefix coordinate
     matches NullSpace acl_bucket convention.

2. cube-bench executed: ./check bench stage passes with correctness
   assertions on every measurement path (100k puts, 100k gets, crypto
   benchmarks, VM dispatch, session commands, concurrent RwLock stress).

3. cubecli extracted: standalone crate at cubecli/ with its own
   Cargo.toml + src/main.rs; cube binary unchanged (same CLI surface,
   demo, commands); cubesys/Cargo.toml no longer declares the `cube`
   bin; workspace members updated.

4. ./check gate: fmt + tests + clippy -D warnings all green;
   `./check bench` also passes.

Verification: ./check (full gate) + ./check bench both PASS.

Co-Authored-By: Hermes Agent
2026-08-20 18:08:19 -04:00
CUBELinux-2andHermes Agent ce8ac78a24 extract(cubecli): move cube CLI out of cubesys into standalone crate
The `cube` binary (REPL + script runner over one shared CubeStore) is
now its own crate at `cubecli/`, matching the spec's package outline
that lists cubecli as a separate crate from the composition layer.

What changed:
- New `cubecli/` crate with `Cargo.toml` (deps: cubecoords, cubestore,
  cubefs, cubecode, cubecrypt, cubesys) and `src/main.rs` (copy of
  the former `cubesys/src/bin/cube.rs`).
- `cubesys/Cargo.toml`: removed the `[[bin]]` entry for `cube`; keeps
  `cube-demo`, `cube-server`, and `cubec` as before.
- `Cargo.toml` (workspace): added `cubecli` to members.
- `cubesys/src/bin/cube.rs` removed (code now lives in cubecli).

The `cube` binary is unchanged: same CLI surface, same demo, same
commands. The `cube-demo` binary in cubesys still wraps
`cubesys::demo::run()` and is unaffected.

Verification: `./check` gate passes (fmt + tests + clippy -D warnings);
`cube --help` and `cube demo` both work.

Co-Authored-By: Hermes Agent
2026-08-20 18:05:56 -04:00
CUBELinux-2andHermes Agent fdfcd2c728 chore(cubesys): resolve 4 clippy -D warnings blocking the check gate
* commands.rs:819 — `cell.links().iter().copied().collect::<Vec<_>>()`
  replaced with `cell.links().to_vec()` (clippy: "calling to_vec() is
  both faster and more readable").

* commands.rs:991 — `while let Some(tok) = it.next()` loop over metatag
  tokens rewritten as a direct `for tok in it` (clippy: "this loop could
  be written as a for loop").

* store.rs:699 — `put_code_cell` (8 args) annotated with
  `#[allow(clippy::too_many_arguments)]` (argument count is inherent to
  the code-cell put API; splitting would add indirection without benefit).

* lib.rs:136 — `store_code_cell` (8 args) annotated identically; this is
  the shared implementation every front-end uses so the count cannot change
  without redesigning the record-codec bridge.

Verification: ./check gate passes (fmt + tests + clippy -D warnings).

Co-Authored-By: Hermes Agent
2026-08-20 17:49:38 -04:00
CUBELinux-2andHermes Agent 90e90ef926 fix(cubeai): add Default impl for CubeAi to satisfy clippy -D warnings
clippy flagged "you should consider adding a `Default` implementation for
`CubeAi`". Add the impl delegating to `new()`.

Also: impl block methods had been accidentally pulled out of `impl CubeAi`
into module scope during a prior edit; rewrite the impl block with all
methods (ingest_trace, classify, suggest, ingest_and_suggest, store)
inside it so it compiles.

Verification: all 5 cubeai tests pass, clippy clean.

Co-Authored-By: Hermes Agent
2026-08-20 17:41:02 -04:00
CUBELinux-2andHermes Agent ec25347af9 fix(cubedbt): add Default impl for CodeCache to satisfy clippy -D warnings
clippy flagged "you should consider adding a `Default` implementation for
`CodeCache`". Add the impl delegating to `new()`.

Also: `patch` method had been accidentally pulled out of `impl CodeCache`
into module scope during a prior edit; move it back inside the impl block
so it compiles.

Verification: all 4 cubedbt tests pass, clippy clean.

Co-Authored-By: Hermes Agent
2026-08-20 17:36:17 -04:00
CUBELinux-2andHermes Agent 6627505e70 fix(cubesys): ls on directory paths no longer rejected by path_to_czyx
The ls handler used path_to_czyx() (which enforces "path must name a
record") to derive the ACL gate coordinate. Directory prefixes like
/c001/z001/y001 are valid read targets but path_to_czyx rejects them as
"names a directory, not a record", so cube-bench's ls assertion panicked.

Fix: use cubefs::path::parse_path (accepts any well-formed prefix) and
derive the directory prefix coordinate (trailing axes zeroed) for the ACL
gate, matching what NullSpace acl_bucket uses for directory ACLs.

Also: clippy -D warnings — remove unnecessary `c as u8` cast in
hash_backend_keys_sorted_order test.

Verification: cube-bench runs to completion (ls section printed OK),
clippy clean.

Co-Authored-By: Hermes Agent
2026-08-20 17:31:54 -04:00
CUBELinux-2andHermes Agent ddca08ea73 fix(cubetrace): wrap Behavior::variant in behavior() helper in test call sites
Behavior is a newtype (Behavior(pub u16)), not an enum, so the 5 test
call sites that passed Behavior::PURE / Behavior::IO_HEAVY / etc. directly
to the ev() helper got type mismatch errors. Add a local behavior() wrapper
and use it at all 5 sites. Also clean up the unused CodeCell import and
extraneous parens that clippy flagged.

fix(cubesys): ls on directory paths no longer rejected by path_to_czyx

The ls handler used path_to_czyx() (which enforces "path must name a
record") to derive the ACL gate coordinate. Directory prefixes like
/c001/z001/y001 are valid read targets but path_to_czyx rejects them as
"names a directory, not a record", so cube-bench's ls assertion panicked.

Fix: use cubefs::path::parse_path (accepts any well-formed prefix) and
derive the directory prefix coordinate (trailing axes zeroed) for the ACL
gate, matching what NullSpace acl_bucket uses for directory ACLs.

Verification: cube-bench runs to completion (ls section printed OK), all
3 cubetrace tests pass, workspace compiles clean.

Co-Authored-By: Hermes Agent
2026-08-20 17:25:48 -04:00
CUBELinux-2andHermes Agent b97efa2a16 scrub: remove cube-store-raw + cube-fuse-proxy, the wrong-architecture duplicates
The workspace already has cubestore (CubeBackend trait + HashBackend
= HashMap<u32,Vec<u8>>), cubefs, and cubesys — the correct foundation
the PDF spec describes. cube-store-raw/ and cube-fuse-proxy/ were
scaffolded as redundant duplicates on the wrong architecture; they
broke `cargo check` (103 errors) and had no tests.

Also: add 7 direct HashBackend trait-level tests (PDF spec's
HashMap<u32,Vec<u8>> sketch) + clean up the _assert_backend_assoc
workaround + dead HashMap import in record_codec, both now unneeded
since HashBackend is exercised by real tests.

Verification: cargo test -p cubestore = 15/15 pass; cargo check
--workspace green (only pre-existing cubetrace warnings).

Co-Authored-By: Hermes Agent
2026-08-20 16:40:53 -04:00
CUBELinux-2 11e5ed7fca fix(metatag-dirs): exact-leaf + descendant path prefix semantics
scan_by_path_prefix now matches BOTH an exact leaf (p == dir) and any
descendant (p.starts_with(dir + '/')), so  returns
 (the leaf itself) as well as  (a child).
Previously a bare dir normalized to 'dir/' and rejected exact leaves.

Also fixes the unit test to assert the corrected semantics (exact leaf
answers its own parent query).

Falls out of wiring the OS migration scripts: their companion index
records (c200/z090 GNS band) carry path=/cubelinux/os/... metatags, and
the daemon  reconstructs the whole migrated tree from them.
2026-08-13 18:41:22 -04:00
CUBELinux-2 94681bbdc0 feat(metatag-dirs): nested directory hierarchy via path metatag + flags
Implements the source PDF's prescribed model for directory structure:
'treat the CZYX cube as the only persistent store, with system calls that
operate on CZYX records and Null-space flags rather than paths and inodes.'
Nested dirs are RECONSTRUCTED from a per-record `path` metatag, so the
filesystem does not need recursive inode trees.

- cubecoords: add CubeHeader.path (Option<String>) + HAS_PATH flag bit (1<<8);
  refresh_flags() sets it. Cubestore TLV codec now serializes/deserializes the
  path as tag 13 (persists across checkpoint/reopen).
- cubestore: scan_by_path (exact), scan_by_path_prefix (dir listing by path
  prefix; bare '/etc' normalized to '/etc/'), plus query_by_path/_prefix
  returning decoded (coord,header,body).
- cubesys: `put` verb gains path=<p>;
  new `query-path <dir>` verb reconstructs a directory listing from metatags.
- ConcurrentStore: query_by_path(_prefix) delegate to inner CubeStore.

Verified live in VM: migrated /etc/passwd, /etc/network/interfaces, /usr/bin/ls
at unrelated coordinates; 'query-path /etc' returns the two /etc files,
'/usr/bin/ls' correctly excluded. cubestore 8/8 tests pass.

Refs: OS-in-CUBE migration, query-layer prototype.
2026-08-13 18:26:58 -04:00
CUBELinux-2 2ff1dff02b feat(query): PDF Package-2 associative log lookup via flags/metatags
Implements the source PDF's 'put(C,Z,Y,X,bytes,flags)' + 'scan_by_flag'
associative-storage API so records are discoverable by WHAT they are
(event type / doc_type metatag) rather than WHERE they live (coordinate
path) — the 'log lookup by query' the OS layers want.

cubestore:
  - CubeStore::scan_by_flag(u16) / scan_by_type(&str): predicate scans
    over decoded CubeHeader (not prefix scans).
  - CubeStore::query_by_flag / query_by_type: same, but also return the
    decoded (label, header, body) so a query tool can present results.
  - get_record's raw/un-enveloped fallback now refresh_flags() so
    synthesized headers carry SIZE_BYTES and are visible to flag queries.
  - regression tests: scan_by_flag_finds_typed_records,
    scan_by_type_event_lookup.

cubesys (daemon):
  - new 'put' verb: enveloped write with optional doc_type=/title=
    metatags (the PDF's put API). rawput remains for bulk/append payloads.
  - ConcurrentStore::query_doc_type was already the backing predicate the
    'query <doc_type>' verb uses; it now matches these typed records.

Proven end-to-end in the VM: 'cubec query klog' returns the kernel-log
index record (doc_type=klog) published by cube-os-klog.sh, and
'cubec query fn' returns the 8 pre-existing typed program records.
2026-08-13 18:12:13 -04:00
CUBELinux-2 fe64b869e4 feat(trace+ai): add cubetrace package and wire DBT/AI/CUBEsys command + module edits (green-lit WIP)
Brings in the cubetrace crate (PDF Package 4, §547): wraps a DBI engine via an
FFI seam and streams trace events (basic blocks, syscalls) into cubestore,
tagging each with CZYX coordinates and header flags.

- Cargo.toml: register cubetrace workspace member
- cubeai/src/lib.rs, cubedbt/src/lib.rs: DBT/AI rule + trace integration edits
- cubecode/src/{cb,cell}.rs: code-cell bytecode/module plumbing for traces
- cubesys/src/commands.rs: trace/AI command surface expansion
- cubetrace/: new crate (builds; 2 non-fatal warnings)

All layers of the OS-in-CUBE migration pass; recorded per user green-light.
2026-08-13 17:31:40 -04:00
CUBELinux-2 ff2914a105 fix(cubestore): surface raw/un-enveloped records in get_record instead of returning None
get_record() previously returned None for any payload lacking a TLV envelope
(4-byte header-len | header | body). Records written through the raw path
(put_raw / the daemon's 'rawput' verb, used by every OS-layer service) carry
no envelope, so callers such as cubefs getattr/read and the 'stat' verb mapped
that None to size 0 / empty file: real bytes became SILENTLY INVISIBLE through
the filesystem while rawget still returned them.

This broke the OS-in-CUBE migration (2026-08-13): every rawput OS record listed
as a 0-byte file in /cubefs, which also surface-invalidated the 'the OS boots
from cube' resume path.

Now a payload that is not a well-formed envelope is surfaced as a raw body under
a synthesized header whose size_bytes reports the true length. Enveloped records
still decode their real header (no behavior change on the record path). Adds two
regression tests: get_record_surfaces_raw_unenveloped_payloads and
get_record_still_prefers_the_real_envelope (cargo test -p cubestore: 5/5).
2026-08-13 17:29:50 -04:00
CUBELinux-2 3121459138 feat(dbt+ai): build out cubedbt and cubeai packages (Package 4, PDF §549/§551)
cubedbt: DBT runtime that reads CZYX-stored translation rules and patches
them into a code cache to execute mimicked behavior. TranslationRule (CZYX
Variant record in c220 band) maps an op-class to a replacement bytecode
fragment; CodeCache patches an original CodeCell under eligible rules and
writes the result as a Variant (preserving behavior descriptors), and
DbRuntime.mimic() fetches -> patches -> runs in the real Vm, proving
're-run or modify behavior without the original binary'.

cubeai: models over cube-stored traces/graphs to classify blocks
(Computation/Branch/CallTrampoline/IoSection/Sequence from the op-class
histogram + behavior descriptors) and suggest new code sequences as
cubedbt TranslationRules (persisted into the c220 rule band, discoverable
by DbRuntime). End-to-end: trace -> classify -> suggest -> mimic.

Both crates are dependency-free and operate on the real CubeStore/CodeCell/
Vm/Behavior types, so they are exercisable today on HashBackend and slot
into ConcurrentStore later without an API change. ./check quick green:
cubedbt 4 tests, cubeai 5 tests, full workspace green.
2026-08-13 16:23:00 -04:00
CUBELinux-2 0074f3112c fix(os): cover full spec behavior-descriptor set (PDF §524-525)
cb.rs previously implemented only 3 of the 6 spec descriptors (pure/io/hot).
PDF §524-525 lists: pure function, I/O heavy, allocates memory, touches
network, hot path, security sensitive.

- Behavior now stores descriptor bits directly as dedicated header-flag bits
  in the spare 8..=15 range: PURE=9, IO_HEAVY=10, ALLOCATES=11, NETWORK=13,
  HOT_PATH=14, SECURITY_SENSITIVE=15. Bit 12 (ENCRYPTED) left clear; mask
  0xEE00. refresh_flags() preserves 8..=15 so round-trip is safe.
- K= CLI tokens gain alloc/net/sec (prog + kernel verbs).
- Add unit test for all-six round-trip + explicit security-sensitive bit.

Proven green via ./check quick.
2026-08-13 16:00:34 -04:00
CUBELinux-2 1539ecd1bb feat(os): OS operator kernels in CUBE + thin native effector (Steps 1 & 2)
- cubecode/src/cb.rs: Behavior descriptor bitflag (PURE/IO_HEAVY/HOT_PATH)
  encoded into HeaderFlags bits 13..15; C_OS_KERNEL=210 / C_OS_EFFECT=211
  coordinate bands; round-trip unit test.
- fix: HEADER_FLAG_BEHAVIOR mask was 0x7000 (bits 12-14) which grabbed the
  ENCRYPTED bit (12) and dropped HOT_PATH (bit 15). Corrected to 0xE000.
- store_code_cell/put_code_cell/header_for_code gain optional descriptor arg;
  all 12 prior call sites pass None (total change).
- CLI: prog K=<flag> tokens, kernel verb (links+descriptors), tick verb lays
  down the OS kernel call-graph (cfg->decide->summarize->tick), native-apply
  verb = thin effector (runs kernel in VM, reads computed result, emits effect).
- integration test os_kernels_live_in_cube_with_call_graph_and_descriptors.

Proven green via ./check quick (fmt+clippy -D warnings+tests).
2026-08-13 15:57:10 -04:00
43 changed files with 5715 additions and 113 deletions
+3
View File
@@ -1,3 +1,6 @@
/target
Cargo.lock
target/
# stray build dir (not source)
/src/CUBELinux-build/
+5
View File
@@ -2,12 +2,17 @@
resolver = "2"
members = [
"cubecoords",
"cube-mvw",
"cubestore",
"cubefs",
"cubecode",
"cubecrypt",
"cubesys",
"cubedbt",
"cubeai",
"cubetrace",
"cube-bench",
"cubecli",
]
[workspace.package]
-17
View File
@@ -1,17 +0,0 @@
CUBELinux LICENSE — source-available, non-commercial by default
Copyright (c) 2026 CUBELinux (CUBELinux.com) / DULRobotics (DULRobotics.com)
1. Permission is granted to use, copy, modify, and distribute this software,
in source or binary form, with or without modification, FOR PERSONAL AND
NON-COMMERCIAL DEVELOPMENT AND TESTING, provided this license and the
copyright notice are retained.
2. COMMERCIAL USE (any use to obtain a commercial benefit, including selling
the software or a service built on it, or use inside a commercial product)
is permitted ONLY under a separate written COMMERCIAL LICENSE from the
copyright holder. Unlicensed commercial use is prohibited.
3. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND.
4. The copyright holder may later release it under an open-source license or
grant commercial licenses at its discretion.
+50
View File
@@ -0,0 +1,50 @@
# RESUME POINT — HashBackend (HashMap<u32, Vec<u8>>) work, 2026-08-20 → 2026-08-21
## WHAT WAS DONE THIS SESSION (continuation)
### Prior session's 4 tasks — all closed and verified
1. clippy -D warnings in cubesys — 4 lints fixed
2. cube-bench executed — correctness-gated microbenchmarks
3. cubecli extracted — standalone crate
4. go live (./check gate) — passes
### THIS session: ran all 4 `./check` opt-in stages in logical order
1. **./check mount** — PASS (57 assertions, 0 failed)
- In-memory `--seed` mount: full regression suite
- Daemon-backed `--socket` mount: full regression suite
- Durability across daemon restart: verified (record survives kill+restart)
- Cross-user ACLs (root ↔ luulu): verified
2. **./check daemon** — PASS (3 tests, 0 failed)
- `cubefs_create_write_reaches_daemon`
- `daemon_backend_envelope_roundtrip`
- `daemon_backend_put_get_roundtrip`
- All 3 were `#[ignore]`d, now exercised against live cube-server
3. **./check stress** — PASS (~150s, 57075 pairs driven)
- Daemon alive throughout, ~380 prog+run pairs/s sustained
- 156 records in C=77 namespace (command dedup by coordinate)
- No crashes, no latency spikes
4. **cube-bench at scale 200k** — FAILED initially, then FIXED and PASS
- **Bug found**: bench's expected-value math assumed all records beyond c=0 land in c=1, but `coord_for` spreads across c=0,1,2,3 as `i` grows past 65536. At scale=200k, c=1 should have 65536 records (not 134464).
- **Fix**: compute expected counts per C bucket from the `coord_for` mapping (c=0: min(n, 65536); c=1: next 65536; c=2: next 65536; c=3: remainder). Added expected_c2 + expected_c3 to the total-coverage assertion.
- **Verified**: `cargo fmt --all -- --check` clean, `cargo clippy -p cube-bench -- -D warnings` clean, 100k + 200k both pass.
- **Numbers at 200k (release)**: put_raw 236ns/op, get_raw 109ns/op, scan_prefix 12.4ms for 65536 coords, ~4237k put/s, crypto ~1.4-2.6μs/seal, VM 276ns entry / 142ns leaf, session 1.4μs prog / 2.6μs run.
### Git state
- Working tree: `cube-bench/src/main.rs` modified (the fix above)
- Branch: `feat/os-kernel-in-cube`
- All prior session commits intact (7b0cf6f is HEAD)
## CURRENT STATE
- Branch: feat/os-kernel-in-cube
- cube-bench at 200k: fixed + verified (fmt + clippy + 100k + 200k all pass)
- ./check mount/daemon/stress: all green
- cubecli: standalone crate, functionally identical
## NEXT STEPS
1. Commit the cube-bench fix (27 insertions, 5 deletions)
2. README/docs sync — reflect cubecli as standalone crate (./check doesn't check docs)
3. Spec reading resumption from CUBELinux.txt (~offset 360+) — Package 2 use cases
+33
View File
@@ -0,0 +1,33 @@
# Resume Marker — 2026-08-20
# Status: kernel ready for compile, workhorse ready for Xfce switching, stopped before heavy build
## What was done this session
1. **Kernel build environment — READY** (stopped before heavy compile per user request)
- Kernel: vanilla v6.19.3 + ipu4 drivers (GRUB entry LOCALVERSION=-ipu4p)
- `./check` green, git clean
- Rust toolchain: stable 1.97.1 (default) + nightly 1.100.0 installed via rustup
- CONFIG_RUST=y set in .config, CONFIG_RUST_IS_AVAILABLE=y
- `scripts/target.json` has `rustc-abi: x86-softfloat` (rejected by stable, OK with nightly)
- Build fix identified: `RUSTC=/root/.cargo/bin/rustc RUSTLIB_SRC=... make ...` to use nightly
- Build NOT started — user wants new session for the heavy compile
2. **GUI session switching — READY** (workhorse has Xfce, GDM shows it)
- Xfce 4.20.2 + xfce4-goodies installed on workhorse
- `/usr/share/xsessions/xfce.desktop` present, `Exec=startxfce4`
- GDM (gdm3) active, shows xfce.desktop + xfce-wayland.desktop at login gear menu
- User can switch GNOME↔Xfce at GDM login by clicking gear icon → "Xfce Session"
- lightdm package removed (was stuck half-configured; cleaned up via dpkg purge)
- No lightdm; gdm3 is the DM; /etc/X11/default-display-manager = /usr/sbin/gdm3
## What remains (for next session / new session)
- **Kernel compile**: `make -j$(nproc) bzImage modules` with nightly rustc override
- Command pattern: `cd /home/CUBELinux/CUBELinux-2/src/CUBELinux-build && RUSTC=/root/.cargo/bin/rustc HOSTRUSTC=/root/.cargo/bin/rustc make -j$(nproc) bzImage modules`
- Need to confirm RUST_LIB_SRC path for nightly toolchain
- User explicitly asked NOT to auto-launch — new session should decide timing
- **GRUB**: new entry for -cube kernel; verify ipu4p entry still works
- **Post-boot verification**: cube fs mount, daemon startup, camera (ipu4) if desired
## Notes
- User restarted the session; this marker captures where we left off
- User directive: "get project to point of compiling new kernel, then stop so we can use new session for that"
- Heavy compile (make -j8) saturates all cores + memory — do NOT start without user go
+72
View File
@@ -0,0 +1,72 @@
# Resume Marker — 2026-08-20 (kernel build + GUI session switching)
## Context
User restarted the session. Two parallel tasks were in progress when interrupted:
1. Kernel Rust build config issue (kbuild + nightly rustc)
2. GUI session switching — install Xfce on workhorse so user can switch GNOME↔Xfce at login
---
## Task 1: Kernel Rust Build Config
**Status: unresolved — root cause found, fix not yet applied**
**What we know:**
- Kernel source: `/home/CUBELinux/CUBELinux-2/src/CUBELinux-build/` (vanilla v6.19.3 + ipu4 out-of-tree)
- `CONFIG_RUST=y` set in `.config`
- rustc 1.97.1 (stable) is default; rustc 1.100.0-nightly also installed via rustup (default host: x86_64-unknown-linux-gnu)
- `scripts/target.json` exists with `rustc-abi: x86-softfloat`, `llvm-target: x86_64-linux-gnu`
- `make rust/core.o` **fails**: `error loading target specification: custom targets are unstable and require -Zunstable-options`
- The rustc_library command in `rust/Makefile:559` does NOT pass `-Zunstable-options`
- `rust/Makefile` already uses `-Zunstable-options` in rustdoc paths (lines 136, 324, 333) but NOT in the core rustc_library invocation
- `KBUILD_RUSTFLAGS` (from `arch/x86/Makefile`) sets `--target=.../scripts/target.json` but no `-Zunstable-options`
- The `rustc_library` rule (`rust/Makefile:555-567`) builds the command as:
```
$(RUSTC) $(filter-out $(skip_flags),$(rust_flags)) $(rustc_target_flags) --emit=...
```
where `rust_flags` = `KBUILD_RUSTFLAGS` + `rust_common_flags` and `rustc_target_flags` = `core-flags` (which is `--edition=...` + cfgs). Neither carries `-Zunstable-options`.
**What needs to happen:**
- Option A: Patch `rust/Makefile` to add `-Zunstable-options` to the rustc_library command (line 559 area) — minimal change, mirrors what rustdoc already does
- Option B: Use nightly rustc as RUSTC and ensure it's invoked with `-Zunstable-options` — but kbuild doesn't add it automatically
- The real fix is Option A: add `-Zunstable-options` to the rustc invocation in the rustc_library rule, since the target.json is a custom target spec that requires it
- After that: `make rust/all` or `make rust/core.o` should succeed, then full kernel build
**Files involved:**
- `/home/CUBELinux/CUBELinux-2/src/CUBELinux-build/rust/Makefile` (line 555-567, rustc_library rule)
- `/home/CUBELinux/CUBELinux-2/src/CUBELinux-build/arch/x86/Makefile` (KBUILD_RUSTFLAGS)
- `/home/CUBELinux/CUBELinux-2/src/CUBELinux-build/scripts/target.json` (custom target spec, needs -Zunstable-options)
---
## Task 2: GUI Session Switching (Xfce on workhorse)
**Status: dpkg interrupted — lightdm stuck half-configured**
**What we know:**
- Workhorse runs GNOME (gdm3 + gnome-shell) currently
- User wants GNOME ↔ Xfce switching at the login screen (like Cinnamon is available)
- `apt-get install -y task-xfce-desktop` timed out (exit 124) and left dpkg in interrupted state
- `dpkg --configure -a` fails: lightdm is half-configured, depends on itself being configured first
- lightdm package status: `iF` (half-configured), `light-locker` and `task-xfce-desktop` also stuck
- Xfce components were likely downloaded but not configured due to the dpkg blockage
**What needs to happen:**
1. Fix dpkg: either `dpkg --remove --force-remove-reinstreq lightdm` then `dpkg --configure -a`, or manually remove the lock/status entries
2. Re-run `apt-get install -y task-xfce-desktop` (or just `xfce4` + `lightdm` if minimal)
3. Verify `xfce4-session` works: `xfce4-session --version`
4. Verify GDM shows Xfce session: check `/usr/share/xsessions/xfce.desktop` exists, `gdm3` is running
5. Test: log in via Xfce session
**Files/dirs involved:**
- `/usr/share/xsessions/` (GDM session definitions)
- `/usr/share/wayland-sessions/` (if Wayland Xfce session needed)
- `/etc/gdm3/` (GDM config)
---
## General Notes
- User explicitly does NOT want auto-launching heavy kernel compiles while they're using the machine
- Kernel build with `make -j$(nproc)` will saturate CPU — must ask before launching
- The workhorse is Debian-based, root access available
- rustup is installed at `/root/.rustup`, toolchains in `/root/.rustup/toolchains/`
+96
View File
@@ -0,0 +1,96 @@
# Resume Marker — 2026-08-21 (kernel Rust build: IRQ lifetime fixes + ready for compile)
## Context
The session after RESUME-20260820. Fixed the kernel Rust build errors and verified `make rust/core.o` compiles cleanly. Heavy kernel compile not started (per user directive — will run tomorrow with explicit go).
---
## What was done this session
### 1. Kernel Rust build: IRQ lifetime errors fixed
**File changed:** `rust/kernel/irq/request.rs`
**Root cause:** `rust/core.o` failed with 3× `E0310: the parameter type T may not live long enough` errors. The extern "C" IRQ callback functions were missing `+ 'static` lifetime bounds on their generic `T` parameters.
**Fixes applied (4 total):**
| Function | Line | Change |
|----------|------|--------|
| `handle_irq_callback` | 264 | `T: Handler``T: Handler + 'static` |
| `handle_threaded_irq_callback` | 483 | `T: ThreadedHandler``T: ThreadedHandler + 'static` |
| `thread_fn_callback` | 499 | `T: ThreadedHandler``T: ThreadedHandler + 'static` |
The `impl` blocks already had `T: Handler + 'static` and `T: ThreadedHandler + 'static` on lines 185, 198, 403, 416 — the three extern C callback functions and their callers were the missing ones.
**Verification:** `python3 /tmp/hermes-verify-kernel-rust-build.py` — 16/16 checks passed:
- Source file checks: all 3 fixed signatures present, no old patterns remain
- Build: `make rust/core.o` exit 0, no E0310 errors, no "error: aborting"
- Object file: 344,464 bytes at `rust/core.o`
- Only 2 pre-existing warnings (unused import `flags::*`, unused feature `used_with_arg`)
### 2. ./check gate — ALL CHECKS PASSED
```
./check → ALL CHECKS PASSED (run: ./check mount | bench | stress | daemon for those stages)
```
fmt clean, 223 tests, clippy -D clean. No regressions from the IRQ fix.
### 3. Build config (carried over from prior session, still valid)
- Nightly rustc 1.100.0 (`rustup default nightly`)
- `scripts/target.json`: `rustc-abi: softfloat` (patched from `x86-softfloat` in prior session — rustc 1.100 rejects `x86-softfloat`)
- `-Zunstable-options` added to `rust/Makefile:559` (rustc_library rule)
- `CONFIG_RUST=y`, `CONFIG_RUSTC_VERSION=110000` in `.config`
- `rust-src` component installed for nightly toolchain
- `generate_rust_target.rs` patched to emit `softfloat` (not `x86-softfloat`)
---
## What remains for tomorrow
### Kernel compile (heavy — ask before launching)
- Command: `cd /home/CUBELinux/CUBELinux-2/src/CUBELinux-build && make -j4 bzImage modules`
- Use `-j4` not `-j8` — 7.4 GB RAM, only 3.7 GB free; rustc is memory-hungry, -j8 risks swapping
- Expected time (incremental, current state with 28,447 .o files already present): **~515 minutes**
- If clean build needed: **4590 minutes** on this hardware (Intel i5-1035G4, 4c/8t, 1.1 GHz base, NVMe)
- After build: install modules, update GRUB, reboot into new kernel
### GRUB
- New kernel entry: `LOCALVERSION=-cube` (already in .config)
- Verify `LOCALVERSION=-ipu4p` entry still works (camera kernel, currently active)
### Post-boot verification
- Cube filesystem mount
- Daemon startup
- Camera (ipu4) if desired
---
## Xfce (separate track — not touched this session)
- dpkg state clean: lightdm, light-locker, task-xfce-desktop all removed/purged
- Xfce 4.20.2 intact: `xfce4-session`, `xfce4-panel`, `xfce4-goodies` all `ii`
- Session files: `/usr/share/xsessions/xfce.desktop`, `/usr/share/wayland-sessions/xfce-wayland.desktop`
- GDM active, default DM `/usr/sbin/gdm3`
- Issue reported (external monitor oversized, both monitors showing in Xfce vs GNOME's correct single monitor): root cause is `xfsettingsd` not running during Xfce session — Xfce inherits GNOME's RandR state without managing it. The fix is ensuring `xfsettingsd` starts (autostart entry at `/etc/xdg/autostart/xfsettingsd.desktop` exists, not disabled). No action taken — user will test after reboot.
---
## Files
- Workspace: `/home/CUBELinux/CUBELinux-2/`
- Changed file: `src/CUBELinux-build/rust/kernel/irq/request.rs`
- Canonical gate: `./check` (run from workspace root)
- Build dir: `src/CUBELinux-build/`
- Kernel .config: `src/CUBELinux-build/.config` (CONFIG_RUST=y, CONFIG_LOCALVERSION="-cube")
- Rust target.json: `src/CUBELinux-build/scripts/target.json` (rustc-abi: softfloat)
- System daemon socket: `/run/cube/cube.sock`
- luulu session daemon socket: `/run/user/1000/cube/cube.sock`
- Verification script used: `/tmp/hermes-verify-kernel-rust-build.py` (cleaned up after run)
## Git state
- Branch `feat/os-kernel-in-cube`, working tree clean (verify with `git status` before committing)
- Prior commits on HEAD: `c090233` (bench fix), `ae1a62a` (docs), `3f007a1` (bench saturating_sub) + 3 older commits
---
## Reminder
- User explicitly does NOT want auto-launching heavy kernel compiles while they're using the machine
- `make -j8` saturates all 8 cores + memory on this Surface Pro 7 (i5-1035G4, 7.4 GB RAM)
- Must get user go before launching the kernel compile tomorrow
+70
View File
@@ -0,0 +1,70 @@
# RESUME POINT — 2026-08-21: all ./check stages green, cube-bench fixed, docs synced
## WHERE WE ARE (verified on disk this session)
### Git state
Branch `feat/os-kernel-in-cube`, clean working tree. Commits this session:
- `c090233` bench(cube-bench): fix scan_prefix expected-value math for scale > 65536
- `ae1a62a` docs: reflect cubecli as standalone crate in README + integration docs
- `3f007a1` bench(cube-bench): use saturating_sub for expected-value arithmetic
Plus 3 prior-session commits still on HEAD (7b0cf6f, ce8ac78, fdfcd2c).
### ./check gate
`./check` → ALL CHECKS PASSED (fmt clean, 223 tests, clippy -D clean).
### ./check opt-in stages (all run this session, all green)
- `./check mount` — 57/57 FUSE assertions (in-memory --seed + daemon-backed --socket + durability-across-restart + cross-user ACLs root↔luulu)
- `./check daemon` — 3/3 live daemon tests (were #[ignore]d, now exercised)
- `./check stress` — ~150s, 57075 prog+run pairs, daemon alive throughout, ~380 pairs/s
- `cube-bench --release 200000` — FAILED once (real bug in bench's expected-value math), then FIXED and PASS at 1k/10k/50k/100k/200k
### Bug found + fixed this session
cube-bench assumed all records beyond c=0 land in c=1. `coord_for` spreads across c=0,1,2,3 as i grows past 65536, so at 200k the expected c=1 count was wrong (134464 vs actual 65536). Fix: compute per-bucket expected counts from the coord_for mapping; extend total-coverage assertion to include c=2 and c=3. Verified at all scales with bench's own correctness assertions. clippy -D warnings clean on cube-bench after replacing manual `-` with `saturating_sub`.
### Docs synced
- `STARTUP-README.md`: cube CLI now references `cubecli/src/main.rs` (was `cubesys/src/bin/cube.rs`)
- `cubesys/docs/integration.md`: Binaries section notes cube is now a standalone crate (`cubecli/`)
## ARCHITECTURAL QUESTION: where should CUBE live when it becomes the OS?
Two-layer model (already partially deployed — this is the right shape):
**Layer 1 — Kernel (the "utilize CUBE" part).** CUBE core crates — `cubecoords`, `cubestore` (HashMap + FileBackedStore + WAL/checkpoint), `cubecode` (bytecode + VM), `cubecrypt` (seal/open + keyinit) — get compiled into the kernel or as a kernel module. Kernel gets direct syscalls: `open_by_czyz()`, `write_czyz()`, `seal_czyz()`, `run_czyz()`. The cube becomes the backing store the VFS and process accounting talk to directly — not a userspace daemon over a socket. This is the spec's Phase 3 ("OS services talk directly to the cube store; syscalls like open by CZYX + flags").
**Layer 2 — Userspace daemon (`cube-server`).** Stays as the durable anchor: WAL, checkpoint, socket for external/remote clients, the "resume pointer" writer. Survives when the kernel module isn't loaded; what remote clients talk to. Currently at `/run/cube/cube.sock` (system) + `/run/user/1000/cube/` (luulu session). Unify under `/usr/lib/cube/` when it's the OS — daemon binary, libs, socket path, store dir all there.
**Filesystem placement when it's the OS:**
- Kernel module/built-in: lives in the kernel source tree (copy of core crates, adapted for kernel constraints — `no_std`, no alloc assumptions that conflict with kernel alloc) OR as out-of-tree module alongside your existing Surface/ipu4 kernel.
- Userspace daemon + libs: `/usr/lib/cube/`. Socket at `/run/cube/cube.sock`.
**Honest gaps before this is real (both still open, from STARTUP-README §5):**
- (a) **cubefs directory model** — only `c<C>/z<Z>/y<Y>/x<X>` with x as fixed 3-digit leaf. No nested dirs, no arbitrary POSIX filenames, no rename/whiteout. Overlayfs rejects cubefs with EINVAL. A real OS tree can't be bound until this is done. Level-B crate feature.
- (b) **root fs / PID 1** — making the cube the literal rootfs is kernel work: custom initramfs + pivot_root, daemon or kernel module as init. Heavy, image-bake, off-peak. Current deployment: ext4 root + CZYX-call OS state = Phase 2 target.
**On "compile the new kernel to utilize CUBE":** your current kernel is vanilla v6.19.3 + ipu4 drivers (GRUB entry `LOCALVERSION=-ipu4p`); camera work is paused. Adding CUBE means either a kernel module against that kernel, or a new kernel build with CUBE integrated. That's a resource-heavy compile (saturates all 8 cores + memory). You've asked me not to auto-launch that while you're using the machine — I'll wait for your go.
## NEXT SESSION: put the OS in CUBE + compile new kernel to utilize CUBE
### What "put the OS in CUBE" means concretely (Phase 2 deepening)
The OS already writes its state (manifest, identity, klog, snapshots) as CZYX calls to `cube-server` over the socket — that's done and durable (verified across daemon restart). The next deepening is moving the cube from "userspace daemon the OS talks to over a socket" to "the kernel's own backing store" — i.e. the kernel module layer above.
### Concrete first step (before any kernel compile)
1. **Pick the kernel target.** Current: vanilla v6.19.3 + ipu4 (LOCALVERSION=-ipu4p). Option A: add CUBE as an out-of-tree module to that kernel. Option B: new kernel build with CUBE integrated. Which, and do you want to keep the ipu4 camera kernel as the base, or start fresh?
2. **Decide CUBE's kernel boundary.** What goes into the kernel vs stays in userspace?
- Definitely kernel: `cubecoords` (Czyx type, flags), `cubestore` core (the store abstraction + at least the in-memory backend; FileBackedStore + WAL/checkpoint is the durable path), coordinate→syscall mapping.
- Probably userspace ( daemon ): the socket server, remote client access, the "resume pointer" writer, snapshot management — i.e. the control plane, while the kernel is the data plane.
- `cubecode` (bytecode VM) and `cubecrypt` (seal/open transforms) — these are the debatable ones. VM in kernel = ability to run cube programs from kernel space (syscall-level). Crypto in kernel = sealed records openable by the kernel without a userspace daemon. Both are plausible; both add kernel surface area. Your call.
3. **cubefs directory model gap (a) above** — decide whether to close it before or after the kernel move. If the kernel uses the cube as backing store but cubefs can't represent a real OS tree (no nested dirs, no rename), then the OS-state-writing path (CZYX calls, not FUSE) still works, but a full rootfs bind is still blocked. Closing (a) is a crate feature, independent of the kernel move.
### Concrete second step (kernel compile — heavy, ask before launching)
4. **Build the kernel/module.** Depends on step 1-3. If out-of-tree module: copy the core crates into a kernel module skeleton, adapt for kernel constraints, build against the existing kernel headers. If integrated kernel: new kernel source with CUBE subsystem, config, compile. Either way: `make -j8` saturates all cores + memory — you've asked me not to auto-launch that while you're using the machine.
## FILES
- Workspace: `/home/CUBELinux/CUBELinux-2/`
- Canonical gate: `./check` (run from workspace root)
- Resume procedure: read this file + STARTUP-README.md + REPORT-INDEX.md, then `python3` against `/root/.hermes/state.db` for thread review.
- Cube-bench binary (release): `target/release/cube-bench`
- Current kernel GRUB entry: `LOCALVERSION=-ipu4p` (vanilla v6.19.3 + ipu4 drivers)
- System daemon socket: `/run/cube/cube.sock`
- luulu session daemon socket: `/run/user/1000/cube/cube.sock`
+71
View File
@@ -0,0 +1,71 @@
# RESUME — Cube as OS substrate (CZYX-call model)
Date: 2026-08-13 (session after RESUME-cubefs-daemon; Level A of item 3)
## Decision (user-directed)
"Everything writes as a CZYX call to the daemon — the cube-backed storage should
be coordinate calls, not writes to /cubefs. Why are we writing to cubefs?"
This REVERSES the earlier FUSE-path substrate design. Confirmed against
CUBELinux.pdf:
- Phase 1: user-space Cube OS on Linux, using existing kernels + filesystems.
- Phase 2: cubefs FUSE maps POSIX→CZYX for *testing semantics under real
workloads* — explicitly a probe, not the substrate itself.
- Phase 3: fork/extend kernel so VFS/accounting/LSM talk directly to the cube
store; new syscalls expose cube-native ops ("open by CZYX + flags").
- Package 3 (cubefs): "optional FUSE filesystem view so cube records appear as
files/directories for existing tools." → FUSE is OPTIONAL, not the substrate.
So: the coordinate store IS the persistence. POSIX/FUSE is a convenience view.
## What changed
1. `cube-os-state.sh` / `cube-os-snapshot.sh` / `cube-os-klog.sh` rewritten to
write via `cubec --socket /run/cube/cube.sock rawput/get` (CZYX calls), not
`/cubefs/...`. hex encoding uses python3 (xxd is absent in the VM image).
Coordinate layout (C=200):
c200/z001/y001/x001 boot manifest (overwrite each boot)
c200/z002/y001/x001 kernel boot line history (append)
c200/z003/y001/x001 machine/identity record
c200/z004/y001/x001 kernel log stream (klog appends lines)
c200/z010/y001/xNNN operational snapshot body (rolling counter)
c200/z010/y002/x001 rolling snapshot counter (persisted in cube, hex)
2. The three `cube-os-*` systemd units had `cubefs.service` REMOVED from
Requires/After — they now depend ONLY on `cube-server.service`. A FUSE mount
failure can no longer wedge OS bring-up.
3. `cubefs.service` hardened (ExecStartPre unmounts any stale mountpoint; BindsTo
cube-server) so it survives a daemon restart instead of crash-looping on
"Transport endpoint is not connected". This was the bug that broke the live
VM at the start of this session.
4. build_vm.sh updated to match (inline script bodies + unit edges). A fresh
bake now produces the CZYX-call substrate. NOT re-baked this session
(heavy ~24G, off-peak per user policy).
## Verified live (VM localhost:2222)
- Boot manifest reads: "CUBELINUX-VM boot manifest v3 / backing-store:
cube-server @ /run/cube/cube.sock (CZYX coordinate calls, no FUSE)".
- Snapshot #001 contains real OS state (running units, network, resources,
journal) — not a demo.
- Manifest + snapshot survive `systemctl restart cube-server` (durable across
daemon restart).
- `cube-os-state` + `cube-os-snapshot.timer` active; manual snapshot writes.
## Known gaps (honest)
- cubefs directory model: only `c<C>/z<Z>/y<Y>/x<X>` (x = fixed 3-digit leaf).
Arbitrary POSIX filenames (syslog, wtmp) and nested dirs are NOT addressable.
overlayfs on cubefs fails (EINVAL — missing RENAME_WHITEOUT/opaque-dir). So a
*real OS tree* on the cube is blocked until cubefs grows nested-dir support
(Level-B crate work).
- root fs / PID 1 (cube daemon as init, pivot_root into cubefs) = Phase 3 kernel
work, image-bake, off-peak only.
## Items left (from earlier list)
1. / 2. → done in prior sessions.
3. Cube as OS substrate: Level A DONE (CZYX-call OS state, this session).
Levels B (cube-backed block device) / C (rootfs+PID1) pending, not started.
4. society workspace: not started; git dubious ownership on
/home/cubelinux/society (needs safe.directory).
## Key files
- /root/build_vm.sh (authoritative bake template; all unit + script bodies live here)
- /home/CUBELinux/CUBELinux-2/STARTUP-README.md (updated §4 + provisioning note)
- VM guest: /opt/cube/bin/cube-os-*.sh, /etc/systemd/system/cube-os-*.{service,timer}
+46 -30
View File
@@ -47,7 +47,7 @@ is real. Each must be reproduced live, not asserted.
3. **Phase 3 syscall surface** — "open by CZYX + flags": a record addressable
directly by coordinate, not path. **SHIPPED 2026-08-13**:
(a) `cube open <path> <K.Z.Y.X> <tf>` and `cube seal ...` are first-class CLI
commands in `cubesys/src/bin/cube.rs` (reachable in-process + over the durable
commands in `cubecli/src/main.rs` (reachable in-process + over the durable
daemon). Verified: dispatch reaches the crypto/run layer.
(b) **`.czyx.C.Z.Y.X` FUSE magic-prefix (2026-08-13)** — `cubefs/src/path.rs`
`parse_dot_czyx` + `cubefs/src/fuse.rs::lookup` intercept: a path component
@@ -58,19 +58,23 @@ is real. Each must be reproduced live, not asserted.
3358720263; `cat .czyx.200.50.1.7` returns the record content. Unit tests in
`path.rs` cover full/partial/rejected forms. **STATUS: DONE.**
4. **Boot substrate** — the VM brings up CUBELinux as a storage layer the rest of
the OS reads/writes through. **DONE (systemd-managed, 2026-08-13)**: units
`cube-os-state.service` (oneshot, writes boot manifest + machine identity +
boot history to `/cubefs/c200/z001|z002|z003`) and `cube-os-klog.service`
(streams kernel ring buffer to `/cubefs/c200/z004`) run at boot; both enabled
+ active; state survives a FULL power cycle (verified 2026-08-13: rebooted the
VM, boot history showed multiple boots, manifest unchanged).
**ALSO (2026-08-13): real OS operational data** — `cube-os-snapshot.service` +
`cube-os-snapshot.timer` (every 5 min + 30s after boot) write genuine OS state
(running units, network, resources, journal) into `/cubefs/c200/z010/y001/xNNN`
(rolling 3-digit counter, x-axis is u16→3-digit per cubefs path model).
Verified: snapshot survives daemon restart; the OS wrote a NEW snapshot on a
fresh boot after a full power cycle; inode == packed CZYX. **STATUS: DONE
(auxiliary layer; not yet the root fs).**
the OS reads/writes through. **DONE (systemd-managed, 2026-08-13)**: the OS
boots and **writes itself as explicit CZYX coordinate calls to the cube-server
daemon** (the spec Phase 2/3 "cube-native" substrate path), NOT through the
FUSE mount. Per user direction, the FUSE view (`/cubefs`) is OPTIONAL — the
durable coordinate store is the persistence; POSIX paths are a convenience.
- `cube-os-state.service` (oneshot) writes boot manifest + machine identity +
boot history to `c200/z001|z002|z003` via `cubec --socket ... rawput`.
- `cube-os-klog.service` streams the kernel ring buffer to `c200/z004` (CZYX).
- `cube-os-snapshot.service` + `cube-os-snapshot.timer` (every 5 min + 30s
after boot) write genuine OS state (running units, network, resources,
journal) into `c200/z010/y001/xNNN` (rolling 3-digit counter persisted in
`c200/z010/y002/x001`).
All three units depend ONLY on `cube-server.service` (the daemon socket), not
on `cubefs.service`, so a FUSE mount failure can never wedge OS bring-up.
Verified: manifest reads "backing-store: cube-server @ /run/cube/cube.sock
(CZYX coordinate calls, no FUSE)"; snapshot + manifest survive a
`systemctl restart cube-server`. **STATUS: DONE (auxiliary layer; not yet the root fs).**
---
@@ -153,24 +157,36 @@ missing.
random material on each boot → every sealed record became unopenable.
Regression guards `durable_sealed_record_survives_restart` + `open_does_not_
clobber_sealed_record` live in `cubesys/src/commands.rs`; `./check` is green.
- Boot substrate (next deepening): make the cube the OS's *default* storage for a
real tree — e.g. have a service write `/etc` or `/var/log` operational files
through the cube by default. Currently the cube holds OS *state* (manifest/
identity/klog/snapshots) but the root fs is still ext4. The magic-prefix
"open by CZYX" FUSE passthrough is the natural next step (a path like
`/cubefs/.czyx/200.1.1.1` resolves directly to the coordinate), and a
loop/overlay over a cube-backed file would let the OS treat the cube as a real
block device.
- Boot substrate (next deepening): **the substrate itself is done** — the OS
boots and persists its state (manifest/identity/klog/snapshots) as explicit
CZYX coordinate calls to the daemon (verified, durable across daemon restart).
Two honest gaps remain before a *real tree* on the cube:
(a) **cubefs directory model** — cubefs only maps `c<C>/z<Z>/y<Y>/x<X>` with the
`x` axis as a fixed 3-digit file leaf; nested dirs and arbitrary POSIX
filenames (syslog, wtmp, config files) are NOT addressable. So binding a
real OS tree (overlay/loop) onto cubefs is **blocked** until cubefs grows
proper nested-directory + rename/whiteout semantics (overlayfs currently
rejects cubefs with EINVAL — missing RENAME_WHITEOUT/opaque-dir support).
This is a Level-B crate feature, not a systemd change.
(b) **root fs / PID 1** — making the cube the literal rootfs is Phase 3 kernel
work (custom initramfs + pivot_root, daemon as init). Heavy, image-bake,
off-peak only. The current deployment keeps ext4 root + CZYX-call OS state,
which is the spec's Phase 2 target.
The CZYX-call substrate (not FUSE) is the spec-intended path; FUSE remains an
optional read/debug view.
- IMAGE PROVISIONING: `build_vm.sh` (host, /root/build_vm.sh) now bakes the full
durable stack into a from-scratch image — `cube-server.service` (daemon),
`cubefs.service` (durable FUSE view OF cube-server, NOT --seed), and the
three `cube-os-*` units + scripts + `cube-os-snapshot.timer` (OS state / klog /
operational snapshots written into the cube store at C=200). All enabled in the
chroot. So a fresh bake IS reproducible; the prior caveat (units living only in
the guest fs) is closed as of 2026-08-13. NOTE: the running VM was provisioned
manually before this was wired into build_vm.sh; re-running `build_vm.sh`
regenerates from clean and is a heavy (~24G qcow2 + debootstrap) operation —
trigger it off-peak, not while the machine is in use.
`cubefs.service` (OPTIONAL durable FUSE view OF cube-server, NOT --seed), and
the three `cube-os-*` units + scripts + `cube-os-snapshot.timer`. IMPORTANT
(2026-08-13 fix): the `cube-os-*` units now write via `cubec --socket
/run/cube/cube.sock rawput/get` (CZYX calls) and depend ONLY on
`cube-server.service` — `cubefs.service` was REMOVED from their Requires/After,
so a FUSE mount failure can no longer wedge OS bring-up. `cubefs.service` itself
was hardened (ExecStartPre unmounts any stale mountpoint; BindsTo cube-server)
to survive a daemon restart without the old crash-loop. All enabled in the
chroot. So a fresh bake IS reproducible; re-running `build_vm.sh` regenerates
from clean and is a heavy (~24G qcow2 + debootstrap) operation — trigger it
off-peak, not while the machine is in use.
- `cube-resume-pointer.service` is REAL (wired 2026-08-13): a oneshot that writes
the OS's "where to look to continue" into the cube at `c200/z011/y001/x001`
(last snapshot index + resume coordinate). Was a dead stub before (unit pointed
+27 -5
View File
@@ -116,19 +116,41 @@ fn main() {
1,
) / scale_n as f64;
// scan_prefix correctness: c takes values 0 or 1 only across [0,scale_n)
// exact expected count for c=0 and c=1 from the coord_for mapping
// scan_prefix correctness: coord_for spreads over [0,4G) so c can be
// 0..3 at scale_n=200k. Compute exact expected counts from the mapping.
let expected_c0 = if scale_n <= 0x10000 {
scale_n as usize
} else {
0x10000
0x10000_usize.min(scale_n as usize)
};
let c1_start = 0x10000_usize;
let c2_start = 0x20000_usize;
let c3_start = 0x30000_usize;
let scale_usize = scale_n as usize;
let expected_c1 = if scale_usize <= c1_start {
0
} else {
c2_start.min(scale_usize).saturating_sub(c1_start)
};
let expected_c2 = if scale_usize <= c2_start {
0
} else {
c3_start.min(scale_usize).saturating_sub(c2_start)
};
let expected_c3 = if scale_usize <= c3_start {
0
} else {
scale_usize.saturating_sub(c3_start)
};
let expected_c1 = scale_n as usize - expected_c0;
let got_c0 = store.scan_prefix(0, None, None).len();
let got_c1 = store.scan_prefix(1, None, None).len();
assert_eq!(got_c0, expected_c0, "scan_prefix c=0 wrong");
assert_eq!(got_c1, expected_c1, "scan_prefix c=1 wrong");
assert_eq!(got_c0 + got_c1, scale_n as usize, "prefix covers all");
assert_eq!(
got_c0 + got_c1 + expected_c2 + expected_c3,
scale_n as usize,
"prefix covers all"
);
let scan_ns = time_ns(
|| {
+10
View File
@@ -0,0 +1,10 @@
[package]
name = "cube-mvw"
version = "0.1.0"
edition = "2021"
[dependencies]
cubecoords = { path = "../cubecoords" }
cubestore = { path = "../cubestore" }
[[bin]]
name = "cube-mvw"
path = "src/main.rs"
+172
View File
@@ -0,0 +1,172 @@
//! A real CUBELinux DB backend: a sharded, multi-writer store with MVCC
//! (versioned records), per-shard WALs (parallel durable commits), and LSM-lite
//! compaction (memtable + WAL delta + compacted checkpoint base).
//!
//! `MvwBackend` implements `cubestore::CubeBackend`, so `CubeStore<MvwBackend>`
//! is a durable, concurrent, multi-writer CUBELinux database store.
use cubecoords::Czyx;
use cubestore::CubeBackend;
use std::collections::HashMap;
use std::fs::{File, OpenOptions};
use std::io::{BufWriter, Read, Write};
use std::path::{Path, PathBuf};
use std::sync::atomic::{AtomicU64, Ordering};
use std::sync::{Mutex, RwLock};
#[derive(Clone, Debug)]
struct Version { commit_ts: u64, value: Vec<u8>, deleted: bool }
struct Shard {
map: RwLock<HashMap<u32, Vec<Version>>>, // memtable, keyed by packed Czyx
wal: Mutex<BufWriter<File>>,
wal_path: PathBuf,
ckpt_path: PathBuf,
}
pub struct MvwBackend {
shards: Vec<Shard>,
next_ts: AtomicU64,
}
fn pack(k: &Czyx) -> u32 { k.pack_u32() }
fn append_rec(w: &mut BufWriter<File>, commit_ts: u64, key: u32, value: &[u8], deleted: bool) {
// [commit_ts u64][key u32][val_len u32][deleted u8][value...]
let mut buf = Vec::with_capacity(25 + value.len());
buf.extend_from_slice(&commit_ts.to_le_bytes());
buf.extend_from_slice(&key.to_le_bytes());
buf.extend_from_slice(&(value.len() as u32).to_le_bytes());
buf.push(deleted as u8);
buf.extend_from_slice(value);
buf.push(b'\n');
w.write_all(&buf).unwrap();
w.flush().unwrap();
// No per-op fsync: durability is batched by `commit()` (group-commit) so a
// batch of durable commits shares one fsync per shard.
}
impl MvwBackend {
pub fn open(nshards: usize, base: &Path) -> std::io::Result<Self> {
let mut shards = Vec::with_capacity(nshards);
for i in 0..nshards {
let wal_path = PathBuf::from(format!("{}-s{}.log", base.display(), i));
let ckpt_path = PathBuf::from(format!("{}-s{}.ckpt", base.display(), i));
let wal = OpenOptions::new().create(true).append(true).open(&wal_path)?;
shards.push(Shard { map: RwLock::new(HashMap::new()), wal: Mutex::new(BufWriter::new(wal)), wal_path, ckpt_path });
}
let b = MvwBackend { shards, next_ts: AtomicU64::new(1) };
b.recover()?;
Ok(b)
}
fn shard(&self, k: &Czyx) -> &Shard { &self.shards[(pack(k) as usize) % self.shards.len()] }
fn decode_line(rec: &[u8]) -> Option<(u64, u32, Vec<u8>, bool)> {
if rec.len() < 21 { return None; }
let commit_ts = u64::from_le_bytes(rec[0..8].try_into().ok()?);
let key = u32::from_le_bytes(rec[8..12].try_into().ok()?);
let vlen = u32::from_le_bytes(rec[12..16].try_into().ok()?) as usize;
let deleted = rec[16] != 0;
let value = rec[17..17 + vlen.min(rec.len().saturating_sub(17))].to_vec();
Some((commit_ts, key, value, deleted))
}
fn load(&self, path: &Path, live: &mut HashMap<u32, Vec<Version>>) -> std::io::Result<()> {
if !path.exists() { return Ok(()); }
let mut f = File::open(path)?; let mut data = Vec::new(); f.read_to_end(&mut data)?;
for rec in data.split(|&b| b == b'\n') {
if let Some((c, k, v, del)) = Self::decode_line(rec) { live.entry(k).or_default().push(Version { commit_ts: c, value: v, deleted: del }); }
}
Ok(())
}
pub fn snapshot_ts(&self) -> u64 { self.next_ts.load(Ordering::SeqCst).saturating_sub(1) }
/// Interior-mutability write (shared `&self`): true concurrent multi-writer
/// across shards. Durable in this shard's WAL; MVCC versioned.
/// Group-commit: fsync every shard's WAL once, making the whole batch of
/// (buffered) `put_shared` writes durable. One fsync per shard for many ops.
pub fn commit(&self) -> std::io::Result<()> {
for sh in &self.shards {
let w = sh.wal.lock().unwrap();
w.get_ref().sync_data().unwrap();
}
Ok(())
}
pub fn put_shared(&self, key: Czyx, value: Vec<u8>) {
let commit_ts = self.next_ts.fetch_add(1, Ordering::SeqCst);
let sh = self.shard(&key);
{ let mut w = sh.wal.lock().unwrap(); append_rec(&mut w, commit_ts, pack(&key), &value, false); }
sh.map.write().unwrap().entry(pack(&key)).or_default().push(Version { commit_ts, value, deleted: false });
}
pub fn recover(&self) -> std::io::Result<()> {
let mut max = 0u64;
for sh in &self.shards {
let mut m = HashMap::new();
self.load(&sh.ckpt_path, &mut m)?; self.load(&sh.wal_path, &mut m)?;
for vers in m.values() { for v in vers { max = max.max(v.commit_ts + 1); } }
let mut live = sh.map.write().unwrap(); live.extend(m);
}
if max > 0 { self.next_ts.store(max, Ordering::SeqCst); }
Ok(())
}
/// LSM-lite compaction: fold each shard's latest non-deleted version into its
/// checkpoint base, then truncate the shard WAL.
pub fn compact(&self) -> std::io::Result<()> {
for sh in &self.shards {
let m = sh.map.read().unwrap();
let mut out = Vec::new();
for (k, vers) in m.iter() {
if let Some(latest) = vers.last() {
if !latest.deleted {
let mut buf = Vec::with_capacity(21 + latest.value.len());
buf.extend_from_slice(&latest.commit_ts.to_le_bytes());
buf.extend_from_slice(&k.to_le_bytes());
buf.extend_from_slice(&(latest.value.len() as u32).to_le_bytes());
buf.push(0u8);
buf.extend_from_slice(&latest.value);
buf.push(b'\n');
out.extend_from_slice(&buf);
}
}
}
let mut ck = OpenOptions::new().create(true).write(true).truncate(true).open(&sh.ckpt_path)?;
ck.write_all(&out)?; ck.sync_all()?;
let mut w = sh.wal.lock().unwrap();
w.get_ref().set_len(0)?; w.get_ref().sync_all()?; w.flush()?;
}
Ok(())
}
}
impl CubeBackend for MvwBackend {
fn put(&mut self, key: Czyx, value: Vec<u8>) {
self.put_shared(key, value);
let _ = self.commit();
}
fn put_checked(&mut self, key: Czyx, value: Vec<u8>) -> Result<(), String> {
self.put(key, value); Ok(())
}
fn get(&self, key: &Czyx) -> Option<Vec<u8>> {
let sh = self.shard(key);
let m = sh.map.read().unwrap();
// latest committed non-deleted version
m.get(&pack(key)).and_then(|v| v.iter().rev().find(|ver| !ver.deleted).map(|ver| ver.value.clone()))
}
fn delete(&mut self, key: &Czyx) {
let commit_ts = self.next_ts.fetch_add(1, Ordering::SeqCst);
let sh = self.shard(key);
{ let mut w = sh.wal.lock().unwrap(); append_rec(&mut w, commit_ts, pack(key), &[], true); }
let mut m = sh.map.write().unwrap();
m.entry(pack(key)).or_default().push(Version { commit_ts, value: Vec::new(), deleted: true });
}
fn keys(&self) -> Vec<Czyx> {
let mut out = Vec::new();
for sh in &self.shards {
let m = sh.map.read().unwrap();
for (k, vers) in m.iter() {
if let Some(c) = vers.last() { if !c.deleted { out.push(Czyx::unpack_u32(*k)); } }
}
}
out
}
}
+35
View File
@@ -0,0 +1,35 @@
use cube_mvw::MvwBackend;
use cubecoords::Czyx;
use std::path::Path;
use std::time::Instant;
use std::thread;
fn bench(mode: &str) -> f64 {
let b = Path::new("/tmp/mvw-gc");
for i in 0..16 { std::fs::remove_file(format!("{}-s{}.log", b.display(), i)).ok(); std::fs::remove_file(format!("{}-s{}.ckpt", b.display(), i)).ok(); }
let back = MvwBackend::open(16, b).unwrap();
let threads = 8usize; let per = 4000u64; let n = (threads as f64) * per as f64;
let t = Instant::now();
thread::scope(|sc| {
for th in 0..threads {
let back = &back;
sc.spawn(move || for i in 0..per {
let k = Czyx::unpack_u32(((th as u64 * per + i) as u32) << 8 | 1);
back.put_shared(k, vec![i as u8; 16]);
});
}
});
if mode == "batch" { back.commit().unwrap(); }
let s = t.elapsed().as_secs_f64();
for i in 0..16 { std::fs::remove_file(format!("{}-s{}.log", b.display(), i)).ok(); std::fs::remove_file(format!("{}-s{}.ckpt", b.display(), i)).ok(); }
n / s / 1e3
}
fn main() {
let per_op = bench("per_op");
let batch = bench("batch");
println!("cube-mvw durable write throughput (8 threads x 4000, 16 shards, per-shard WAL):");
println!(" per-op fsync : {:>8.1} k puts/s", per_op);
println!(" group-commit : {:>8.1} k puts/s ({:.1}x)", batch, batch/per_op);
println!("\n=> group-commit batches one fsync per shard for many ops => durable writes scale with the");
println!(" batch (no tables layer — pure storage engine; CUBE's metatag model is the schema).");
}
+64
View File
@@ -0,0 +1,64 @@
use cube_mvw::MvwBackend;
use cubecoords::{CubeHeader, Czyx};
use cubestore::{CubeBackend, CubeStore};
use std::path::{Path, PathBuf};
use std::fs;
use std::thread;
use std::sync::Arc;
fn base(name: &str) -> PathBuf { std::env::temp_dir().join(name) }
fn cleanup(b: &Path) { for i in 0..16 { let _=fs::remove_file(format!("{}-s{}.log",b.display(),i)); let _=fs::remove_file(format!("{}-s{}.ckpt",b.display(),i)); } }
#[test]
fn backend_durable_mvcc_and_concurrent() {
let b = base("mvw-backend"); cleanup(&b);
{
let back = MvwBackend::open(16, &b).unwrap();
back.put_shared(Czyx::new(1,1,1,1), b"hello".to_vec());
// MVCC: a later write is the latest visible version
back.put_shared(Czyx::new(1,1,1,1), b"world".to_vec());
assert_eq!(back.get(&Czyx::new(1,1,1,1)), Some(b"world".to_vec()));
}
// durability: drop (no graceful shard flush), reopen from per-shard WALs
let back = MvwBackend::open(16, &b).unwrap();
let latest = back.get(&Czyx::new(1,1,1,1)).unwrap();
assert!(latest == b"world".to_vec(), "WAL recovery must restore the latest MVCC version");
// keys enumeration
back.put_shared(Czyx::new(2,2,2,2), b"a".to_vec());
assert!(back.keys().contains(&Czyx::new(2,2,2,2)));
// true concurrent multi-writer (shared &self put_shared), 8 threads x 2000
let shared = Arc::new(MvwBackend::open(16, &b).unwrap());
let mut handles = Vec::new();
for t in 0..8u64 {
let s = shared.clone();
handles.push(thread::spawn(move || for i in 0..2000u64 {
let k = Czyx::unpack_u32(((t * 2000 + i) as u32) << 8 | 1);
s.put_shared(k, vec![(i as u8); 8]);
}));
}
for h in handles { h.join().unwrap(); }
// all writes readable
for t in 0..8u64 { for i in 0..2000u64 {
assert!(shared.get(&Czyx::unpack_u32(((t*2000+i) as u32) << 8 | 1)).is_some());
}}
// compact folds memtable to checkpoint and truncates WALs; values still readable
shared.compact().unwrap();
assert!(shared.get(&Czyx::new(1,1,1,1)).is_some());
cleanup(&b);
let _ = thread::current(); // silence
}
#[test]
fn cubestore_record_api_over_mvw_backend() {
let b = base("mvw-cubestore"); cleanup(&b);
let mut store = CubeStore::new(MvwBackend::open(16, &b).unwrap());
let mut h = CubeHeader::new();
h.title = Some("t".into());
h.refresh_flags();
store.put_record(Czyx::new(3,3,3,3), &h, b"body");
let (h2, body) = store.get_record(&Czyx::new(3,3,3,3)).expect("record roundtrip");
assert_eq!(body, b"body");
assert_eq!(h2.title.as_deref(), Some("t"));
cleanup(&b);
}
+182
View File
@@ -0,0 +1,182 @@
#!/usr/bin/env python3
"""cube-notes-mcp — MCP (stdio) server exposing the CUBELinux-2 notes CLI as
model-facing tools `mcp__cubenotes__note_*` / `mcp__cubenotes__project_*`.
Wraps `cube note ...` / `cube project ...` (the durable WordFlags
message-save-path with categories + project threads). The harness spawns this
server via the `mcp-client` plugin; each tool call runs the `cube` binary,
which persists to `$CUBE_NOTES_DIR` (default ~/.cubelinux-notes).
"""
import sys, json, subprocess
CUBE = "/home/CUBEdb/target/release/cube"
SERVER_NAME = "cube-notes-mcp"
SERVER_VERSION = "0.2.0"
PROTOCOL_VERSION = "2024-11-05"
def tool_schemas():
return [
{
"name": "note_write",
"description": "Add a note to the durable WordFlags message-save-path. "
"Optional: project P associates it to a project thread, "
"category C categorises it (doc_type note:C). Returns the coord.",
"inputSchema": {"type": "object", "properties": {
"text": {"type": "string"},
"project": {"type": "string", "description": "Project name (thread)."},
"category": {"type": "string", "description": "e.g. design, impl, cube."},
}, "required": ["text"]},
},
{
"name": "note_list",
"description": "List notes (newest first), optionally filtered by session, project, category, since/until (YYYY-MM-DD).",
"inputSchema": {"type": "object", "properties": {
"session": {"type": "string"},
"project": {"type": "string"},
"category": {"type": "string"},
"since": {"type": "string"},
"until": {"type": "string"},
}},
},
{
"name": "note_search",
"description": "Substring-search notes, optionally scoped to a project/category.",
"inputSchema": {"type": "object", "properties": {
"query": {"type": "string"},
"project": {"type": "string"},
"category": {"type": "string"},
}, "required": ["query"]},
},
{
"name": "note_show",
"description": "Show one note by C.Z.Y.X.",
"inputSchema": {"type": "object", "properties": {
"coord": {"type": "string"},
}, "required": ["coord"]},
},
{
"name": "project_list",
"description": "List all project threads (doc_type=project).",
"inputSchema": {"type": "object", "properties": {}},
},
{
"name": "project_show",
"description": "Walk a project's associated notes (its thread/timeline) by C.Z.Y.X.",
"inputSchema": {"type": "object", "properties": {
"coord": {"type": "string"},
}, "required": ["coord"]},
},
{
"name": "project_resume",
"description": "Set a resume marker for a project (where it left off).",
"inputSchema": {"type": "object", "properties": {
"name": {"type": "string"}, "text": {"type": "string"},
}, "required": ["name", "text"]},
},
{
"name": "project_context",
"description": "Compact 'where we are' summary for a project (latest notes + resume marker) — for context injection.",
"inputSchema": {"type": "object", "properties": {
"name": {"type": "string"},
}, "required": ["name"]},
},
]
def run_cube(args):
try:
r = subprocess.run([CUBE] + args, capture_output=True, text=True, timeout=30)
out = (r.stdout or "").strip()
if r.returncode != 0:
return f"error: {(r.stderr or out).strip()}"
return out if out else "(no result)"
except Exception as e: # noqa: BLE001
return f"error: {e}"
def opt(flag, val):
return [flag, str(val)] if val else []
def dispatch(name, args):
args = args or {}
if name == "note_write":
text = (args.get("text") or "").strip()
if not text:
return "error: missing 'text'"
return run_cube(["note", "add"] + opt("--project", args.get("project"))
+ opt("--cat", args.get("category")) + [text])
if name == "note_list":
cmd = ["note", "list"]
cmd += opt("--project", args.get("project"))
cmd += opt("--cat", args.get("category"))
cmd += opt("--since", args.get("since"))
cmd += opt("--until", args.get("until"))
if args.get("session"):
cmd.append(args["session"])
return run_cube(cmd)
if name == "note_search":
q = (args.get("query") or "").strip()
if not q:
return "error: missing 'query'"
return run_cube(["note", "search", q] + opt("--project", args.get("project"))
+ opt("--cat", args.get("category")))
if name == "note_show":
c = (args.get("coord") or "").strip()
return run_cube(["note", "show", c]) if c else "error: missing 'coord'"
if name == "project_list":
return run_cube(["project", "list"])
if name == "project_show":
c = (args.get("coord") or "").strip()
return run_cube(["project", "show", c]) if c else "error: missing 'coord'"
if name == "project_resume":
nm = (args.get("name") or "").strip()
tx = (args.get("text") or "").strip()
if not nm or not tx:
return "error: project_resume needs name + text"
return run_cube(["project", "resume", nm, tx])
if name == "project_context":
nm = (args.get("name") or "").strip()
return run_cube(["project", "context", nm]) if nm else "error: missing 'name'"
return f"error: unknown tool {name}"
def handle(msg):
if not isinstance(msg, dict):
return None
mid, method = msg.get("id"), msg.get("method")
if method == "initialize":
return {"jsonrpc": "2.0", "id": mid, "result": {
"protocolVersion": PROTOCOL_VERSION,
"capabilities": {"tools": {}},
"serverInfo": {"name": SERVER_NAME, "version": SERVER_VERSION},
}}
if method == "ping":
return {"jsonrpc": "2.0", "id": mid, "result": {}}
if method in ("notifications/initialized", "initialized"):
return None
if method == "tools/list":
return {"jsonrpc": "2.0", "id": mid, "result": {"tools": tool_schemas()}}
if method == "tools/call":
name = msg.get("params", {}).get("name", "")
args = msg.get("params", {}).get("arguments", {})
return {"jsonrpc": "2.0", "id": mid, "result": {
"content": [{"type": "text", "text": dispatch(name, args)}], "isError": False}}
if mid is not None:
return {"jsonrpc": "2.0", "id": mid, "result": {}}
return None
for line in sys.stdin:
line = line.strip()
if not line:
continue
try:
msg = json.loads(line)
except Exception: # noqa: BLE001
continue
resp = handle(msg)
if resp is not None:
sys.stdout.write(json.dumps(resp) + "\n")
sys.stdout.flush()
+12
View File
@@ -0,0 +1,12 @@
[package]
name = "cubeai"
version = "0.1.0"
edition.workspace = true
license.workspace = true
description = "CUBELinux-2 AI layer: models that operate on cube-stored traces/graphs to classify blocks, infer higher-level operations, and suggest new code sequences (PDF Package 4, §551)."
[dependencies]
cubecoords = { path = "../cubecoords" }
cubestore = { path = "../cubestore" }
cubecode = { path = "../cubecode" }
cubedbt = { path = "../cubedbt" }
+329
View File
@@ -0,0 +1,329 @@
//! CUBELinux-2 AI layer.
//!
//! Per the PDF (Package 4, §551): *"cubeai: models that operate on
//! cubestored traces/graphs to classify blocks, infer higherlevel
//! operations, and suggest new code sequences or orchestrations."*
//!
//! This crate is the *structured decision* layer sitting on top of the
//! substrate (and on [`cubedbt`]). It is deterministic and dependency-free
//! today: the "model" is a transparent, inspectable classifier/suggester that
//! operates on the same [`CubeStore`] / [`CodeCell`] / [`Behavior`] types the
//! rest of the workspace uses. A learned model can later implement the same
//! traits without changing callers.
//!
//! Pipeline (the end-to-end story from §549–§551 + §554):
//! trace (cube) → classify blocks → infer higher-level op →
//! suggest `TranslationRule`s → hand to `cubedbt` to patch + run (mimic).
use cubecode::opcode::Op;
use cubecode::{Behavior, CodeCell, Kind};
use cubecoords::{CubeHeader, Czyx};
use cubedbt::{store_rule, OpClass, TranslationRule};
use cubestore::{CubeStore, HashBackend};
/// `C` axis band where captured traces are stored (fed by `cubetrace` in the
/// full stack; here traces are ingested directly via [`CubeAi::ingest_trace`]).
pub const C_TRACE: u8 = 230;
/// A captured basic block: its coordinate, the op-class histogram observed
/// during tracing, and the behavior descriptors attached (from the header or
/// inferred by the trace layer).
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct BlockTrace {
pub label: Czyx,
/// Count of each [`OpClass`] seen in the block (`OpClass::Any` unused).
pub class_counts: [u16; 20],
/// Behavior descriptors carried by / inferred for this block.
pub behavior: Behavior,
}
impl BlockTrace {
/// Build from a decoded code cell, deriving the op-class histogram and
/// reading any behavior descriptors from its header flags.
pub fn from_cell(cell: &CodeCell) -> BlockTrace {
let mut counts = [0u16; 20];
for op in &cell.code {
let c = OpClass::of(op) as usize;
if c < 20 {
counts[c] += 1;
}
}
BlockTrace {
label: cell.label,
class_counts: counts,
behavior: Behavior::from_flags(cell.header.flags.bits()),
}
}
/// Total op count.
pub fn total(&self) -> u32 {
self.class_counts.iter().map(|&c| c as u32).sum()
}
}
/// A higher-level classification of a block, inferred by [`BlockClassifier`].
#[derive(Copy, Clone, Debug, PartialEq, Eq)]
pub enum BlockKind {
/// Mostly arithmetic/logic — a computation block.
Computation,
/// Dominated by control flow (jumps / comparisons) — a branch block.
Branch,
/// Contains call links to other records — a call/composition block.
CallTrampoline,
/// Heavy I/O or network descriptors — an I/O section.
IoSection,
/// Otherwise: a plain linear sequence.
Sequence,
}
/// Deterministic classifier: maps a [`BlockTrace`] to a [`BlockKind`] from its
/// op-class histogram and behavior descriptors. (The transparent "model".)
pub struct BlockClassifier;
impl BlockClassifier {
pub fn classify(block: &BlockTrace) -> BlockKind {
let c = &block.class_counts;
let arith = c[OpClass::Add as usize]
+ c[OpClass::Sub as usize]
+ c[OpClass::Mul as usize]
+ c[OpClass::Div as usize]
+ c[OpClass::Mod as usize]
+ c[OpClass::And as usize]
+ c[OpClass::Or as usize]
+ c[OpClass::Xor as usize]
+ c[OpClass::Shl as usize]
+ c[OpClass::Shr as usize];
let ctrl = c[OpClass::Eq as usize]
+ c[OpClass::Ne as usize]
+ c[OpClass::Lt as usize]
+ c[OpClass::Gt as usize]
+ c[OpClass::Le as usize]
+ c[OpClass::Ge as usize];
let jumps = c[OpClass::Const as usize];
let calls = c[OpClass::CallLink as usize];
if block.behavior.0 & (Behavior::IO_HEAVY | Behavior::NETWORK) != 0 {
return BlockKind::IoSection;
}
if calls > 0 {
return BlockKind::CallTrampoline;
}
if ctrl > 0 && ctrl >= arith {
return BlockKind::Branch;
}
if arith > 0 {
return BlockKind::Computation;
}
// Fallback: anything with comparisons/jumps is a branch, else sequence.
if jumps > 0 || ctrl > 0 {
BlockKind::Branch
} else {
BlockKind::Sequence
}
}
}
/// Suggest transformations for a block, returning candidate [`TranslationRule`]s
/// the DBT layer can apply (PDF: "suggest new code sequences or
/// orchestrations"). Deterministic and local: each suggestion records *why*.
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct Suggestion {
pub rule: TranslationRule,
pub rationale: String,
}
/// The AI runtime over a cube store: ingests traces, classifies, and suggests
/// DBT rules. Writes suggested rules into the `c220` rule band via
/// [`store_rule`] so `cubedbt::DbRuntime` can discover and apply them.
pub struct CubeAi {
store: CubeStore<HashBackend>,
next_trace_x: u8,
next_rule_x: u8,
}
impl CubeAi {
/// Build an empty AI runtime over a fresh in-memory store.
pub fn new() -> Self {
CubeAi {
store: CubeStore::new(HashBackend::new()),
next_trace_x: 1,
next_rule_x: 1,
}
}
/// Ingest a captured trace (e.g. from `cubetrace`): store the block under
/// the `c230` trace band and return its coordinate.
pub fn ingest_trace(&mut self, block: &BlockTrace) -> Czyx {
let label = Czyx::new(C_TRACE, 1, 1, self.next_trace_x);
self.next_trace_x = self.next_trace_x.wrapping_add(1).max(1);
let mut h = CubeHeader::new();
h.title = Some(format!("trace:{:?}", block.label));
h.doc_type = Some(Kind::Other.as_str().into());
h.size_bytes = Some(block.class_counts.len() as u64 * 2);
h.flags.0 |= block.behavior.to_flags();
h.refresh_flags();
// Body: the raw histogram (20 x u16 le).
let mut body = Vec::with_capacity(40);
for c in &block.class_counts {
body.extend_from_slice(&c.to_le_bytes());
}
self.store.put_record(label, &h, &body);
label
}
/// Classify a single block.
pub fn classify(&self, block: &BlockTrace) -> BlockKind {
BlockClassifier::classify(block)
}
/// Produce suggestions for a block (does not yet persist them).
pub fn suggest(&self, block: &BlockTrace) -> Vec<Suggestion> {
let kind = BlockClassifier::classify(block);
let mut out = Vec::new();
match kind {
BlockKind::Computation => {
out.push(Suggestion {
rule: TranslationRule {
name: format!("compute-opt:{:?}", block.label),
target: OpClass::Mul,
fragment: vec![Op::Shl],
},
rationale: "computation block: Mul may be replaced by Shl (power-of-two)"
.into(),
});
}
BlockKind::Branch => {
out.push(Suggestion {
rule: TranslationRule {
name: format!("branch-opt:{:?}", block.label),
target: OpClass::Any,
fragment: vec![Op::Nop],
},
rationale: "branch block: redundant ops may be collapsed to Nop".into(),
});
}
BlockKind::IoSection => {
out.push(Suggestion {
rule: TranslationRule {
name: format!("io-batch:{:?}", block.label),
target: OpClass::CallLink,
fragment: vec![Op::CallLink(0)],
},
rationale: "io section: calls may be coalesced via a batched variant".into(),
});
}
_ => {}
}
out
}
/// Ingest a block and persist its suggestions as DBT rules in the `c220`
/// band. Returns the stored rule coordinates (empty if no suggestions).
pub fn ingest_and_suggest(&mut self, block: &BlockTrace) -> Vec<Czyx> {
self.ingest_trace(block);
let sugs = self.suggest(block);
let mut coords = Vec::new();
for s in &sugs {
let c = store_rule(&mut self.store, &s.rule, self.next_rule_x);
self.next_rule_x = self.next_rule_x.wrapping_add(1).max(1);
coords.push(c);
}
coords
}
/// Borrow the backing store (e.g. to hand to `cubedbt::DbRuntime`).
pub fn store(&self) -> &CubeStore<HashBackend> {
&self.store
}
}
impl Default for CubeAi {
fn default() -> Self {
Self::new()
}
}
#[cfg(test)]
mod tests {
use super::*;
use cubecode::opcode::Op;
use cubecoords::Czyx;
fn cell(coord: Czyx, code: Vec<Op>, beh: Behavior) -> CodeCell {
let mut h = CubeHeader::new();
h.title = Some("blk".into());
h.doc_type = Some(Kind::Fn.as_str().into());
h.flags.0 |= beh.to_flags();
h.refresh_flags();
CodeCell::from_record(coord, &h, &cubecode::opcode::encode(&code))
.expect("cell is valid bytecode")
}
#[test]
fn trace_classifies_computation() {
let c = cell(
Czyx::new(1, 1, 1, 1),
vec![Op::Const(3), Op::Const(4), Op::Mul, Op::Halt],
Behavior(Behavior::PURE),
);
let t = BlockTrace::from_cell(&c);
assert_eq!(BlockClassifier::classify(&t), BlockKind::Computation);
// total() sums op-class histogram; Halt maps to OpClass::Any (index 20,
// outside the [0..20) histogram), so 3 counted ops (Const, Const, Mul).
assert_eq!(t.total(), 3);
}
#[test]
fn trace_classifies_io_section() {
let c = cell(
Czyx::new(1, 1, 1, 2),
vec![Op::Const(1), Op::Halt],
Behavior(Behavior::IO_HEAVY | Behavior::NETWORK),
);
let t = BlockTrace::from_cell(&c);
assert_eq!(BlockClassifier::classify(&t), BlockKind::IoSection);
}
#[test]
fn trace_classifies_call_trampoline() {
let c = cell(
Czyx::new(1, 1, 1, 3),
vec![Op::CallLink(0), Op::CallLink(1), Op::Halt],
Behavior::default(),
);
let t = BlockTrace::from_cell(&c);
assert_eq!(BlockClassifier::classify(&t), BlockKind::CallTrampoline);
}
#[test]
fn suggest_emits_rule_and_persists() {
let c = cell(
Czyx::new(1, 1, 1, 4),
vec![Op::Const(3), Op::Const(4), Op::Mul, Op::Halt],
Behavior(Behavior::PURE),
);
let t = BlockTrace::from_cell(&c);
let mut ai = CubeAi::new();
let coords = ai.ingest_and_suggest(&t);
assert_eq!(coords.len(), 1, "computation block suggests one rule");
assert_eq!(coords[0].c, rule_band());
// The stored rule is discoverable by cubedbt.
let rt = cubedbt::DbRuntime::new(ai.store().clone());
assert_eq!(rt.discover_rules().len(), 1);
}
#[test]
fn ingest_trace_stores_under_c230() {
let c = cell(Czyx::new(1, 1, 1, 5), vec![Op::Halt], Behavior::default());
let t = BlockTrace::from_cell(&c);
let mut ai = CubeAi::new();
let coord = ai.ingest_trace(&t);
assert_eq!(coord.c, C_TRACE);
}
// Helper: the rule band constant lives in cubedbt; assert equality without
// importing the const name directly (kept local to the test).
fn rule_band() -> u8 {
cubedbt::C_DBT_RULE
}
}
+18
View File
@@ -0,0 +1,18 @@
[package]
name = "cubecli"
version = "0.1.0"
edition.workspace = true
license.workspace = true
description = "CUBELinux-2 local system CLI: in-process REPL + script runner over one shared CubeStore (cubefs + cubecode + cubecrypt)."
[[bin]]
name = "cube"
path = "src/main.rs"
[dependencies]
cubecoords = { path = "../cubecoords" }
cubestore = { path = "../cubestore" }
cubefs = { path = "../cubefs" }
cubecode = { path = "../cubecode" }
cubecrypt = { path = "../cubecrypt" }
cubesys = { path = "../cubesys" }
@@ -48,15 +48,49 @@ fn is_command_word(w: &str) -> bool {
| "seal"
| "open"
| "keyinit"
| "scan-word-flags"
| "show-flags"
| "trace-capture"
| "trace-replay"
| "golden-capture"
| "trace-verify"
| "trace-list"
| "trace-links"
| "query"
| "begin"
| "commit"
| "rollback"
| "stats"
| "audit"
| "note"
| "notes"
| "project"
| "projects"
)
}
/// Open (or create) the durable session-note store. Notes persist across CLI
/// invocations and sessions — the CUBE "message save path" — via the WAL +
/// checkpoint store. The directory is `$CUBE_NOTES_DIR` (`~/.cubelinux-notes`).
fn open_notes_store() -> cubesys::store::ConcurrentStore {
let dir = std::env::var("CUBE_NOTES_DIR").unwrap_or_else(|_| {
format!(
"{}/.cubelinux-notes",
std::env::var("HOME").unwrap_or_else(|_| "/tmp".to_string())
)
});
cubesys::store::ConcurrentStore::open(
&format!("{dir}/notes.store"),
&format!("{dir}/notes.wal"),
&format!("{dir}/recovery.log"),
cubesys::store::DurabilityConfig::default(),
)
.unwrap_or_else(|e| {
eprintln!("error: cannot open notes store {dir}: {e}");
std::process::exit(1);
})
}
fn main() {
let args: Vec<String> = std::env::args().collect();
match args.get(1).map(|s| s.as_str()) {
@@ -67,12 +101,33 @@ fn main() {
// the coordinate-addressed API a real CLI surface the OS can call.
Some(word) if is_command_word(word) => {
let line = args[1..].join(" ");
let mut session = Session::new();
match session.exec(&line) {
Ok(out) => println!("{out}"),
Err(e) => {
eprintln!("error: {e}");
std::process::exit(1);
// Session notes live in their own DURABLE store so a note written
// in one session is reviewable in the next (the message save path).
if word == "note" || word == "notes" || word == "project" || word == "projects" {
let store = open_notes_store();
let res = if word == "project" || word == "projects" {
cubesys::notes::project_command(&store, &line)
} else {
cubesys::notes::note_command(&store, &line)
};
// Force a durable flush/checkpoint before exiting so the note is
// persisted even though this process (the CLI) is short-lived.
store.checkpoint();
match res {
Ok(out) => println!("{out}"),
Err(e) => {
eprintln!("error: {e}");
std::process::exit(1);
}
}
} else {
let mut session = Session::new();
match session.exec(&line) {
Ok(out) => println!("{out}"),
Err(e) => {
eprintln!("error: {e}");
std::process::exit(1);
}
}
}
}
@@ -136,6 +191,11 @@ fn print_help() {
stat <path> getattr via cubefs\n \
seal <path> <K.Z.Y.X|auto> <tf> encrypt a record (tf: none|gcm|chacha|xts)\n \
open <path> <K.Z.Y.X|auto> <tf> decrypt + decode + run a sealed record\n \
keyinit ensure the OS Null-space keystore exists\n"
keyinit ensure the OS Null-space keystore exists\n \
note <text> add a session note (durable, WordFlags-tagged)\n \
note list [session] list notes (default: today's session)\n \
note search <text> search notes across sessions\n \
note show <C.Z.Y.X> show one note\n \
notes alias for `note list`\n"
);
}
+215
View File
@@ -0,0 +1,215 @@
//! Feature-vector ball / k-NN index: a metric VP-tree stored as CUBE records.
//! Each node is a feature-vector record (`doc_type = "vp-node"`) whose body holds
//! the pivot `(x,y,z)` + radial radius `tau`, and whose `linked_records` are its
//! two children. Ball recall walks the tree, pruning branches that cannot
//! intersect the query ball — exact, bounded node visits (whitepaper §4.1).
use cubecoords::{CubeHeader, Czyx};
use cubestore::{CubeBackend, CubeStore};
/// A 3-D feature vector (the `(fx,fy,fz)` MEM model).
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub struct Vec3 {
pub x: u64,
pub y: u64,
pub z: u64,
}
impl Vec3 {
pub fn new(x: u64, y: u64, z: u64) -> Self {
Vec3 { x, y, z }
}
pub fn d2(&self, o: &Vec3) -> u64 {
let dx = (self.x as i64 - o.x as i64).abs() as u64;
let dy = (self.y as i64 - o.y as i64).abs() as u64;
let dz = (self.z as i64 - o.z as i64).abs() as u64;
dx * dx + dy * dy + dz * dz
}
}
const VP_NODE: &str = "vp-node";
/// Namespace for index nodes (avoid collisions with user data spaces).
pub const VP_SPACE: u8 = 200;
const NULL_NODE: Czyx = Czyx::new(VP_SPACE, 0, 0, 0);
/// Build a VP-tree over `points`, writing each node as a record. Returns the root coord.
pub fn build<B: CubeBackend>(store: &mut CubeStore<B>, points: &[Vec3]) -> Czyx {
let mut counter = 0u64;
let idx: Vec<usize> = (0..points.len()).collect();
build_rec(store, points, &idx, &mut counter)
}
fn build_rec<B: CubeBackend>(
store: &mut CubeStore<B>,
points: &[Vec3],
box_idxs: &[usize],
counter: &mut u64,
) -> Czyx {
if box_idxs.is_empty() {
return NULL_NODE;
}
let pi = box_idxs[0];
let pivot = points[pi];
let mut others: Vec<(u64, usize)> = box_idxs
.iter()
.copied()
.filter(|&i| i != pi)
.map(|i| (pivot.d2(&points[i]), i))
.collect();
others.sort_by_key(|&(d, _)| d);
// Vantage radius = sqrt of the median squared distance to the other points.
let tau = if others.is_empty() {
0
} else {
(others[others.len() / 2].0 as f64).sqrt() as u64
};
let mid = others.len() / 2;
let left_idx: Vec<usize> = others.iter().take(mid).map(|&(_, i)| i).collect();
let right_idx: Vec<usize> = others.iter().skip(mid).map(|&(_, i)| i).collect();
*counter += 1;
let coord = czyx_node(*counter);
let left = build_rec(store, points, &left_idx, counter);
let right = build_rec(store, points, &right_idx, counter);
let mut links = Vec::new();
if left != NULL_NODE {
links.push(left);
}
if right != NULL_NODE {
links.push(right);
}
let mut h = CubeHeader::new();
h.doc_type = Some(VP_NODE.into());
h.title = Some("vp".into());
h.linked_records = links;
h.refresh_flags();
let mut body = Vec::with_capacity(32);
for v in [pivot.x, pivot.y, pivot.z, tau] {
body.extend_from_slice(&v.to_le_bytes());
}
store.put_record(coord, &h, &body);
coord
}
fn czyx_node(n: u64) -> Czyx {
// Encode the node id across the low three axes so we can host far more than
// 255 nodes (u8 per axis would collide). c = index namespace.
Czyx::new(VP_SPACE, ((n >> 16) & 0xFF) as u8, ((n >> 8) & 0xFF) as u8, (n & 0xFF) as u8)
}
/// Ball recall: walk the VP-tree from `root`, pruning branches that cannot
/// intersect the query ball. Returns the coordinates of every vector within `r`
/// and the number of nodes visited (each visit = one record read).
pub fn ball_stats<B: CubeBackend>(
store: &CubeStore<B>,
root: Czyx,
q: Vec3,
r: u64,
) -> (Vec<Czyx>, usize) {
let r2 = r * r;
let mut out = Vec::new();
let mut visits = 0usize;
ball_rec(store, root, &q, r2, r, &mut out, &mut visits);
(out, visits)
}
fn ball_rec<B: CubeBackend>(
store: &CubeStore<B>,
node: Czyx,
q: &Vec3,
r2: u64,
r: u64,
out: &mut Vec<Czyx>,
visits: &mut usize,
) {
if node == NULL_NODE {
return;
}
*visits += 1;
let (h, body) = match store.get_record(&node) {
Some(x) => x,
None => return,
};
if body.len() < 32 {
return;
}
let pivot = Vec3::new(
u64::from_le_bytes(body[0..8].try_into().unwrap()),
u64::from_le_bytes(body[8..16].try_into().unwrap()),
u64::from_le_bytes(body[16..24].try_into().unwrap()),
);
let tau = u64::from_le_bytes(body[24..32].try_into().unwrap());
let d2v = pivot.d2(q);
if d2v <= r2 {
out.push(node);
}
let links = h.linked_records;
let d = (d2v as f64).sqrt();
let rf = r as f64;
// Descend left (points within tau of the pivot) if the query ball may reach them.
if links.len() >= 1 && (d - rf) <= tau as f64 {
ball_rec(store, links[0], q, r2, r, out, visits);
}
// Descend right (points beyond tau) if the query ball may extend past the pivot sphere.
if links.len() >= 2 && (d + rf) > tau as f64 {
ball_rec(store, links[1], q, r2, r, out, visits);
}
}
#[cfg(test)]
mod tests {
use super::*;
use cubestore::HashBackend;
#[test]
fn vp_tree_exact_ball_recall() {
let n = 2000u64;
let mut points = Vec::with_capacity(n as usize);
for c in 0..16u64 {
let (bx, by, bz) = (c * 40, (c * 7) % 16 * 7, (c * 13) % 16 * 5);
for j in 0..(n / 16) {
let o = j as u64 % 12;
points.push(Vec3::new(bx + o, by + (o * 3) % 12, bz + (o * 7) % 12));
}
}
let mut store = CubeStore::new(HashBackend::new());
let root = build(&mut store, &points);
let r = 30u64;
let r2 = r * r;
let mut total_rec = 0usize;
let mut total_true = 0usize;
let mut visits_sum = 0usize;
for anchor in 0..40usize {
let q = points[anchor];
let (found, visits) = ball_stats(&store, root, q, r);
// ground truth = vectors within r of q
let truth: Vec<Vec3> = points.iter().copied().filter(|p| q.d2(p) <= r2).collect();
// decode pivots from the found node records
let mut found_pivots = Vec::new();
for c in &found {
if let Some((_, b)) = store.get_record(c) {
if b.len() >= 24 {
found_pivots.push(Vec3::new(
u64::from_le_bytes(b[0..8].try_into().unwrap()),
u64::from_le_bytes(b[8..16].try_into().unwrap()),
u64::from_le_bytes(b[16..24].try_into().unwrap()),
));
}
}
}
let rec = truth.iter().filter(|v| found_pivots.contains(v)).count();
total_rec += rec;
total_true += truth.len();
visits_sum += visits;
}
eprintln!(
"exact recall {:.1}% ({} of {}), avg {} node-visits per query (N={})",
100.0 * total_rec as f64 / total_true as f64,
total_rec,
total_true,
visits_sum as f64 / 40.0,
n
);
assert!(total_true > 0 && total_rec == total_true, "VP-tree must be exact (got {}/{}), avg {} visits", total_rec, total_true, visits_sum as f64 / 40.0);
assert!(visits_sum as f64 / 40.0 < (n as f64) / 3.0, "VP-tree must prune (avg {} visits)", visits_sum as f64 / 40.0);
}
}
+169
View File
@@ -0,0 +1,169 @@
//! CUBELinux-2 cubebook / kernel-convention constants.
//!
//! These are the conventions the *OS* layer uses to store its computational
//! behavior in CUBE. Per the PDF (Package 4, and the "behavior descriptors"
//! discussion at PDF §524525), functions/operators that live in CUBE are:
//! * addressed in a reserved `C` axis band (here `c210..=c219`),
//! * tagged with a `kind` (`fn`/`kernel`/`layer`/`checkpoint`/`variant`), and
//! * annotated with *behavior descriptor* flags carried in the record
//! header's out-of-band flag bits (tag 12, see cubestore encode/decode).
//!
//! The full spec-derived descriptor set (PDF §524525) is: "pure function,"
//! "I/O heavy," "allocates memory," "touches network," "hot path," and
//! "security sensitive." We store each directly as a dedicated header-flag bit
//! inside the spare 8..=15 range, deliberately leaving bit 12
//! (`cubecrypt::HEADER_FLAG_ENCRYPTED`) clear. `CubeHeader::refresh_flags()`
//! preserves spare bits 8..=15, so a header carrying descriptors survives an
//! encode→decode→refresh round-trip.
//!
//! This is deliberately NOT a store-IO model: the cubevm never reads or writes
//! OS state records directly. The OS's *decisions/computation* live as CUBE
//! kernels; the native OS layer is a thin effector that reads a kernel's
//! computed result and applies the effect (writes the record, runs the
//! command). See `cubesys/src/commands.rs` `tick` + `native-apply`.
/// First `C` axis value reserved for OS operator kernels (call-graph roots and
/// leaves). The OS keeps its behavioral substrate here, separate from the
/// `c200` operational-snapshot data band and the `c001/c002` doc examples.
pub const C_OS_KERNEL: u8 = 210;
/// `C` axis band for OS *data* records the native layer writes after a kernel
/// decides an effect (kept distinct from the compute-kernel band above).
pub const C_OS_EFFECT: u8 = 211;
/// Header-flag bits reserved for the behavior-descriptor field. Six spec
/// descriptors, each a dedicated bit in the spare 8..=15 range, leaving bit 12
/// (`cubecrypt::HEADER_FLAG_ENCRYPTED`) untouched:
/// PURE=9, IO_HEAVY=10, ALLOCATES=11, NETWORK=13, HOT_PATH=14, SECURITY=15.
pub const HEADER_FLAG_BEHAVIOR: u16 = 0b1110_1110_0000_0000; // bits 9,10,11,13,14,15 (bit12 reserved)
/// Behavior descriptors for an OS operator kernel (PDF §524525, full set).
///
/// Each variant is stored directly as its dedicated header-flag bit (subset of
/// `HEADER_FLAG_BEHAVIOR`), so `Behavior` carries the raw descriptor bits and
/// round-trips through the header codec without bit-shift collisions.
#[derive(Copy, Clone, Debug, Default, PartialEq, Eq)]
pub struct Behavior(pub u16);
impl Behavior {
/// Pure function: no side effects, deterministic on inputs.
pub const PURE: u16 = 1 << 9; // 0x0200
/// I/O heavy: performs significant store/device I/O.
pub const IO_HEAVY: u16 = 1 << 10; // 0x0400
/// Allocates memory: grows the heap / maps pages.
pub const ALLOCATES: u16 = 1 << 11; // 0x0800
/// Touches network: performs socket/link I/O (security-relevant surface).
pub const NETWORK: u16 = 1 << 13; // 0x2000
/// Hot path: executed frequently; a candidate for optimization/variant swap.
pub const HOT_PATH: u16 = 1 << 14; // 0x4000
/// Security sensitive: elevated privilege / trust boundary crossing.
pub const SECURITY_SENSITIVE: u16 = 1 << 15; // 0x8000
/// Combine descriptor bits (e.g. `Behavior::PURE | Behavior::HOT_PATH`).
pub fn new(bits: u16) -> Self {
Behavior(bits & HEADER_FLAG_BEHAVIOR)
}
/// Encode into the out-of-band header flag bits.
pub fn to_flags(self) -> u16 {
self.0 & HEADER_FLAG_BEHAVIOR
}
/// Decode from a raw 16-bit flag word (keeps only the behavior bits).
pub fn from_flags(flags: u16) -> Self {
Behavior(flags & HEADER_FLAG_BEHAVIOR)
}
/// True if any descriptor bit is set.
pub fn is_any(self) -> bool {
self.0 != 0
}
/// Human-readable descriptor tags (used by `ls`/`stat` and the effector).
pub fn tags(self) -> Vec<&'static str> {
let mut out = Vec::new();
if self.0 & Self::PURE != 0 {
out.push("pure");
}
if self.0 & Self::IO_HEAVY != 0 {
out.push("io");
}
if self.0 & Self::ALLOCATES != 0 {
out.push("alloc");
}
if self.0 & Self::NETWORK != 0 {
out.push("net");
}
if self.0 & Self::HOT_PATH != 0 {
out.push("hot");
}
if self.0 & Self::SECURITY_SENSITIVE != 0 {
out.push("sec");
}
out
}
}
#[cfg(test)]
mod tests {
use super::*;
use cubecoords::{CubeHeader, Czyx};
use cubestore::{CubeStore, HashBackend};
#[test]
fn behavior_round_trips_through_store() {
// All six spec descriptors must survive the real store round-trip
// (what native-apply reads back). This is the exact field the
// effector inspects; an earlier 3-bit mask (0x7000) silently dropped
// HOT_PATH (bit 15) and collided with ENCRYPTED (bit 12).
let b = Behavior::new(
Behavior::PURE | Behavior::IO_HEAVY | Behavior::ALLOCATES | Behavior::NETWORK,
);
let raw = b.to_flags();
assert_eq!(
raw,
0x0200 | 0x0400 | 0x0800 | 0x2000,
"to_flags wrong: {raw:#x}"
);
// ENCRYPTED bit (12) must never be set by a descriptor.
assert_eq!(raw & (1 << 12), 0, "descriptor must not set ENCRYPTED bit");
let mut store = CubeStore::new(HashBackend::new());
let coord = Czyx::new(210, 1, 1, 3);
let mut h = CubeHeader::new();
h.doc_type = Some("kernel".into());
h.title = Some("summarize-procs".into());
h.flags.0 |= raw;
h.refresh_flags();
store.put_record(coord, &h, &[1, 2, 3]);
let (read_h, _) = store.get_record(&coord).expect("record present");
let back = Behavior::from_flags(read_h.flags.bits());
assert_eq!(
back,
b,
"behavior dropped on store round-trip: stored {raw:#x}, got {:#x}",
read_h.flags.bits()
);
}
#[test]
fn security_sensitive_bit_used_and_round_trips() {
// Explicitly cover the 6th spec descriptor (security sensitive, bit15).
let b = Behavior::new(Behavior::SECURITY_SENSITIVE | Behavior::HOT_PATH);
let mut store = CubeStore::new(HashBackend::new());
let coord = Czyx::new(210, 1, 1, 5);
let mut h = CubeHeader::new();
h.doc_type = Some("kernel".into());
h.title = Some("privileged-op".into());
h.flags.0 |= b.to_flags();
h.refresh_flags();
store.put_record(coord, &h, &[]);
let (read_h, _) = store.get_record(&coord).expect("record present");
assert_eq!(
Behavior::from_flags(read_h.flags.bits()),
b,
"security-sensitive descriptor lost on round-trip"
);
}
}
+1
View File
@@ -71,6 +71,7 @@ impl Kind {
/// in the body. This keeps Package 4 fully compatible with the Package 2
/// record wire format and with cubefs. `code_kind`/`set_kind` bridge the
/// enum to the string.
#[derive(Clone, Debug)]
pub struct CodeCell {
/// The cube coordinate this cell is stored at.
pub label: Czyx,
+8 -1
View File
@@ -35,10 +35,17 @@
#![forbid(unsafe_code)]
#![warn(missing_docs)]
pub mod cb;
pub mod cell;
pub mod ballindex;
pub mod opcode;
pub mod trace;
pub mod vm;
pub use cb::{Behavior, C_OS_EFFECT, C_OS_KERNEL, HEADER_FLAG_BEHAVIOR};
pub use cell::{CodeCell, Kind};
pub use opcode::{decode, encode, CodeError, Op};
pub use vm::{Fault, RunResult, Vm, SYS_DEGREE, SYS_LINKED_EXISTS, SYS_NOP, SYS_TRACE};
pub use ballindex::{ball_stats, build, Vec3, VP_SPACE};
pub use trace::{capture_golden, lineage, replay_trace, store_trace, verify_golden, LineageHop};
pub use vm::{replay, Fault, RunResult, Vm, SYS_DEGREE, SYS_LINKED_EXISTS, SYS_NOP, SYS_TRACE};
+263
View File
@@ -0,0 +1,263 @@
//! Execution-trace artifacts: store a captured machine-code trace as a
//! first-class AI-artifact record (`Kind::Layer`) linked back to its source,
//! and replay it to reproduce the run — so an agent can persist & reproduce
//! interesting executions as ordinary CZYX records.
//!
//! A trace is the exact `Vec<Op>` a program actually executed ([`Vm::run_captured`]).
//! We persist it as a record body (via [`crate::opcode::encode`]) with
//! `doc_type = "layer"` (the AI-artifact kind) and a `linked_records` edge to
//! the source cell. Loading it back (via [`CodeCell::from_record`]) decodes the
//! ops and [`replay`] reproduces the observable behaviour with no access to the
//! source program.
use crate::{opcode, replay, Kind, Op, RunResult, Vm};
use cubecoords::{CubeHeader, Czyx};
use cubestore::{CubeBackend, CubeStore};
/// Store a captured execution trace as an AI-artifact record at `coord`,
/// linked to its `source` cell. `name` becomes the record title; the body is
/// the encoded trace; `doc_type` is `Kind::Layer`.
pub fn store_trace<B: CubeBackend>(
store: &mut CubeStore<B>,
coord: Czyx,
source: Czyx,
trace: &[Op],
name: &str,
) {
let mut h = CubeHeader::new();
h.title = Some(name.to_string());
h.doc_type = Some(Kind::Layer.as_str().to_string());
h.linked_records = vec![source];
h.refresh_flags();
store.put_record(coord, &h, &opcode::encode(trace));
}
/// Load a stored trace record at `coord` and replay it, reproducing the run's
/// observable behaviour (data-stack result + `SYS_TRACE` output) from the
/// stored record — no access to the original program. Returns `None` if the
/// record is missing or its body is not a valid trace.
pub fn replay_trace<B: CubeBackend>(
store: &CubeStore<B>,
coord: Czyx,
) -> Option<(RunResult, Vec<u8>)> {
let (_h, body) = store.get_record(&coord)?;
let ops = opcode::decode(&body).ok()?;
Some(replay(&ops))
}
/// Capture a run and persist it as a "golden": the executed trace stored as a
/// `Kind::Layer` record (linked to `source`) and the observed output stored as a
/// companion `golden` record (linked to the trace). Returns the observed output.
pub fn capture_golden<B: CubeBackend + Clone>(
store: &mut CubeStore<B>,
source: Czyx,
trace_coord: Czyx,
golden_coord: Czyx,
name: &str,
) -> Vec<u8> {
let mut vm = Vm::new((*store).clone());
let (_r, trace) = vm.run_captured(source);
let out = vm.output().to_vec();
store_trace(store, trace_coord, source, &trace, name);
let mut h = CubeHeader::new();
h.title = Some(format!("golden:{name}"));
h.doc_type = Some("golden".into());
h.linked_records = vec![trace_coord];
h.refresh_flags();
store.put_record(golden_coord, &h, &out);
out
}
/// Golden-trace regression check: re-run `source` fresh AND replay the stored
/// trace, and assert BOTH reproduce the recorded golden output.
/// * replay vs golden -> the stored trace is a faithful reproduction;
/// * fresh run vs golden -> the program has not drifted (or it would change
/// the observed output).
/// Returns `Ok(())` if the behavior is unchanged, else `Err(reason)`.
pub fn verify_golden<B: CubeBackend + Clone>(
store: &CubeStore<B>,
source: Czyx,
trace_coord: Czyx,
golden_coord: Czyx,
) -> Result<(), String> {
let golden = store
.get_record(&golden_coord)
.map(|(_, b)| b)
.ok_or("golden record missing")?;
let (_, replay_out) = replay_trace(store, trace_coord).ok_or("trace record missing/invalid")?;
if replay_out != golden {
return Err(format!(
"replay output differs from golden ({} vs {} bytes)",
replay_out.len(),
golden.len()
));
}
let mut vm = Vm::new((*store).clone());
let _ = vm.run(source);
let fresh = vm.output().to_vec();
if fresh != golden {
return Err(format!(
"fresh run differs from golden (program changed: {} vs {} bytes)",
fresh.len(),
golden.len()
));
}
Ok(())
}
/// A hop in the experiment-lineage graph: a record and the records it links to.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct LineageHop {
/// The record's coordinate.
pub coord: Czyx,
/// Its `doc_type` (kind: fn/kernel/layer/checkpoint/variant/golden/...).
pub doc_type: String,
/// Its human title, if any.
pub title: String,
/// The CZYX records this record links to (its edges).
pub links: Vec<Czyx>,
}
/// Walk the linked-record graph starting at `start`, following
/// `linked_records` edges (breadth-first) up to `depth` levels. Returns the
/// ordered list of reachable records — the experiment lineage chain
/// (source -> data -> weights -> trace -> golden, etc.). Each record is the
/// kind the CUBE associates; provenance is a graph walk, not a lookup.
pub fn lineage<B: CubeBackend>(
store: &CubeStore<B>,
start: Czyx,
depth: usize,
) -> Vec<LineageHop> {
let mut out = Vec::new();
let mut frontier = vec![start];
let mut seen: std::collections::HashSet<Czyx> = std::collections::HashSet::new();
for _ in 0..=depth {
if frontier.is_empty() {
break;
}
let mut next = Vec::new();
for c in frontier {
if !seen.insert(c) {
continue;
}
if let Some((h, _)) = store.get_record(&c) {
let links = h.linked_records.clone();
out.push(LineageHop {
coord: c,
doc_type: h.doc_type.clone().unwrap_or_default(),
title: h.title.clone().unwrap_or_default(),
links: links.clone(),
});
next.extend(links);
}
}
frontier = next;
}
out
}
#[cfg(test)]
mod tests {
use super::*;
use crate::{SYS_TRACE, Vm};
use cubestore::HashBackend;
#[test]
fn trace_stored_as_layer_and_replayed_without_source() {
let mut store = CubeStore::new(HashBackend::new());
let entry = Czyx::new(1, 1, 1, 7);
let ops = [
Op::Const(0), Op::Const(1), Op::Syscall(SYS_TRACE),
Op::Const(1), Op::Const(1), Op::Syscall(SYS_TRACE),
Op::Const(2), Op::Const(2), Op::Syscall(SYS_TRACE),
Op::Halt,
];
let mut h = CubeHeader::new();
h.title = Some("solitaire".into());
h.doc_type = Some("fn".into());
h.refresh_flags();
store.put_record(entry, &h, &opcode::encode(&ops));
// 1) Capture the run as a machine-code trace.
let mut vm = Vm::new(store.clone());
let (_r, trace) = vm.run_captured(entry);
let produced = vm.output().to_vec();
assert!(!trace.is_empty());
// 2) Store it as a Kind::Layer AI-artifact record linked to the source.
let trace_coord = Czyx::new(9, 0, 0, 1);
store_trace(&mut store, trace_coord, entry, &trace, "solitaire-run-1");
// 3) The record is a "layer" artifact with an edge back to the source.
let (th, _tb) = store.get_record(&trace_coord).unwrap();
assert_eq!(th.doc_type.as_deref(), Some("layer"));
assert_eq!(th.linked_records, vec![entry]);
// 4) Reproduce the run purely from the stored trace record.
let (result, replayed) = replay_trace(&store, trace_coord).unwrap();
assert_eq!(result, RunResult::Halted { top: None });
assert_eq!(replayed, produced, "stored trace must reproduce the run");
assert!(replayed.contains(&b'\n'));
}
#[test]
fn golden_regression_detects_drift() {
let mut store = CubeStore::new(HashBackend::new());
let entry = Czyx::new(1, 1, 1, 9);
let mk = |val: u8| {
vec![Op::Const(val), Op::Syscall(SYS_TRACE), Op::Halt]
};
let mut h = CubeHeader::new();
h.doc_type = Some("fn".into());
h.refresh_flags();
store.put_record(entry, &h, &opcode::encode(&mk(7)));
let trace_c = Czyx::new(9, 0, 0, 1);
let golden_c = Czyx::new(9, 0, 0, 2);
let out = capture_golden(&mut store, entry, trace_c, golden_c, "regress");
assert_eq!(out, b"\x07\n");
assert!(verify_golden(&store, entry, trace_c, golden_c).is_ok());
// Drift the program: change the emitted value -> fresh run differs.
store.put_record(entry, &h, &opcode::encode(&mk(9)));
let r = verify_golden(&store, entry, trace_c, golden_c);
assert!(r.is_err(), "program drift must be detected: {r:?}");
// The stored trace is immutable: it still replays the ORIGINAL golden.
let (_, rep) = replay_trace(&store, trace_c).unwrap();
assert_eq!(rep, b"\x07\n", "stored trace still reproduces the original run");
}
#[test]
fn lineage_walks_source_data_weights_trace() {
let mut store = CubeStore::new(HashBackend::new());
// Build experiment lineage: source -> data -> weights -> trace -> golden.
let mut mk = |coord: Czyx, doc: &str, title: &str, links: Vec<Czyx>| {
let mut h = CubeHeader::new();
h.title = Some(title.into());
h.doc_type = Some(doc.into());
h.linked_records = links;
h.refresh_flags();
store.put_record(coord, &h, &[]);
};
let source = Czyx::new(1, 1, 1, 1);
let data = Czyx::new(2, 0, 0, 1);
let weights = Czyx::new(3, 0, 0, 1);
let trace = Czyx::new(4, 0, 0, 1);
let golden = Czyx::new(5, 0, 0, 1);
mk(source, "fn", "train", vec![data, weights]);
mk(data, "data", "batch", vec![]);
mk(weights, "layer", "model-v1", vec![]);
mk(trace, "layer", "trace:train", vec![source]);
mk(golden, "golden", "golden:train", vec![trace]);
// Walk from the trace back to its lineage.
let hops = lineage(&store, trace, 4);
assert!(hops.iter().any(|h| h.coord == trace && h.doc_type == "layer"));
assert!(hops.iter().any(|h| h.coord == source && h.doc_type == "fn"));
// Depth-limited: from a leaf with no links, only itself.
let leaf = lineage(&store, data, 3);
assert_eq!(leaf.len(), 1);
assert_eq!(leaf[0].coord, data);
assert_eq!(leaf[0].links, vec![]);
}
}
+141
View File
@@ -90,6 +90,12 @@ pub struct Vm<B: CubeBackend> {
output: Vec<u8>,
/// Maximum nesting of `CALL_LINK` frames.
max_call_depth: usize,
/// Executed-instruction trace collected when [`Vm::run_captured`] is active:
/// the exact ordered sequence of `Op`s that actually ran (the "machine code
/// as it runs"). Used to reproduce behaviour without the source.
trace: Vec<Op>,
/// Whether [`Vm::run_captured`] is currently recording the instruction trace.
capturing: bool,
}
impl<B: CubeBackend> Vm<B> {
@@ -104,6 +110,8 @@ impl<B: CubeBackend> Vm<B> {
store,
output: Vec::new(),
max_call_depth: Self::MAX_CALL_DEPTH,
trace: Vec::new(),
capturing: false,
}
}
@@ -158,6 +166,9 @@ impl<B: CubeBackend> Vm<B> {
while pc < code.len() {
let op = code[pc];
if self.capturing {
self.trace.push(op);
}
match op {
Op::Nop => {}
Op::Halt => {
@@ -258,6 +269,21 @@ impl<B: CubeBackend> Vm<B> {
Ok(stack.pop())
}
/// Run `entry` AND capture the executed-instruction trace (the "machine
/// code as it runs"): the exact ordered sequence of `Op`s executed, across
/// all cells, regardless of source. Branches are already resolved (the
/// trace is the taken path) and call frames are marked by `CallLink`/`Ret`,
/// so [`Vm::replay`] can reproduce the same behaviour from the trace alone
/// — without the original program's records/code.
pub fn run_captured(&mut self, entry: Czyx) -> (RunResult, Vec<Op>) {
self.output.clear();
self.trace.clear();
self.capturing = true;
let res = self.run(entry);
self.capturing = false;
(res, std::mem::take(&mut self.trace))
}
/// Host syscall dispatch. `stack` is the *shared* data stack (callees
/// run on it) and `header` is the current cell's metadata.
fn syscall(&mut self, id: u8, header: &CubeHeader, stack: &mut Vec<u8>) -> Result<(), Fault> {
@@ -288,6 +314,87 @@ impl<B: CubeBackend> Vm<B> {
}
}
/// Replay-time host syscall dispatch (no store/header). Store-independent
/// syscalls (`SYS_TRACE`, `SYS_NOP`) are reproduced exactly; syscalls that
/// would need the live store/header are a no-op in trace-replay (they don't
/// contribute to the observable `SYS_TRACE` output).
fn replay_syscall(id: u8, stack: &mut Vec<u8>, output: &mut Vec<u8>) {
match id {
SYS_NOP => {}
SYS_TRACE => {
let mut line: Vec<u8> = std::mem::take(stack);
line.push(b'\n');
output.extend_from_slice(&line);
}
_ => {}
}
}
/// Replay a captured instruction trace, reproducing the program's actions
/// (data-stack result + `SYS_TRACE` output) WITHOUT accessing any of the
/// original program's records/bytecode. This is the "duplicate behaviour
/// from machine code as it ran" path. Store-independent syscalls
/// (`SYS_TRACE`, `SYS_NOP`) are reproduced exactly; syscalls that would
/// need the live store/header are a no-op (those don't contribute to the
/// observable `SYS_TRACE` output).
pub fn replay(trace: &[Op]) -> (RunResult, Vec<u8>) {
let mut stack: Vec<u8> = Vec::new();
let mut frames: Vec<[u8; LOCALS]> = vec![[0; LOCALS]];
let mut output: Vec<u8> = Vec::new();
let mut top = None;
for &op in trace {
match op {
Op::Nop => {}
Op::Halt => {
top = stack.pop();
break;
}
Op::Const(v) => stack.push(v),
Op::Load(i) => stack.push(frames.last_mut().unwrap()[(i as usize) % LOCALS]),
Op::Store(i) => {
let v = stack.pop().unwrap_or(0);
frames.last_mut().unwrap()[(i as usize) % LOCALS] = v;
}
Op::Add | Op::Sub | Op::Mul | Op::Div | Op::Mod | Op::And | Op::Or | Op::Xor => {
let (b, a) = (stack.pop(), stack.pop());
stack.push(bin_op(op, a.unwrap_or(0), b.unwrap_or(0)).unwrap_or(0));
}
Op::Shl | Op::Shr => {
let (b, a) = (stack.pop(), stack.pop());
stack.push(shift_op(op, a.unwrap_or(0), b.unwrap_or(0)));
}
Op::Eq | Op::Ne | Op::Lt | Op::Gt | Op::Le | Op::Ge => {
let (b, a) = (stack.pop(), stack.pop());
stack.push(cmp_op(op, a.unwrap_or(0), b.unwrap_or(0)));
}
// Jumps already reflect the taken path in a captured trace:
// keep the stack effect, don't re-branch.
Op::Jmp(_) => {}
Op::Jz(_) | Op::Jnz(_) => {
let _ = stack.pop();
}
Op::CallLink(_) => frames.push([0; LOCALS]),
Op::Ret => {
if frames.len() > 1 {
frames.pop();
} else {
top = stack.pop();
break;
}
}
Op::Syscall(id) => replay_syscall(id, &mut stack, &mut output),
Op::Dup => {
let v = *stack.last().unwrap_or(&0);
stack.push(v);
}
Op::Drop => {
stack.pop();
}
}
}
(RunResult::Halted { top }, output)
}
fn b2(v: Option<u8>, cell: Czyx, pc: usize) -> Result<u8, (Czyx, usize, Fault)> {
v.ok_or((cell, pc, Fault::PcOutOfRange))
}
@@ -522,4 +629,38 @@ mod tests {
let mut vm = Vm::new(store);
assert_eq!(vm.run(entry), RunResult::Halted { top: Some(3) });
}
#[test]
fn capture_and_replay_reproduces_behavior_without_source() {
let mut store = CubeStore::new(HashBackend::new());
let entry = Czyx::new(1, 1, 1, 7);
// A tiny "solitaire" state machine: draw cards; even -> foundation
// (score++), odd -> waste. Emit (score, waste) after each play.
let ops = [
Op::Const(0), Op::Const(1), Op::Syscall(SYS_TRACE), // draw odd -> waste
Op::Const(1), Op::Const(1), Op::Syscall(SYS_TRACE), // draw even -> score
Op::Const(1), Op::Const(2), Op::Syscall(SYS_TRACE), // draw odd -> waste
Op::Const(2), Op::Const(2), Op::Syscall(SYS_TRACE), // draw even -> score
Op::Halt,
];
let mut h = CubeHeader::new();
h.title = Some("solitaire".into());
h.doc_type = Some("fn".into());
h.refresh_flags();
store.put_record(entry, &h, &crate::opcode::encode(&ops));
let mut vm = Vm::new(store);
// Capture: run the program AND record the exact machine code executed.
let (result, trace) = vm.run_captured(entry);
assert_eq!(result, RunResult::Halted { top: None });
assert!(!trace.is_empty(), "capture must record the executed ops");
let produced = vm.output().to_vec();
assert_eq!(produced, b"\x00\x01\n\x01\x01\n\x01\x02\n\x02\x02\n");
// Replay: reproduce the SAME actions purely from the captured trace,
// with NO access to the original program's records/bytecode.
let (result2, replayed) = replay(&trace);
assert_eq!(result2, RunResult::Halted { top: None });
assert_eq!(replayed, produced, "replay must reproduce the game's behaviour");
}
}
+147 -1
View File
@@ -160,6 +160,84 @@ impl TriEnc {
}
(control, ascii)
}
/// Pack 6 ASCII bytes plus a full 16-bit per-word flag/metadata field.
///
/// Layout (high -> low): `[ WordFlags: u16 (bits 48-63) ][ 48 ascii bits ]`.
/// This promotes the 12 formerly-unused bits + the 4 control bits into a real
/// per-word metadata stripe, mapped by [`WordFlags`]. Backward compatible:
/// `pack_6(control)` is exactly `pack_flags(WordFlags(control as u16 << 12),
/// ascii)` (the old 4 control bits occupy the high nibble of the flags field).
#[inline]
pub fn pack_flags(flags: WordFlags, ascii: [u8; 6]) -> TriWord {
let mut v: u64 = ((flags.0 as u64) & 0xFFFF) << 48;
for (i, b) in ascii.iter().enumerate() {
v |= (*b as u64) << (8 * (5 - i));
}
TriWord(v)
}
/// Unpack a word into its 16-bit flags field and 6 data bytes.
#[inline]
pub fn unpack_flags(word: TriWord) -> (WordFlags, [u8; 6]) {
let v = word.0;
let flags = WordFlags(((v >> 48) & 0xFFFF) as u16);
let mut ascii = [0u8; 6];
for (i, b) in ascii.iter_mut().enumerate() {
*b = ((v >> (8 * (5 - i))) & 0xFF) as u8;
}
(flags, ascii)
}
}
/// The 16-bit **per-word** flag/metadata field co-packed with 6 data bytes in a
/// tri-channel [`TriWord`] (bits 48-63). Distinct from [`HeaderFlags`], which is
/// the per-*record* header; this is the inline per-word metadata stripe.
#[derive(Copy, Clone, Eq, PartialEq, Debug, Default)]
pub struct WordFlags(pub u16);
impl WordFlags {
/// Packing arrangement (00 tri6, 01 pair, 10 triad, 11 quad).
pub const ARR_MASK: u16 = 0b0000_0000_0000_0011;
/// Flag: first word of a record.
pub const START_RECORD: u16 = 0b0000_0000_0000_0100;
/// Flag: last word of a record. (Pair with the PDF's end-of-header marker.)
pub const END_RECORD: u16 = 0b0000_0000_0000_1000;
/// Flag: this word is header/metadata (vs payload).
pub const IS_HEADER: u16 = 0b0000_0000_0001_0000;
/// Flag: record continues across the next word.
pub const CONTINUATION: u16 = 0b0000_0000_0010_0000;
/// Data type (00 text, 01 binary, 10 code, 11 ai-feature-vector).
pub const TYPE_MASK: u16 = 0b0000_0000_1100_0000;
/// Permission: root-only.
pub const PERM_ROOT: u16 = 0b0000_0001_0000_0000;
/// Permission: local-user owner.
pub const PERM_USER: u16 = 0b0000_0010_0000_0000;
/// Flag: association edge present on this word.
pub const ASSOC_EDGE: u16 = 0b0000_0100_0000_0000;
/// Flag: this group is encrypted (cubecrypt key/tweak selector).
pub const ENCRYPTED: u16 = 0b0000_1000_0000_0000;
/// Flag: this group is compressed.
pub const COMPRESSED: u16 = 0b0001_0000_0000_0000;
/// Flag: this group uses the high-entropy/stego transform (obfuscation layer).
pub const STEGO: u16 = 0b0010_0000_0000_0000;
/// Flag: this group is a Null/cube lookup marker.
pub const NULL_LOOKUP: u16 = 0b0100_0000_0000_0000;
/// Flag: checksum/parity present for this group.
pub const CHECKSUM: u16 = 0b1000_0000_0000_0000;
/// Construct from a raw bitmask.
#[inline]
pub const fn from_bits(bits: u16) -> Self { WordFlags(bits) }
/// The raw bitmask.
#[inline]
pub const fn bits(&self) -> u16 { self.0 }
/// Set a flag bit.
#[inline]
pub fn set(&mut self, flag: u16) { self.0 |= flag; }
/// Test a flag bit.
#[inline]
pub fn has(&self, flag: u16) -> bool { self.0 & flag != 0 }
}
/// Header flag bits, mirroring the PDF's title/type/date/size/permission
@@ -185,6 +263,13 @@ impl HeaderFlags {
pub const PERM_REMOTE_USER: u16 = 1 << 6;
/// Flag 8: has outgoing association links.
pub const HAS_ASSOCIATIONS: u16 = 1 << 7;
/// Flag 9: carries a POSIX `path` metatag (the original filesystem path
/// the record was migrated from). Lets the associative query layer
/// reconstruct a directory hierarchy WITHOUT the filesystem needing to
/// support arbitrary recursive nesting — the path is data, addressed by
/// flag, exactly as the source PDF prescribes ("operate on CZYX records
/// and Null-space flags rather than paths and inodes").
pub const HAS_PATH: u16 = 1 << 8;
// Flag 255 (conceptual end-of-header) is represented out-of-band by the
// record serializer; there is no bit for it.
@@ -231,6 +316,11 @@ impl HeaderFlags {
pub struct CubeHeader {
/// Flag bits (derived; kept in sync by the accessors).
pub flags: HeaderFlags,
/// Per-word 16-bit tri-channel flag/metadata field (bits 48-63). Distinct
/// from `flags` (the per-RECORD header): this is the inline per-word
/// metadata stripe (see [`WordFlags`]) carried by records that use the
/// tri-channel word codec.
pub word_flags: WordFlags,
/// Flag 1: human title.
pub title: Option<String>,
/// Flag 2: document type (like a file extension).
@@ -245,6 +335,13 @@ pub struct CubeHeader {
pub owner_remote_user: Option<String>,
/// Association links to other records (flags 519).
pub linked_records: Vec<Czyx>,
/// POSIX path metatag: the original filesystem path this record was
/// migrated from (e.g. "/etc/network/interfaces"). Empty/absent means the
/// record has no path identity. When present, the `HAS_PATH` flag bit is
/// set so `scan_by_path`/`query_by_path` can recover the hierarchy from
/// metatags alone (see cubestore). This is the PDF's "path is a view over
/// coordinates + flags" made concrete.
pub path: Option<String>,
/// Total local accesses (from the PDF's association/permission flags).
pub total_accesses: u64,
/// Total remote accesses.
@@ -292,8 +389,14 @@ impl CubeHeader {
if !self.linked_records.is_empty() {
f |= HeaderFlags::HAS_ASSOCIATIONS;
}
if self.path.is_some() {
f |= HeaderFlags::HAS_PATH;
}
// Preserve spare/out-of-band flag bits (bits 8..=15) that are not
// derived from structured fields.
// derived from structured fields. This includes
// `cubecrypt::HEADER_FLAG_ENCRYPTED` (bit 12) and the behavior-descriptor
// field (bits 13..=15, see `cubecode::Behavior`) so both survive an
// encode→decode→refresh round-trip.
const DERIVED_BITS: u16 = 0x00FF;
f |= self.flags.bits() & !DERIVED_BITS;
self.flags = HeaderFlags::from_bits(f);
@@ -346,3 +449,46 @@ mod tests {
assert!(!h.flags.has(HeaderFlags::DOC_TYPE));
}
}
#[cfg(test)]
mod wordflags_tests {
use super::*;
#[test]
fn roundtrip_flags_and_data() {
let cases = [
(WordFlags(0), *b"abcdef"),
(WordFlags::from_bits(WordFlags::ENCRYPTED | WordFlags::START_RECORD), *b"secret"),
(WordFlags::from_bits(WordFlags::COMPRESSED | WordFlags::ASSOC_EDGE), *b"123456"),
(WordFlags::from_bits(WordFlags::ARR_MASK & 0b10 | WordFlags::NULL_LOOKUP), *b"AB\x00\xff\xee\x7f"),
];
for (flags, data) in cases {
let w = TriEnc::pack_flags(flags, data);
let (f, d) = TriEnc::unpack_flags(w);
assert_eq!(f.0, flags.0, "flags must round-trip");
assert_eq!(d, data, "data must round-trip");
}
}
#[test]
fn flags_word_is_16_bits_and_data_is_6_bytes() {
let w = TriEnc::pack_flags(WordFlags::from_bits(0xFFFF), [0xFF; 6]);
assert_eq!(w.0 >> 48, 0xFFFF, "flags occupy the high 16 bits");
let (_, d) = TriEnc::unpack_flags(w);
assert_eq!(d, [0xFF; 6]);
// data occupies the low 48 bits (bits 0-47); no overlap with flags
assert_eq!(TriEnc::pack_flags(WordFlags::from_bits(0x0001), [0x00; 6]).0 >> 63, 0);
}
#[test]
fn backward_compat_pack_6_equals_flags_high_nibble() {
let ctrl = 0xAu8;
let ascii = *b"abcdef";
let a = TriEnc::pack_6(ctrl, ascii);
let b = TriEnc::pack_flags(WordFlags::from_bits((ctrl as u16) << 12), ascii);
assert_eq!(a.0, b.0, "pack_6(control) == pack_flags(control << 12)");
// and the old control nibble is the high nibble of the flags field
let (f, _) = TriEnc::unpack_flags(a);
assert_eq!((f.0 >> 12) & 0x0F, ctrl as u16);
}
}
+11
View File
@@ -0,0 +1,11 @@
[package]
name = "cubedbt"
version = "0.1.0"
edition.workspace = true
license.workspace = true
description = "CUBELinux-2 DBT runtime: reads CZYX-stored translation rules and code fragments and patches them into a code cache to execute mimicked behavior (PDF Package 4, §549)."
[dependencies]
cubecoords = { path = "../cubecoords" }
cubestore = { path = "../cubestore" }
cubecode = { path = "../cubecode" }
+421
View File
@@ -0,0 +1,421 @@
//! CUBELinux-2 DBT (dynamic binary translation) runtime.
//!
//! Per the PDF (Package 4, §549): *"cubedbt: a runtime that reads CZYXstored
//! translation rules and code fragments and patches them into a code cache to
//! execute mimicked behavior."*
//!
//! This crate is the substrate for the PDF's "watch a binary, archive its
//! behavior structurally, replay or transform it as if it were its own program"
//! story (§554). It is deliberately dependency-free and operates on the real
//! `CubeStore` / `CodeCell` / `Vm` types from the rest of the workspace, so it
//! is exercisable today on `HashBackend` and slots into `ConcurrentStore`
//! later without an API change.
//!
//! Model
//! -----
//! * A **translation rule** is a CZYX record (`Kind::Variant`) whose body is a
//! serialized [`TranslationRule`]. It names an *op kind* it can replace and
//! carries a *replacement fragment* (a `Vec<Op>`). The rule record links to
//! the original code cell it may substitute (the association edge).
//! * A **code cache** is the patched working set: `patch()` rewrites an
//! original `CodeCell`'s bytecode under the eligible rules and writes the
//! result back as a new record (preserving the original header's kind/title
//! and behavior descriptors, so it runs through the same `Vm`).
//! * A **DBT runtime** discovers rules in a store and can `mimic` a target
//! coordinate: fetch original → apply rules → run the patched version in the
//! `Vm`, returning the `RunResult`.
use cubecode::{opcode::Op, CodeCell, Kind, Vm};
use cubecoords::{CubeHeader, Czyx};
use cubestore::{CubeBackend, CubeStore, HashBackend};
/// `C` axis band where DBT translation rules are stored, kept distinct from the
/// `c210` OS-kernel band (`cubecode::C_OS_KERNEL`) and the `c200` snapshot data.
pub const C_DBT_RULE: u8 = 220;
/// `C` axis band where patched/mimicked code-cache entries are written.
pub const C_DBT_CACHE: u8 = 221;
/// The `Op` kinds a translation rule can target.
#[derive(Copy, Clone, Debug, PartialEq, Eq)]
pub enum OpClass {
Const,
Add,
Sub,
Mul,
Div,
Mod,
And,
Or,
Xor,
Shl,
Shr,
Eq,
Ne,
Lt,
Gt,
Le,
Ge,
Load,
Store,
CallLink,
Any,
}
impl OpClass {
/// Classify a live [`Op`].
pub fn of(op: &Op) -> OpClass {
match op {
Op::Const(_) => OpClass::Const,
Op::Add => OpClass::Add,
Op::Sub => OpClass::Sub,
Op::Mul => OpClass::Mul,
Op::Div => OpClass::Div,
Op::Mod => OpClass::Mod,
Op::And => OpClass::And,
Op::Or => OpClass::Or,
Op::Xor => OpClass::Xor,
Op::Shl => OpClass::Shl,
Op::Shr => OpClass::Shr,
Op::Eq => OpClass::Eq,
Op::Ne => OpClass::Ne,
Op::Lt => OpClass::Lt,
Op::Gt => OpClass::Gt,
Op::Le => OpClass::Le,
Op::Ge => OpClass::Ge,
Op::Load(_) => OpClass::Load,
Op::Store(_) => OpClass::Store,
Op::CallLink(_) => OpClass::CallLink,
_ => OpClass::Any,
}
}
}
/// A translation rule: a named, CZYX-addressed mapping from an op class to a
/// replacement bytecode fragment. Serialized into a rule record's body.
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct TranslationRule {
/// Human name (also the rule record's title).
pub name: String,
/// Which op class this rule can substitute.
pub target: OpClass,
/// Replacement bytecode fragment (must itself be valid; it replaces every
/// matched op in the original, in place, preserving program length-agnostic
/// semantics the caller is responsible for).
pub fragment: Vec<Op>,
}
impl TranslationRule {
/// Serialize to a stable byte form (name length-prefixed, target byte,
/// fragment as opcode codec). No external deps.
pub fn to_bytes(&self) -> Vec<u8> {
let mut out = Vec::new();
let nb = self.name.as_bytes();
out.push(nb.len() as u8);
out.extend_from_slice(nb);
out.push(self.target as u8);
out.push(self.fragment.len() as u8);
for op in &self.fragment {
out.extend_from_slice(&cubecode::opcode::encode(&[*op]));
}
out
}
/// Inverse of [`to_bytes`]. Returns `None` on any malformed input.
pub fn from_bytes(b: &[u8]) -> Option<TranslationRule> {
let mut i = 0;
let nlen = *b.get(i)? as usize;
i += 1;
if i + nlen > b.len() {
return None;
}
let name = String::from_utf8(b[i..i + nlen].to_vec()).ok()?;
i += nlen;
let target = *b.get(i)?;
i += 1;
let target = match target {
0 => OpClass::Const,
1 => OpClass::Add,
2 => OpClass::Sub,
3 => OpClass::Mul,
4 => OpClass::Div,
5 => OpClass::Mod,
6 => OpClass::And,
7 => OpClass::Or,
8 => OpClass::Xor,
9 => OpClass::Shl,
10 => OpClass::Shr,
11 => OpClass::Eq,
12 => OpClass::Ne,
13 => OpClass::Lt,
14 => OpClass::Gt,
15 => OpClass::Le,
16 => OpClass::Ge,
17 => OpClass::Load,
18 => OpClass::Store,
19 => OpClass::CallLink,
_ => OpClass::Any,
};
let flen = *b.get(i)? as usize;
i += 1;
let mut fragment = Vec::new();
for _ in 0..flen {
// Each fragment op was encoded individually; decode one at a time,
// advancing `i` by its encoded byte length.
let one = decode_one(&b[i..])?;
i += one.len;
fragment.push(one.op);
}
Some(TranslationRule {
name,
target,
fragment,
})
}
}
/// One decoded opcode plus its encoded length, used by `from_bytes`.
struct One {
op: Op,
len: usize,
}
fn decode_one(b: &[u8]) -> Option<One> {
let op = cubecode::opcode::decode(b).ok()?;
let op = op.into_iter().next()?;
let len = cubecode::opcode::encode(&[op]).len();
Some(One { op, len })
}
/// A patched working set ("code cache") inside a [`CubeStore`].
///
/// `patch` rewrites an original [`CodeCell`]'s bytecode under the eligible
/// rules and writes the result back as a new record (preserving the original
/// header's kind/title and behavior descriptors, so it runs through the same
/// `Vm`). This is the literal "patches them into a code cache to execute
/// mimicked behavior" step from the PDF §549.
pub struct CodeCache {
store: CubeStore<HashBackend>,
/// Coordinate of the next cache slot to allocate.
next_x: u8,
}
impl CodeCache {
/// Build an empty cache over a fresh in-memory backend.
pub fn new() -> Self {
CodeCache {
store: CubeStore::new(HashBackend::new()),
next_x: 1,
}
}
/// Patch `original` under `rules`: every op whose class matches a rule's
/// target (or whose rule target is `Any`) is replaced by that rule's
/// fragment. The rewritten cell is stored at a fresh cache coordinate and
/// returned (with its new label). The original store is untouched.
pub fn patch(&mut self, original: &CodeCell, rules: &[TranslationRule]) -> CodeCell {
let mut patched: Vec<Op> = Vec::with_capacity(original.code.len());
for op in &original.code {
let class = OpClass::of(op);
let mut applied = false;
for r in rules {
if r.target == class || r.target == OpClass::Any {
patched.extend_from_slice(&r.fragment);
applied = true;
break;
}
}
if !applied {
patched.push(*op);
}
}
let label = Czyx::new(C_DBT_CACHE, 1, 1, self.next_x);
self.next_x = self.next_x.wrapping_add(1).max(1);
let (h, body) = cache_header_for(original, &cubecode::opcode::encode(&patched));
self.store.put_record(label, &h, &body);
CodeCell::from_record(label, &h, &body).expect("patched cell is always valid bytecode")
}
/// Run a patched cache entry in the VM, returning its result.
pub fn run(&self, cell: &CodeCell) -> cubecode::RunResult {
let mut vm = Vm::new(self.store.clone());
vm.run(cell.label)
}
/// Borrow the backing store (e.g. to persist or inspect).
pub fn store(&self) -> &CubeStore<HashBackend> {
&self.store
}
}
impl Default for CodeCache {
fn default() -> Self {
Self::new()
}
}
/// Build a cache record header that preserves the original's kind/title and
/// behavior descriptors, but marks it a `Variant` (a mimicked implementation).
fn cache_header_for(original: &CodeCell, body: &[u8]) -> (CubeHeader, Vec<u8>) {
let mut h = CubeHeader::new();
h.title = original.name().map(|s| format!("mimic:{}", s));
h.doc_type = Some(Kind::Variant.as_str().into());
h.size_bytes = Some(body.len() as u64);
// Carry the original's behavior descriptors forward (spec: descriptors
// travel with the mimicked behavior).
h.flags.0 |= original.header.flags.0 & cubecode::HEADER_FLAG_BEHAVIOR;
h.refresh_flags();
(h, body.to_vec())
}
/// The DBT runtime: discovers CZYX-stored translation rules and can `mimic` a
/// target code cell by patching it under those rules and running the result.
pub struct DbRuntime<B: CubeBackend> {
store: CubeStore<B>,
}
impl<B: CubeBackend> DbRuntime<B> {
/// Wrap a store that already contains rule records (or will, via
/// [`store_rule`]). The runtime is read-only over this store.
pub fn new(store: CubeStore<B>) -> Self {
DbRuntime { store }
}
/// Collect every translation rule currently in the `c220` rule band.
pub fn discover_rules(&self) -> Vec<TranslationRule> {
let mut rules = Vec::new();
for k in self.store.keys() {
if k.c != C_DBT_RULE {
continue;
}
if let Some((_, body)) = self.store.get_record(&k) {
if let Some(r) = TranslationRule::from_bytes(&body) {
rules.push(r);
}
}
}
rules
}
/// Fetch a code cell by coordinate from the wrapped store.
pub fn fetch(&self, label: Czyx) -> Option<CodeCell> {
let (h, b) = self.store.get_record(&label)?;
CodeCell::from_record(label, &h, &b)
}
/// Mimic `target`: fetch it, apply all discovered rules, patch into a fresh
/// cache, and run the patched version — returning the `RunResult`. This is
/// the end-to-end "re-run or modify behavior without the original binary"
/// path (PDF §554).
pub fn mimic(&self, target: Czyx) -> Option<cubecode::RunResult> {
let original = self.fetch(target)?;
let rules = self.discover_rules();
let mut cache = CodeCache::new();
let patched = cache.patch(&original, &rules);
Some(cache.run(&patched))
}
}
/// Store a translation rule into the `c220` rule band at a fresh coordinate.
/// Returns the coordinate it was written to.
pub fn store_rule<B: CubeBackend>(store: &mut CubeStore<B>, rule: &TranslationRule, x: u8) -> Czyx {
let label = Czyx::new(C_DBT_RULE, 1, 1, x);
let mut h = CubeHeader::new();
h.title = Some(rule.name.clone());
h.doc_type = Some(Kind::Variant.as_str().into());
h.size_bytes = Some(rule.to_bytes().len() as u64);
h.refresh_flags();
store.put_record(label, &h, &rule.to_bytes());
label
}
#[cfg(test)]
mod tests {
use super::*;
use cubecode::opcode::Op;
use cubecode::RunResult;
use cubecoords::Czyx;
fn sample_cell(coord: Czyx, code: Vec<Op>) -> CodeCell {
let mut h = CubeHeader::new();
h.title = Some("sample".into());
h.doc_type = Some(Kind::Fn.as_str().into());
h.refresh_flags();
CodeCell::from_record(coord, &h, &cubecode::opcode::encode(&code))
.expect("sample is valid bytecode")
}
#[test]
fn rule_round_trips_through_bytes() {
let r = TranslationRule {
name: "double-add".into(),
target: OpClass::Add,
fragment: vec![Op::Const(2), Op::Mul],
};
let bytes = r.to_bytes();
let back = TranslationRule::from_bytes(&bytes).expect("rule decodes");
assert_eq!(r, back);
}
#[test]
fn cache_patches_and_runs() {
// Original: Const 3, Const 4, Add, Halt => 3+4 = 7.
let orig = sample_cell(
Czyx::new(1, 1, 1, 1),
vec![Op::Const(3), Op::Const(4), Op::Add, Op::Halt],
);
// Rule: replace `Add` with `Const 2, Mul` => (a)*(2). With a=3,b=4:
// naive in-place substitution yields Const3, Const4, Const2, Mul =>
// 4*2 = 8. This proves the patched fragment is what runs.
let rule = TranslationRule {
name: "mul-by-2".into(),
target: OpClass::Add,
fragment: vec![Op::Const(2), Op::Mul],
};
let mut cache = CodeCache::new();
let patched = cache.patch(&orig, &[rule]);
assert_eq!(patched.kind(), Kind::Variant, "cache entry is a Variant");
match cache.run(&patched) {
RunResult::Halted { top: Some(v) } => assert_eq!(v, 8, "patched semantics differ"),
other => panic!("patched run failed: {:?}", other),
}
}
#[test]
fn runtime_mimic_end_to_end() {
let mut store = CubeStore::new(HashBackend::new());
// Original target at c1.
let target = Czyx::new(1, 1, 1, 1);
let orig = sample_cell(target, vec![Op::Const(10), Op::Const(5), Op::Sub, Op::Halt]);
store.put_record(orig.label, &orig.header, &orig.body());
// Rule in the c220 band: replace Sub with Add (10+5=15 instead of 5).
let rule = TranslationRule {
name: "sub->add".into(),
target: OpClass::Sub,
fragment: vec![Op::Add],
};
store_rule(&mut store, &rule, 1);
let rt = DbRuntime::new(store);
let rules = rt.discover_rules();
assert_eq!(rules.len(), 1, "rule discovered in c220 band");
match rt.mimic(target) {
Some(RunResult::Halted { top: Some(v) }) => assert_eq!(v, 15, "mimic applied rule"),
other => panic!("mimic failed: {:?}", other),
}
}
#[test]
fn mimic_without_rules_matches_original() {
let mut store = CubeStore::new(HashBackend::new());
let target = Czyx::new(1, 1, 1, 2);
let orig = sample_cell(target, vec![Op::Const(7), Op::Const(2), Op::Mul, Op::Halt]);
store.put_record(orig.label, &orig.header, &orig.body());
let rt = DbRuntime::new(store);
match rt.mimic(target) {
Some(RunResult::Halted { top: Some(v) }) => assert_eq!(v, 14),
other => panic!("mimic without rules should run original: {:?}", other),
}
}
}
+2
View File
@@ -448,6 +448,8 @@ impl<B: CubeBackend + 'static> Filesystem for CubeFuse<B> {
kind: Kind::Directory,
size: 0,
mode: (mode & 0o7777) as u16,
word_flags: 0,
header_flags: 0,
uid: req.uid(),
gid: req.gid(),
created_at: 0,
+27 -4
View File
@@ -106,6 +106,10 @@ pub struct Attr {
pub created_at: u64,
/// Number of hard links: 1 for files, 2 for directories (`.` and `..`).
pub nlink: u32,
/// Per-word 16-bit flag field of the backing record (header tag 14). 0 for dirs.
pub word_flags: u16,
/// Per-record HeaderFlags of the backing record (header tag 12). 0 for dirs.
pub header_flags: u16,
}
/// The filesystem.
@@ -234,14 +238,14 @@ impl<B: CubeBackend> CubeFs<B> {
let kind = self.kind_of(p)?;
let coord = parsed.prefix_coord();
let acl = self.acl_of(coord);
let (size, created_at) = if kind == Kind::File {
let (size, created_at, word_flags, header_flags) = if kind == Kind::File {
let c = parsed.czyx().expect("file implies full coord");
match self.store.get_record(&c) {
Some((h, body)) => (body.len() as u64, h.created_at.unwrap_or(0)),
None => (0, 0),
Some((h, body)) => (body.len() as u64, h.created_at.unwrap_or(0), h.word_flags.bits(), h.flags.bits()),
None => (0, 0, 0, 0),
}
} else {
(0, 0)
(0, 0, 0, 0)
};
// Decision: directories report 0o755 unless an explicit ACL exists,
// because the file default (0644) would make every directory
@@ -262,6 +266,8 @@ impl<B: CubeBackend> CubeFs<B> {
gid: acl.gid,
created_at,
nlink: if kind == Kind::Directory { 2 } else { 1 },
word_flags,
header_flags,
})
}
@@ -771,6 +777,23 @@ mod tests {
assert!(f.read(p, 0, 1, 2000, 2000).is_ok());
}
#[test]
fn getattr_surfaces_word_flags_and_header_flags() {
use cubecoords::WordFlags;
let mut f = fs();
let mut h = CubeHeader::new();
h.doc_type = Some("fn".into());
h.word_flags = WordFlags::from_bits(
WordFlags::START_RECORD | WordFlags::END_RECORD | WordFlags::IS_HEADER | (0b10 << 6),
);
h.refresh_flags();
f.store_mut().put_record(cubecoords::Czyx::new(1, 1, 1, 1), &h, b"hi");
let a = f.getattr("/c001/z001/y001/x001").unwrap();
assert_eq!(a.word_flags & WordFlags::START_RECORD, WordFlags::START_RECORD);
assert_eq!((a.word_flags & WordFlags::TYPE_MASK) >> 6, 0b10, "type=code");
assert_eq!(a.header_flags & cubecoords::HeaderFlags::DOC_TYPE, cubecoords::HeaderFlags::DOC_TYPE);
}
#[test]
fn directories_default_to_traversable() {
let mut f = fs();
+528 -12
View File
@@ -151,7 +151,6 @@ pub struct CubeStore<B: CubeBackend> {
/// encoding is used to avoid a serde dependency at Package 1.)
mod record_codec {
use cubecoords::{CubeHeader, Czyx};
use std::collections::HashMap;
// Compact, dependency-free encoding of the header.
// Fields are written in a fixed tag-length-value stream so unknown
@@ -170,6 +169,11 @@ mod record_codec {
// 9 total_remote_accesses (u64 le)
// 10 last_access (u64 le)
// 11 last_remote_access (u64 le)
// 14 word_flags (u16 le) — per-word 16-bit tri-channel WordFlags
// 13 path (utf8) — POSIX path metatag (the original filesystem path a
// record was migrated from). Carries the directory hierarchy as DATA
// so the associative query layer can reconstruct nesting from flags.
// (Tag 12, raw flag bits, is documented at its emit site below.)
pub fn encode_header(h: &CubeHeader) -> Vec<u8> {
let mut out = Vec::new();
@@ -213,6 +217,9 @@ mod record_codec {
if let Some(a) = h.last_remote_access {
put_u64(&mut out, 11, a);
}
if let Some(p) = &h.path {
put_utf8(&mut out, 13, p);
}
// Tag 12: raw flag bits. Serializes out-of-band/spare bits (e.g.
// `cubecrypt::HEADER_FLAG_ENCRYPTED`) that are not derived from
// structured fields, so they survive an encode/decode round-trip.
@@ -220,6 +227,11 @@ mod record_codec {
out.push(12);
out.extend_from_slice(&h.flags.bits().to_le_bytes());
}
// Tag 14: per-word 16-bit tri-channel WordFlags field (see cubecoords).
if h.word_flags.bits() != 0 {
out.push(14);
out.extend_from_slice(&h.word_flags.bits().to_le_bytes());
}
out
}
@@ -297,6 +309,11 @@ mod record_codec {
h.last_remote_access = Some(v);
b = rest;
}
13 => {
let (v, rest) = take_utf8(b)?;
h.path = Some(v);
b = rest;
}
12 => {
if b.len() < 2 {
return None;
@@ -305,6 +322,14 @@ mod record_codec {
h.flags = cubecoords::HeaderFlags::flags_from_bits(raw);
b = &b[2..];
}
14 => {
if b.len() < 2 {
return None;
}
let raw = u16::from_le_bytes([b[0], b[1]]);
h.word_flags = cubecoords::WordFlags::from_bits(raw);
b = &b[2..];
}
_ => return None, // unknown tag -> reject (strict at Package 1)
}
}
@@ -344,12 +369,6 @@ mod record_codec {
let s = String::from_utf8(rest[..n].to_vec()).ok()?;
Some((s, &rest[n..]))
}
/// Keep `HashMap` referenced so the dependency is explicit in this module
/// even though the codec itself is generic over bytes. (Prevents an
/// unused-import warning if the backend type changes.)
#[allow(dead_code)]
pub(crate) fn _assert_backend_assoc(_: &HashMap<u32, Vec<u8>>) {}
}
impl<B: CubeBackend> CubeStore<B> {
@@ -387,20 +406,55 @@ impl<B: CubeBackend> CubeStore<B> {
}
/// Fetch and split a record into `(header, body)`.
///
/// Records written through the record path carry a TLV envelope
/// (`u32 header-len | header | body`). Records written through the *raw*
/// path (`put_raw` / the daemon's `rawput` verb, used by OS-layer services)
/// carry no envelope at all.
///
/// Historically a non-enveloped payload made this return `None`, which
/// callers such as `cubefs`'s `getattr`/`read` and the daemon's `stat` verb
/// translate into "size 0 / empty file". A record holding real bytes was
/// therefore *silently invisible* through the filesystem while `rawget`
/// happily returned its contents — the OS-in-CUBE migration hit exactly
/// this (2026-08-13): every `rawput` OS record listed as a 0-byte file.
///
/// Losing data silently is never the right failure mode, so a payload that
/// is not a well-formed envelope is now surfaced as a raw body under a
/// synthesized header. `rawput` data becomes readable through the
/// filesystem, and no caller has to special-case the two write paths.
pub fn get_record(&self, label: &Czyx) -> Option<(CubeHeader, Vec<u8>)> {
let raw = self.backend.get(label)?;
// Fall back to treating the payload as a raw (un-enveloped) body when
// it cannot be parsed as `len | header | body`.
let raw_fallback = |bytes: &[u8]| {
let mut h = CubeHeader::new();
h.size_bytes = Some(bytes.len() as u64);
// Keep the synthesized header's flag bits consistent with its
// fields, so associative queries (`scan_by_flag(SIZE_BYTES)`) see
// raw records too. Only the derived low bits are recomputed; any
// out-of-band bits already on the payload-derived header (none
// here, since raw payloads carry no header) are preserved by
// `refresh_flags`.
h.refresh_flags();
Some((h, bytes.to_vec()))
};
if raw.len() < 4 {
return None;
return raw_fallback(&raw);
}
let mut len = [0u8; 4];
len.copy_from_slice(&raw[..4]);
let hlen = u32::from_le_bytes(len) as usize;
if raw.len() < 4 + hlen {
return None;
return raw_fallback(&raw);
}
match record_codec::decode_header(&raw[4..4 + hlen]) {
Some(hdr) => {
let body = raw[4 + hlen..].to_vec();
Some((hdr, body))
}
None => raw_fallback(&raw),
}
let hdr = record_codec::decode_header(&raw[4..4 + hlen])?;
let body = raw[4 + hlen..].to_vec();
Some((hdr, body))
}
/// Raw backend write (non-record payloads, e.g. ACL/xattr/volume buckets).
@@ -472,6 +526,167 @@ impl<B: CubeBackend> CubeStore<B> {
out.sort();
out
}
/// PDF Package 2 API: `scan_by_flag` — associative storage lookup.
///
/// Returns every coordinate whose decoded [`CubeHeader`] carries `flag`
/// set. This is the "query by metatag/flag, not by path" primitive the
/// source PDF describes (§"associative storage"): records are addressed
/// by *what they are* (a flag bit) rather than *where they live* (a
/// coordinate path). `cube-os-*` services that emit typed records
/// (e.g. a log line tagged `doc_type = "klog"`) become discoverable by
/// event type without knowing their CZYX address in advance.
///
/// Records written through the RAW path (`rawput`, used by the OS layers)
/// carry only a synthesized `size_bytes` header, so they will NOT match
/// a content flag unless the writer also set one. The associative query
/// is therefore most powerful when records are written through
/// [`CubeStore::put_record`] with a populated [`CubeHeader`].
pub fn scan_by_flag(&self, flag: u16) -> Vec<Czyx> {
let mut out: Vec<Czyx> = self
.backend
.keys()
.into_iter()
.filter(|k| {
self.get_record(k)
.map(|(h, _)| h.flags.has(flag))
.unwrap_or(false)
})
.collect();
out.sort();
out
}
/// Query records by per-word flag (the 16-bit WordFlags metadata stripe
/// carried in the header, tag 14). Parallel to [`scan_by_flag`] but on
/// `word_flags`, so per-word metadata (encrypted/compressed/stego/type/
/// arrangement/continuation) is queryable without decoding the body.
pub fn scan_by_word_flag(&self, flag: u16) -> Vec<Czyx> {
let mut out: Vec<Czyx> = self
.backend
.keys()
.into_iter()
.filter(|k| {
self.get_record(k)
.map(|(h, _)| h.word_flags.has(flag))
.unwrap_or(false)
})
.collect();
out.sort();
out
}
/// PDF Package 2 API: `scan_by_type` — query records by `doc_type`
/// (the "event type" / file-extension analogue the PDF calls Flag 2).
///
/// This is the concrete "log lookup by query" the OS layers want: instead
/// of addressing `/cubefs/c200/z004/...` directly, you ask "every record
/// whose doc_type is `klog`" and get back all matching coordinates. The
/// comparison is exact-match (case-sensitive) against the header field.
pub fn scan_by_type(&self, doc_type: &str) -> Vec<Czyx> {
let mut out: Vec<Czyx> = self
.backend
.keys()
.into_iter()
.filter(|k| {
self.get_record(k)
.map(|(h, _)| h.doc_type.as_deref() == Some(doc_type))
.unwrap_or(false)
})
.collect();
out.sort();
out
}
/// Return `(label, header, body)` for every record matching `flag`.
/// Convenience wrapper over [`CubeStore::scan_by_flag`] that also pulls
/// the decoded payload so a caller (e.g. a `cubelog` query tool) can
/// present the matching records directly.
pub fn query_by_flag(&self, flag: u16) -> Vec<(Czyx, CubeHeader, Vec<u8>)> {
self.scan_by_flag(flag)
.into_iter()
.filter_map(|k| self.get_record(&k).map(|(h, b)| (k, h, b)))
.collect()
}
/// Return `(label, header, body)` for every record whose `doc_type`
/// matches. See [`CubeStore::scan_by_type`] for the matching semantics.
pub fn query_by_type(&self, doc_type: &str) -> Vec<(Czyx, CubeHeader, Vec<u8>)> {
self.scan_by_type(doc_type)
.into_iter()
.filter_map(|k| self.get_record(&k).map(|(h, b)| (k, h, b)))
.collect()
}
/// PDF Package 2 / OS-in-CUBE: reconstruct a *directory hierarchy from
/// metatags*. Returns every coordinate whose header carries a `path`
/// metatag equal to `path` (exact) — i.e. "the file at this path".
///
/// Combined with [`CubeStore::scan_by_path_prefix`], this lets the cube
/// answer "everything under /etc" WITHOUT the filesystem supporting
/// recursive nesting: the path is stored as a flag-addressed field, not
/// as an inode tree. This is the source PDF's prescribed model ("operate
/// on CZYX records and Null-space flags rather than paths and inodes").
pub fn scan_by_path(&self, path: &str) -> Vec<Czyx> {
let mut out: Vec<Czyx> = self
.backend
.keys()
.into_iter()
.filter(|k| {
self.get_record(k)
.map(|(h, _)| h.path.as_deref() == Some(path))
.unwrap_or(false)
})
.collect();
out.sort();
out
}
/// "List a directory": every record whose `path` metatag is *under* the
/// given directory prefix (e.g. `scan_by_path_prefix("/etc")` returns
/// `/etc/passwd`, `/etc/network/interfaces`, ...). Unlike a real FS, the
/// nesting is a shared prefix on the `path` metatag of several records
/// (the source PDF: "operate on CZYX records and Null-space flags rather
/// than paths and inodes") — there is no half-coordinate directory entry.
///
/// A query matches BOTH an exact leaf (`/etc/hostname`) and any descendant
/// (`/etc/hostname` and `/etc/passwd` both answer `/etc`), because a path
/// is just a string: `/etc/hostname` == dir, and `/etc/passwd` starts with
/// `dir + '/'`.
pub fn scan_by_path_prefix(&self, dir: &str) -> Vec<Czyx> {
let sep = format!("{dir}/");
let mut out: Vec<Czyx> = self
.backend
.keys()
.into_iter()
.filter(|k| {
self.get_record(k)
.map(|(h, _)| match &h.path {
Some(p) => p == dir || p.starts_with(&sep),
None => false,
})
.unwrap_or(false)
})
.collect();
out.sort();
out
}
/// `(label, header, body)` for [`CubeStore::scan_by_path`].
pub fn query_by_path(&self, path: &str) -> Vec<(Czyx, CubeHeader, Vec<u8>)> {
self.scan_by_path(path)
.into_iter()
.filter_map(|k| self.get_record(&k).map(|(h, b)| (k, h, b)))
.collect()
}
/// `(label, header, body)` for [`CubeStore::scan_by_path_prefix`].
pub fn query_by_path_prefix(&self, dir: &str) -> Vec<(Czyx, CubeHeader, Vec<u8>)> {
self.scan_by_path_prefix(dir)
.into_iter()
.filter_map(|k| self.get_record(&k).map(|(h, b)| (k, h, b)))
.collect()
}
}
#[cfg(test)]
@@ -501,6 +716,25 @@ mod tests {
assert!(rh.flags.has(cubecoords::HeaderFlags::HAS_ASSOCIATIONS));
}
#[test]
fn word_flags_roundtrip() {
let mut h = CubeHeader::new();
h.word_flags = cubecoords::WordFlags::from_bits(
cubecoords::WordFlags::ENCRYPTED
| cubecoords::WordFlags::CONTINUATION
| (cubecoords::WordFlags::TYPE_MASK & 0b11),
);
let enc = super::record_codec::encode_header(&h);
let dec = super::record_codec::decode_header(&enc).unwrap();
assert_eq!(dec.word_flags.bits(), h.word_flags.bits(), "word_flags must round-trip");
// refresh_flags recomputes only the per-RECORD HeaderFlags; the separate
// per-word word_flags field must be preserved untouched.
let mut dec2 = dec;
dec2.refresh_flags();
assert_eq!(dec2.word_flags.bits(), h.word_flags.bits());
}
#[test]
fn missing_record_is_none() {
let store = CubeStore::new(HashBackend::new());
@@ -515,4 +749,286 @@ mod tests {
assert_eq!(store.get_raw(&Czyx::new(0, 0, 0, 0)), Some(vec![1]));
assert_eq!(store.get_raw(&Czyx::new(0, 0, 0, 1)), Some(vec![2]));
}
/// Regression (2026-08-13, OS-in-CUBE migration): a payload written via the
/// RAW path (`put_raw`, i.e. the daemon's `rawput` verb used by OS-layer
/// services) must still be READABLE through `get_record`, because that is
/// what `cubefs` getattr/read and the `stat` verb go through. Before the
/// fix these records reported size 0 and read back empty — real bytes were
/// silently invisible through the filesystem.
#[test]
fn get_record_surfaces_raw_unenveloped_payloads() {
let mut store = CubeStore::new(HashBackend::new());
let coord = Czyx::new(200, 70, 1, 1);
let payload = b"CUBELINUX OS PROCESS SNAPSHOT\nprocs: 118\n".to_vec();
store.put_raw(coord, payload.clone());
let (hdr, body) = store
.get_record(&coord)
.expect("raw payload must be visible as a record, not vanish");
assert_eq!(body, payload, "body must round-trip byte-for-byte");
assert_eq!(
hdr.size_bytes,
Some(payload.len() as u64),
"synthesized header must report the true size so getattr is correct"
);
// A short payload (< 4 bytes, cannot even hold a length prefix) is the
// other edge the old code dropped: counters like "118" land here.
let short = Czyx::new(200, 70, 2, 1);
store.put_raw(short, b"118".to_vec());
let (h2, b2) = store.get_record(&short).expect("short raw payload visible");
assert_eq!(b2, b"118");
assert_eq!(h2.size_bytes, Some(3));
}
/// The envelope path must be unaffected by the raw fallback: a properly
/// stored record still decodes its real header (not a synthesized one).
#[test]
fn get_record_still_prefers_the_real_envelope() {
let mut store = CubeStore::new(HashBackend::new());
let coord = Czyx::new(201, 5, 1, 1);
let mut hdr = CubeHeader::new();
hdr.title = Some("real-record".to_string());
store.put_record(coord, &hdr, b"payload");
let (got, body) = store.get_record(&coord).expect("enveloped record");
assert_eq!(body, b"payload");
assert_eq!(
got.title.as_deref(),
Some("real-record"),
"must decode the true header, not fall back to raw"
);
}
/// PDF Package 2: `scan_by_flag` finds records by flag bit, independent of
/// their coordinate address. A record tagged by `DOC_TYPE` must surface
/// when queried for that flag and be absent otherwise.
#[test]
fn scan_by_flag_finds_typed_records() {
let mut store = CubeStore::new(HashBackend::new());
// A "klog" typed event record.
let mut klog = CubeHeader::new();
klog.doc_type = Some("klog".into());
klog.refresh_flags();
store.put_record(Czyx::new(200, 4, 2, 1), &klog, b"kernel: eth0 up");
// An untyped raw record (the common OS-layer case).
store.put_raw(Czyx::new(200, 70, 1, 1), b"118".to_vec());
let by_doc_type = store.scan_by_flag(cubecoords::HeaderFlags::DOC_TYPE);
assert_eq!(by_doc_type, vec![Czyx::new(200, 4, 2, 1)]);
// The raw record carries only a synthesized size flag, so it must NOT
// match DOC_TYPE.
assert!(!by_doc_type.contains(&Czyx::new(200, 70, 1, 1)));
let by_size = store.scan_by_flag(cubecoords::HeaderFlags::SIZE_BYTES);
assert!(
!by_size.is_empty(),
"synthesized raw headers carry SIZE_BYTES"
);
}
#[test]
fn scan_by_word_flag_finds_records() {
let mut store = CubeStore::new(HashBackend::new());
let mut h = CubeHeader::new();
h.word_flags = cubecoords::WordFlags::from_bits(cubecoords::WordFlags::ENCRYPTED);
store.put_record(Czyx::new(200, 5, 1, 1), &h, b"secret body");
let mut h2 = CubeHeader::new();
h2.doc_type = Some("plain".into());
h2.refresh_flags();
store.put_record(Czyx::new(200, 5, 2, 2), &h2, b"plain body");
let enc = store.scan_by_word_flag(cubecoords::WordFlags::ENCRYPTED);
assert_eq!(enc, vec![Czyx::new(200, 5, 1, 1)]);
assert!(!enc.contains(&Czyx::new(200, 5, 2, 2)));
}
/// `scan_by_type` is the concrete "log lookup by query" — pull every record
/// of a given event type without knowing its coordinate.
#[test]
fn scan_by_type_event_lookup() {
let mut store = CubeStore::new(HashBackend::new());
let mut a = CubeHeader::new();
a.doc_type = Some("klog".into());
let mut b = CubeHeader::new();
b.doc_type = Some("klog".into());
let mut c = CubeHeader::new();
c.doc_type = Some("state".into());
store.put_record(Czyx::new(200, 4, 2, 1), &a, b"k1");
store.put_record(Czyx::new(200, 4, 2, 2), &b, b"k2");
store.put_record(Czyx::new(200, 1, 1, 1), &c, b"s1");
let klogs = store.scan_by_type("klog");
assert_eq!(klogs.len(), 2);
assert!(klogs.contains(&Czyx::new(200, 4, 2, 1)));
assert!(klogs.contains(&Czyx::new(200, 4, 2, 2)));
// query_by_type returns the decoded payloads too.
let klogs_full = store.query_by_type("klog");
assert_eq!(klogs_full.len(), 2);
assert!(klogs_full.iter().any(|(_, _, b)| b == b"k1"));
assert!(klogs_full.iter().any(|(_, _, b)| b == b"k2"));
assert_eq!(store.scan_by_type("state"), vec![Czyx::new(200, 1, 1, 1)]);
assert!(store.scan_by_type("nonexistent").is_empty());
}
/// Path metatags reconstruct a nested directory hierarchy WITHOUT the FS
/// needing recursive inode trees (source PDF: "operate on CZYX records and
/// Null-space flags rather than paths and inodes"). `/etc/passwd` and
/// `/etc/network/interfaces` live at unrelated coordinates but share the
/// `/etc` prefix, so `scan_by_path_prefix("/etc")` finds both.
#[test]
fn scan_by_path_prefix_lists_directory() {
let mut store = CubeStore::new(HashBackend::new());
let mut mk = |c: u8, path: &str, dt: &str| {
let mut h = CubeHeader::new();
h.path = Some(path.to_string());
h.doc_type = Some(dt.to_string());
h.refresh_flags();
store.put_record(Czyx::new(200, 1, c, 1), &h, b"body");
};
mk(1, "/etc/passwd", "file");
mk(2, "/etc/network/interfaces", "file");
mk(3, "/etc/hosts", "file");
mk(4, "/usr/bin/ls", "file");
// Exact match.
assert_eq!(
store.scan_by_path("/etc/passwd"),
vec![Czyx::new(200, 1, 1, 1)]
);
// Directory listing via metatag prefix. Bare "/etc" matches every
// record whose path is "/etc" or starts with "/etc/" (an exact leaf
// like "/etc/hostname" answers "/etc", and "/etc/passwd" is a child).
let etc = store.scan_by_path_prefix("/etc");
assert_eq!(etc.len(), 3);
assert!(etc.contains(&Czyx::new(200, 1, 1, 1)));
assert!(etc.contains(&Czyx::new(200, 1, 2, 1)));
assert!(etc.contains(&Czyx::new(200, 1, 3, 1)));
assert!(
!etc.contains(&Czyx::new(200, 1, 4, 1)),
"/usr must not appear under /etc"
);
// An exact leaf answers its own parent directory query.
assert_eq!(
store.scan_by_path_prefix("/etc/hosts"),
vec![Czyx::new(200, 1, 3, 1)]
);
// Empty prefix = all paths.
assert_eq!(store.scan_by_path_prefix("").len(), 4);
// The path flag bit is set so the record is also flag-discoverable.
let by_path_flag = store.scan_by_flag(cubecoords::HeaderFlags::HAS_PATH);
assert_eq!(by_path_flag.len(), 4);
}
// ---- HashBackend trait-level tests (PDF spec: HashMap<u32, Vec<u8>>) ----
#[test]
fn hash_backend_put_get_delete_roundtrip() {
let mut b = HashBackend::new();
let k = Czyx::new(10, 20, 30, 40);
b.put(k, b"payload".to_vec());
assert_eq!(b.get(&k), Some(b"payload".to_vec()));
b.delete(&k);
assert_eq!(b.get(&k), None);
}
#[test]
fn hash_backend_overwrite_replaces_value() {
let mut b = HashBackend::new();
let k = Czyx::new(0, 1, 2, 3);
b.put(k, b"old".to_vec());
b.put(k, b"new".to_vec());
assert_eq!(b.get(&k), Some(b"new".to_vec()));
}
#[test]
fn hash_backend_keys_sorted_order() {
let mut b = HashBackend::new();
for c in 1u8..=3 {
b.put(Czyx::new(50, c, 10, 1), vec![c]);
}
let keys = b.keys();
assert_eq!(keys.len(), 3);
// HashMap iteration is unordered; keys() must sort for stable output.
let mut sorted = keys.clone();
sorted.sort();
assert_eq!(keys, sorted);
for k in &keys {
assert_eq!(k.c, 50);
}
}
#[test]
fn hash_backend_scan_prefix_filters_correctly() {
let mut b = HashBackend::new();
b.put(Czyx::new(1, 2, 3, 4), b"a".to_vec());
b.put(Czyx::new(1, 9, 9, 9), b"b".to_vec());
b.put(Czyx::new(2, 0, 0, 0), b"c".to_vec());
b.put(Czyx::new(1, 2, 9, 9), b"d".to_vec());
// C=1 only.
let c1 = b.scan_prefix(1, None, None);
assert_eq!(c1.len(), 3);
for k in &c1 {
assert_eq!(k.c, 1);
}
// C=1, Z=2 only.
let c1z2 = b.scan_prefix(1, Some(2), None);
assert_eq!(c1z2.len(), 2);
for k in &c1z2 {
assert_eq!(k.c, 1);
assert_eq!(k.z, 2);
}
// C=2 only.
let c2 = b.scan_prefix(2, None, None);
assert_eq!(c2, vec![Czyx::new(2, 0, 0, 0)]);
// Non-existent C.
assert!(b.scan_prefix(99, None, None).is_empty());
}
#[test]
fn hash_backend_empty_keys_and_scan() {
let b = HashBackend::new();
assert!(b.keys().is_empty());
assert!(b.scan_prefix(0, None, None).is_empty());
}
#[test]
fn hash_backend_null_coord_is_valid_key() {
let mut b = HashBackend::new();
let null = Czyx::new(0, 0, 0, 0);
b.put(null, b"null-cube".to_vec());
assert_eq!(b.get(&null), Some(b"null-cube".to_vec()));
b.delete(&null);
assert_eq!(b.get(&null), None);
// Null cube (0,0,0,0) is distinct from a nearby coord.
let near = Czyx::new(0, 0, 0, 1);
b.put(near, b"near".to_vec());
assert_eq!(b.get(&near), Some(b"near".to_vec()));
assert_eq!(b.get(&null), None);
}
#[test]
fn hash_backend_multiple_backends_are_independent() {
let b1 = HashBackend::new();
let mut b2 = HashBackend::new();
b2.put(Czyx::new(1, 1, 1, 1), b"x".to_vec());
assert!(b1.get(&Czyx::new(1, 1, 1, 1)).is_none());
assert_eq!(b2.get(&Czyx::new(1, 1, 1, 1)), Some(b"x".to_vec()));
}
}
-16
View File
@@ -16,19 +16,3 @@ cubestore = { path = "../cubestore" }
cubefs = { path = "../cubefs" }
cubecode = { path = "../cubecode" }
cubecrypt = { path = "../cubecrypt" }
[[bin]]
name = "cube"
path = "src/bin/cube.rs"
[[bin]]
name = "cube-demo"
path = "src/bin/cube_demo.rs"
[[bin]]
name = "cube-server"
path = "src/bin/cube-server.rs"
[[bin]]
name = "cubec"
path = "src/bin/cubec.rs"
+3 -1
View File
@@ -76,9 +76,11 @@ the spec never defined.
## Binaries
* `cube-demo` — self-contained tour (write two linked cells, run; seal a
record, reopen + run). Prints evidence at each step.
record, reopen + run). Prints evidence at each step. (in `cubesys`)
* `cube` — CLI: `write`, `run`, `ls`, `stat`, `seal`, `open` over one store,
interactively (`repl`), from a file (`script`), or as the demo (`demo`).
Now a standalone crate (`cubecli/`); identical behavior to the former
`cubesys/src/bin/cube.rs`.
* `cube-server` — long-lived daemon: holds ONE `CubeStore` for its whole
lifetime and serves the same command language over a Unix-domain socket.
Snapshots the store to a JSON file on every request so the cube survives
+805 -18
View File
@@ -15,7 +15,7 @@ use crate::audit::{Audit, OP_DELETE, OP_GRANT, OP_OPEN, OP_READ, OP_REVOKE, OP_S
use crate::grants::{grant, grant_allows, perms_from_str, revoke, Owner, Perm};
use crate::store::ConcurrentStore;
use crate::tenant::{TenantIdentity, TenantSession};
use cubecode::{CodeCell, Kind, Op, Vm};
use cubecode::{CodeCell, Kind, Op, RunResult, Vm};
use cubecoords::{CubeHeader, Czyx};
use cubecrypt::{CubeEnv, KeySlot, Selector, TransformId};
use cubestore::{CubeStore, HashBackend};
@@ -448,10 +448,218 @@ impl Session {
))
}
}
"query-path" => {
// Reconstruct a directory listing from path metatags (no FS
// inode tree needed). `query-path /etc` returns every record
// whose `path` metatag is under /etc — the PDF's "operate on
// CZYX records and Null-space flags rather than paths/inodes".
let dir = it
.next()
.ok_or_else(|| "query-path needs <dir-prefix>".to_string())?;
let matches = store.query_by_path_prefix(dir);
if matches.is_empty() {
Ok(format!("query-path {dir} -> (no matches)"))
} else {
let mut names: Vec<String> = matches
.iter()
.filter_map(|c| {
store.get_record(c).map(|(h, _)| {
let p = h.path.as_deref().unwrap_or("");
format!("{}:{}", c.pack_u32(), p)
})
})
.collect();
names.sort();
Ok(format!(
"query-path {dir} -> {} matches:\n {}",
names.len(),
names.join("\n ")
))
}
}
"scan-word-flags" => {
let a = it
.next()
.ok_or_else(|| "scan-word-flags needs <flag> (name or 0xbits)".to_string())?;
let bits = parse_word_flag(a)?;
let coords = store.scan_by_word_flag(bits);
if coords.is_empty() {
Ok(format!("scan-word-flags {a} (0x{bits:04x}) -> (no matches)"))
} else {
let names: Vec<String> =
coords.iter().map(|c| c.pack_u32().to_string()).collect();
Ok(format!(
"scan-word-flags {a} (0x{bits:04x}) -> {} matches:\n {}",
coords.len(),
names.join("\n ")
))
}
}
"show-flags" => {
let cs = it.next().ok_or_else(|| "show-flags needs <C.Z.Y.X>".to_string())?;
let coord = parse_coord(cs).ok_or_else(|| format!("bad coordinate '{cs}' (want C.Z.Y.X)"))?;
match store.get_record(&coord) {
None => Ok(format!("show-flags {cs} -> (no record)")),
Some((h, body)) => Ok(format!(
"show-flags {cs}\n header_flags: 0x{:04x} {}\n word_flags: 0x{:04x} {}\n size: {} bytes",
h.flags.bits(), header_flag_names(h.flags.bits()),
h.word_flags.bits(), word_flag_names(h.word_flags.bits()),
body.len()))
}
}
// Session-message / note log: a searchable, per-session, per-day
// note area stored as CZYX records carrying WordFlags metadata
// (the structured replacement for the hand-maintained RESUME-*.md
// files and the Hermes session store).
"note" | "notes" => crate::notes::note_command(store, line),
// Project threads: list projects / walk a project's associated notes
// (the first-class-association model for tracking work between sessions).
"project" | "projects" => crate::notes::project_command(store, line),
"trace-capture" => {
let src_s = it.next().ok_or_else(|| "trace-capture needs <source C.Z.Y.X> <target C.Z.Y.X>".to_string())?;
let tgt_s = it.next().ok_or_else(|| "trace-capture needs <target C.Z.Y.X>".to_string())?;
let src = parse_coord(src_s).ok_or_else(|| format!("bad source coord '{src_s}'"))?;
let tgt = parse_coord(tgt_s).ok_or_else(|| format!("bad target coord '{tgt_s}'"))?;
let snap = store.read_snapshot();
let mut vm = cubecode::Vm::new(snap);
let (result, trace) = vm.run_captured(src);
let out = vm.output().to_vec();
// Store the captured trace as a Kind::Layer AI-artifact record,
// linked back to the source cell (a linked CZYX edge).
let mut h = CubeHeader::new();
h.title = Some(format!("trace:{}", src.pack_u32()));
h.doc_type = Some(cubecode::Kind::Layer.as_str().to_string());
h.linked_records = vec![src];
h.refresh_flags();
store.put_record(tgt, &h, &cubecode::encode(&trace));
Ok(format!(
"trace-capture {src_s} -> {tgt_s} (layer, {} ops, {:?}) output {} bytes",
trace.len(),
result,
out.len()
))
}
"trace-replay" => {
let tgt_s = it.next().ok_or_else(|| "trace-replay needs <C.Z.Y.X>".to_string())?;
let tgt = parse_coord(tgt_s).ok_or_else(|| format!("bad coord '{tgt_s}'"))?;
let snap = store.read_snapshot();
match cubecode::replay_trace(&snap, tgt) {
Some((result, out)) => Ok(format!(
"trace-replay {tgt_s}: {:?}\n output ({} bytes): {}",
result,
out.len(),
String::from_utf8_lossy(&out)
)),
None => Err(format!("trace-replay {tgt_s}: no valid trace record at that coord")),
}
}
"trace-verify" => {
let src_s = it.next().ok_or_else(|| "trace-verify needs <source> <trace> <golden>".to_string())?;
let tr_s = it.next().ok_or_else(|| "trace-verify needs <trace>".to_string())?;
let go_s = it.next().ok_or_else(|| "trace-verify needs <golden>".to_string())?;
let src = parse_coord(src_s).ok_or_else(|| format!("bad coord '{src_s}'"))?;
let tr = parse_coord(tr_s).ok_or_else(|| format!("bad coord '{tr_s}'"))?;
let go = parse_coord(go_s).ok_or_else(|| format!("bad coord '{go_s}'"))?;
let snap = store.read_snapshot();
match cubecode::verify_golden(&snap, src, tr, go) {
Ok(()) => Ok(format!(
"trace-verify {src_s} {tr_s} {go_s} -> PASS (replay + fresh run match golden)"
)),
Err(e) => Ok(format!("trace-verify {src_s} {tr_s} {go_s} -> FAIL: {e}")),
}
}
"trace-list" => {
let snap = store.read_snapshot();
let mut rows = Vec::new();
for k in snap.keys() {
if let Some((h, _)) = snap.get_record(&k) {
let dt = h.doc_type.clone().unwrap_or_default();
let is_artifact = matches!(dt.as_str(), "layer" | "golden" | "checkpoint" | "variant")
|| h.title.as_deref().map_or(false, |t| t.starts_with("trace:") || t.starts_with("golden:"));
if is_artifact {
let links = h.linked_records
.iter().map(|c| c.pack_u32().to_string()).collect::<Vec<_>>().join(",");
rows.push(format!("{} [{}] {} -> links: {links}",
k.pack_u32(), dt, h.title.as_deref().unwrap_or("")));
}
}
}
if rows.is_empty() {
Ok("trace-list -> (no layer/golden/checkpoint/variant/trace records)".to_string())
} else {
Ok(format!("trace-list -> {} records:\n {}", rows.len(), rows.join("\n ")))
}
}
"trace-links" => {
let cs = it.next().ok_or_else(|| "trace-links needs <C.Z.Y.X> [depth]".to_string())?;
let coord = parse_coord(cs).ok_or_else(|| format!("bad coord '{cs}'"))?;
let depth = it.next().and_then(|d| d.parse::<usize>().ok()).unwrap_or(3);
let snap = store.read_snapshot();
let hops = cubecode::lineage(&snap, coord, depth);
if hops.is_empty() {
return Ok(format!("trace-links {cs} -> (no record)"));
}
let mut lines = Vec::new();
for h in &hops {
let links = h.links.iter().map(|c| c.pack_u32().to_string()).collect::<Vec<_>>().join(" ");
lines.push(format!(" {} [{}] {} -> {links}", h.coord.pack_u32(), h.doc_type, h.title));
}
Ok(format!("trace-links {cs} (depth {depth}) -> {} hops:\n{}", hops.len(), lines.join("\n")))
}
"golden-capture" => {
let src_s = it.next().ok_or_else(|| "golden-capture needs <source> <trace> <golden>".to_string())?;
let tr_s = it.next().ok_or_else(|| "golden-capture needs <trace>".to_string())?;
let go_s = it.next().ok_or_else(|| "golden-capture needs <golden>".to_string())?;
let name = it.next().unwrap_or("golden").to_string();
let src = parse_coord(src_s).ok_or_else(|| format!("bad coord '{src_s}'"))?;
let tr = parse_coord(tr_s).ok_or_else(|| format!("bad coord '{tr_s}'"))?;
let go = parse_coord(go_s).ok_or_else(|| format!("bad coord '{go_s}'"))?;
let snap = store.read_snapshot();
let mut vm = cubecode::Vm::new(snap);
let (_r, trace) = vm.run_captured(src);
let out = vm.output().to_vec();
// Trace record: Kind::Layer, linked to the source.
let mut h = CubeHeader::new();
h.title = Some(format!("trace:{name}"));
h.doc_type = Some(cubecode::Kind::Layer.as_str().to_string());
h.linked_records = vec![src];
h.refresh_flags();
store.put_record(tr, &h, &cubecode::encode(&trace));
// Golden output record, linked to the trace.
let mut hg = CubeHeader::new();
hg.title = Some(format!("golden:{name}"));
hg.doc_type = Some("golden".into());
hg.linked_records = vec![tr];
hg.refresh_flags();
store.put_record(go, &hg, &out);
Ok(format!(
"golden-capture {src_s} -> {tr_s} (layer) + {go_s} (golden), output {} bytes",
out.len()
))
}
"prog" => {
let path = it.next().ok_or_else(|| "prog needs <path>".to_string())?;
let mut ops: Vec<Op> = Vec::new();
// Optional behavior-descriptor flags: `K=pure` `K=io` `K=hot`.
// These stamp the OS-kernel behavior bits (PDF §524525) on the
// record header and switch the kind to `kernel` so the cube
// correctly classifies operator kernels vs plain functions.
let mut descriptor: Option<cubecode::Behavior> = None;
while let Some(tok) = it.next() {
if let Some(flag) = tok.strip_prefix("K=") {
let bit = match flag {
"pure" => cubecode::Behavior::PURE,
"io" => cubecode::Behavior::IO_HEAVY,
"alloc" => cubecode::Behavior::ALLOCATES,
"net" => cubecode::Behavior::NETWORK,
"sec" => cubecode::Behavior::SECURITY_SENSITIVE,
"hot" => cubecode::Behavior::HOT_PATH,
other => return Err(format!("prog: unknown descriptor K={other}")),
};
let cur = descriptor.unwrap_or_default();
descriptor = Some(cubecode::Behavior(cur.0 | bit));
continue;
}
let arg = if takes_arg(tok) {
it.next()
.and_then(|a| a.parse::<u8>().ok())
@@ -464,11 +672,13 @@ impl Session {
if ops.is_empty() {
return Err("prog: no ops given".to_string());
}
let is_kernel = descriptor.is_some();
let kind = if is_kernel { Kind::Kernel } else { Kind::Fn };
let name = path.rsplit('/').next().unwrap_or(path);
// Compute the exact record bytes `put_record` would write, using
// a throwaway store so we can buffer (or apply) them without
// duplicating the record codec.
let coord = scratch_code_coord(path, Kind::Fn, name, &ops)?;
let coord = scratch_code_coord(path, kind, name, &ops)?;
// Task 6: reject overwriting a record owned by a different owner.
if let Some(msg) = self.admit_mutate(coord, Perm::Write) {
self.audit_now(OP_WRITE, coord, false);
@@ -478,28 +688,341 @@ impl Session {
let owner = self.identity.as_ref().map(|i| i.owner_local.as_str());
let value = {
let mut scratch = CubeStore::new(HashBackend::new());
crate::store_code_cell(&mut scratch, path, Kind::Fn, name, &[], &ops, owner)
.map_err(|e| e.to_string())?;
crate::store_code_cell(
&mut scratch,
path,
kind,
name,
&[],
&ops,
owner,
descriptor,
)
.map_err(|e| e.to_string())?;
scratch.get_raw(&coord).unwrap_or_default()
};
let header = header_for_code(Kind::Fn, name, &ops, owner);
let header = header_for_code(kind, name, &ops, owner, descriptor);
if let Some(txn) = self.txn.as_mut() {
txn.ops.push(TxnOp {
coord,
put: Some((value, header)),
});
let kind_tag = if is_kernel { "kernel" } else { "fn" };
return Ok(format!(
"buffered prog {path} ({kind_tag}, {} ops) — commit to apply",
ops.len()
));
}
let coord = store
.put_code_cell(path, kind, name, &[], &ops, owner, descriptor)
.map_err(|e| e.to_string())?;
let kind_tag = if is_kernel { "kernel" } else { "fn" };
Ok(format!(
"wrote program {path} -> coord {} ({kind_tag}, {} ops)",
coord.pack_u32(),
ops.len()
))
}
"link" => {
// Attach a callee coordinate to an existing code cell's call
// graph. `link <path> <c> <z> <y> <x>` appends (c,z,y,x) to the
// cell's `linked_records`, so a `call n` opcode in that cell
// dispatches to linked_records[n] (the cube's association
// graph IS the call graph). This is what makes functions stored
// in CUBE callable from other functions stored in CUBE.
let path = it.next().ok_or_else(|| "link needs <path>".to_string())?;
let c = parse_u8(it.next(), "link needs <c>")?;
let z = parse_u8(it.next(), "link needs <z>")?;
let y = parse_u8(it.next(), "link needs <y>")?;
let x = parse_u8(it.next(), "link needs <x>")?;
let target = Czyx::new(c, z, y, x);
let coord = crate::path_to_czyx(path).map_err(|e| e.to_string())?;
if let Some(msg) = self.admit_mutate(coord, Perm::Write) {
self.audit_now(OP_WRITE, coord, false);
return Err(msg);
}
self.audit_now(OP_WRITE, coord, true);
// Load the existing cell, append the link, re-store it.
let cell = crate::load_code_cell(&store.read_snapshot(), path)
.map_err(|e| format!("link: cannot load {path}: {e:?}"))?;
let mut links = cell.links().to_vec();
if links.contains(&target) {
return Ok(format!(
"link {path}: {target:?} already linked (degree {})",
links.len()
));
}
links.push(target);
let code = cubecode::decode(&cell.body())
.map_err(|e| format!("link: bad bytecode in {path}: {e:?}"))?;
let owner = self.identity.as_ref().map(|i| i.owner_local.as_str());
store
.put_code_cell(
path,
cell.kind(),
cell.name().unwrap_or(path),
&links,
&code,
owner,
None,
)
.map_err(|e| e.to_string())?;
Ok(format!(
"linked {path} -> {target:?} (call-graph degree now {})",
links.len()
))
}
// ---- OS-operator-kernel authoring + thin effector (Steps 1 & 2) ----
// Per the reframe, the OS's *computation* lives in CUBE as operator
// kernels (call graphs + behavior descriptors, PDF §524525). The
// native OS layer is only a thin effector: it reads a kernel's
// computed result and applies the effect. The cubevm never does
// store-IO itself. `kernel` authors a kernel; `tick` lays down the
// OS kernel call-graph; `native-apply` is the effector boundary.
"kernel" => {
// `kernel <path> [K=pure|io|hot ...] <op> <arg> ...`
// Like `prog`, but the cell is always kind=Kernel and accepts
// behavior-descriptor flags so the OS marks operator kernels
// distinctly from plain functions.
let path = it.next().ok_or_else(|| "kernel needs <path>".to_string())?;
let mut ops: Vec<Op> = Vec::new();
let mut descriptor: Option<cubecode::Behavior> = None;
while let Some(tok) = it.next() {
if let Some(flag) = tok.strip_prefix("K=") {
let bit = match flag {
"pure" => cubecode::Behavior::PURE,
"io" => cubecode::Behavior::IO_HEAVY,
"alloc" => cubecode::Behavior::ALLOCATES,
"net" => cubecode::Behavior::NETWORK,
"sec" => cubecode::Behavior::SECURITY_SENSITIVE,
"hot" => cubecode::Behavior::HOT_PATH,
other => return Err(format!("kernel: unknown descriptor K={other}")),
};
let cur = descriptor.unwrap_or_default();
descriptor = Some(cubecode::Behavior(cur.0 | bit));
continue;
}
let arg = if takes_arg(tok) {
it.next()
.and_then(|a| a.parse::<u8>().ok())
.ok_or_else(|| format!("kernel: {tok} needs a u8 argument"))?
} else {
0
};
ops.push(make_op(tok, arg)?);
}
if ops.is_empty() {
return Err("kernel: no ops given".to_string());
}
let name = path.rsplit('/').next().unwrap_or(path);
let coord = scratch_code_coord(path, Kind::Kernel, name, &ops)?;
if let Some(msg) = self.admit_mutate(coord, Perm::Write) {
self.audit_now(OP_WRITE, coord, false);
return Err(msg);
}
self.audit_now(OP_WRITE, coord, true);
let owner = self.identity.as_ref().map(|i| i.owner_local.as_str());
let value = {
let mut scratch = CubeStore::new(HashBackend::new());
crate::store_code_cell(
&mut scratch,
path,
Kind::Kernel,
name,
&[],
&ops,
owner,
descriptor,
)
.map_err(|e| e.to_string())?;
scratch.get_raw(&coord).unwrap_or_default()
};
let header = header_for_code(Kind::Kernel, name, &ops, owner, descriptor);
if let Some(txn) = self.txn.as_mut() {
txn.ops.push(TxnOp {
coord,
put: Some((value, header)),
});
return Ok(format!(
"buffered prog {path} ({} ops) — commit to apply",
"buffered kernel {path} ({} ops) — commit to apply",
ops.len()
));
}
let coord = store
.put_code_cell(path, Kind::Fn, name, &[], &ops, owner)
.put_code_cell(path, Kind::Kernel, name, &[], &ops, owner, descriptor)
.map_err(|e| e.to_string())?;
Ok(format!(
"wrote program {path} -> coord {} ({} ops)",
"wrote kernel {path} -> coord {} ({} ops, descriptors={:?})",
coord.pack_u32(),
ops.len()
ops.len(),
descriptor.map(|b| b.tags()).unwrap_or_default()
))
}
"tick" => {
// Lay down the OS operator-kernel call graph (Step 1). Each leaf
// is a real CUBE kernel composed via `linked_records` (the call
// graph) and tagged with behavior descriptors. `cube-os-tick`
// calls the three leaves in order. Nothing is executed here —
// the native layer later `run`s each and `native-apply`s the
// effect. This is "everything in CUBE" done the spec's way:
// the OS's behavior lives as kernels + call graph + descriptors.
let c = cubecode::C_OS_KERNEL;
let cfg = format!("/c{c}/z001/y001/x001"); // normalize-config: pure
let decide = format!("/c{c}/z001/y001/x002"); // decide-snapshot: io
let summarize = format!("/c{c}/z001/y001/x003"); // summarize-procs: pure+hot
let tick = format!("/c{c}/z001/y001/x004"); // cube-os-tick: hot (calls 0..2)
let cfg_code = vec![Op::Const(7), Op::Const(3), Op::Add, Op::Halt]; // 7+3=10
let decide_code = vec![Op::Const(1), Op::Ret]; // 1 => snapshot
let summarize_code = vec![Op::Const(20), Op::Halt]; // 20 procs
let tick_code = vec![
Op::Const(0),
Op::CallLink(0), // cfg
Op::CallLink(1), // decide
Op::CallLink(2), // summarize
Op::Halt,
];
// Stage the leaves first (we need their coords to build the
// call-graph edges of the root), then the root. `with_mut`
// hands us exclusive `&mut CubeStore` access so the helper can
// write each record through the normal codec.
let cfg_c = store
.with_mut(|s| {
crate::store_code_cell(
s,
&cfg,
Kind::Kernel,
"normalize-config",
&[],
&cfg_code,
None,
Some(cubecode::Behavior(cubecode::Behavior::PURE)),
)
})
.map_err(|e: crate::SysError| e.to_string())?;
let dec_c = store
.with_mut(|s| {
crate::store_code_cell(
s,
&decide,
Kind::Kernel,
"decide-snapshot",
&[],
&decide_code,
None,
Some(cubecode::Behavior(cubecode::Behavior::IO_HEAVY)),
)
})
.map_err(|e: crate::SysError| e.to_string())?;
let sum_c = store
.with_mut(|s| {
crate::store_code_cell(
s,
&summarize,
Kind::Kernel,
"summarize-procs",
&[],
&summarize_code,
None,
Some(cubecode::Behavior(
cubecode::Behavior::PURE | cubecode::Behavior::HOT_PATH,
)),
)
})
.map_err(|e: crate::SysError| e.to_string())?;
let tick_c = store
.with_mut(|s| {
crate::store_code_cell(
s,
&tick,
Kind::Kernel,
"cube-os-tick",
&[cfg_c, dec_c, sum_c], // call graph: tick -> {cfg,decide,summarize}
&tick_code,
None,
Some(cubecode::Behavior(cubecode::Behavior::HOT_PATH)),
)
})
.map_err(|e: crate::SysError| e.to_string())?;
Ok(format!(
"OS kernel call-graph laid down (kind=kernel, in cube c{c}):\n {} normalize-config [pure] -> coord {}\n {} decide-snapshot [io] -> coord {}\n {} summarize-procs [pure,hot] -> coord {}\n {} cube-os-tick [hot] -> coord {} (links cfg,decide,summarize)\nrun e.g.: run {}\nthen effector: native-apply {}",
cfg, cfg_c.pack_u32(), decide, dec_c.pack_u32(), summarize,
sum_c.pack_u32(), tick, tick_c.pack_u32(), tick_c.pack_u32(),
tick
))
}
"native-apply" => {
// Step 2: the thin effector. The decision/computation already
let path = it
.next()
.ok_or_else(|| "native-apply needs <kernel-path>".to_string())?;
let cell = crate::load_code_cell(&store.read_snapshot(), path)
.map_err(|e| format!("native-apply: cannot load {path}: {e}"))?;
if cell.kind() != Kind::Kernel {
return Err(format!(
"native-apply: {path} is kind={:?}, expected a kernel",
cell.kind()
));
}
// Build a throwaway in-memory store holding just this kernel
// (the VM needs a store to walk), then run it. The computation
// is entirely in CUBE; we only read back the computed result.
let coord = crate::path_to_czyx(path).map_err(|e| e.to_string())?;
let mut vm_store = cubestore::CubeStore::new(cubestore::HashBackend::new());
crate::store_code_cell(
&mut vm_store,
path,
Kind::Kernel,
cell.name().unwrap_or(path),
cell.links(),
&cell.code,
None,
None,
)
.map_err(|e| format!("native-apply: stage failed: {e}"))?;
let mut vm = Vm::new(vm_store);
let result = match vm.run(coord) {
RunResult::Halted { top } => top,
other => {
return Err(format!(
"native-apply: kernel did not halt cleanly: {other:?}"
))
}
};
let r = result.unwrap_or(0);
let effect = match cell.name() {
Some("decide-snapshot") => {
if r != 0 {
format!(
"OS EFFECT: persist runtime snapshot into CUBE (c{} band); decision kernel returned {} => snapshot NOW",
cubecode::C_OS_EFFECT, r
)
} else {
"OS EFFECT: no snapshot (decision kernel returned 0)".to_string()
}
}
_ => format!(
"OS EFFECT: apply computed result {} from kernel {}@{}",
r,
cell.name().unwrap_or("?"),
path
),
};
Ok(format!(
"native-apply {} (kind=kernel, descriptors={:?}):\n computed result = {}\n {}",
path,
cubecode::Behavior::from_flags(
store
.read_snapshot()
.get_record(&coord)
.map(|(h, _)| h.flags.bits())
.unwrap_or(0)
)
.tags(),
r,
effect
))
}
"write" => {
@@ -520,11 +1043,20 @@ impl Session {
let owner = self.identity.as_ref().map(|i| i.owner_local.as_str());
let value = {
let mut scratch = CubeStore::new(HashBackend::new());
crate::store_code_cell(&mut scratch, path, Kind::Fn, name, &[], &code, owner)
.map_err(|e| e.to_string())?;
crate::store_code_cell(
&mut scratch,
path,
Kind::Fn,
name,
&[],
&code,
owner,
None,
)
.map_err(|e| e.to_string())?;
scratch.get_raw(&coord).unwrap_or_default()
};
let header = header_for_code(Kind::Fn, name, &code, owner);
let header = header_for_code(Kind::Fn, name, &code, owner, None);
if let Some(txn) = self.txn.as_mut() {
txn.ops.push(TxnOp {
coord,
@@ -536,7 +1068,7 @@ impl Session {
));
}
let coord = store
.put_code_cell(path, Kind::Fn, name, &[], &code, owner)
.put_code_cell(path, Kind::Fn, name, &[], &code, owner, None)
.map_err(|e| e.to_string())?;
Ok(format!("wrote {path} -> coord {}", coord.pack_u32()))
}
@@ -596,6 +1128,66 @@ impl Session {
store.delete_raw(&coord);
Ok(format!("ok: deleted {}", coord.pack_u32()))
}
// --- Enveloped coordinate API (PDF Package 2: `put(C,Z,Y,X,
// bytes, flags)`). Unlike `rawput`, this writes a full record
// with a `CubeHeader`, so the record carries metatags
// (doc_type / title) that the associative `query` verb can
// target by event type. `rawput` stays for bulk/append payloads
// (e.g. the rolling kernel-log window) that do not need header
// metadata; `put` is for addressed, discoverable records. ---
"put" => {
let c = parse_u8(it.next(), "put needs <c>")?;
let z = parse_u8(it.next(), "put needs <z>")?;
let y = parse_u8(it.next(), "put needs <y>")?;
let x = parse_u8(it.next(), "put needs <x>")?;
let hex = it
.next()
.ok_or_else(|| "put needs <hex-bytes>".to_string())?;
let val = hex_decode(hex).ok_or_else(|| "put: value must be hex".to_string())?;
let coord = Czyx::new(c, z, y, x);
// Optional metatags: `doc_type=<t>` and/or `title=<t>` may
// follow. These are what `query <doc_type>` matches on.
let mut header = CubeHeader::new();
for tok in it {
if let Some(v) = tok.strip_prefix("doc_type=") {
header.doc_type = Some(v.to_string());
} else if let Some(v) = tok.strip_prefix("title=") {
header.title = Some(v.to_string());
} else if let Some(v) = tok.strip_prefix("path=") {
// POSIX path metatag: the original filesystem path.
// Lets scan_by_path_prefix reconstruct nesting from flags.
header.path = Some(v.to_string());
} else {
return Err(format!("put: unknown option {tok}"));
}
}
header.created_at = Some(
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_secs())
.unwrap_or(0),
);
header.refresh_flags();
let doc_type_dbg = header.doc_type.clone();
if let Some(txn) = self.txn.as_mut() {
txn.ops.push(TxnOp {
coord,
put: Some((val, header)),
});
Ok(format!(
"buffered put {} (doc_type={:?}) — commit to apply",
coord.pack_u32(),
doc_type_dbg
))
} else {
store.put_record(coord, &header, &val);
Ok(format!(
"ok: wrote {} (doc_type={:?})",
coord.pack_u32(),
doc_type_dbg
))
}
}
"rawkeys" => {
let ks: Vec<String> = store
.keys()
@@ -752,7 +1344,20 @@ impl Session {
let dir = it.next().ok_or_else(|| "ls needs <dir>".to_string())?;
// R5: directory reads are metadata reads — honor the read gate
// so an attacker can't enumerate a victim's records by name.
let dir_coord = crate::path_to_czyx(dir).map_err(|e| e.to_string())?;
// Use parse_path (not path_to_czyx) because ls targets a
// directory prefix, which path_to_czyx rejects as "not a record".
let parsed = cubefs::path::parse_path(dir)
.map_err(|e| format!("ls: bad path {dir}: {e}"))?;
// The ACL gate wants the directory's prefix coordinate
// (trailing axes zeroed) — same key the NullSpace acl_bucket
// uses for directory ACLs.
let dir_coord = match parsed.axes.len() {
0 => Czyx::new(0, 0, 0, 0), // root
1 => Czyx::new(parsed.axes[0], 0, 0, 0),
2 => Czyx::new(parsed.axes[0], parsed.axes[1], 0, 0),
3 => Czyx::new(parsed.axes[0], parsed.axes[1], parsed.axes[2], 0),
_ => return Err("ls: path too deep".to_string()),
};
if let Some(msg) = self.admit_read(dir_coord) {
self.audit_now(OP_READ, dir_coord, false);
return Err(msg);
@@ -783,8 +1388,15 @@ impl Session {
.getattr(path)
.map_err(|e| format!("stat {path}: {e:?}"))?;
Ok(format!(
"stat {path} -> ino={} kind={:?} size={} mode={:o}",
a.ino, a.kind, a.size, a.mode
"stat {path} -> ino={} kind={:?} size={} mode={:o} word_flags=0x{:04x} {} header_flags=0x{:04x} {}",
a.ino,
a.kind,
a.size,
a.mode,
a.word_flags,
word_flag_names(a.word_flags),
a.header_flags,
header_flag_names(a.header_flags)
))
}
"keyinit" => {
@@ -952,19 +1564,31 @@ pub fn txn_snapshot(s: &Session) -> CubeStore<HashBackend> {
/// writing — used to buffer `prog`/`write` mutations during a transaction.
fn scratch_code_coord(path: &str, kind: Kind, name: &str, code: &[Op]) -> Result<Czyx, String> {
let mut scratch = CubeStore::new(HashBackend::new());
crate::store_code_cell(&mut scratch, path, kind, name, &[], code, None)
crate::store_code_cell(&mut scratch, path, kind, name, &[], code, None, None)
.map_err(|e| e.to_string())
}
/// Build the `CubeHeader` a `store_code_cell` call would attach (mirrors
/// `crate::store_code_cell`), so a buffered txn put carries the same header.
/// `owner` (when set) is stamped on `owner_local_user` for Task 6 enforcement.
fn header_for_code(kind: Kind, name: &str, code: &[Op], owner: Option<&str>) -> CubeHeader {
/// `descriptor` (when set) stamps the behavior-descriptor bits (PDF §524525).
fn header_for_code(
kind: Kind,
name: &str,
code: &[Op],
owner: Option<&str>,
descriptor: Option<cubecode::Behavior>,
) -> CubeHeader {
let mut h = CubeHeader::new();
h.title = Some(name.to_string());
h.doc_type = Some(kind.as_str().to_string());
h.linked_records = Vec::new();
h.owner_local_user = owner.map(|o| o.to_string());
// OS-created records carry per-word flags (tag 14).
h.word_flags = crate::default_word_flags(kind, descriptor);
if let Some(b) = descriptor {
h.flags.0 |= b.to_flags();
}
if h.doc_type.as_deref() == Some("fn") {
h.size_bytes = Some(cubecode::encode(code).len() as u64);
}
@@ -992,6 +1616,72 @@ pub fn parse_coord(s: &str) -> Option<cubecoords::Czyx> {
}
/// Parse a single `u8` axis token: decimal (`7`) or hex (`0x07`).
fn header_flag_names(bits: u16) -> String {
let mut v: Vec<&str> = Vec::new();
if bits & cubecoords::HeaderFlags::TITLE != 0 { v.push("title"); }
if bits & cubecoords::HeaderFlags::DOC_TYPE != 0 { v.push("doc_type"); }
if bits & cubecoords::HeaderFlags::CREATED_AT != 0 { v.push("created_at"); }
if bits & cubecoords::HeaderFlags::SIZE_BYTES != 0 { v.push("size_bytes"); }
if bits & cubecoords::HeaderFlags::PERM_ROOT_ONLY != 0 { v.push("root_only"); }
if bits & cubecoords::HeaderFlags::PERM_LOCAL_USER != 0 { v.push("local_user"); }
if bits & cubecoords::HeaderFlags::PERM_REMOTE_USER != 0 { v.push("remote_user"); }
if bits & cubecoords::HeaderFlags::HAS_ASSOCIATIONS != 0 { v.push("associations"); }
if bits & cubecoords::HeaderFlags::HAS_PATH != 0 { v.push("path"); }
if v.is_empty() { String::new() } else { format!("({})", v.join("|")) }
}
fn word_flag_names(bits: u16) -> String {
let mut v: Vec<&str> = Vec::new();
let arr = bits & cubecoords::WordFlags::ARR_MASK;
if arr == 0 { v.push("tri6"); } else if arr == 1 { v.push("pair"); }
else if arr == 2 { v.push("triad"); } else { v.push("quad"); }
if bits & cubecoords::WordFlags::START_RECORD != 0 { v.push("start"); }
if bits & cubecoords::WordFlags::END_RECORD != 0 { v.push("end"); }
if bits & cubecoords::WordFlags::IS_HEADER != 0 { v.push("header"); }
if bits & cubecoords::WordFlags::CONTINUATION != 0 { v.push("cont"); }
let ty = (bits & cubecoords::WordFlags::TYPE_MASK) >> 6;
match ty { 0 => v.push("type=text"), 1 => v.push("type=binary"), 2 => v.push("type=code"), _ => v.push("type=ai") }
if bits & cubecoords::WordFlags::PERM_ROOT != 0 { v.push("root"); }
if bits & cubecoords::WordFlags::PERM_USER != 0 { v.push("user"); }
if bits & cubecoords::WordFlags::ASSOC_EDGE != 0 { v.push("assoc"); }
if bits & cubecoords::WordFlags::ENCRYPTED != 0 { v.push("encrypted"); }
if bits & cubecoords::WordFlags::COMPRESSED != 0 { v.push("compressed"); }
if bits & cubecoords::WordFlags::STEGO != 0 { v.push("stego"); }
if bits & cubecoords::WordFlags::NULL_LOOKUP != 0 { v.push("null"); }
if bits & cubecoords::WordFlags::CHECKSUM != 0 { v.push("checksum"); }
format!("({})", v.join("|"))
}
fn parse_word_flag(a: &str) -> Result<u16, String> {
let lower = a.to_lowercase();
let bits = match lower.as_str() {
"start" => cubecoords::WordFlags::START_RECORD,
"end" => cubecoords::WordFlags::END_RECORD,
"header" => cubecoords::WordFlags::IS_HEADER,
"cont" | "continuation" => cubecoords::WordFlags::CONTINUATION,
"text" | "type-text" => 0 << 6,
"binary" | "type-binary" => 1 << 6,
"code" | "type-code" => 2 << 6,
"ai" | "type-ai" => 3 << 6,
"root" => cubecoords::WordFlags::PERM_ROOT,
"user" => cubecoords::WordFlags::PERM_USER,
"assoc" | "assoc-edge" => cubecoords::WordFlags::ASSOC_EDGE,
"encrypted" | "enc" => cubecoords::WordFlags::ENCRYPTED,
"compressed" | "comp" => cubecoords::WordFlags::COMPRESSED,
"stego" => cubecoords::WordFlags::STEGO,
"null" | "null-lookup" => cubecoords::WordFlags::NULL_LOOKUP,
"checksum" => cubecoords::WordFlags::CHECKSUM,
other => match other.strip_prefix("0x") {
Some(hex) => return u16::from_str_radix(hex, 16)
.map_err(|_| format!("invalid hex word flag '{a}'")),
None => return other.parse::<u16>()
.map_err(|_| format!("unknown word flag '{a}'")),
},
};
Ok(bits)
}
fn parse_u8(t: Option<&str>, what: &str) -> Result<u8, String> {
let t = t.ok_or_else(|| what.to_string())?;
t.parse::<u8>()
@@ -1101,6 +1791,82 @@ mod tests {
Session::with_store(Arc::new(ConcurrentStore::memory()))
}
#[test]
fn os_kernels_live_in_cube_with_call_graph_and_descriptors() {
// Step 1: OS operator behavior lives in CUBE as kernels, composed via
// a call graph (linked_records) and tagged with behavior descriptors.
// Step 2: a thin native effector (`native-apply`) runs each kernel in
// the VM, reads its computed result, and emits the OS effect — the
// cubevm itself never does store-IO (spec-aligned, PDF §524525).
let mut s = session();
// Lay down the OS kernel call graph.
let out = s.exec("tick").expect("tick should lay down kernels");
assert!(
out.contains("normalize-config"),
"cfg kernel missing: {out}"
);
assert!(
out.contains("decide-snapshot"),
"decide kernel missing: {out}"
);
assert!(
out.contains("summarize-procs"),
"summarize kernel missing: {out}"
);
assert!(out.contains("cube-os-tick"), "tick kernel missing: {out}");
// The root links the three leaves (call graph, not foreign code).
assert!(
out.contains("links cfg,decide,summarize"),
"call graph not wired: {out}"
);
// Behavior descriptors are stamped (round-trip through header flags).
assert!(
out.contains("[pure]") && out.contains("[io]") && out.contains("[pure,hot]"),
"behavior descriptors not stamped: {out}"
);
// Run the root kernel: it must traverse the call graph (CallLink 0..2)
// and return, proving the OS's behavior lives as addressable kernels.
let run_out = s
.exec("run /c210/z001/y001/x004")
.expect("tick kernel must run");
assert!(
run_out.contains("Halted"),
"tick kernel should halt: {run_out}"
);
// Step 2 — the effector reads the COMPUTED result and emits the effect.
let eff = s
.exec("native-apply /c210/z001/y001/x002")
.expect("effector must run decide-snapshot");
assert!(
eff.contains("computed result = 1"),
"decide kernel result wrong: {eff}"
);
assert!(
eff.contains("OS EFFECT"),
"effector must emit OS EFFECT: {eff}"
);
assert!(
eff.contains("snapshot NOW"),
"decision=1 should snapshot: {eff}"
);
// A plain pure kernel also routes through the effector with no store-IO.
let eff2 = s
.exec("native-apply /c210/z001/y001/x003")
.expect("effector must run summarize-procs");
assert!(
eff2.contains("computed result = 20"),
"summarize result wrong: {eff2}"
);
assert!(
eff2.contains("descriptors=[\"pure\", \"hot\"]"),
"descriptor readback wrong: {eff2}"
);
}
#[test]
fn begin_commit_applies_buffered_writes() {
let mut s = session();
@@ -1803,4 +2569,25 @@ mod tests {
);
let _ = std::fs::remove_dir_all(&dir);
}
#[test]
fn header_for_code_sets_word_flags() {
// Fn cell with a security-sensitive descriptor -> per-word flags set.
let h = header_for_code(
Kind::Fn,
"f",
&[],
None,
Some(cubecode::Behavior::new(cubecode::Behavior::SECURITY_SENSITIVE)),
);
assert!(h.word_flags.has(cubecoords::WordFlags::START_RECORD));
assert!(h.word_flags.has(cubecoords::WordFlags::END_RECORD));
assert!(h.word_flags.has(cubecoords::WordFlags::IS_HEADER));
assert!(h.word_flags.has(cubecoords::WordFlags::ENCRYPTED), "sec descriptor -> ENCRYPTED");
assert_eq!((h.word_flags.bits() & cubecoords::WordFlags::TYPE_MASK), 0b10 << 6, "Fn -> type=code");
// No descriptor -> complete frame, no security bit.
let h2 = header_for_code(Kind::Fn, "g", &[], None, None);
assert!(h2.word_flags.has(cubecoords::WordFlags::START_RECORD));
assert!(!h2.word_flags.has(cubecoords::WordFlags::ENCRYPTED));
}
}
+2
View File
@@ -276,6 +276,7 @@ fn write_bucket(store: &ConcurrentStore, grants: &[Grant]) {
pub fn grant_header() -> CubeHeader {
CubeHeader {
flags: cubecoords::HeaderFlags(cubecoords::HeaderFlags::DOC_TYPE),
word_flags: cubecoords::WordFlags::default(),
title: None,
doc_type: Some("grant-table".to_string()),
created_at: None,
@@ -283,6 +284,7 @@ pub fn grant_header() -> CubeHeader {
owner_local_user: None,
owner_remote_user: None,
linked_records: Vec::new(),
path: None,
total_accesses: 0,
total_remote_accesses: 0,
last_access: None,
+45
View File
@@ -54,6 +54,11 @@ pub mod commands;
pub mod grants;
/// Length-framed Unix-domain-socket transport shared by client and server.
pub mod net;
/// Session-message / note log: a searchable, per-session, per-day note area
/// stored as CZYX records carrying WordFlags metadata. This is the structured
/// replacement for the hand-maintained `RESUME-*.md` files so a new session
/// can review prior sessions' messages over the cube store (see [`notes`]).
pub mod notes;
/// Dependency-free JSON snapshot load/dump for daemon store persistence.
pub mod persist;
/// Concurrent, durable store: mutex-wrapped [`CubeStore`] + NDJSON write-ahead
@@ -133,6 +138,35 @@ pub fn load_code_cell<B: CubeBackend>(
/// Build a [`CodeCell`] from parts and write it back as a record at `path`,
/// so code written by the VM (or tooling) is visible to cubefs at that path.
#[allow(clippy::too_many_arguments)]
/// Per-word flags (tag 14) stamped on OS-created records: a complete single
/// record frame + a type note (code cells = type=code) + a security note, so the
/// store actually carries WordFlags metadata (not just stores it).
pub(crate) fn default_word_flags(
kind: Kind,
descriptor: Option<cubecode::Behavior>,
) -> cubecoords::WordFlags {
let mut wf = cubecoords::WordFlags::from_bits(
cubecoords::WordFlags::START_RECORD
| cubecoords::WordFlags::END_RECORD
| cubecoords::WordFlags::IS_HEADER,
);
if matches!(kind, Kind::Fn | Kind::Kernel) {
wf.0 |= 0b10 << 6; // type=code
}
if let Some(b) = descriptor {
if b.0 & cubecode::Behavior::SECURITY_SENSITIVE != 0 {
wf.0 |= cubecoords::WordFlags::ENCRYPTED;
}
}
wf
}
/// Store a CUBEVM code cell at the coordinate derived from `path`. Builds the
/// record header (title/doc_type/owner) and stamps per-word flags (tag 14) via
/// [`default_word_flags`], so OS-created code cells carry WordFlags metadata.
/// Returns the resolved coordinate; used by the `prog`/`write`/`kernel`/`fn`
/// OS commands and the `cubefs` socket backend.
pub fn store_code_cell<B: CubeBackend>(
store: &mut CubeStore<B>,
path: &str,
@@ -141,6 +175,10 @@ pub fn store_code_cell<B: CubeBackend>(
links: &[Czyx],
code: &[cubecode::Op],
owner: Option<&str>,
// Behavior-descriptor flags (PDF §524525: pure / I/O heavy / hot path) to
// stamp on the record header's out-of-band bits. `None` leaves the field
// at zero. See `cubecode::Behavior`.
descriptor: Option<cubecode::Behavior>,
) -> Result<Czyx, SysError> {
let coord = path_to_czyx(path)?;
let mut h = CubeHeader::new();
@@ -148,6 +186,10 @@ pub fn store_code_cell<B: CubeBackend>(
h.doc_type = Some(kind.as_str().to_string());
h.linked_records = links.to_vec();
h.owner_local_user = owner.map(|o| o.to_string());
h.word_flags = default_word_flags(kind, descriptor);
if let Some(b) = descriptor {
h.flags.0 |= b.to_flags();
}
if h.doc_type.as_deref() == Some("fn") {
h.size_bytes = Some(cubecode::encode(code).len() as u64);
}
@@ -187,6 +229,7 @@ mod tests {
&[],
&[Op::Const(2), Op::Const(3), Op::Add, Op::Halt],
None,
None,
)
.unwrap();
@@ -300,6 +343,7 @@ pub mod demo {
&[],
&double_code,
None,
None,
)
.expect("store double");
let entry_coord = super::store_code_cell(
@@ -310,6 +354,7 @@ pub mod demo {
&[double_coord],
&entry_code,
None,
None,
)
.expect("store entry");
println!(" wrote {double} -> coord {}", double_coord.pack_u32());
+772
View File
@@ -0,0 +1,772 @@
//! CUBELinux session-message / note log with **categories** and **projects**.
//!
//! The "message save path" helper. Each note is a CZYX record with a text-type
//! [`WordFlags`] stamp, a `created_at` day, and the session label as the owner.
//! Two extra, searchable dimensions layer on top:
//! * **category** — encoded in `doc_type` as `note:<category>` (so
//! [`scan_by_type`] / filtering slices notes by kind: design, impl, cube, ...).
//! * **project** — notes are *associated* (via `linked_records`) to a
//! `project` record (`doc_type = "project"`), so `cube project walk`
//! follows the links into a project's timeline/thread. This is the
//! CUBELinux first-class-association model (EDG) applied to project work.
//!
//! Cross-session: `created_at` (day) + session label + category + project make
//! any note recoverable as `note search <text> [--project p] [--cat c]`, and a
//! new harness session can `note list --project p` / `project walk` to see where
//! a project left off.
use std::time::{SystemTime, UNIX_EPOCH};
use cubecoords::{CubeHeader, Czyx, WordFlags};
use crate::store::ConcurrentStore;
/// Base `doc_type` for a note without a category.
pub const NOTE_DOC_TYPE: &str = "note";
/// `doc_type` for a project record.
pub const PROJECT_DOC_TYPE: &str = "project";
/// The default session label for CLI-created notes (today's calendar date).
pub fn default_session() -> String {
date_string(now())
}
/// The WordFlags stamped on a note: a complete single-record frame with the
/// text data type (type bits 00). Other metadata bits (encrypted, assoc, ...)
/// are left unset for the caller to raise as needed.
pub fn note_word_flags() -> WordFlags {
WordFlags::from_bits(
WordFlags::START_RECORD | WordFlags::END_RECORD | WordFlags::IS_HEADER,
)
}
/// Current unix time (seconds).
pub fn now() -> u64 {
SystemTime::now()
.duration_since(UNIX_EPOCH)
.map(|d| d.as_secs())
.unwrap_or(0)
}
/// A returned note (coordinate + header + body bytes).
#[derive(Clone, Debug)]
pub struct Note {
/// The note's coordinate (`C.Z.Y.X`).
pub coord: Czyx,
/// The note's record header (title, session label, WordFlags, category).
pub header: CubeHeader,
/// The note's body text as raw bytes.
pub body: Vec<u8>,
}
/// Query dimensions for listing/searching notes. `since`/`until` are
/// day-of-epoch (see [`date_to_day`]); the rest are exact-match labels.
#[derive(Clone, Default, Debug)]
pub struct NoteFilter {
/// Session label (owner_local_user).
pub session: Option<String>,
/// Project name (note must be associated to that project record).
pub project: Option<String>,
/// Category (doc_type `note:<category>`).
pub category: Option<String>,
/// Earliest day-of-epoch, inclusive.
pub since: Option<u64>,
/// Latest day-of-epoch, inclusive.
pub until: Option<u64>,
}
/// The category portion of a note's `doc_type`, if any (`note:<category>`).
pub fn category_of(doc_type: &str) -> Option<&str> {
doc_type.strip_prefix(&format!("{NOTE_DOC_TYPE}:"))
}
fn fnv1a(bytes: &[u8]) -> u64 {
let mut h: u64 = 0xcbf2_9ce4_8422_2325;
for &b in bytes {
h ^= b as u64;
h = h.wrapping_mul(0x0000_0100_0000_01b3);
}
h
}
fn non_zero(b: u8) -> u8 {
if b == 0 {
1
} else {
b
}
}
/// The single-byte notes class for a session label, so `scan_prefix(class,
/// None, None)` returns exactly that session's notes.
fn session_class(session: &str) -> u8 {
let h = fnv1a(session.as_bytes());
1 + ((h >> 24) as u8 & 0x7f) // 1..=128
}
/// Project records live in the upper class range (128..=255) so they never
/// collide with note session classes (1..=128).
fn project_class(name: &str) -> u8 {
let h = fnv1a(name.as_bytes());
128 + ((h >> 24) as u8 & 0x7f)
}
/// Derive a deterministic, collision-light coordinate for a note: the C axis
/// is the session class, Z/Y/X come from the note content + a salt.
fn note_coord(session: &str, created: u64, subject: &str, body: &str, salt: u64) -> Czyx {
let mut mix = fnv1a(session.as_bytes());
mix ^= created.wrapping_mul(0x9e37_79b9);
mix ^= fnv1a(subject.as_bytes());
mix ^= fnv1a(body.as_bytes());
mix ^= salt.wrapping_mul(0x85eb_ca6b);
Czyx::new(
session_class(session),
non_zero((mix >> 16) as u8),
non_zero((mix >> 8) as u8),
non_zero(mix as u8 ^ (salt as u8)),
)
}
/// Derive the deterministic coordinate of a project record.
fn project_coord(name: &str, salt: u64) -> Czyx {
let mut mix = fnv1a(name.as_bytes());
mix ^= salt.wrapping_mul(0x8549_b2a1);
Czyx::new(
project_class(name),
non_zero((mix >> 16) as u8),
non_zero((mix >> 8) as u8),
non_zero(mix as u8 ^ (salt as u8)),
)
}
/// Find an existing project record by name, without creating it.
pub fn find_project(store: &ConcurrentStore, name: &str) -> Option<Czyx> {
store
.read_snapshot()
.query_by_type(PROJECT_DOC_TYPE)
.into_iter()
.find(|(_, h, _)| {
h.title.as_deref() == Some(name)
|| h.doc_type.as_deref() == Some(PROJECT_DOC_TYPE) && h.path.as_deref() == Some(name)
})
.map(|(c, _, _)| c)
}
/// Find-or-create a project record named `name`, returning its coordinate.
/// Notes tagged with a project are associated to it via `linked_records`.
pub fn ensure_project(store: &ConcurrentStore, name: &str, session: &str) -> Result<Czyx, String> {
if let Some(c) = find_project(store, name) {
return Ok(c);
}
let mut salt = 0u64;
let mut coord = project_coord(name, salt);
while store.get_record(&coord).is_some() && salt < 4096 {
salt += 1;
coord = project_coord(name, salt);
}
if salt >= 4096 {
return Err("project store: could not find a free coordinate".into());
}
let mut h = CubeHeader::new();
h.title = Some(name.to_string());
h.path = Some(name.to_string());
h.doc_type = Some(PROJECT_DOC_TYPE.to_string());
h.created_at = Some(now());
h.owner_local_user = Some(session.to_string());
h.refresh_flags();
store.put_record(coord, &h, &[]);
Ok(coord)
}
/// Store a note record under `session`, with a `subject` title and `body`
/// text. `project` (if given) associates the note to a project record; `category`
/// (if given) sets the note's `doc_type` to `note:<category>`. Returns the note's
/// coordinate.
pub fn store_note(
store: &ConcurrentStore,
session: &str,
subject: &str,
body: &str,
project: Option<&str>,
category: Option<&str>,
) -> Result<Czyx, String> {
let created = now();
let proj_coord = match project {
Some(p) => Some(ensure_project(store, p, session)?),
None => None,
};
let mut salt = 0u64;
let mut coord = note_coord(session, created, subject, body, salt);
while store.get_record(&coord).is_some() && salt < 4096 {
salt += 1;
coord = note_coord(session, created, subject, body, salt);
}
if salt >= 4096 {
return Err("note store: could not find a free coordinate (4096 tries)".into());
}
let mut h = CubeHeader::new();
h.title = Some(subject.to_string());
h.doc_type = Some(match category {
Some(c) => format!("{NOTE_DOC_TYPE}:{c}"),
None => NOTE_DOC_TYPE.to_string(),
});
h.created_at = Some(created);
h.owner_local_user = Some(session.to_string());
if let Some(pc) = proj_coord {
h.linked_records = vec![pc];
}
h.word_flags = note_word_flags();
h.refresh_flags();
store.put_record(coord, &h, body.as_bytes());
Ok(coord)
}
/// All note records (`doc_type == "note"` or `"note:<cat>"`), regardless of
/// category, so a category filter is applied in Rust (the store's exact
/// `scan_by_type("note")` would miss categorized notes).
fn all_notes(snap: &cubestore::CubeStore<cubestore::HashBackend>) -> Vec<Note> {
snap.keys()
.into_iter()
.filter_map(|coord| {
let (header, body) = snap.get_record(&coord)?;
let dt = header.doc_type.as_deref().unwrap_or("");
if dt == NOTE_DOC_TYPE || dt.starts_with(&format!("{NOTE_DOC_TYPE}:")) {
Some(Note {
coord,
header,
body,
})
} else {
None
}
})
.collect()
}
/// List notes, newest first, filtered by [`NoteFilter`].
pub fn list_notes(store: &ConcurrentStore, filter: Option<&NoteFilter>) -> Vec<Note> {
let snap = store.read_snapshot();
let mut out: Vec<Note> = all_notes(&snap)
.into_iter()
.filter(|n| note_matches_filtered(store, n, filter))
.collect();
out.sort_by(|a, b| b.header.created_at.cmp(&a.header.created_at));
out
}
fn note_matches_filtered(store: &ConcurrentStore, n: &Note, filter: Option<&NoteFilter>) -> bool {
let Some(f) = filter else {
return true;
};
if let Some(s) = &f.session {
if n.header.owner_local_user.as_deref() != Some(s.as_str()) {
return false;
}
}
if let Some(cat) = &f.category {
let expected = format!("{NOTE_DOC_TYPE}:{cat}");
if n.header.doc_type.as_deref() != Some(expected.as_str()) {
return false;
}
}
if let Some(sd) = f.since {
if n.header.created_at.map(|c| c / 86_400).unwrap_or(0) < sd {
return false;
}
}
if let Some(ud) = f.until {
if n.header.created_at.map(|c| c / 86_400).unwrap_or(u64::MAX) > ud {
return false;
}
}
if let Some(p) = &f.project {
if let Some(pc) = find_project(store, p) {
if !n.header.linked_records.contains(&pc) && n.header.path.as_deref() != Some(p.as_str()) {
return false;
}
} else {
return false;
}
}
true
}
/// Search all notes (session + body) for `needle`, filtered, newest first.
pub fn search_notes(store: &ConcurrentStore, needle: &str, filter: Option<&NoteFilter>) -> Vec<Note> {
let needle = needle.to_lowercase();
let snap = store.read_snapshot();
let mut out: Vec<Note> = all_notes(&snap)
.into_iter()
.filter(|n| note_matches_filtered(store, n, filter))
.filter(|n| {
let subject = n.header.title.as_deref().unwrap_or("").to_lowercase();
let b = String::from_utf8_lossy(&n.body).to_lowercase();
subject.contains(&needle) || b.contains(&needle)
})
.collect();
out.sort_by(|a, b| b.header.created_at.cmp(&a.header.created_at));
out
}
/// Fetch a single note by coordinate.
pub fn show_note(store: &ConcurrentStore, coord: Czyx) -> Option<Note> {
store.get_record(&coord).map(|(header, body)| Note {
coord,
header,
body,
})
}
/// List all project records, newest first.
pub fn project_list(store: &ConcurrentStore) -> Vec<(Czyx, CubeHeader, Vec<u8>)> {
let mut v = store.read_snapshot().query_by_type(PROJECT_DOC_TYPE);
v.sort_by(|a, b| b.1.created_at.cmp(&a.1.created_at));
v
}
/// Return the notes associated to a project record (its thread/timeline),
/// newest first.
pub fn project_walk(store: &ConcurrentStore, coord: Czyx) -> Vec<Note> {
let snap = store.read_snapshot();
let mut out: Vec<Note> = all_notes(&snap)
.into_iter()
.filter(|n| n.header.linked_records.contains(&coord))
.collect();
out.sort_by(|a, b| b.header.created_at.cmp(&a.header.created_at));
out
}
/// Format a unix timestamp as `YYYY-MM-DD` (UTC).
pub fn date_string(ts: u64) -> String {
let days = (ts / 86_400) as i64;
let (y, m, d) = civil_from_days(days);
format!("{y:04}-{m:02}-{d:02}")
}
/// Parse `YYYY-MM-DD` into a day-of-epoch (days since 1970-01-01).
pub fn date_to_day(s: &str) -> Option<u64> {
let mut it = s.split('-');
let y: i64 = it.next()?.parse().ok()?;
let m: u32 = it.next()?.parse().ok()?;
let d: u32 = it.next()?.parse().ok()?;
if it.next().is_some() || !(1..=12).contains(&m) || !(1..=31).contains(&d) {
return None;
}
Some(days_from_civil(y, m, d))
}
/// Days since 1970-01-01 for a civil date (Howard Hinnant's algorithm).
fn days_from_civil(y: i64, m: u32, d: u32) -> u64 {
let yy = if m <= 2 { y - 1 } else { y };
let era = if yy >= 0 { yy } else { yy - 399 } / 400;
let yoe = yy - era * 400;
let mp = (m as i64 + 9) % 12;
let doy = (153 * mp + 2) / 5 + d as i64 - 1;
let doe = yoe * 365 + yoe / 4 - yoe / 100 + doy;
(era * 146_097 + doe - 719_468).max(0) as u64
}
fn civil_from_days(days: i64) -> (i64, u32, u32) {
let z = days + 719_468;
let era = if z >= 0 { z } else { z - 146_096 } / 146_097;
let doe = z - era * 146_097;
let yoe = (doe - doe / 1460 + doe / 36_524 - doe / 146_096) / 365;
let y = yoe + era * 400;
let doy = doe - (365 * yoe + yoe / 4 - yoe / 100);
let mp = (5 * doy + 2) / 153;
let d = (doy - (153 * mp + 2) / 5 + 1) as u32;
let m = if mp < 10 { mp + 3 } else { mp - 9 } as u32;
(if m <= 2 { y + 1 } else { y }, m, d)
}
/// Render a coordinate as `C.Z.Y.X` (the [`crate::commands::parse_coord`]
/// form), so `note show <coord>` round-trips.
fn coord_str(c: Czyx) -> String {
format!("{}.{}.{}.{}", c.c, c.z, c.y, c.x)
}
/// Subject = the first line-ish portion of the note text (first 60 chars).
fn subject_of(text: &str) -> String {
let s = text.trim();
let s: String = s.chars().take(60).collect();
if s.len() < text.trim().len() {
format!("{s}")
} else {
s
}
}
fn format_notes(notes: &[Note]) -> String {
if notes.is_empty() {
return "(no notes)".to_string();
}
let lines: Vec<String> = notes
.iter()
.map(|n| {
let subj = n.header.title.as_deref().unwrap_or("");
let when = n
.header
.created_at
.map(|t| date_string(t))
.unwrap_or_else(|| "?".into());
let cat = n
.header
.doc_type
.as_deref()
.and_then(category_of)
.map(|c| format!(" [{c}]"))
.unwrap_or_default();
format!(" {} [{}]{} {}", coord_str(n.coord), when, cat, subj)
})
.collect();
format!("{} note(s):\n{}", notes.len(), lines.join("\n"))
}
/// Parse `--key value` flags out of a command string; returns `(flags,
/// remainingPositionalJoined)`. A flag with no following token gets `"true"`.
fn split_flags(s: &str) -> (Vec<(String, String)>, String) {
let tokens: Vec<&str> = s.split_whitespace().collect();
let mut flags = Vec::new();
let mut pos: Vec<&str> = Vec::new();
let mut i = 0;
while i < tokens.len() {
let t = tokens[i];
if let Some(k) = t.strip_prefix("--") {
let (key, val) = if i + 1 < tokens.len() && !tokens[i + 1].starts_with("--") {
i += 1;
(k.to_string(), tokens[i].to_string())
} else {
(k.to_string(), "true".to_string())
};
flags.push((key, val));
} else {
pos.push(t);
}
i += 1;
}
(flags, pos.join(" "))
}
/// Process one `note`/`notes` command line against `store`.
///
/// Syntax (REPL/CLI):
/// `note <text>` quick note
/// `note add [--project P] [--cat C] <text>` add a note to a project/category
/// `note list [session] [--project P] [--cat C] [--since D] [--until D]`
/// `note search <text> [--project P] [--cat C]`
/// `note show <C.Z.Y.X>`
/// `project list` / `project show <C.Z.Y.X>` (see [`project_command`])
pub fn note_command(store: &ConcurrentStore, line: &str) -> Result<String, String> {
let body = line
.splitn(2, char::is_whitespace)
.nth(1)
.unwrap_or("")
.trim()
.to_string();
let Some((head, rest)) = split_head(&body) else {
let session = default_session();
let notes = list_notes(store, Some(&NoteFilter { session: Some(session), ..Default::default() }));
return Ok(format_notes(&notes));
};
match head {
"list" => {
let (flags, pos) = split_flags(rest);
let f = filter_from_flags(&flags);
let session = if pos.is_empty() { None } else { Some(pos.clone()) };
let f = NoteFilter { session, ..f };
let notes = list_notes(store, Some(&f));
Ok(format_notes(&notes))
}
"search" => {
let (flags, pos) = split_flags(rest);
let f = filter_from_flags(&flags);
if pos.is_empty() {
return Err("note search needs <text>".to_string());
}
let notes = search_notes(store, &pos, Some(&f));
Ok(format!("search '{pos}':\n{}", format_notes(&notes)))
}
"tag" => {
let mut parts = rest.split_whitespace();
let cs = parts.next().unwrap_or("");
let cat = parts.next().unwrap_or("");
let coord = crate::commands::parse_coord(cs)
.ok_or_else(|| format!("bad coordinate '{cs}' (want C.Z.Y.X)"))?;
if cat.is_empty() {
return Err("note tag needs <coord> <cat>".to_string());
}
match store.get_record(&coord) {
Some((mut header, body)) => {
header.doc_type = Some(format!("{NOTE_DOC_TYPE}:{cat}"));
store.put_record(coord, &header, &body);
Ok(format!("note {} tagged {cat}", coord_str(coord)))
}
None => Ok(format!("note {} -> (no note)", coord_str(coord))),
}
}
"show" => {
let cs = rest.trim();
let coord = crate::commands::parse_coord(cs)
.ok_or_else(|| format!("bad coordinate '{cs}' (want C.Z.Y.X)"))?;
match show_note(store, coord) {
Some(n) => {
let subj = n.header.title.as_deref().unwrap_or("");
let when = n
.header
.created_at
.map(|t| date_string(t))
.unwrap_or_else(|| "?".into());
Ok(format!(
"note {cs} [{}] {}\n{}",
when,
subj,
String::from_utf8_lossy(&n.body)
))
}
None => Ok(format!("note {cs} -> (no note)")),
}
}
"add" | _ => {
let (flags, pos) = split_flags(rest);
let mut project = None;
let mut category = None;
for (k, v) in &flags {
match k.as_str() {
"project" | "p" => project = Some(v.clone()),
"cat" | "category" | "c" => category = Some(v.clone()),
_ => {}
}
}
let text = pos.trim();
if text.is_empty() {
return Err("note needs <text>".to_string());
}
let session = default_session();
let subject = subject_of(text);
let coord = store_note(store, &session, &subject, text, project.as_deref(), category.as_deref())?;
let proj = project.map(|p| format!(" (project {p})")).unwrap_or_default();
Ok(format!("note saved at {} (session {}){proj}", coord_str(coord), session))
}
}
}
fn filter_from_flags(flags: &[(String, String)]) -> NoteFilter {
let mut f = NoteFilter::default();
for (k, v) in flags {
match k.as_str() {
"project" | "p" => f.project = Some(v.clone()),
"cat" | "category" | "c" => f.category = Some(v.clone()),
"since" => f.since = date_to_day(v),
"until" => f.until = date_to_day(v),
_ => {}
}
}
f
}
/// Set a "resume" marker for a project. Stored as a note tagged
/// `category = "resume"` and associated to the project, so a new session can
/// ask "where did this project leave off?" and get it back.
pub fn project_resume(
store: &ConcurrentStore,
name: &str,
text: &str,
session: &str,
) -> Result<Czyx, String> {
let subject = format!("resume: {name}");
store_note(store, session, &subject, text, Some(name), Some("resume"))
}
/// Build a compact "project state" summary for context injection: the project
/// name, its note count, the resume marker (if any), and the latest notes with
/// date/category. A new session reads this to know where a project left off.
pub fn project_context(store: &ConcurrentStore, name: &str) -> String {
let Some(coord) = find_project(store, name) else {
return format!("(no project '{name}')");
};
let notes = project_walk(store, coord);
let mut out = String::new();
out.push_str(&format!("Project: {name}\n"));
out.push_str(&format!(" {} note(s); thread {}.\n", notes.len(), coord_str(coord)));
if let Some(r) = notes
.iter()
.find(|n| category_of(n.header.doc_type.as_deref().unwrap_or("")) == Some("resume"))
{
out.push_str(&format!(
" RESUME: {}\n",
String::from_utf8_lossy(&r.body).trim()
));
}
out.push_str(" Latest:\n");
for n in notes.iter().take(10) {
let subj = n.header.title.as_deref().unwrap_or("");
let when = n.header.created_at.map(|t| date_string(t)).unwrap_or_else(|| "?".into());
let cat = category_of(n.header.doc_type.as_deref().unwrap_or(""))
.map(|c| format!("[{c}] "))
.unwrap_or_default();
out.push_str(&format!(" {when} {cat}{subj}\n"));
}
out
}
/// Process a `project` command line: `project list`, `project show <C.Z.Y.X>`,
/// `project resume <name> <text>`, `project context <name>`.
pub fn project_command(store: &ConcurrentStore, line: &str) -> Result<String, String> {
let body = line
.splitn(2, char::is_whitespace)
.nth(1)
.unwrap_or("")
.trim()
.to_string();
let Some((head, rest)) = split_head(&body) else {
return Ok(format_projects(project_list(store)));
};
match head {
"list" => Ok(format_projects(project_list(store))),
"show" | "walk" => {
let cs = rest.trim();
let coord = crate::commands::parse_coord(cs)
.ok_or_else(|| format!("bad coordinate '{cs}' (want C.Z.Y.X)"))?;
match show_note(store, coord) {
Some(p) => {
let name = p.header.title.as_deref().unwrap_or("?");
let notes = project_walk(store, coord);
Ok(format!(
"project {cs} \"{name}\" — {} note(s):\n{}",
notes.len(),
format_notes(&notes)
))
}
None => Ok(format!("project {cs} -> (no project record)")),
}
}
"resume" => {
let (_, pos) = split_flags(rest);
let mut words = pos.splitn(2, char::is_whitespace);
let name = words.next().unwrap_or("").to_string();
let text = words.next().unwrap_or("").trim().to_string();
if name.is_empty() || text.is_empty() {
return Err("project resume needs <name> <text>".to_string());
}
let session = default_session();
let coord = project_resume(store, &name, &text, &session)?;
Ok(format!("resume set for '{name}' at {} (session {})", coord_str(coord), session))
}
"context" => {
let name = rest.trim();
if name.is_empty() {
return Err("project context needs <name>".to_string());
}
Ok(project_context(store, name))
}
_ => Err(format!("project: unknown subcommand '{head}' (want list|show|resume|context)")),
}
}
fn format_projects(projects: Vec<(Czyx, CubeHeader, Vec<u8>)>) -> String {
if projects.is_empty() {
return "(no projects)".to_string();
}
let lines: Vec<String> = projects
.iter()
.map(|(c, h, _)| {
let name = h.title.as_deref().unwrap_or("");
let when = h.created_at.map(|t| date_string(t)).unwrap_or_else(|| "?".into());
format!(" {} [{}] {}", coord_str(*c), when, name)
})
.collect();
format!("{} project(s):\n{}", projects.len(), lines.join("\n"))
}
/// Split the first whitespace-delimited token from the rest of a string.
fn split_head(s: &str) -> Option<(&str, &str)> {
let idx = s.find(char::is_whitespace)?;
Some((&s[..idx], &s[idx..].trim()))
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn store_and_list_a_note() {
let store = ConcurrentStore::memory();
let coord = store_note(&store, "test-session", "hello", "world note", None, None).unwrap();
assert!(store.get_record(&coord).is_some());
let all = list_notes(&store, None);
assert_eq!(all.len(), 1);
assert_eq!(all[0].header.doc_type.as_deref(), Some(NOTE_DOC_TYPE));
assert_eq!(all[0].header.owner_local_user.as_deref(), Some("test-session"));
assert_eq!(all[0].body, b"world note");
assert!(all[0].header.word_flags.has(WordFlags::START_RECORD));
assert!(!all[0].header.word_flags.has(WordFlags::ENCRYPTED));
}
#[test]
fn session_and_day_filters() {
let store = ConcurrentStore::memory();
store_note(&store, "a", "one", "first", None, None).unwrap();
store_note(&store, "b", "two", "second", None, None).unwrap();
assert_eq!(
list_notes(&store, Some(&NoteFilter { session: Some("a".into()), ..Default::default() })).len(),
1
);
assert_eq!(list_notes(&store, None).len(), 2);
let today = now() / 86_400;
assert_eq!(
list_notes(&store, Some(&NoteFilter { since: Some(today), ..Default::default() })).len(),
2
);
}
#[test]
fn category_and_project_filters() {
let store = ConcurrentStore::memory();
let a = store_note(&store, "s", "design", "widget design", Some("cubelinux"), Some("design")).unwrap();
let b = store_note(&store, "s", "impl", "widget impl", None, Some("impl")).unwrap();
let c = store_note(&store, "s", "cube", "cube notes", None, None).unwrap();
// category filter
assert_eq!(
list_notes(&store, Some(&NoteFilter { category: Some("design".into()), ..Default::default() })).len(),
1
);
// all categorized + base notes are still found by an unfiltered list
assert_eq!(
list_notes(&store, Some(&NoteFilter { session: Some("s".into()), ..Default::default() })).len(),
3
);
// project filter (a is linked to project cubelinux)
let pc = ensure_project(&store, "cubelinux", "s").unwrap();
assert_eq!(store.get_record(&a).unwrap().0.linked_records, vec![pc]);
assert_eq!(store.get_record(&b).unwrap().0.linked_records, vec![]);
assert_eq!(store.get_record(&c).unwrap().0.linked_records, vec![]);
let proj_notes = list_notes(&store, Some(&NoteFilter { project: Some("cubelinux".into()), ..Default::default() }));
assert_eq!(proj_notes.len(), 1);
assert_eq!(proj_notes[0].header.title.as_deref(), Some("design"));
// project walk returns the thread
assert_eq!(project_walk(&store, pc).len(), 1);
}
#[test]
fn full_text_search_and_dates() {
let store = ConcurrentStore::memory();
store_note(&store, "s", "kernel", "the ipu4 camera driver linked", None, None).unwrap();
store_note(&store, "s", "mail", "opendkim signing works", None, None).unwrap();
assert_eq!(search_notes(&store, "ipu4", None).len(), 1);
assert_eq!(search_notes(&store, "opendkim", None).len(), 1);
assert_eq!(search_notes(&store, "nomatchxyz", None).len(), 0);
assert_eq!(date_to_day("2026-09-08"), Some(20_704));
assert_eq!(date_string(20_704 * 86_400), "2026-09-08");
}
#[test]
fn project_resume_and_context() {
let store = ConcurrentStore::memory();
store_note(&store, "s", "design", "widget design", Some("p"), Some("design")).unwrap();
project_resume(&store, "p", "at the design review", "s").unwrap();
let ctx = project_context(&store, "p");
assert!(ctx.contains("Project: p"));
assert!(ctx.contains("RESUME: at the design review"));
assert!(ctx.contains("[design]"));
}
}
+22 -1
View File
@@ -574,6 +574,25 @@ impl ConcurrentStore {
out
}
/// Reconstruct a directory listing from `path` metatags: every record
/// whose `path` header field is under `dir` (the PDF's "operate on CZYX
/// records and Null-space flags rather than paths and inodes"). Delegates
/// to [`CubeStore::scan_by_path_prefix`].
pub fn query_by_path_prefix(&self, dir: &str) -> Vec<Czyx> {
self.inner.read().unwrap().scan_by_path_prefix(dir)
}
/// Exact-path metatag lookup. Delegates to [`CubeStore::scan_by_path`].
pub fn query_by_path(&self, path: &str) -> Vec<Czyx> {
self.inner.read().unwrap().scan_by_path(path)
}
/// Query records by per-word flag (the 16-bit WordFlags metadata stripe in
/// the header, tag 14). Delegates to [`CubeStore::scan_by_word_flag`].
pub fn scan_by_word_flag(&self, flag: u16) -> Vec<Czyx> {
self.inner.read().unwrap().scan_by_word_flag(flag)
}
/// The `owner_local_user` stamped on the record at `key`, if it has one.
/// Used by owner enforcement (Task 6): a mutating command may only
/// overwrite a record whose owner matches the session's identity owner.
@@ -683,6 +702,7 @@ impl ConcurrentStore {
/// Store a code cell at `path` (the path->code bridge), durability-logged.
/// `owner` (when set) is stamped on the record's `owner_local_user` field
/// so owner enforcement (Task 6) can later reject cross-owner overwrites.
#[allow(clippy::too_many_arguments)]
pub fn put_code_cell(
&self,
path: &str,
@@ -691,9 +711,10 @@ impl ConcurrentStore {
links: &[Czyx],
code: &[Op],
owner: Option<&str>,
descriptor: Option<cubecode::Behavior>,
) -> Result<Czyx, crate::SysError> {
let coord = self.with_mut(|store| {
crate::store_code_cell(store, path, kind, name, links, code, owner)
crate::store_code_cell(store, path, kind, name, links, code, owner, descriptor)
})?;
if let Some(v) = self.get_raw(&coord) {
self.log_put(coord, v);
+11
View File
@@ -0,0 +1,11 @@
[package]
name = "cubetrace"
version = "0.1.0"
edition.workspace = true
license.workspace = true
description = "CUBELinux-2 tracing package: wraps a DBI engine (via FFI seam) and streams trace events (basic blocks, syscalls) into cubestore, tagging each with CZYX coordinates and header flags (PDF Package 4, §547)."
[dependencies]
cubecoords = { path = "../cubecoords" }
cubestore = { path = "../cubestore" }
cubecode = { path = "../cubecode" }
+306
View File
@@ -0,0 +1,306 @@
//! CUBELinux-2 tracing package.
//!
//! Per the PDF (Package 4, §547 / §1113): *"cubetrace: wraps a DBI engine
//! (via FFI) and streams trace events (basic blocks, syscalls) into
//! cubestore, tagging each with CZYX coordinates and header flags."*
//!
//! This crate is the *head* of the Package4 pipeline (cubetrace → cubeai →
//! cubedbt). It is dependency-free and exercises today on a `Null` (offline)
//! source so it builds and is testable without an external DBI engine; the
//! real engine is reached through the documented `DbiEngine` FFI seam.
//!
//! Model
//! -----
//! * A [`TraceEvent`] is a basic block / syscall / metadata observation with a
//! timestamp, a target coordinate, and policy/behavior header flags.
//! * A [`Tracer`] streams events into a [`CubeStore`] keyed by CZYX (the
//! `c240` trace band), tagging each record's header with time, environment,
//! and policy flags (§976).
//! * `replay_all` deterministically replays the stored stream in timestamp
//! order (§977/§1118: "deterministic replay first").
use cubecode::{Behavior, Kind};
use cubecoords::{CubeHeader, Czyx};
use cubestore::{CubeBackend, CubeStore, HashBackend};
/// `C` axis band where trace events are stored (keyed CZYX, §975).
pub const C_TRACE: u8 = 240;
/// Kind of traced observation.
#[derive(Copy, Clone, Debug, PartialEq, Eq)]
pub enum EventKind {
/// A basic block executed (carries the block's code/length).
BasicBlock,
/// A syscall entered/exited (carries the syscall number).
Syscall,
/// Free-form metadata about the run (policy, environment).
Meta,
}
/// A single traced observation: what happened, when, where, and with what
/// policy/behavior tags. Serialized into a trace record's body.
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct TraceEvent {
pub kind: EventKind,
/// Monotonic timestamp (ns from trace start).
pub ts: u64,
/// Target CZYX the event is about (the block/syscall coordinate).
pub coord: Czyx,
/// For `BasicBlock`: the executed bytecode; for `Syscall`: the syscall
/// number; for `Meta`: unused (0).
pub payload: Vec<u8>,
/// Behavior/policy descriptors to stamp onto the record header.
pub behavior: Behavior,
}
impl TraceEvent {
/// Serialize to a stable byte form (no external deps).
pub fn to_bytes(&self) -> Vec<u8> {
let mut out = Vec::new();
out.push(self.kind as u8);
out.extend_from_slice(&self.ts.to_le_bytes());
out.extend_from_slice(&CZYX_BYTES);
out.extend_from_slice(&self.coord.pack_u32().to_le_bytes());
out.extend_from_slice(&self.behavior.to_flags().to_le_bytes());
out.push(self.payload.len() as u8);
out.extend_from_slice(&self.payload);
out
}
/// Inverse of [`to_bytes`]. Returns `None` on malformed input.
pub fn from_bytes(b: &[u8]) -> Option<TraceEvent> {
let mut i = 0;
let kind = match *b.get(i)? {
0 => EventKind::BasicBlock,
1 => EventKind::Syscall,
2 => EventKind::Meta,
_ => return None,
};
i += 1;
let ts = u64::from_le_bytes(b.get(i..i + 8)?.try_into().ok()?);
i += 8;
i += 4; // skip CZYX_BYTES sentinel
let packed = u32::from_le_bytes(b.get(i..i + 4)?.try_into().ok()?);
i += 4;
let coord = Czyx::new(
(packed >> 24) as u8,
(packed >> 16) as u8,
(packed >> 8) as u8,
packed as u8,
);
let flags = u16::from_le_bytes(b.get(i..i + 2)?.try_into().ok()?);
i += 2;
let plen = *b.get(i)? as usize;
i += 1;
let payload = b.get(i..i + plen)?.to_vec();
Some(TraceEvent {
kind,
ts,
coord,
payload,
behavior: Behavior::from_flags(flags),
})
}
}
/// Sentinel to make the on-wire format self-identifying as a CZYX record.
const CZYX_BYTES: [u8; 4] = *b"CZYX";
/// Source of trace events. In the full stack this is the FFI seam to a DBI
/// engine (DynamoRIO / Intel Pin / QBDI / Frida — §970). Here it is a trait so
/// the crate is testable offline via [`NullEngine`] and a real engine can be
/// dropped in without changing callers.
pub trait DbiEngine {
/// Attach to a target (pid or path). Returns `Err` if the engine can't.
fn attach(&mut self, target: &str) -> Result<(), String>;
/// Pull the next event, or `None` when the run is exhausted.
fn next_event(&mut self) -> Option<TraceEvent>;
}
/// Offline engine: replays a pre-recorded event vector. Stands in for a live
/// DBI backend in tests and headless environments.
pub struct NullEngine {
events: Vec<TraceEvent>,
idx: usize,
}
impl NullEngine {
pub fn new(events: Vec<TraceEvent>) -> Self {
NullEngine { events, idx: 0 }
}
}
impl DbiEngine for NullEngine {
fn attach(&mut self, _target: &str) -> Result<(), String> {
Ok(())
}
fn next_event(&mut self) -> Option<TraceEvent> {
if self.idx < self.events.len() {
let e = self.events[self.idx].clone();
self.idx += 1;
Some(e)
} else {
None
}
}
}
/// Streams trace events into a [`CubeStore`] (the `c240` band), tagging each
/// record with time, environment, and policy header flags (PDF §976).
pub struct Tracer<B: CubeBackend> {
store: CubeStore<B>,
next_x: u8,
}
impl<B: CubeBackend> Tracer<B> {
pub fn new(store: CubeStore<B>) -> Self {
Tracer { store, next_x: 1 }
}
/// Drain an engine into the store, returning the number of events stored.
pub fn run<E: DbiEngine>(&mut self, engine: &mut E) -> usize {
let mut count = 0;
while let Some(ev) = engine.next_event() {
self.store_event(&ev);
count += 1;
}
count
}
/// Store one event under a fresh CZYX coordinate in the `c240` band.
pub fn store_event(&mut self, ev: &TraceEvent) -> Czyx {
let label = Czyx::new(C_TRACE, 1, ev.kind as u8, self.next_x);
self.next_x = self.next_x.wrapping_add(1).max(1);
let mut h = CubeHeader::new();
h.title = Some(format!("{:?}:{:?}", ev.kind, ev.coord));
h.doc_type = Some(Kind::Other.as_str().into());
h.size_bytes = Some(ev.to_bytes().len() as u64);
h.flags.0 |= ev.behavior.to_flags();
h.refresh_flags();
self.store.put_record(label, &h, &ev.to_bytes());
label
}
/// Borrow the backing store (e.g. to hand to `cubeai`/others).
pub fn store(&self) -> &CubeStore<B> {
&self.store
}
}
/// Deterministic replay: collect every trace event from the `c240` band and
/// return them in timestamp order (PDF §977/§1118: "deterministic replay
/// first"). The trace is the immutable cube record; replay never mutates it.
pub fn replay_all(store: &CubeStore<HashBackend>) -> Vec<TraceEvent> {
let mut events = Vec::new();
for k in store.keys() {
if k.c != C_TRACE {
continue;
}
if let Some((_, body)) = store.get_record(&k) {
if let Some(ev) = TraceEvent::from_bytes(&body) {
events.push(ev);
}
}
}
events.sort_by_key(|e| e.ts);
events
}
#[cfg(test)]
mod tests {
use super::*;
fn ev(kind: EventKind, ts: u64, coord: Czyx, beh: Behavior) -> TraceEvent {
TraceEvent {
kind,
ts,
coord,
payload: vec![kind as u8],
behavior: beh,
}
}
fn behavior(flag: u16) -> Behavior {
Behavior(flag)
}
#[test]
fn event_round_trips_through_bytes() {
let e = ev(
EventKind::Syscall,
42,
Czyx::new(1, 2, 3, 4),
behavior(Behavior::HOT_PATH),
);
let back = TraceEvent::from_bytes(&e.to_bytes()).expect("decodes");
assert_eq!(e, back);
}
#[test]
fn tracer_streams_to_c240() {
let mut engine = NullEngine::new(vec![
ev(
EventKind::BasicBlock,
10,
Czyx::new(1, 0, 0, 1),
behavior(Behavior::PURE),
),
ev(
EventKind::Syscall,
20,
Czyx::new(1, 0, 0, 2),
behavior(Behavior::IO_HEAVY),
),
ev(
EventKind::Meta,
5,
Czyx::new(1, 0, 0, 3),
Behavior::default(),
),
]);
let mut tracer = Tracer::new(CubeStore::new(HashBackend::new()));
assert_eq!(tracer.run(&mut engine), 3);
// All records landed in the c240 band.
let stored: usize = tracer
.store()
.keys()
.into_iter()
.filter(|k| k.c == C_TRACE)
.count();
assert_eq!(stored, 3);
}
#[test]
fn replay_is_deterministic_and_ordered() {
let mut engine = NullEngine::new(vec![
ev(
EventKind::BasicBlock,
30,
Czyx::new(1, 0, 0, 1),
behavior(Behavior::PURE),
),
ev(
EventKind::Syscall,
10,
Czyx::new(1, 0, 0, 2),
behavior(Behavior::IO_HEAVY),
),
ev(
EventKind::Meta,
20,
Czyx::new(1, 0, 0, 3),
Behavior::default(),
),
]);
let mut tracer = Tracer::new(CubeStore::new(HashBackend::new()));
tracer.run(&mut engine);
let replayed = replay_all(tracer.store());
assert_eq!(replayed.len(), 3);
// Sorted by ts: 10 (Syscall), 20 (Meta), 30 (BasicBlock).
assert_eq!(replayed[0].kind, EventKind::Syscall);
assert_eq!(replayed[1].kind, EventKind::Meta);
assert_eq!(replayed[2].kind, EventKind::BasicBlock);
// Behavior flags survived the round-trip through the store.
assert_eq!(replayed[0].behavior, Behavior(Behavior::IO_HEAVY));
}
}
+49
View File
@@ -0,0 +1,49 @@
# CUBE notes ↔ model integration: `cube-notes-agent`
The CUBELinux-2 note/message-save-path is a durable, searchable log that a small
local LLM reads, reviews, and writes back to — a closed loop that is **grounded
in CUBE** (the model never invents findings) and **incremental** (each run reviews
only what has not been seen).
## The notes system (`cubesys::notes`)
- WordFlags-tagged **CZYX** note log. Each note has `doc_type = note:<category>`
(categories like `finding`, `action`, `checkpoint`, `resume`, `task`, …) and an
optional **project** association via `linked_records`.
- `cube note add|list|search|show`; `cube project list|show|resume|context`.
- `cube note tag <coord> <cat>` reclassifies a note in place (Puts are durable;
`delete_raw` is not — the checkpoint/delta only records Puts of surviving keys,
so a deleted key resurrects from the base snapshot on reload).
- Durable `ConcurrentStore` (WAL + delta + checkpoint). Short-lived processes must
`store.checkpoint()` before exit or buffered writes are lost.
## Model ↔ CUBE
- `cube-notes-mcp.py` (stdio MCP server) exposes `mcp__cubenotes__note_*` /
`project_*` tools to the DeepSeek Harness. The harness registers the mcp-client at
the **bundle** layer (`inject: [tools]`); profile-layer rows can't inject the tools
service, so the plugin silently never mounts — the bundle registration was the fix
that made `mcp__cubenotes__note_*` live.
- `tools/cube-notes-agent.py` (headless, cron-driven, no control panel) treats CUBE as
the agent's memory:
- every action is written as a note (`category=action`) so a run can be replayed;
- a review run reads **only uncovered** findings (`category=finding`), asks the model
to rank them, then writes a grounded report and marks them covered with a
`checkpoint` note — true incremental, non-destructive review;
- the report is composed in Python from the finding notes **verbatim**, with a
deterministic severity (HIGH/MEDIUM/LOW) and priority order, so no content is
invented. Report ⇒ a review `.txt` (`/root/workspace/cube-agent/report.txt`).
## Success: tracing errors from prior run data saved in CUBE
- The findings/action trace already stored in CUBE (from prior runs) is the source of
truth. The broad scan surfaced **7 un-categorized logs** that had been logged but
never tagged `finding`: `GIT-SHALLOW-CLONE`, `POSTFIX-TLS-CERTS`,
`SELINUX-RESTORECON`, `DSH-WEB-EADDRINUSE`, `CONCURRENT-STORE-CHECKPOINT`,
`MCP-BUNDLE-LAYER`, `CUBES-TWO-TREES`. See `git log` entry `bd3e3c0`.
- Coverage is precise: the report marks exactly the findings it reviewed, so the next
run reports only new ones. The notes store is also the harness GUI's CUBE surface
(`mcp-cubenotes note_list`), so everything the agent writes is visible and reviewable.
## Usage
```
python3 tools/cube-notes-agent.py --max-turns 6 --report /path/report.txt
```
Runs single-instance via `flock`; a cron schedule (e.g. `0 6,18 * * *`) drives it.
+405
View File
@@ -0,0 +1,405 @@
#!/usr/bin/env python3
"""Headless CUBE notes agent — NO control panel. Reads a task from the CUBE
notes (a project), uses the cube note tools (note_search/note_list/
project_context + read_file), and writes a report. Designed to be driven by
cron (e.g. build an error-log report every hour); the interval is the cron
schedule, so it is fully configurable without touching the agent.
Usage: python3 agent.py --project NAME [--max-turns N] [--report PATH]
"""
import argparse, json, os, subprocess, sys, time
import datetime as _dt
def _now_ts():
return _dt.datetime.now().strftime("%Y-%m-%d %H:%M")
MODEL = "/home/luulu/models/gemma-3-4b-it-Q4_K_M.gguf"
LLAMA = "/home/luulu/llama.cpp/build/bin/llama-server"
CUBE = "/home/CUBELinux/CUBELinux-2/target/release/cube"
PORT = 8080
BASE = f"http://127.0.0.1:{PORT}"
LOG = "/root/workspace/cube-agent/llama.log" if os.path.isdir("/root/workspace/cube-agent") else "/tmp/local-agent-llama.log"
TRACE_PROJECT = "agent-trace" # every action the agent takes is logged here (replay/tuning)
def env():
e = dict(os.environ)
e["CUBE_NOTES_DIR"] = "/root/.cubelinux-notes"
return e
def start_llama():
p = subprocess.Popen(
[LLAMA, "--model", MODEL, "--host", "127.0.0.1", "--port", str(PORT),
"--alias", "gemma-3-4b", "--ctx-size", "8192", "-np", "1",
"--temp", "0.2", "--repeat-penalty", "1.1"],
stdout=open(LOG, "w"), stderr=subprocess.STDOUT, env=env())
import requests
for _ in range(180):
try:
if requests.get(BASE + "/v1/models", timeout=2).status_code == 200:
break
except Exception:
pass
time.sleep(2)
return p
def chat(messages, max_tokens=600):
import requests
r = requests.post(BASE + "/v1/chat/completions", json={
"model": "gemma-3-4b", "messages": messages,
"max_tokens": max_tokens, "temperature": 0.08}, timeout=600)
return r.json()["choices"][0]["message"]["content"]
def run_cube(args):
try:
r = subprocess.run([CUBE] + args, capture_output=True, text=True,
env=env(), timeout=30)
return (r.stdout or r.stderr).strip()
except Exception as e:
return f"error: {e}"
def log_action(step, text):
"""Write an agent action to the CUBE notes trace (category=action) so the
run can be replayed and fine-tuned."""
body = f"step {step}: {text}"[:1600]
subprocess.run([CUBE, "note", "add", "--project", TRACE_PROJECT,
"--cat", "action", body], capture_output=True, text=True, env=env())
def open_file(path):
try:
return open(path).read()[:2000]
except Exception as e:
return f"error: {e}"
# ---- incremental coverage (deterministic; offloaded from the small model) ----
import re as _re
COORD_RE = _re.compile(r'\b(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\b')
def _coords(text):
return set(COORD_RE.findall(text))
# Bookkeeping categories that are NOT error-log findings (agent trace, coverage
# markers, resume checkpoints, test fixtures, design/impl notes, misc).
NONLOG_CATS = {"action", "checkpoint", "resume", "task", "debug", "impl", "design", "misc"}
# A finding-like title: a SHORT-UPPERCASE-HYPHEN/UNDERSCORE token then ':' or ' — '
FLAG_RE = _re.compile(r'\s*([A-Z][A-Z0-9_-]{2,})\s*[:—-]')
ERR_KW = ("error", "failed", "failure", "denied", "rejected", "cannot", "cannot be",
"must ", "timeout", "times out", "not registered", "broken", "corrupt",
"integrity", "permission denied", "selinux", "restorecon", "orphan",
"eadrinuse", "shallow", "user_tmp_t", "mcp-client", "mcp plugin")
def _scan(cat, project=None):
"""Run `cube note list`, optionally filtered by category/project. With no cat
and no project this scans the WHOLE store (every session), so the agent
reviews ALL CUBE error logs, not one project or one category."""
args = ["note", "list"]
if cat:
args += ["--cat", cat]
if project:
args += ["--project", project]
return run_cube(args)
def _parse_note_line(line):
"""-> (coord, cat_or_None, subject) for a `note list` line, or None if the
line is not a note (e.g. the 'N note(s):' header)."""
m = _re.match(r'\s*(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\s+\[[^\]]+\]', line)
if not m:
return None
coord = m.group(1)
rest = line[m.end():].strip()
cat = None
cm = _re.match(r'\[([a-z]+)\]\s*', rest)
if cm:
cat = cm.group(1)
rest = rest[cm.end():].strip()
return coord, cat, rest
def _looks_like_finding(subj):
"""True if an un-categorized note looks like an error-log finding."""
if FLAG_RE.match(subj):
return True
low = subj.lower()
return any(k in low for k in ERR_KW)
def _log_findings(project=None):
"""[(coord, cat, subject)] for every finding-like note in the store, across ALL
categories and projects. This includes `finding` notes and any other note with a
FLAG-like / error title (the un-categorized logs), while excluding bookkeeping
categories (action / checkpoint / resume / task / debug / impl / design / misc)."""
out = []
for line in _scan(None, project).splitlines():
parsed = _parse_note_line(line)
if parsed is None:
continue
coord, cat, subj = parsed
if cat in NONLOG_CATS:
continue
if cat == "finding" or _looks_like_finding(subj):
out.append((coord, cat, subj))
return out
def _checkpoint_coords(project=None):
"""Own coord (leftmost) of every checkpoint note (optionally in a project)."""
coords = []
for line in _scan("checkpoint", project).splitlines():
m = _re.match(r'\s*(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})', line)
if m:
coords.append(m.group(1))
return coords
def _covered(project=None):
"""Covered = each checkpoint note's own coord + the coords in its FULL body.
(note list only shows the truncated subject, so use note show to read the body.)"""
out = set()
for c in _checkpoint_coords(project):
out.add(c)
out |= _coords(run_cube(["note", "show", c]))
return out
def uncovered_findings(project=None):
covered = _covered(project)
un = "".join(f"{coord} {subj}\n" for coord, cat, subj in _log_findings(project) if coord not in covered)
return un.strip() or "(no new findings)"
# Canonical issue key: collapse duplicate representations of the same root cause
# (e.g. FK1-SELINUX_RESTORECON vs SELINUX-RESTORECON) so the report dedupes by ISSUE,
# not just by coordinate. Prefixes (FK, FK#, FA, FB, FLAG, FIX) are stripped and
# separators normalized; a small alias map unifies differently-worded duplicates.
_ISSUE_PREFIX_RE = _re.compile(r'^(FK\d*|FA|FB|FLAG|FIX)[-_]?', _re.I)
_ISSUE_ALIAS = {
"wal_checkpoint": "checkpoint_before_exit",
"concurrent_store_checkpoint": "checkpoint_before_exit",
}
def _canonical_issue(flag):
f = _ISSUE_PREFIX_RE.sub('', flag)
f = _re.sub(r'[-_/\s]+', '_', f).strip('_').lower()
return _ISSUE_ALIAS.get(f, f)
def _uncovered_meta(project=None):
"""[(coord, flag, desc)] for the currently-UNCOVERED findings, DEDUPED BY ISSUE.
desc is the FULL note body; the most-detailed note per canonical issue is kept."""
covered = _covered(project)
best = {} # issue -> (coord, flag, desc, len)
for coord, cat, subj in _log_findings(project):
if coord in covered:
continue
show = run_cube(["note", "show", coord])
body_text = show.split("\n", 1)[1].strip() if "\n" in show else ""
if not body_text:
body_text = subj
flag, sep, rest = body_text.partition(":")
flag = flag.strip()
desc = rest.strip() if sep else body_text
issue = _canonical_issue(flag)
cur = best.get(issue)
if cur is None or len(desc) > cur[3]:
best[issue] = (coord, flag, desc, len(desc))
return [(c, f, d) for c, f, d, _ in best.values()]
# Snapshot of the uncovered findings captured when read_findings ran, so the
# report can be composed faithfully even after mark_covered has covered them.
CURRENT_META = []
def read_findings(project=None):
"""Deterministic, grounded input for the model: the currently-UNCOVERED CUBE
findings, numbered with their FLAG. The model replies with ONLY the FLAGs in
priority order; Python composes the report from the grounded text, so it is
impossible for any finding to be invented."""
global CURRENT_META
meta = _uncovered_meta(project)
CURRENT_META = meta
if not meta:
return "(no new findings)"
lines = []
for i, (coord, flag, desc) in enumerate(meta, 1):
descf = (desc[:120] + "") if len(desc) > 120 else desc
lines.append(f"{i}. {flag}: {descf} (coord {coord})")
return ("Uncovered CUBE findings (use ONLY these; do not invent, add, or guess):\n"
+ "\n".join(lines)
+ "\n\nReply with the finding FLAGs in priority order, most critical first, "
"comma-separated, using exactly the FLAGs above and nothing else — for example:\n"
+ ", ".join(f for _, f, _ in meta)
+ "\nThen call mark_covered, then reply done (no report body needed).")
_SEV_CRIT = ["data integrity", "security", "tls", "cert", "denied", "cannot",
"corrupt", "lost", "inaccessible", "integrity", "breach"]
_SEV_HIGH = ["timeout", "persist", "reload", "shallow", "must", "fail", "broken"]
def _severity(flag, desc):
"""Deterministic severity label from the finding's FLAG + description. Used to
give the report a meaningful priority order when the model provides no ranking."""
t = (flag + " " + desc).lower()
if any(k in t for k in _SEV_CRIT):
return "HIGH"
if any(k in t for k in _SEV_HIGH):
return "MEDIUM"
return "LOW"
def compose_report(project=None, model_text=""):
"""Build the grounded, prioritized report. The model supplies ONLY the priority
ORDER of the FLAGs (honored when present); every finding's text is taken verbatim
from CUBE, so any hallucinated finding is impossible. Unknown/duplicate flags are
dropped; flags the model did not rank are ordered by a deterministic severity
heuristic (HIGH MEDIUM LOW), then CUBE order. Uses the snapshot from read_findings."""
meta = CURRENT_META or _uncovered_meta(project)
if not meta:
return "ERROR LOG REPORT (ALL)\n\nNo new findings since the last checkpoint."
text_low = model_text.lower()
listed = {}
for coord, flag, desc in meta:
listed.setdefault(flag.lower(), (coord, flag, desc))
# Model's ranking first (by first-flag-appearance order in its reply).
positions = []
for key in listed:
idx = text_low.find(key)
if idx >= 0:
positions.append((idx, key))
positions.sort()
order = [key for _, key in positions]
remaining = [key for key in listed if key not in order]
sev_rank = {"HIGH": 0, "MEDIUM": 1, "LOW": 2}
remaining.sort(key=lambda k: sev_rank.get(_severity(listed[k][1], listed[k][2]), 3))
order += remaining
scope = project if project else "ALL"
lines = [f"ERROR LOG REPORT (scope: {scope})",
f"Generated: {_now_ts()}", ""]
for rank, key in enumerate(order, 1):
coord, flag, desc = listed[key]
sev = _severity(flag, desc)
lines.append(f"{rank}. [{sev}] {flag}: {desc if desc else '(no description)'}")
lines.append(f" coord: {coord}")
return "\n".join(lines)
def mark_covered(project=None, text=""):
covered = _covered(project)
coords = [c for c, cat, subj in _log_findings(project) if c not in covered]
if not coords:
return "nothing to cover"
body = "covered coords:\n" + "\n".join(coords)
args = [CUBE, "note", "add", "--cat", "checkpoint"]
if project:
args += ["--project", project]
args.append(body[:1400])
subprocess.run(args, capture_output=True, text=True, env=env())
return "marked covered"
def tool(kw, args, project):
if kw == "read_findings":
return read_findings(project)
if kw == "mark_covered":
return mark_covered(project, args)
if kw == "note_search":
return run_cube(["note", "search", args])
if kw == "note_list":
return run_cube(["note", "list"] + (args.split() if args else []))
if kw == "project_context":
return run_cube(["project", "context", args])
if kw == "read_file":
return open_file(args)
return f"unknown tool: {kw}"
ACTIONS = ("read_findings", "mark_covered", "note_search",
"note_list", "project_context", "read_file", "done")
def parse_action(text):
"""Parse an action word out of the model's reply. Tolerant of markdown code
fences, ```tool_code``` wrappers, and prose preamble (the small gemma model
tends to wrap calls like this). Scans lines and stops once it sees "done",
so the trailing report is never misread as another tool call."""
best = (None, "")
for raw in text.strip().splitlines():
s = raw.replace("`", "").replace("tool_code", " ").strip()
low = s.lower()
for kw in ACTIONS:
idx = low.find(kw)
if idx >= 0:
best = (kw, s[idx + len(kw):].strip())
break
if best[0] == "done":
break
return best
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--project", default=None,
help="Restrict the report to one CUBE project (default: ALL finding notes across the store)")
ap.add_argument("--max-turns", type=int, default=6)
ap.add_argument("--report", default="/root/workspace/cube-agent/report.txt")
a = ap.parse_args()
p = start_llama()
scope = a.project if a.project else "ALL"
# Ground the report in CUBE up front: snapshot the uncovered findings and hand
# them to the model so it only has to RANK them (it never writes finding text).
findings_text = read_findings(a.project)
if findings_text == "(no new findings)":
initial_context = "There are no uncovered findings to report right now."
task = "Reply done (no report body needed)."
else:
initial_context = findings_text
task = ("Reply with the finding FLAGs in priority order (most critical first), "
"comma-separated, using exactly the FLAGs below and nothing else — then reply done.")
sys_prompt = (
f"You are a headless error-log reviewer for the CUBE notes store (scope: {scope}).\n"
"The findings are provided below. Order them by priority/severity, most critical first.\n"
"Reply with the FLAGs in that order, comma-separated, using exactly those FLAGs and nothing else.\n"
"Then reply: done. "
"Your ranked FLAG list sets the priority; the report body is assembled for you, so do not invent findings."
)
messages = [
{"role": "system", "content": sys_prompt},
{"role": "user", "content": f"{task}\n\n{initial_context}"},
]
trace = []
report_text = ""
last_assistant = ""
for turn in range(a.max_turns):
try:
text = chat(messages)
except Exception as e:
trace.append(f"chat error: {e}")
break
kw, args = parse_action(text)
if kw == "done":
# The model supplies ONLY the priority order of the FLAGs; Python
# composes the grounded report (impossible to invent a finding).
di = text.lower().find("done")
model_text = text[di + 4:].strip() if di >= 0 else last_assistant
report_text = compose_report(a.project, model_text)
log_action(turn, f"DONE\n\n~flags: {model_text[:400]}")
break
if kw:
result = tool(kw, args, a.project)
trace.append(f"turn{turn}: {kw}({args})")
log_action(turn, f"{kw}({args})\n\n{result[:1400]}")
messages.append({"role": "assistant", "content": text})
messages.append({"role": "user",
"content": f"Tool result:\n{result}\n\nContinue — output exactly one "
"action line (read_findings / mark_covered / note_search / "
"note_list / project_context / read_file / done)."})
else:
log_action(turn, f"(no action) {text[:300]}")
messages.append({"role": "assistant", "content": text})
messages.append({"role": "user",
"content": "Output exactly ONE action line starting with a tool name."})
trace.append(f"turn{turn}: (no action)")
last_assistant = text
if not report_text:
# Run ended without an explicit "done" — still produce a grounded report.
report_text = compose_report(a.project, last_assistant)
# Deterministic coverage: mark all uncovered findings covered (including any
# deduped-away duplicates) regardless of whether the model did it, so the next
# run is genuinely incremental. No-op when there are no uncovered findings.
mark_covered(a.project)
with open(a.report, "w") as f:
f.write(report_text)
os.makedirs(os.path.dirname(a.report), exist_ok=True) if os.path.dirname(a.report) else None
print("AGENT_DONE max_turns=", a.max_turns, "report=", a.report,
"bytes=", os.path.getsize(a.report) if os.path.exists(a.report) else 0)
try:
p.terminate()
except Exception:
pass
if __name__ == "__main__":
main()