Compare commits

24 Commits
Author SHA1 Message Date
admin 8c66478a42 cubesys/cube-notes-mcp: project resume + context. 'cube project resume <name> <text>' sets a resume marker (note category=resume, associated to the project); 'cube project context <name>' emits a compact 'where we are' summary (latest notes + resume marker) for context injection into a new session. MCP tools project_resume/project_context added. 5 notes tests pass. 2026-09-08 05:34:58 -04:00
admin 18f8eb9b4a cubesys/cubecli/cube-notes-mcp: note categories + project threads. notes use doc_type 'note:<category>' for categories and associate to a 'project' record (linked_records) for project threads; cube note add|list|search accept --project/--cat/--since/--until; new 'cube project list' + 'cube project show <coord>' (walk a project's thread). MCP server exposes note_write/list/search/show + project_list/project_show. 4 notes tests pass; workspace check green. 2026-09-08 05:29:06 -04:00
admin 15dd4c48a4 cube-notes-mcp: stdio MCP server exposing the cubesys::notes CLI (cube note add|list|search|show) as model-facing mcp__cubenotes__note_* tools for the DeepSeek Harness <-> CUBE session message path. 2026-09-08 04:02:21 -04:00
admin a601710d50 cubesys/cubecli: session note/message-save-path (notes) — WordFlags-tagged CZYX note log (doc_type=note, per-session owner + created_at day), store_note/list/search/show + note_command dispatcher, durable CubeStore+WAL+checkpoint; cube note add|list|search|show. New cubesys::notes module w/ 3 tests; workspace check green. 2026-09-08 04:01:06 -04:00
CUBELinux-2 cc896fc336 CUBELinux-2: WordFlags/tri-channel 16-bit flag field, tag-14 + scan_by_word_flag, cubetrace capture→replay→golden→lineage, CUBE VP-tree balling index (cubecode::ballindex), cubefs-visible WordFlags, and cube-mvw (sharded multi-writer MVCC store with per-shard WALs + group-commit + compaction, impl CubeBackend => CubeStore<MvwBackend> DB). Adds show-flags/scan-word-flags/trace-*/golden-capture CLI; ~ tests green. 2026-09-06 21:20:04 -04:00
CUBELinux-2andHermes Agent (upstage/solar-pro4:free) 3f007a1f38 bench(cube-bench): use saturating_sub for expected-value arithmetic
clippy -D warnings flagged implicit_saturating_sub on the c1/c2/c3
expected-count calculations. Replace manual - with saturating_sub so
the lint gate stays clean at any scale.

Co-Authored-By: Hermes Agent (upstage/solar-pro4:free)
2026-08-20 19:03:48 -04:00
CUBELinux-2andHermes Agent (upstage/solar-pro4:free) ae1a62a44e docs: reflect cubecli as standalone crate in README + integration docs
The  binary was extracted to  in a prior session; the
docs still referenced  as its home.

Co-Authored-By: Hermes Agent (upstage/solar-pro4:free)
2026-08-20 18:59:28 -04:00
CUBELinux-2andHermes Agent (upstage/solar-pro4:free) c090233056 bench(cube-bench): fix scan_prefix expected-value math for scale > 65536
coord_for spreads records across C=0,1,2,3 as i grows past 65536,
but the bench assumed everything beyond c=0 landed in c=1. At
scale=200k this asserted (got 65536, expected 134464). Fix: compute
per-bucket expected counts from the coord_for mapping and extend the
total-coverage assertion to include c=2 and c=3.

Verified: fmt clean, clippy -D clean, 100k + 200k both pass.

Co-Authored-By: Hermes Agent (upstage/solar-pro4:free)
2026-08-20 18:56:20 -04:00
CUBELinux-2andHermes Agent 7b0cf6fda5 chore(this session): land all 4 tasks — fixes, cube-bench run, cubecli extract, check gate passed
1. Fixes applied this session:
   - cubesys/src/lib.rs:136 — `store_code_cell` annotated with
     `#[allow(clippy::too_many_arguments)]` (8 args is inherent to the
     record-codec bridge; clippy -D warnings gate).
   - cubesys/src/store.rs:699 — `put_code_cell` annotated identically.
   - cubesys/src/commands.rs:819 — `cell.links().iter().copied().collect()`
     replaced with `cell.links().to_vec()`.
   - cubesys/src/commands.rs:991 — `while let Some(tok) = it.next()` loop
     rewritten as direct `for tok in it`.
   - cubesys/src/commands.rs:1187 — ls handler uses `parse_path` (not
     `path_to_czyx`) for directory ACL gate; derived prefix coordinate
     matches NullSpace acl_bucket convention.

2. cube-bench executed: ./check bench stage passes with correctness
   assertions on every measurement path (100k puts, 100k gets, crypto
   benchmarks, VM dispatch, session commands, concurrent RwLock stress).

3. cubecli extracted: standalone crate at cubecli/ with its own
   Cargo.toml + src/main.rs; cube binary unchanged (same CLI surface,
   demo, commands); cubesys/Cargo.toml no longer declares the `cube`
   bin; workspace members updated.

4. ./check gate: fmt + tests + clippy -D warnings all green;
   `./check bench` also passes.

Verification: ./check (full gate) + ./check bench both PASS.

Co-Authored-By: Hermes Agent
2026-08-20 18:08:19 -04:00
CUBELinux-2andHermes Agent ce8ac78a24 extract(cubecli): move cube CLI out of cubesys into standalone crate
The `cube` binary (REPL + script runner over one shared CubeStore) is
now its own crate at `cubecli/`, matching the spec's package outline
that lists cubecli as a separate crate from the composition layer.

What changed:
- New `cubecli/` crate with `Cargo.toml` (deps: cubecoords, cubestore,
  cubefs, cubecode, cubecrypt, cubesys) and `src/main.rs` (copy of
  the former `cubesys/src/bin/cube.rs`).
- `cubesys/Cargo.toml`: removed the `[[bin]]` entry for `cube`; keeps
  `cube-demo`, `cube-server`, and `cubec` as before.
- `Cargo.toml` (workspace): added `cubecli` to members.
- `cubesys/src/bin/cube.rs` removed (code now lives in cubecli).

The `cube` binary is unchanged: same CLI surface, same demo, same
commands. The `cube-demo` binary in cubesys still wraps
`cubesys::demo::run()` and is unaffected.

Verification: `./check` gate passes (fmt + tests + clippy -D warnings);
`cube --help` and `cube demo` both work.

Co-Authored-By: Hermes Agent
2026-08-20 18:05:56 -04:00
CUBELinux-2andHermes Agent fdfcd2c728 chore(cubesys): resolve 4 clippy -D warnings blocking the check gate
* commands.rs:819 — `cell.links().iter().copied().collect::<Vec<_>>()`
  replaced with `cell.links().to_vec()` (clippy: "calling to_vec() is
  both faster and more readable").

* commands.rs:991 — `while let Some(tok) = it.next()` loop over metatag
  tokens rewritten as a direct `for tok in it` (clippy: "this loop could
  be written as a for loop").

* store.rs:699 — `put_code_cell` (8 args) annotated with
  `#[allow(clippy::too_many_arguments)]` (argument count is inherent to
  the code-cell put API; splitting would add indirection without benefit).

* lib.rs:136 — `store_code_cell` (8 args) annotated identically; this is
  the shared implementation every front-end uses so the count cannot change
  without redesigning the record-codec bridge.

Verification: ./check gate passes (fmt + tests + clippy -D warnings).

Co-Authored-By: Hermes Agent
2026-08-20 17:49:38 -04:00
CUBELinux-2andHermes Agent 90e90ef926 fix(cubeai): add Default impl for CubeAi to satisfy clippy -D warnings
clippy flagged "you should consider adding a `Default` implementation for
`CubeAi`". Add the impl delegating to `new()`.

Also: impl block methods had been accidentally pulled out of `impl CubeAi`
into module scope during a prior edit; rewrite the impl block with all
methods (ingest_trace, classify, suggest, ingest_and_suggest, store)
inside it so it compiles.

Verification: all 5 cubeai tests pass, clippy clean.

Co-Authored-By: Hermes Agent
2026-08-20 17:41:02 -04:00
CUBELinux-2andHermes Agent ec25347af9 fix(cubedbt): add Default impl for CodeCache to satisfy clippy -D warnings
clippy flagged "you should consider adding a `Default` implementation for
`CodeCache`". Add the impl delegating to `new()`.

Also: `patch` method had been accidentally pulled out of `impl CodeCache`
into module scope during a prior edit; move it back inside the impl block
so it compiles.

Verification: all 4 cubedbt tests pass, clippy clean.

Co-Authored-By: Hermes Agent
2026-08-20 17:36:17 -04:00
CUBELinux-2andHermes Agent 6627505e70 fix(cubesys): ls on directory paths no longer rejected by path_to_czyx
The ls handler used path_to_czyx() (which enforces "path must name a
record") to derive the ACL gate coordinate. Directory prefixes like
/c001/z001/y001 are valid read targets but path_to_czyx rejects them as
"names a directory, not a record", so cube-bench's ls assertion panicked.

Fix: use cubefs::path::parse_path (accepts any well-formed prefix) and
derive the directory prefix coordinate (trailing axes zeroed) for the ACL
gate, matching what NullSpace acl_bucket uses for directory ACLs.

Also: clippy -D warnings — remove unnecessary `c as u8` cast in
hash_backend_keys_sorted_order test.

Verification: cube-bench runs to completion (ls section printed OK),
clippy clean.

Co-Authored-By: Hermes Agent
2026-08-20 17:31:54 -04:00
CUBELinux-2andHermes Agent ddca08ea73 fix(cubetrace): wrap Behavior::variant in behavior() helper in test call sites
Behavior is a newtype (Behavior(pub u16)), not an enum, so the 5 test
call sites that passed Behavior::PURE / Behavior::IO_HEAVY / etc. directly
to the ev() helper got type mismatch errors. Add a local behavior() wrapper
and use it at all 5 sites. Also clean up the unused CodeCell import and
extraneous parens that clippy flagged.

fix(cubesys): ls on directory paths no longer rejected by path_to_czyx

The ls handler used path_to_czyx() (which enforces "path must name a
record") to derive the ACL gate coordinate. Directory prefixes like
/c001/z001/y001 are valid read targets but path_to_czyx rejects them as
"names a directory, not a record", so cube-bench's ls assertion panicked.

Fix: use cubefs::path::parse_path (accepts any well-formed prefix) and
derive the directory prefix coordinate (trailing axes zeroed) for the ACL
gate, matching what NullSpace acl_bucket uses for directory ACLs.

Verification: cube-bench runs to completion (ls section printed OK), all
3 cubetrace tests pass, workspace compiles clean.

Co-Authored-By: Hermes Agent
2026-08-20 17:25:48 -04:00
CUBELinux-2andHermes Agent b97efa2a16 scrub: remove cube-store-raw + cube-fuse-proxy, the wrong-architecture duplicates
The workspace already has cubestore (CubeBackend trait + HashBackend
= HashMap<u32,Vec<u8>>), cubefs, and cubesys — the correct foundation
the PDF spec describes. cube-store-raw/ and cube-fuse-proxy/ were
scaffolded as redundant duplicates on the wrong architecture; they
broke `cargo check` (103 errors) and had no tests.

Also: add 7 direct HashBackend trait-level tests (PDF spec's
HashMap<u32,Vec<u8>> sketch) + clean up the _assert_backend_assoc
workaround + dead HashMap import in record_codec, both now unneeded
since HashBackend is exercised by real tests.

Verification: cargo test -p cubestore = 15/15 pass; cargo check
--workspace green (only pre-existing cubetrace warnings).

Co-Authored-By: Hermes Agent
2026-08-20 16:40:53 -04:00
CUBELinux-2 11e5ed7fca fix(metatag-dirs): exact-leaf + descendant path prefix semantics
scan_by_path_prefix now matches BOTH an exact leaf (p == dir) and any
descendant (p.starts_with(dir + '/')), so  returns
 (the leaf itself) as well as  (a child).
Previously a bare dir normalized to 'dir/' and rejected exact leaves.

Also fixes the unit test to assert the corrected semantics (exact leaf
answers its own parent query).

Falls out of wiring the OS migration scripts: their companion index
records (c200/z090 GNS band) carry path=/cubelinux/os/... metatags, and
the daemon  reconstructs the whole migrated tree from them.
2026-08-13 18:41:22 -04:00
CUBELinux-2 94681bbdc0 feat(metatag-dirs): nested directory hierarchy via path metatag + flags
Implements the source PDF's prescribed model for directory structure:
'treat the CZYX cube as the only persistent store, with system calls that
operate on CZYX records and Null-space flags rather than paths and inodes.'
Nested dirs are RECONSTRUCTED from a per-record `path` metatag, so the
filesystem does not need recursive inode trees.

- cubecoords: add CubeHeader.path (Option<String>) + HAS_PATH flag bit (1<<8);
  refresh_flags() sets it. Cubestore TLV codec now serializes/deserializes the
  path as tag 13 (persists across checkpoint/reopen).
- cubestore: scan_by_path (exact), scan_by_path_prefix (dir listing by path
  prefix; bare '/etc' normalized to '/etc/'), plus query_by_path/_prefix
  returning decoded (coord,header,body).
- cubesys: `put` verb gains path=<p>;
  new `query-path <dir>` verb reconstructs a directory listing from metatags.
- ConcurrentStore: query_by_path(_prefix) delegate to inner CubeStore.

Verified live in VM: migrated /etc/passwd, /etc/network/interfaces, /usr/bin/ls
at unrelated coordinates; 'query-path /etc' returns the two /etc files,
'/usr/bin/ls' correctly excluded. cubestore 8/8 tests pass.

Refs: OS-in-CUBE migration, query-layer prototype.
2026-08-13 18:26:58 -04:00
CUBELinux-2 2ff1dff02b feat(query): PDF Package-2 associative log lookup via flags/metatags
Implements the source PDF's 'put(C,Z,Y,X,bytes,flags)' + 'scan_by_flag'
associative-storage API so records are discoverable by WHAT they are
(event type / doc_type metatag) rather than WHERE they live (coordinate
path) — the 'log lookup by query' the OS layers want.

cubestore:
  - CubeStore::scan_by_flag(u16) / scan_by_type(&str): predicate scans
    over decoded CubeHeader (not prefix scans).
  - CubeStore::query_by_flag / query_by_type: same, but also return the
    decoded (label, header, body) so a query tool can present results.
  - get_record's raw/un-enveloped fallback now refresh_flags() so
    synthesized headers carry SIZE_BYTES and are visible to flag queries.
  - regression tests: scan_by_flag_finds_typed_records,
    scan_by_type_event_lookup.

cubesys (daemon):
  - new 'put' verb: enveloped write with optional doc_type=/title=
    metatags (the PDF's put API). rawput remains for bulk/append payloads.
  - ConcurrentStore::query_doc_type was already the backing predicate the
    'query <doc_type>' verb uses; it now matches these typed records.

Proven end-to-end in the VM: 'cubec query klog' returns the kernel-log
index record (doc_type=klog) published by cube-os-klog.sh, and
'cubec query fn' returns the 8 pre-existing typed program records.
2026-08-13 18:12:13 -04:00
CUBELinux-2 fe64b869e4 feat(trace+ai): add cubetrace package and wire DBT/AI/CUBEsys command + module edits (green-lit WIP)
Brings in the cubetrace crate (PDF Package 4, §547): wraps a DBI engine via an
FFI seam and streams trace events (basic blocks, syscalls) into cubestore,
tagging each with CZYX coordinates and header flags.

- Cargo.toml: register cubetrace workspace member
- cubeai/src/lib.rs, cubedbt/src/lib.rs: DBT/AI rule + trace integration edits
- cubecode/src/{cb,cell}.rs: code-cell bytecode/module plumbing for traces
- cubesys/src/commands.rs: trace/AI command surface expansion
- cubetrace/: new crate (builds; 2 non-fatal warnings)

All layers of the OS-in-CUBE migration pass; recorded per user green-light.
2026-08-13 17:31:40 -04:00
CUBELinux-2 ff2914a105 fix(cubestore): surface raw/un-enveloped records in get_record instead of returning None
get_record() previously returned None for any payload lacking a TLV envelope
(4-byte header-len | header | body). Records written through the raw path
(put_raw / the daemon's 'rawput' verb, used by every OS-layer service) carry
no envelope, so callers such as cubefs getattr/read and the 'stat' verb mapped
that None to size 0 / empty file: real bytes became SILENTLY INVISIBLE through
the filesystem while rawget still returned them.

This broke the OS-in-CUBE migration (2026-08-13): every rawput OS record listed
as a 0-byte file in /cubefs, which also surface-invalidated the 'the OS boots
from cube' resume path.

Now a payload that is not a well-formed envelope is surfaced as a raw body under
a synthesized header whose size_bytes reports the true length. Enveloped records
still decode their real header (no behavior change on the record path). Adds two
regression tests: get_record_surfaces_raw_unenveloped_payloads and
get_record_still_prefers_the_real_envelope (cargo test -p cubestore: 5/5).
2026-08-13 17:29:50 -04:00
CUBELinux-2 3121459138 feat(dbt+ai): build out cubedbt and cubeai packages (Package 4, PDF §549/§551)
cubedbt: DBT runtime that reads CZYX-stored translation rules and patches
them into a code cache to execute mimicked behavior. TranslationRule (CZYX
Variant record in c220 band) maps an op-class to a replacement bytecode
fragment; CodeCache patches an original CodeCell under eligible rules and
writes the result as a Variant (preserving behavior descriptors), and
DbRuntime.mimic() fetches -> patches -> runs in the real Vm, proving
're-run or modify behavior without the original binary'.

cubeai: models over cube-stored traces/graphs to classify blocks
(Computation/Branch/CallTrampoline/IoSection/Sequence from the op-class
histogram + behavior descriptors) and suggest new code sequences as
cubedbt TranslationRules (persisted into the c220 rule band, discoverable
by DbRuntime). End-to-end: trace -> classify -> suggest -> mimic.

Both crates are dependency-free and operate on the real CubeStore/CodeCell/
Vm/Behavior types, so they are exercisable today on HashBackend and slot
into ConcurrentStore later without an API change. ./check quick green:
cubedbt 4 tests, cubeai 5 tests, full workspace green.
2026-08-13 16:23:00 -04:00
CUBELinux-2 0074f3112c fix(os): cover full spec behavior-descriptor set (PDF §524-525)
cb.rs previously implemented only 3 of the 6 spec descriptors (pure/io/hot).
PDF §524-525 lists: pure function, I/O heavy, allocates memory, touches
network, hot path, security sensitive.

- Behavior now stores descriptor bits directly as dedicated header-flag bits
  in the spare 8..=15 range: PURE=9, IO_HEAVY=10, ALLOCATES=11, NETWORK=13,
  HOT_PATH=14, SECURITY_SENSITIVE=15. Bit 12 (ENCRYPTED) left clear; mask
  0xEE00. refresh_flags() preserves 8..=15 so round-trip is safe.
- K= CLI tokens gain alloc/net/sec (prog + kernel verbs).
- Add unit test for all-six round-trip + explicit security-sensitive bit.

Proven green via ./check quick.
2026-08-13 16:00:34 -04:00
CUBELinux-2 1539ecd1bb feat(os): OS operator kernels in CUBE + thin native effector (Steps 1 & 2)
- cubecode/src/cb.rs: Behavior descriptor bitflag (PURE/IO_HEAVY/HOT_PATH)
  encoded into HeaderFlags bits 13..15; C_OS_KERNEL=210 / C_OS_EFFECT=211
  coordinate bands; round-trip unit test.
- fix: HEADER_FLAG_BEHAVIOR mask was 0x7000 (bits 12-14) which grabbed the
  ENCRYPTED bit (12) and dropped HOT_PATH (bit 15). Corrected to 0xE000.
- store_code_cell/put_code_cell/header_for_code gain optional descriptor arg;
  all 12 prior call sites pass None (total change).
- CLI: prog K=<flag> tokens, kernel verb (links+descriptors), tick verb lays
  down the OS kernel call-graph (cfg->decide->summarize->tick), native-apply
  verb = thin effector (runs kernel in VM, reads computed result, emits effect).
- integration test os_kernels_live_in_cube_with_call_graph_and_descriptors.

Proven green via ./check quick (fmt+clippy -D warnings+tests).
2026-08-13 15:57:10 -04:00
39 changed files with 5240 additions and 96 deletions
+5
View File
@@ -2,12 +2,17 @@
resolver = "2"
members = [
"cubecoords",
"cube-mvw",
"cubestore",
"cubefs",
"cubecode",
"cubecrypt",
"cubesys",
"cubedbt",
"cubeai",
"cubetrace",
"cube-bench",
"cubecli",
]
[workspace.package]
+50
View File
@@ -0,0 +1,50 @@
# RESUME POINT — HashBackend (HashMap<u32, Vec<u8>>) work, 2026-08-20 → 2026-08-21
## WHAT WAS DONE THIS SESSION (continuation)
### Prior session's 4 tasks — all closed and verified
1. clippy -D warnings in cubesys — 4 lints fixed
2. cube-bench executed — correctness-gated microbenchmarks
3. cubecli extracted — standalone crate
4. go live (./check gate) — passes
### THIS session: ran all 4 `./check` opt-in stages in logical order
1. **./check mount** — PASS (57 assertions, 0 failed)
- In-memory `--seed` mount: full regression suite
- Daemon-backed `--socket` mount: full regression suite
- Durability across daemon restart: verified (record survives kill+restart)
- Cross-user ACLs (root ↔ luulu): verified
2. **./check daemon** — PASS (3 tests, 0 failed)
- `cubefs_create_write_reaches_daemon`
- `daemon_backend_envelope_roundtrip`
- `daemon_backend_put_get_roundtrip`
- All 3 were `#[ignore]`d, now exercised against live cube-server
3. **./check stress** — PASS (~150s, 57075 pairs driven)
- Daemon alive throughout, ~380 prog+run pairs/s sustained
- 156 records in C=77 namespace (command dedup by coordinate)
- No crashes, no latency spikes
4. **cube-bench at scale 200k** — FAILED initially, then FIXED and PASS
- **Bug found**: bench's expected-value math assumed all records beyond c=0 land in c=1, but `coord_for` spreads across c=0,1,2,3 as `i` grows past 65536. At scale=200k, c=1 should have 65536 records (not 134464).
- **Fix**: compute expected counts per C bucket from the `coord_for` mapping (c=0: min(n, 65536); c=1: next 65536; c=2: next 65536; c=3: remainder). Added expected_c2 + expected_c3 to the total-coverage assertion.
- **Verified**: `cargo fmt --all -- --check` clean, `cargo clippy -p cube-bench -- -D warnings` clean, 100k + 200k both pass.
- **Numbers at 200k (release)**: put_raw 236ns/op, get_raw 109ns/op, scan_prefix 12.4ms for 65536 coords, ~4237k put/s, crypto ~1.4-2.6μs/seal, VM 276ns entry / 142ns leaf, session 1.4μs prog / 2.6μs run.
### Git state
- Working tree: `cube-bench/src/main.rs` modified (the fix above)
- Branch: `feat/os-kernel-in-cube`
- All prior session commits intact (7b0cf6f is HEAD)
## CURRENT STATE
- Branch: feat/os-kernel-in-cube
- cube-bench at 200k: fixed + verified (fmt + clippy + 100k + 200k all pass)
- ./check mount/daemon/stress: all green
- cubecli: standalone crate, functionally identical
## NEXT STEPS
1. Commit the cube-bench fix (27 insertions, 5 deletions)
2. README/docs sync — reflect cubecli as standalone crate (./check doesn't check docs)
3. Spec reading resumption from CUBELinux.txt (~offset 360+) — Package 2 use cases
+33
View File
@@ -0,0 +1,33 @@
# Resume Marker — 2026-08-20
# Status: kernel ready for compile, workhorse ready for Xfce switching, stopped before heavy build
## What was done this session
1. **Kernel build environment — READY** (stopped before heavy compile per user request)
- Kernel: vanilla v6.19.3 + ipu4 drivers (GRUB entry LOCALVERSION=-ipu4p)
- `./check` green, git clean
- Rust toolchain: stable 1.97.1 (default) + nightly 1.100.0 installed via rustup
- CONFIG_RUST=y set in .config, CONFIG_RUST_IS_AVAILABLE=y
- `scripts/target.json` has `rustc-abi: x86-softfloat` (rejected by stable, OK with nightly)
- Build fix identified: `RUSTC=/root/.cargo/bin/rustc RUSTLIB_SRC=... make ...` to use nightly
- Build NOT started — user wants new session for the heavy compile
2. **GUI session switching — READY** (workhorse has Xfce, GDM shows it)
- Xfce 4.20.2 + xfce4-goodies installed on workhorse
- `/usr/share/xsessions/xfce.desktop` present, `Exec=startxfce4`
- GDM (gdm3) active, shows xfce.desktop + xfce-wayland.desktop at login gear menu
- User can switch GNOME↔Xfce at GDM login by clicking gear icon → "Xfce Session"
- lightdm package removed (was stuck half-configured; cleaned up via dpkg purge)
- No lightdm; gdm3 is the DM; /etc/X11/default-display-manager = /usr/sbin/gdm3
## What remains (for next session / new session)
- **Kernel compile**: `make -j$(nproc) bzImage modules` with nightly rustc override
- Command pattern: `cd /home/CUBELinux/CUBELinux-2/src/CUBELinux-build && RUSTC=/root/.cargo/bin/rustc HOSTRUSTC=/root/.cargo/bin/rustc make -j$(nproc) bzImage modules`
- Need to confirm RUST_LIB_SRC path for nightly toolchain
- User explicitly asked NOT to auto-launch — new session should decide timing
- **GRUB**: new entry for -cube kernel; verify ipu4p entry still works
- **Post-boot verification**: cube fs mount, daemon startup, camera (ipu4) if desired
## Notes
- User restarted the session; this marker captures where we left off
- User directive: "get project to point of compiling new kernel, then stop so we can use new session for that"
- Heavy compile (make -j8) saturates all cores + memory — do NOT start without user go
+72
View File
@@ -0,0 +1,72 @@
# Resume Marker — 2026-08-20 (kernel build + GUI session switching)
## Context
User restarted the session. Two parallel tasks were in progress when interrupted:
1. Kernel Rust build config issue (kbuild + nightly rustc)
2. GUI session switching — install Xfce on workhorse so user can switch GNOME↔Xfce at login
---
## Task 1: Kernel Rust Build Config
**Status: unresolved — root cause found, fix not yet applied**
**What we know:**
- Kernel source: `/home/CUBELinux/CUBELinux-2/src/CUBELinux-build/` (vanilla v6.19.3 + ipu4 out-of-tree)
- `CONFIG_RUST=y` set in `.config`
- rustc 1.97.1 (stable) is default; rustc 1.100.0-nightly also installed via rustup (default host: x86_64-unknown-linux-gnu)
- `scripts/target.json` exists with `rustc-abi: x86-softfloat`, `llvm-target: x86_64-linux-gnu`
- `make rust/core.o` **fails**: `error loading target specification: custom targets are unstable and require -Zunstable-options`
- The rustc_library command in `rust/Makefile:559` does NOT pass `-Zunstable-options`
- `rust/Makefile` already uses `-Zunstable-options` in rustdoc paths (lines 136, 324, 333) but NOT in the core rustc_library invocation
- `KBUILD_RUSTFLAGS` (from `arch/x86/Makefile`) sets `--target=.../scripts/target.json` but no `-Zunstable-options`
- The `rustc_library` rule (`rust/Makefile:555-567`) builds the command as:
```
$(RUSTC) $(filter-out $(skip_flags),$(rust_flags)) $(rustc_target_flags) --emit=...
```
where `rust_flags` = `KBUILD_RUSTFLAGS` + `rust_common_flags` and `rustc_target_flags` = `core-flags` (which is `--edition=...` + cfgs). Neither carries `-Zunstable-options`.
**What needs to happen:**
- Option A: Patch `rust/Makefile` to add `-Zunstable-options` to the rustc_library command (line 559 area) — minimal change, mirrors what rustdoc already does
- Option B: Use nightly rustc as RUSTC and ensure it's invoked with `-Zunstable-options` — but kbuild doesn't add it automatically
- The real fix is Option A: add `-Zunstable-options` to the rustc invocation in the rustc_library rule, since the target.json is a custom target spec that requires it
- After that: `make rust/all` or `make rust/core.o` should succeed, then full kernel build
**Files involved:**
- `/home/CUBELinux/CUBELinux-2/src/CUBELinux-build/rust/Makefile` (line 555-567, rustc_library rule)
- `/home/CUBELinux/CUBELinux-2/src/CUBELinux-build/arch/x86/Makefile` (KBUILD_RUSTFLAGS)
- `/home/CUBELinux/CUBELinux-2/src/CUBELinux-build/scripts/target.json` (custom target spec, needs -Zunstable-options)
---
## Task 2: GUI Session Switching (Xfce on workhorse)
**Status: dpkg interrupted — lightdm stuck half-configured**
**What we know:**
- Workhorse runs GNOME (gdm3 + gnome-shell) currently
- User wants GNOME ↔ Xfce switching at the login screen (like Cinnamon is available)
- `apt-get install -y task-xfce-desktop` timed out (exit 124) and left dpkg in interrupted state
- `dpkg --configure -a` fails: lightdm is half-configured, depends on itself being configured first
- lightdm package status: `iF` (half-configured), `light-locker` and `task-xfce-desktop` also stuck
- Xfce components were likely downloaded but not configured due to the dpkg blockage
**What needs to happen:**
1. Fix dpkg: either `dpkg --remove --force-remove-reinstreq lightdm` then `dpkg --configure -a`, or manually remove the lock/status entries
2. Re-run `apt-get install -y task-xfce-desktop` (or just `xfce4` + `lightdm` if minimal)
3. Verify `xfce4-session` works: `xfce4-session --version`
4. Verify GDM shows Xfce session: check `/usr/share/xsessions/xfce.desktop` exists, `gdm3` is running
5. Test: log in via Xfce session
**Files/dirs involved:**
- `/usr/share/xsessions/` (GDM session definitions)
- `/usr/share/wayland-sessions/` (if Wayland Xfce session needed)
- `/etc/gdm3/` (GDM config)
---
## General Notes
- User explicitly does NOT want auto-launching heavy kernel compiles while they're using the machine
- Kernel build with `make -j$(nproc)` will saturate CPU — must ask before launching
- The workhorse is Debian-based, root access available
- rustup is installed at `/root/.rustup`, toolchains in `/root/.rustup/toolchains/`
+96
View File
@@ -0,0 +1,96 @@
# Resume Marker — 2026-08-21 (kernel Rust build: IRQ lifetime fixes + ready for compile)
## Context
The session after RESUME-20260820. Fixed the kernel Rust build errors and verified `make rust/core.o` compiles cleanly. Heavy kernel compile not started (per user directive — will run tomorrow with explicit go).
---
## What was done this session
### 1. Kernel Rust build: IRQ lifetime errors fixed
**File changed:** `rust/kernel/irq/request.rs`
**Root cause:** `rust/core.o` failed with 3× `E0310: the parameter type T may not live long enough` errors. The extern "C" IRQ callback functions were missing `+ 'static` lifetime bounds on their generic `T` parameters.
**Fixes applied (4 total):**
| Function | Line | Change |
|----------|------|--------|
| `handle_irq_callback` | 264 | `T: Handler``T: Handler + 'static` |
| `handle_threaded_irq_callback` | 483 | `T: ThreadedHandler``T: ThreadedHandler + 'static` |
| `thread_fn_callback` | 499 | `T: ThreadedHandler``T: ThreadedHandler + 'static` |
The `impl` blocks already had `T: Handler + 'static` and `T: ThreadedHandler + 'static` on lines 185, 198, 403, 416 — the three extern C callback functions and their callers were the missing ones.
**Verification:** `python3 /tmp/hermes-verify-kernel-rust-build.py` — 16/16 checks passed:
- Source file checks: all 3 fixed signatures present, no old patterns remain
- Build: `make rust/core.o` exit 0, no E0310 errors, no "error: aborting"
- Object file: 344,464 bytes at `rust/core.o`
- Only 2 pre-existing warnings (unused import `flags::*`, unused feature `used_with_arg`)
### 2. ./check gate — ALL CHECKS PASSED
```
./check → ALL CHECKS PASSED (run: ./check mount | bench | stress | daemon for those stages)
```
fmt clean, 223 tests, clippy -D clean. No regressions from the IRQ fix.
### 3. Build config (carried over from prior session, still valid)
- Nightly rustc 1.100.0 (`rustup default nightly`)
- `scripts/target.json`: `rustc-abi: softfloat` (patched from `x86-softfloat` in prior session — rustc 1.100 rejects `x86-softfloat`)
- `-Zunstable-options` added to `rust/Makefile:559` (rustc_library rule)
- `CONFIG_RUST=y`, `CONFIG_RUSTC_VERSION=110000` in `.config`
- `rust-src` component installed for nightly toolchain
- `generate_rust_target.rs` patched to emit `softfloat` (not `x86-softfloat`)
---
## What remains for tomorrow
### Kernel compile (heavy — ask before launching)
- Command: `cd /home/CUBELinux/CUBELinux-2/src/CUBELinux-build && make -j4 bzImage modules`
- Use `-j4` not `-j8` — 7.4 GB RAM, only 3.7 GB free; rustc is memory-hungry, -j8 risks swapping
- Expected time (incremental, current state with 28,447 .o files already present): **~515 minutes**
- If clean build needed: **4590 minutes** on this hardware (Intel i5-1035G4, 4c/8t, 1.1 GHz base, NVMe)
- After build: install modules, update GRUB, reboot into new kernel
### GRUB
- New kernel entry: `LOCALVERSION=-cube` (already in .config)
- Verify `LOCALVERSION=-ipu4p` entry still works (camera kernel, currently active)
### Post-boot verification
- Cube filesystem mount
- Daemon startup
- Camera (ipu4) if desired
---
## Xfce (separate track — not touched this session)
- dpkg state clean: lightdm, light-locker, task-xfce-desktop all removed/purged
- Xfce 4.20.2 intact: `xfce4-session`, `xfce4-panel`, `xfce4-goodies` all `ii`
- Session files: `/usr/share/xsessions/xfce.desktop`, `/usr/share/wayland-sessions/xfce-wayland.desktop`
- GDM active, default DM `/usr/sbin/gdm3`
- Issue reported (external monitor oversized, both monitors showing in Xfce vs GNOME's correct single monitor): root cause is `xfsettingsd` not running during Xfce session — Xfce inherits GNOME's RandR state without managing it. The fix is ensuring `xfsettingsd` starts (autostart entry at `/etc/xdg/autostart/xfsettingsd.desktop` exists, not disabled). No action taken — user will test after reboot.
---
## Files
- Workspace: `/home/CUBELinux/CUBELinux-2/`
- Changed file: `src/CUBELinux-build/rust/kernel/irq/request.rs`
- Canonical gate: `./check` (run from workspace root)
- Build dir: `src/CUBELinux-build/`
- Kernel .config: `src/CUBELinux-build/.config` (CONFIG_RUST=y, CONFIG_LOCALVERSION="-cube")
- Rust target.json: `src/CUBELinux-build/scripts/target.json` (rustc-abi: softfloat)
- System daemon socket: `/run/cube/cube.sock`
- luulu session daemon socket: `/run/user/1000/cube/cube.sock`
- Verification script used: `/tmp/hermes-verify-kernel-rust-build.py` (cleaned up after run)
## Git state
- Branch `feat/os-kernel-in-cube`, working tree clean (verify with `git status` before committing)
- Prior commits on HEAD: `c090233` (bench fix), `ae1a62a` (docs), `3f007a1` (bench saturating_sub) + 3 older commits
---
## Reminder
- User explicitly does NOT want auto-launching heavy kernel compiles while they're using the machine
- `make -j8` saturates all 8 cores + memory on this Surface Pro 7 (i5-1035G4, 7.4 GB RAM)
- Must get user go before launching the kernel compile tomorrow
+70
View File
@@ -0,0 +1,70 @@
# RESUME POINT — 2026-08-21: all ./check stages green, cube-bench fixed, docs synced
## WHERE WE ARE (verified on disk this session)
### Git state
Branch `feat/os-kernel-in-cube`, clean working tree. Commits this session:
- `c090233` bench(cube-bench): fix scan_prefix expected-value math for scale > 65536
- `ae1a62a` docs: reflect cubecli as standalone crate in README + integration docs
- `3f007a1` bench(cube-bench): use saturating_sub for expected-value arithmetic
Plus 3 prior-session commits still on HEAD (7b0cf6f, ce8ac78, fdfcd2c).
### ./check gate
`./check` → ALL CHECKS PASSED (fmt clean, 223 tests, clippy -D clean).
### ./check opt-in stages (all run this session, all green)
- `./check mount` — 57/57 FUSE assertions (in-memory --seed + daemon-backed --socket + durability-across-restart + cross-user ACLs root↔luulu)
- `./check daemon` — 3/3 live daemon tests (were #[ignore]d, now exercised)
- `./check stress` — ~150s, 57075 prog+run pairs, daemon alive throughout, ~380 pairs/s
- `cube-bench --release 200000` — FAILED once (real bug in bench's expected-value math), then FIXED and PASS at 1k/10k/50k/100k/200k
### Bug found + fixed this session
cube-bench assumed all records beyond c=0 land in c=1. `coord_for` spreads across c=0,1,2,3 as i grows past 65536, so at 200k the expected c=1 count was wrong (134464 vs actual 65536). Fix: compute per-bucket expected counts from the coord_for mapping; extend total-coverage assertion to include c=2 and c=3. Verified at all scales with bench's own correctness assertions. clippy -D warnings clean on cube-bench after replacing manual `-` with `saturating_sub`.
### Docs synced
- `STARTUP-README.md`: cube CLI now references `cubecli/src/main.rs` (was `cubesys/src/bin/cube.rs`)
- `cubesys/docs/integration.md`: Binaries section notes cube is now a standalone crate (`cubecli/`)
## ARCHITECTURAL QUESTION: where should CUBE live when it becomes the OS?
Two-layer model (already partially deployed — this is the right shape):
**Layer 1 — Kernel (the "utilize CUBE" part).** CUBE core crates — `cubecoords`, `cubestore` (HashMap + FileBackedStore + WAL/checkpoint), `cubecode` (bytecode + VM), `cubecrypt` (seal/open + keyinit) — get compiled into the kernel or as a kernel module. Kernel gets direct syscalls: `open_by_czyz()`, `write_czyz()`, `seal_czyz()`, `run_czyz()`. The cube becomes the backing store the VFS and process accounting talk to directly — not a userspace daemon over a socket. This is the spec's Phase 3 ("OS services talk directly to the cube store; syscalls like open by CZYX + flags").
**Layer 2 — Userspace daemon (`cube-server`).** Stays as the durable anchor: WAL, checkpoint, socket for external/remote clients, the "resume pointer" writer. Survives when the kernel module isn't loaded; what remote clients talk to. Currently at `/run/cube/cube.sock` (system) + `/run/user/1000/cube/` (luulu session). Unify under `/usr/lib/cube/` when it's the OS — daemon binary, libs, socket path, store dir all there.
**Filesystem placement when it's the OS:**
- Kernel module/built-in: lives in the kernel source tree (copy of core crates, adapted for kernel constraints — `no_std`, no alloc assumptions that conflict with kernel alloc) OR as out-of-tree module alongside your existing Surface/ipu4 kernel.
- Userspace daemon + libs: `/usr/lib/cube/`. Socket at `/run/cube/cube.sock`.
**Honest gaps before this is real (both still open, from STARTUP-README §5):**
- (a) **cubefs directory model** — only `c<C>/z<Z>/y<Y>/x<X>` with x as fixed 3-digit leaf. No nested dirs, no arbitrary POSIX filenames, no rename/whiteout. Overlayfs rejects cubefs with EINVAL. A real OS tree can't be bound until this is done. Level-B crate feature.
- (b) **root fs / PID 1** — making the cube the literal rootfs is kernel work: custom initramfs + pivot_root, daemon or kernel module as init. Heavy, image-bake, off-peak. Current deployment: ext4 root + CZYX-call OS state = Phase 2 target.
**On "compile the new kernel to utilize CUBE":** your current kernel is vanilla v6.19.3 + ipu4 drivers (GRUB entry `LOCALVERSION=-ipu4p`); camera work is paused. Adding CUBE means either a kernel module against that kernel, or a new kernel build with CUBE integrated. That's a resource-heavy compile (saturates all 8 cores + memory). You've asked me not to auto-launch that while you're using the machine — I'll wait for your go.
## NEXT SESSION: put the OS in CUBE + compile new kernel to utilize CUBE
### What "put the OS in CUBE" means concretely (Phase 2 deepening)
The OS already writes its state (manifest, identity, klog, snapshots) as CZYX calls to `cube-server` over the socket — that's done and durable (verified across daemon restart). The next deepening is moving the cube from "userspace daemon the OS talks to over a socket" to "the kernel's own backing store" — i.e. the kernel module layer above.
### Concrete first step (before any kernel compile)
1. **Pick the kernel target.** Current: vanilla v6.19.3 + ipu4 (LOCALVERSION=-ipu4p). Option A: add CUBE as an out-of-tree module to that kernel. Option B: new kernel build with CUBE integrated. Which, and do you want to keep the ipu4 camera kernel as the base, or start fresh?
2. **Decide CUBE's kernel boundary.** What goes into the kernel vs stays in userspace?
- Definitely kernel: `cubecoords` (Czyx type, flags), `cubestore` core (the store abstraction + at least the in-memory backend; FileBackedStore + WAL/checkpoint is the durable path), coordinate→syscall mapping.
- Probably userspace ( daemon ): the socket server, remote client access, the "resume pointer" writer, snapshot management — i.e. the control plane, while the kernel is the data plane.
- `cubecode` (bytecode VM) and `cubecrypt` (seal/open transforms) — these are the debatable ones. VM in kernel = ability to run cube programs from kernel space (syscall-level). Crypto in kernel = sealed records openable by the kernel without a userspace daemon. Both are plausible; both add kernel surface area. Your call.
3. **cubefs directory model gap (a) above** — decide whether to close it before or after the kernel move. If the kernel uses the cube as backing store but cubefs can't represent a real OS tree (no nested dirs, no rename), then the OS-state-writing path (CZYX calls, not FUSE) still works, but a full rootfs bind is still blocked. Closing (a) is a crate feature, independent of the kernel move.
### Concrete second step (kernel compile — heavy, ask before launching)
4. **Build the kernel/module.** Depends on step 1-3. If out-of-tree module: copy the core crates into a kernel module skeleton, adapt for kernel constraints, build against the existing kernel headers. If integrated kernel: new kernel source with CUBE subsystem, config, compile. Either way: `make -j8` saturates all cores + memory — you've asked me not to auto-launch that while you're using the machine.
## FILES
- Workspace: `/home/CUBELinux/CUBELinux-2/`
- Canonical gate: `./check` (run from workspace root)
- Resume procedure: read this file + STARTUP-README.md + REPORT-INDEX.md, then `python3` against `/root/.hermes/state.db` for thread review.
- Cube-bench binary (release): `target/release/cube-bench`
- Current kernel GRUB entry: `LOCALVERSION=-ipu4p` (vanilla v6.19.3 + ipu4 drivers)
- System daemon socket: `/run/cube/cube.sock`
- luulu session daemon socket: `/run/user/1000/cube/cube.sock`
+71
View File
@@ -0,0 +1,71 @@
# RESUME — Cube as OS substrate (CZYX-call model)
Date: 2026-08-13 (session after RESUME-cubefs-daemon; Level A of item 3)
## Decision (user-directed)
"Everything writes as a CZYX call to the daemon — the cube-backed storage should
be coordinate calls, not writes to /cubefs. Why are we writing to cubefs?"
This REVERSES the earlier FUSE-path substrate design. Confirmed against
CUBELinux.pdf:
- Phase 1: user-space Cube OS on Linux, using existing kernels + filesystems.
- Phase 2: cubefs FUSE maps POSIX→CZYX for *testing semantics under real
workloads* — explicitly a probe, not the substrate itself.
- Phase 3: fork/extend kernel so VFS/accounting/LSM talk directly to the cube
store; new syscalls expose cube-native ops ("open by CZYX + flags").
- Package 3 (cubefs): "optional FUSE filesystem view so cube records appear as
files/directories for existing tools." → FUSE is OPTIONAL, not the substrate.
So: the coordinate store IS the persistence. POSIX/FUSE is a convenience view.
## What changed
1. `cube-os-state.sh` / `cube-os-snapshot.sh` / `cube-os-klog.sh` rewritten to
write via `cubec --socket /run/cube/cube.sock rawput/get` (CZYX calls), not
`/cubefs/...`. hex encoding uses python3 (xxd is absent in the VM image).
Coordinate layout (C=200):
c200/z001/y001/x001 boot manifest (overwrite each boot)
c200/z002/y001/x001 kernel boot line history (append)
c200/z003/y001/x001 machine/identity record
c200/z004/y001/x001 kernel log stream (klog appends lines)
c200/z010/y001/xNNN operational snapshot body (rolling counter)
c200/z010/y002/x001 rolling snapshot counter (persisted in cube, hex)
2. The three `cube-os-*` systemd units had `cubefs.service` REMOVED from
Requires/After — they now depend ONLY on `cube-server.service`. A FUSE mount
failure can no longer wedge OS bring-up.
3. `cubefs.service` hardened (ExecStartPre unmounts any stale mountpoint; BindsTo
cube-server) so it survives a daemon restart instead of crash-looping on
"Transport endpoint is not connected". This was the bug that broke the live
VM at the start of this session.
4. build_vm.sh updated to match (inline script bodies + unit edges). A fresh
bake now produces the CZYX-call substrate. NOT re-baked this session
(heavy ~24G, off-peak per user policy).
## Verified live (VM localhost:2222)
- Boot manifest reads: "CUBELINUX-VM boot manifest v3 / backing-store:
cube-server @ /run/cube/cube.sock (CZYX coordinate calls, no FUSE)".
- Snapshot #001 contains real OS state (running units, network, resources,
journal) — not a demo.
- Manifest + snapshot survive `systemctl restart cube-server` (durable across
daemon restart).
- `cube-os-state` + `cube-os-snapshot.timer` active; manual snapshot writes.
## Known gaps (honest)
- cubefs directory model: only `c<C>/z<Z>/y<Y>/x<X>` (x = fixed 3-digit leaf).
Arbitrary POSIX filenames (syslog, wtmp) and nested dirs are NOT addressable.
overlayfs on cubefs fails (EINVAL — missing RENAME_WHITEOUT/opaque-dir). So a
*real OS tree* on the cube is blocked until cubefs grows nested-dir support
(Level-B crate work).
- root fs / PID 1 (cube daemon as init, pivot_root into cubefs) = Phase 3 kernel
work, image-bake, off-peak only.
## Items left (from earlier list)
1. / 2. → done in prior sessions.
3. Cube as OS substrate: Level A DONE (CZYX-call OS state, this session).
Levels B (cube-backed block device) / C (rootfs+PID1) pending, not started.
4. society workspace: not started; git dubious ownership on
/home/cubelinux/society (needs safe.directory).
## Key files
- /root/build_vm.sh (authoritative bake template; all unit + script bodies live here)
- /home/CUBELinux/CUBELinux-2/STARTUP-README.md (updated §4 + provisioning note)
- VM guest: /opt/cube/bin/cube-os-*.sh, /etc/systemd/system/cube-os-*.{service,timer}
+46 -30
View File
@@ -47,7 +47,7 @@ is real. Each must be reproduced live, not asserted.
3. **Phase 3 syscall surface** — "open by CZYX + flags": a record addressable
directly by coordinate, not path. **SHIPPED 2026-08-13**:
(a) `cube open <path> <K.Z.Y.X> <tf>` and `cube seal ...` are first-class CLI
commands in `cubesys/src/bin/cube.rs` (reachable in-process + over the durable
commands in `cubecli/src/main.rs` (reachable in-process + over the durable
daemon). Verified: dispatch reaches the crypto/run layer.
(b) **`.czyx.C.Z.Y.X` FUSE magic-prefix (2026-08-13)** — `cubefs/src/path.rs`
`parse_dot_czyx` + `cubefs/src/fuse.rs::lookup` intercept: a path component
@@ -58,19 +58,23 @@ is real. Each must be reproduced live, not asserted.
3358720263; `cat .czyx.200.50.1.7` returns the record content. Unit tests in
`path.rs` cover full/partial/rejected forms. **STATUS: DONE.**
4. **Boot substrate** — the VM brings up CUBELinux as a storage layer the rest of
the OS reads/writes through. **DONE (systemd-managed, 2026-08-13)**: units
`cube-os-state.service` (oneshot, writes boot manifest + machine identity +
boot history to `/cubefs/c200/z001|z002|z003`) and `cube-os-klog.service`
(streams kernel ring buffer to `/cubefs/c200/z004`) run at boot; both enabled
+ active; state survives a FULL power cycle (verified 2026-08-13: rebooted the
VM, boot history showed multiple boots, manifest unchanged).
**ALSO (2026-08-13): real OS operational data** — `cube-os-snapshot.service` +
`cube-os-snapshot.timer` (every 5 min + 30s after boot) write genuine OS state
(running units, network, resources, journal) into `/cubefs/c200/z010/y001/xNNN`
(rolling 3-digit counter, x-axis is u16→3-digit per cubefs path model).
Verified: snapshot survives daemon restart; the OS wrote a NEW snapshot on a
fresh boot after a full power cycle; inode == packed CZYX. **STATUS: DONE
(auxiliary layer; not yet the root fs).**
the OS reads/writes through. **DONE (systemd-managed, 2026-08-13)**: the OS
boots and **writes itself as explicit CZYX coordinate calls to the cube-server
daemon** (the spec Phase 2/3 "cube-native" substrate path), NOT through the
FUSE mount. Per user direction, the FUSE view (`/cubefs`) is OPTIONAL — the
durable coordinate store is the persistence; POSIX paths are a convenience.
- `cube-os-state.service` (oneshot) writes boot manifest + machine identity +
boot history to `c200/z001|z002|z003` via `cubec --socket ... rawput`.
- `cube-os-klog.service` streams the kernel ring buffer to `c200/z004` (CZYX).
- `cube-os-snapshot.service` + `cube-os-snapshot.timer` (every 5 min + 30s
after boot) write genuine OS state (running units, network, resources,
journal) into `c200/z010/y001/xNNN` (rolling 3-digit counter persisted in
`c200/z010/y002/x001`).
All three units depend ONLY on `cube-server.service` (the daemon socket), not
on `cubefs.service`, so a FUSE mount failure can never wedge OS bring-up.
Verified: manifest reads "backing-store: cube-server @ /run/cube/cube.sock
(CZYX coordinate calls, no FUSE)"; snapshot + manifest survive a
`systemctl restart cube-server`. **STATUS: DONE (auxiliary layer; not yet the root fs).**
---
@@ -153,24 +157,36 @@ missing.
random material on each boot → every sealed record became unopenable.
Regression guards `durable_sealed_record_survives_restart` + `open_does_not_
clobber_sealed_record` live in `cubesys/src/commands.rs`; `./check` is green.
- Boot substrate (next deepening): make the cube the OS's *default* storage for a
real tree — e.g. have a service write `/etc` or `/var/log` operational files
through the cube by default. Currently the cube holds OS *state* (manifest/
identity/klog/snapshots) but the root fs is still ext4. The magic-prefix
"open by CZYX" FUSE passthrough is the natural next step (a path like
`/cubefs/.czyx/200.1.1.1` resolves directly to the coordinate), and a
loop/overlay over a cube-backed file would let the OS treat the cube as a real
block device.
- Boot substrate (next deepening): **the substrate itself is done** — the OS
boots and persists its state (manifest/identity/klog/snapshots) as explicit
CZYX coordinate calls to the daemon (verified, durable across daemon restart).
Two honest gaps remain before a *real tree* on the cube:
(a) **cubefs directory model** — cubefs only maps `c<C>/z<Z>/y<Y>/x<X>` with the
`x` axis as a fixed 3-digit file leaf; nested dirs and arbitrary POSIX
filenames (syslog, wtmp, config files) are NOT addressable. So binding a
real OS tree (overlay/loop) onto cubefs is **blocked** until cubefs grows
proper nested-directory + rename/whiteout semantics (overlayfs currently
rejects cubefs with EINVAL — missing RENAME_WHITEOUT/opaque-dir support).
This is a Level-B crate feature, not a systemd change.
(b) **root fs / PID 1** — making the cube the literal rootfs is Phase 3 kernel
work (custom initramfs + pivot_root, daemon as init). Heavy, image-bake,
off-peak only. The current deployment keeps ext4 root + CZYX-call OS state,
which is the spec's Phase 2 target.
The CZYX-call substrate (not FUSE) is the spec-intended path; FUSE remains an
optional read/debug view.
- IMAGE PROVISIONING: `build_vm.sh` (host, /root/build_vm.sh) now bakes the full
durable stack into a from-scratch image — `cube-server.service` (daemon),
`cubefs.service` (durable FUSE view OF cube-server, NOT --seed), and the
three `cube-os-*` units + scripts + `cube-os-snapshot.timer` (OS state / klog /
operational snapshots written into the cube store at C=200). All enabled in the
chroot. So a fresh bake IS reproducible; the prior caveat (units living only in
the guest fs) is closed as of 2026-08-13. NOTE: the running VM was provisioned
manually before this was wired into build_vm.sh; re-running `build_vm.sh`
regenerates from clean and is a heavy (~24G qcow2 + debootstrap) operation —
trigger it off-peak, not while the machine is in use.
`cubefs.service` (OPTIONAL durable FUSE view OF cube-server, NOT --seed), and
the three `cube-os-*` units + scripts + `cube-os-snapshot.timer`. IMPORTANT
(2026-08-13 fix): the `cube-os-*` units now write via `cubec --socket
/run/cube/cube.sock rawput/get` (CZYX calls) and depend ONLY on
`cube-server.service` — `cubefs.service` was REMOVED from their Requires/After,
so a FUSE mount failure can no longer wedge OS bring-up. `cubefs.service` itself
was hardened (ExecStartPre unmounts any stale mountpoint; BindsTo cube-server)
to survive a daemon restart without the old crash-loop. All enabled in the
chroot. So a fresh bake IS reproducible; re-running `build_vm.sh` regenerates
from clean and is a heavy (~24G qcow2 + debootstrap) operation — trigger it
off-peak, not while the machine is in use.
- `cube-resume-pointer.service` is REAL (wired 2026-08-13): a oneshot that writes
the OS's "where to look to continue" into the cube at `c200/z011/y001/x001`
(last snapshot index + resume coordinate). Was a dead stub before (unit pointed
+27 -5
View File
@@ -116,19 +116,41 @@ fn main() {
1,
) / scale_n as f64;
// scan_prefix correctness: c takes values 0 or 1 only across [0,scale_n)
// exact expected count for c=0 and c=1 from the coord_for mapping
// scan_prefix correctness: coord_for spreads over [0,4G) so c can be
// 0..3 at scale_n=200k. Compute exact expected counts from the mapping.
let expected_c0 = if scale_n <= 0x10000 {
scale_n as usize
} else {
0x10000
0x10000_usize.min(scale_n as usize)
};
let c1_start = 0x10000_usize;
let c2_start = 0x20000_usize;
let c3_start = 0x30000_usize;
let scale_usize = scale_n as usize;
let expected_c1 = if scale_usize <= c1_start {
0
} else {
c2_start.min(scale_usize).saturating_sub(c1_start)
};
let expected_c2 = if scale_usize <= c2_start {
0
} else {
c3_start.min(scale_usize).saturating_sub(c2_start)
};
let expected_c3 = if scale_usize <= c3_start {
0
} else {
scale_usize.saturating_sub(c3_start)
};
let expected_c1 = scale_n as usize - expected_c0;
let got_c0 = store.scan_prefix(0, None, None).len();
let got_c1 = store.scan_prefix(1, None, None).len();
assert_eq!(got_c0, expected_c0, "scan_prefix c=0 wrong");
assert_eq!(got_c1, expected_c1, "scan_prefix c=1 wrong");
assert_eq!(got_c0 + got_c1, scale_n as usize, "prefix covers all");
assert_eq!(
got_c0 + got_c1 + expected_c2 + expected_c3,
scale_n as usize,
"prefix covers all"
);
let scan_ns = time_ns(
|| {
+10
View File
@@ -0,0 +1,10 @@
[package]
name = "cube-mvw"
version = "0.1.0"
edition = "2021"
[dependencies]
cubecoords = { path = "../cubecoords" }
cubestore = { path = "../cubestore" }
[[bin]]
name = "cube-mvw"
path = "src/main.rs"
+172
View File
@@ -0,0 +1,172 @@
//! A real CUBELinux DB backend: a sharded, multi-writer store with MVCC
//! (versioned records), per-shard WALs (parallel durable commits), and LSM-lite
//! compaction (memtable + WAL delta + compacted checkpoint base).
//!
//! `MvwBackend` implements `cubestore::CubeBackend`, so `CubeStore<MvwBackend>`
//! is a durable, concurrent, multi-writer CUBELinux database store.
use cubecoords::Czyx;
use cubestore::CubeBackend;
use std::collections::HashMap;
use std::fs::{File, OpenOptions};
use std::io::{BufWriter, Read, Write};
use std::path::{Path, PathBuf};
use std::sync::atomic::{AtomicU64, Ordering};
use std::sync::{Mutex, RwLock};
#[derive(Clone, Debug)]
struct Version { commit_ts: u64, value: Vec<u8>, deleted: bool }
struct Shard {
map: RwLock<HashMap<u32, Vec<Version>>>, // memtable, keyed by packed Czyx
wal: Mutex<BufWriter<File>>,
wal_path: PathBuf,
ckpt_path: PathBuf,
}
pub struct MvwBackend {
shards: Vec<Shard>,
next_ts: AtomicU64,
}
fn pack(k: &Czyx) -> u32 { k.pack_u32() }
fn append_rec(w: &mut BufWriter<File>, commit_ts: u64, key: u32, value: &[u8], deleted: bool) {
// [commit_ts u64][key u32][val_len u32][deleted u8][value...]
let mut buf = Vec::with_capacity(25 + value.len());
buf.extend_from_slice(&commit_ts.to_le_bytes());
buf.extend_from_slice(&key.to_le_bytes());
buf.extend_from_slice(&(value.len() as u32).to_le_bytes());
buf.push(deleted as u8);
buf.extend_from_slice(value);
buf.push(b'\n');
w.write_all(&buf).unwrap();
w.flush().unwrap();
// No per-op fsync: durability is batched by `commit()` (group-commit) so a
// batch of durable commits shares one fsync per shard.
}
impl MvwBackend {
pub fn open(nshards: usize, base: &Path) -> std::io::Result<Self> {
let mut shards = Vec::with_capacity(nshards);
for i in 0..nshards {
let wal_path = PathBuf::from(format!("{}-s{}.log", base.display(), i));
let ckpt_path = PathBuf::from(format!("{}-s{}.ckpt", base.display(), i));
let wal = OpenOptions::new().create(true).append(true).open(&wal_path)?;
shards.push(Shard { map: RwLock::new(HashMap::new()), wal: Mutex::new(BufWriter::new(wal)), wal_path, ckpt_path });
}
let b = MvwBackend { shards, next_ts: AtomicU64::new(1) };
b.recover()?;
Ok(b)
}
fn shard(&self, k: &Czyx) -> &Shard { &self.shards[(pack(k) as usize) % self.shards.len()] }
fn decode_line(rec: &[u8]) -> Option<(u64, u32, Vec<u8>, bool)> {
if rec.len() < 21 { return None; }
let commit_ts = u64::from_le_bytes(rec[0..8].try_into().ok()?);
let key = u32::from_le_bytes(rec[8..12].try_into().ok()?);
let vlen = u32::from_le_bytes(rec[12..16].try_into().ok()?) as usize;
let deleted = rec[16] != 0;
let value = rec[17..17 + vlen.min(rec.len().saturating_sub(17))].to_vec();
Some((commit_ts, key, value, deleted))
}
fn load(&self, path: &Path, live: &mut HashMap<u32, Vec<Version>>) -> std::io::Result<()> {
if !path.exists() { return Ok(()); }
let mut f = File::open(path)?; let mut data = Vec::new(); f.read_to_end(&mut data)?;
for rec in data.split(|&b| b == b'\n') {
if let Some((c, k, v, del)) = Self::decode_line(rec) { live.entry(k).or_default().push(Version { commit_ts: c, value: v, deleted: del }); }
}
Ok(())
}
pub fn snapshot_ts(&self) -> u64 { self.next_ts.load(Ordering::SeqCst).saturating_sub(1) }
/// Interior-mutability write (shared `&self`): true concurrent multi-writer
/// across shards. Durable in this shard's WAL; MVCC versioned.
/// Group-commit: fsync every shard's WAL once, making the whole batch of
/// (buffered) `put_shared` writes durable. One fsync per shard for many ops.
pub fn commit(&self) -> std::io::Result<()> {
for sh in &self.shards {
let w = sh.wal.lock().unwrap();
w.get_ref().sync_data().unwrap();
}
Ok(())
}
pub fn put_shared(&self, key: Czyx, value: Vec<u8>) {
let commit_ts = self.next_ts.fetch_add(1, Ordering::SeqCst);
let sh = self.shard(&key);
{ let mut w = sh.wal.lock().unwrap(); append_rec(&mut w, commit_ts, pack(&key), &value, false); }
sh.map.write().unwrap().entry(pack(&key)).or_default().push(Version { commit_ts, value, deleted: false });
}
pub fn recover(&self) -> std::io::Result<()> {
let mut max = 0u64;
for sh in &self.shards {
let mut m = HashMap::new();
self.load(&sh.ckpt_path, &mut m)?; self.load(&sh.wal_path, &mut m)?;
for vers in m.values() { for v in vers { max = max.max(v.commit_ts + 1); } }
let mut live = sh.map.write().unwrap(); live.extend(m);
}
if max > 0 { self.next_ts.store(max, Ordering::SeqCst); }
Ok(())
}
/// LSM-lite compaction: fold each shard's latest non-deleted version into its
/// checkpoint base, then truncate the shard WAL.
pub fn compact(&self) -> std::io::Result<()> {
for sh in &self.shards {
let m = sh.map.read().unwrap();
let mut out = Vec::new();
for (k, vers) in m.iter() {
if let Some(latest) = vers.last() {
if !latest.deleted {
let mut buf = Vec::with_capacity(21 + latest.value.len());
buf.extend_from_slice(&latest.commit_ts.to_le_bytes());
buf.extend_from_slice(&k.to_le_bytes());
buf.extend_from_slice(&(latest.value.len() as u32).to_le_bytes());
buf.push(0u8);
buf.extend_from_slice(&latest.value);
buf.push(b'\n');
out.extend_from_slice(&buf);
}
}
}
let mut ck = OpenOptions::new().create(true).write(true).truncate(true).open(&sh.ckpt_path)?;
ck.write_all(&out)?; ck.sync_all()?;
let mut w = sh.wal.lock().unwrap();
w.get_ref().set_len(0)?; w.get_ref().sync_all()?; w.flush()?;
}
Ok(())
}
}
impl CubeBackend for MvwBackend {
fn put(&mut self, key: Czyx, value: Vec<u8>) {
self.put_shared(key, value);
let _ = self.commit();
}
fn put_checked(&mut self, key: Czyx, value: Vec<u8>) -> Result<(), String> {
self.put(key, value); Ok(())
}
fn get(&self, key: &Czyx) -> Option<Vec<u8>> {
let sh = self.shard(key);
let m = sh.map.read().unwrap();
// latest committed non-deleted version
m.get(&pack(key)).and_then(|v| v.iter().rev().find(|ver| !ver.deleted).map(|ver| ver.value.clone()))
}
fn delete(&mut self, key: &Czyx) {
let commit_ts = self.next_ts.fetch_add(1, Ordering::SeqCst);
let sh = self.shard(key);
{ let mut w = sh.wal.lock().unwrap(); append_rec(&mut w, commit_ts, pack(key), &[], true); }
let mut m = sh.map.write().unwrap();
m.entry(pack(key)).or_default().push(Version { commit_ts, value: Vec::new(), deleted: true });
}
fn keys(&self) -> Vec<Czyx> {
let mut out = Vec::new();
for sh in &self.shards {
let m = sh.map.read().unwrap();
for (k, vers) in m.iter() {
if let Some(c) = vers.last() { if !c.deleted { out.push(Czyx::unpack_u32(*k)); } }
}
}
out
}
}
+35
View File
@@ -0,0 +1,35 @@
use cube_mvw::MvwBackend;
use cubecoords::Czyx;
use std::path::Path;
use std::time::Instant;
use std::thread;
fn bench(mode: &str) -> f64 {
let b = Path::new("/tmp/mvw-gc");
for i in 0..16 { std::fs::remove_file(format!("{}-s{}.log", b.display(), i)).ok(); std::fs::remove_file(format!("{}-s{}.ckpt", b.display(), i)).ok(); }
let back = MvwBackend::open(16, b).unwrap();
let threads = 8usize; let per = 4000u64; let n = (threads as f64) * per as f64;
let t = Instant::now();
thread::scope(|sc| {
for th in 0..threads {
let back = &back;
sc.spawn(move || for i in 0..per {
let k = Czyx::unpack_u32(((th as u64 * per + i) as u32) << 8 | 1);
back.put_shared(k, vec![i as u8; 16]);
});
}
});
if mode == "batch" { back.commit().unwrap(); }
let s = t.elapsed().as_secs_f64();
for i in 0..16 { std::fs::remove_file(format!("{}-s{}.log", b.display(), i)).ok(); std::fs::remove_file(format!("{}-s{}.ckpt", b.display(), i)).ok(); }
n / s / 1e3
}
fn main() {
let per_op = bench("per_op");
let batch = bench("batch");
println!("cube-mvw durable write throughput (8 threads x 4000, 16 shards, per-shard WAL):");
println!(" per-op fsync : {:>8.1} k puts/s", per_op);
println!(" group-commit : {:>8.1} k puts/s ({:.1}x)", batch, batch/per_op);
println!("\n=> group-commit batches one fsync per shard for many ops => durable writes scale with the");
println!(" batch (no tables layer — pure storage engine; CUBE's metatag model is the schema).");
}
+64
View File
@@ -0,0 +1,64 @@
use cube_mvw::MvwBackend;
use cubecoords::{CubeHeader, Czyx};
use cubestore::{CubeBackend, CubeStore};
use std::path::{Path, PathBuf};
use std::fs;
use std::thread;
use std::sync::Arc;
fn base(name: &str) -> PathBuf { std::env::temp_dir().join(name) }
fn cleanup(b: &Path) { for i in 0..16 { let _=fs::remove_file(format!("{}-s{}.log",b.display(),i)); let _=fs::remove_file(format!("{}-s{}.ckpt",b.display(),i)); } }
#[test]
fn backend_durable_mvcc_and_concurrent() {
let b = base("mvw-backend"); cleanup(&b);
{
let back = MvwBackend::open(16, &b).unwrap();
back.put_shared(Czyx::new(1,1,1,1), b"hello".to_vec());
// MVCC: a later write is the latest visible version
back.put_shared(Czyx::new(1,1,1,1), b"world".to_vec());
assert_eq!(back.get(&Czyx::new(1,1,1,1)), Some(b"world".to_vec()));
}
// durability: drop (no graceful shard flush), reopen from per-shard WALs
let back = MvwBackend::open(16, &b).unwrap();
let latest = back.get(&Czyx::new(1,1,1,1)).unwrap();
assert!(latest == b"world".to_vec(), "WAL recovery must restore the latest MVCC version");
// keys enumeration
back.put_shared(Czyx::new(2,2,2,2), b"a".to_vec());
assert!(back.keys().contains(&Czyx::new(2,2,2,2)));
// true concurrent multi-writer (shared &self put_shared), 8 threads x 2000
let shared = Arc::new(MvwBackend::open(16, &b).unwrap());
let mut handles = Vec::new();
for t in 0..8u64 {
let s = shared.clone();
handles.push(thread::spawn(move || for i in 0..2000u64 {
let k = Czyx::unpack_u32(((t * 2000 + i) as u32) << 8 | 1);
s.put_shared(k, vec![(i as u8); 8]);
}));
}
for h in handles { h.join().unwrap(); }
// all writes readable
for t in 0..8u64 { for i in 0..2000u64 {
assert!(shared.get(&Czyx::unpack_u32(((t*2000+i) as u32) << 8 | 1)).is_some());
}}
// compact folds memtable to checkpoint and truncates WALs; values still readable
shared.compact().unwrap();
assert!(shared.get(&Czyx::new(1,1,1,1)).is_some());
cleanup(&b);
let _ = thread::current(); // silence
}
#[test]
fn cubestore_record_api_over_mvw_backend() {
let b = base("mvw-cubestore"); cleanup(&b);
let mut store = CubeStore::new(MvwBackend::open(16, &b).unwrap());
let mut h = CubeHeader::new();
h.title = Some("t".into());
h.refresh_flags();
store.put_record(Czyx::new(3,3,3,3), &h, b"body");
let (h2, body) = store.get_record(&Czyx::new(3,3,3,3)).expect("record roundtrip");
assert_eq!(body, b"body");
assert_eq!(h2.title.as_deref(), Some("t"));
cleanup(&b);
}
+182
View File
@@ -0,0 +1,182 @@
#!/usr/bin/env python3
"""cube-notes-mcp — MCP (stdio) server exposing the CUBELinux-2 notes CLI as
model-facing tools `mcp__cubenotes__note_*` / `mcp__cubenotes__project_*`.
Wraps `cube note ...` / `cube project ...` (the durable WordFlags
message-save-path with categories + project threads). The harness spawns this
server via the `mcp-client` plugin; each tool call runs the `cube` binary,
which persists to `$CUBE_NOTES_DIR` (default ~/.cubelinux-notes).
"""
import sys, json, subprocess
CUBE = "/home/CUBELinux/CUBELinux-2/target/release/cube"
SERVER_NAME = "cube-notes-mcp"
SERVER_VERSION = "0.2.0"
PROTOCOL_VERSION = "2024-11-05"
def tool_schemas():
return [
{
"name": "note_write",
"description": "Add a note to the durable WordFlags message-save-path. "
"Optional: project P associates it to a project thread, "
"category C categorises it (doc_type note:C). Returns the coord.",
"inputSchema": {"type": "object", "properties": {
"text": {"type": "string"},
"project": {"type": "string", "description": "Project name (thread)."},
"category": {"type": "string", "description": "e.g. design, impl, cube."},
}, "required": ["text"]},
},
{
"name": "note_list",
"description": "List notes (newest first), optionally filtered by session, project, category, since/until (YYYY-MM-DD).",
"inputSchema": {"type": "object", "properties": {
"session": {"type": "string"},
"project": {"type": "string"},
"category": {"type": "string"},
"since": {"type": "string"},
"until": {"type": "string"},
}},
},
{
"name": "note_search",
"description": "Substring-search notes, optionally scoped to a project/category.",
"inputSchema": {"type": "object", "properties": {
"query": {"type": "string"},
"project": {"type": "string"},
"category": {"type": "string"},
}, "required": ["query"]},
},
{
"name": "note_show",
"description": "Show one note by C.Z.Y.X.",
"inputSchema": {"type": "object", "properties": {
"coord": {"type": "string"},
}, "required": ["coord"]},
},
{
"name": "project_list",
"description": "List all project threads (doc_type=project).",
"inputSchema": {"type": "object", "properties": {}},
},
{
"name": "project_show",
"description": "Walk a project's associated notes (its thread/timeline) by C.Z.Y.X.",
"inputSchema": {"type": "object", "properties": {
"coord": {"type": "string"},
}, "required": ["coord"]},
},
{
"name": "project_resume",
"description": "Set a resume marker for a project (where it left off).",
"inputSchema": {"type": "object", "properties": {
"name": {"type": "string"}, "text": {"type": "string"},
}, "required": ["name", "text"]},
},
{
"name": "project_context",
"description": "Compact 'where we are' summary for a project (latest notes + resume marker) — for context injection.",
"inputSchema": {"type": "object", "properties": {
"name": {"type": "string"},
}, "required": ["name"]},
},
]
def run_cube(args):
try:
r = subprocess.run([CUBE] + args, capture_output=True, text=True, timeout=30)
out = (r.stdout or "").strip()
if r.returncode != 0:
return f"error: {(r.stderr or out).strip()}"
return out if out else "(no result)"
except Exception as e: # noqa: BLE001
return f"error: {e}"
def opt(flag, val):
return [flag, str(val)] if val else []
def dispatch(name, args):
args = args or {}
if name == "note_write":
text = (args.get("text") or "").strip()
if not text:
return "error: missing 'text'"
return run_cube(["note", "add"] + opt("--project", args.get("project"))
+ opt("--cat", args.get("category")) + [text])
if name == "note_list":
cmd = ["note", "list"]
cmd += opt("--project", args.get("project"))
cmd += opt("--cat", args.get("category"))
cmd += opt("--since", args.get("since"))
cmd += opt("--until", args.get("until"))
if args.get("session"):
cmd.append(args["session"])
return run_cube(cmd)
if name == "note_search":
q = (args.get("query") or "").strip()
if not q:
return "error: missing 'query'"
return run_cube(["note", "search", q] + opt("--project", args.get("project"))
+ opt("--cat", args.get("category")))
if name == "note_show":
c = (args.get("coord") or "").strip()
return run_cube(["note", "show", c]) if c else "error: missing 'coord'"
if name == "project_list":
return run_cube(["project", "list"])
if name == "project_show":
c = (args.get("coord") or "").strip()
return run_cube(["project", "show", c]) if c else "error: missing 'coord'"
if name == "project_resume":
nm = (args.get("name") or "").strip()
tx = (args.get("text") or "").strip()
if not nm or not tx:
return "error: project_resume needs name + text"
return run_cube(["project", "resume", nm, tx])
if name == "project_context":
nm = (args.get("name") or "").strip()
return run_cube(["project", "context", nm]) if nm else "error: missing 'name'"
return f"error: unknown tool {name}"
def handle(msg):
if not isinstance(msg, dict):
return None
mid, method = msg.get("id"), msg.get("method")
if method == "initialize":
return {"jsonrpc": "2.0", "id": mid, "result": {
"protocolVersion": PROTOCOL_VERSION,
"capabilities": {"tools": {}},
"serverInfo": {"name": SERVER_NAME, "version": SERVER_VERSION},
}}
if method == "ping":
return {"jsonrpc": "2.0", "id": mid, "result": {}}
if method in ("notifications/initialized", "initialized"):
return None
if method == "tools/list":
return {"jsonrpc": "2.0", "id": mid, "result": {"tools": tool_schemas()}}
if method == "tools/call":
name = msg.get("params", {}).get("name", "")
args = msg.get("params", {}).get("arguments", {})
return {"jsonrpc": "2.0", "id": mid, "result": {
"content": [{"type": "text", "text": dispatch(name, args)}], "isError": False}}
if mid is not None:
return {"jsonrpc": "2.0", "id": mid, "result": {}}
return None
for line in sys.stdin:
line = line.strip()
if not line:
continue
try:
msg = json.loads(line)
except Exception: # noqa: BLE001
continue
resp = handle(msg)
if resp is not None:
sys.stdout.write(json.dumps(resp) + "\n")
sys.stdout.flush()
+12
View File
@@ -0,0 +1,12 @@
[package]
name = "cubeai"
version = "0.1.0"
edition.workspace = true
license.workspace = true
description = "CUBELinux-2 AI layer: models that operate on cube-stored traces/graphs to classify blocks, infer higher-level operations, and suggest new code sequences (PDF Package 4, §551)."
[dependencies]
cubecoords = { path = "../cubecoords" }
cubestore = { path = "../cubestore" }
cubecode = { path = "../cubecode" }
cubedbt = { path = "../cubedbt" }
+329
View File
@@ -0,0 +1,329 @@
//! CUBELinux-2 AI layer.
//!
//! Per the PDF (Package 4, §551): *"cubeai: models that operate on
//! cubestored traces/graphs to classify blocks, infer higherlevel
//! operations, and suggest new code sequences or orchestrations."*
//!
//! This crate is the *structured decision* layer sitting on top of the
//! substrate (and on [`cubedbt`]). It is deterministic and dependency-free
//! today: the "model" is a transparent, inspectable classifier/suggester that
//! operates on the same [`CubeStore`] / [`CodeCell`] / [`Behavior`] types the
//! rest of the workspace uses. A learned model can later implement the same
//! traits without changing callers.
//!
//! Pipeline (the end-to-end story from §549–§551 + §554):
//! trace (cube) → classify blocks → infer higher-level op →
//! suggest `TranslationRule`s → hand to `cubedbt` to patch + run (mimic).
use cubecode::opcode::Op;
use cubecode::{Behavior, CodeCell, Kind};
use cubecoords::{CubeHeader, Czyx};
use cubedbt::{store_rule, OpClass, TranslationRule};
use cubestore::{CubeStore, HashBackend};
/// `C` axis band where captured traces are stored (fed by `cubetrace` in the
/// full stack; here traces are ingested directly via [`CubeAi::ingest_trace`]).
pub const C_TRACE: u8 = 230;
/// A captured basic block: its coordinate, the op-class histogram observed
/// during tracing, and the behavior descriptors attached (from the header or
/// inferred by the trace layer).
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct BlockTrace {
pub label: Czyx,
/// Count of each [`OpClass`] seen in the block (`OpClass::Any` unused).
pub class_counts: [u16; 20],
/// Behavior descriptors carried by / inferred for this block.
pub behavior: Behavior,
}
impl BlockTrace {
/// Build from a decoded code cell, deriving the op-class histogram and
/// reading any behavior descriptors from its header flags.
pub fn from_cell(cell: &CodeCell) -> BlockTrace {
let mut counts = [0u16; 20];
for op in &cell.code {
let c = OpClass::of(op) as usize;
if c < 20 {
counts[c] += 1;
}
}
BlockTrace {
label: cell.label,
class_counts: counts,
behavior: Behavior::from_flags(cell.header.flags.bits()),
}
}
/// Total op count.
pub fn total(&self) -> u32 {
self.class_counts.iter().map(|&c| c as u32).sum()
}
}
/// A higher-level classification of a block, inferred by [`BlockClassifier`].
#[derive(Copy, Clone, Debug, PartialEq, Eq)]
pub enum BlockKind {
/// Mostly arithmetic/logic — a computation block.
Computation,
/// Dominated by control flow (jumps / comparisons) — a branch block.
Branch,
/// Contains call links to other records — a call/composition block.
CallTrampoline,
/// Heavy I/O or network descriptors — an I/O section.
IoSection,
/// Otherwise: a plain linear sequence.
Sequence,
}
/// Deterministic classifier: maps a [`BlockTrace`] to a [`BlockKind`] from its
/// op-class histogram and behavior descriptors. (The transparent "model".)
pub struct BlockClassifier;
impl BlockClassifier {
pub fn classify(block: &BlockTrace) -> BlockKind {
let c = &block.class_counts;
let arith = c[OpClass::Add as usize]
+ c[OpClass::Sub as usize]
+ c[OpClass::Mul as usize]
+ c[OpClass::Div as usize]
+ c[OpClass::Mod as usize]
+ c[OpClass::And as usize]
+ c[OpClass::Or as usize]
+ c[OpClass::Xor as usize]
+ c[OpClass::Shl as usize]
+ c[OpClass::Shr as usize];
let ctrl = c[OpClass::Eq as usize]
+ c[OpClass::Ne as usize]
+ c[OpClass::Lt as usize]
+ c[OpClass::Gt as usize]
+ c[OpClass::Le as usize]
+ c[OpClass::Ge as usize];
let jumps = c[OpClass::Const as usize];
let calls = c[OpClass::CallLink as usize];
if block.behavior.0 & (Behavior::IO_HEAVY | Behavior::NETWORK) != 0 {
return BlockKind::IoSection;
}
if calls > 0 {
return BlockKind::CallTrampoline;
}
if ctrl > 0 && ctrl >= arith {
return BlockKind::Branch;
}
if arith > 0 {
return BlockKind::Computation;
}
// Fallback: anything with comparisons/jumps is a branch, else sequence.
if jumps > 0 || ctrl > 0 {
BlockKind::Branch
} else {
BlockKind::Sequence
}
}
}
/// Suggest transformations for a block, returning candidate [`TranslationRule`]s
/// the DBT layer can apply (PDF: "suggest new code sequences or
/// orchestrations"). Deterministic and local: each suggestion records *why*.
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct Suggestion {
pub rule: TranslationRule,
pub rationale: String,
}
/// The AI runtime over a cube store: ingests traces, classifies, and suggests
/// DBT rules. Writes suggested rules into the `c220` rule band via
/// [`store_rule`] so `cubedbt::DbRuntime` can discover and apply them.
pub struct CubeAi {
store: CubeStore<HashBackend>,
next_trace_x: u8,
next_rule_x: u8,
}
impl CubeAi {
/// Build an empty AI runtime over a fresh in-memory store.
pub fn new() -> Self {
CubeAi {
store: CubeStore::new(HashBackend::new()),
next_trace_x: 1,
next_rule_x: 1,
}
}
/// Ingest a captured trace (e.g. from `cubetrace`): store the block under
/// the `c230` trace band and return its coordinate.
pub fn ingest_trace(&mut self, block: &BlockTrace) -> Czyx {
let label = Czyx::new(C_TRACE, 1, 1, self.next_trace_x);
self.next_trace_x = self.next_trace_x.wrapping_add(1).max(1);
let mut h = CubeHeader::new();
h.title = Some(format!("trace:{:?}", block.label));
h.doc_type = Some(Kind::Other.as_str().into());
h.size_bytes = Some(block.class_counts.len() as u64 * 2);
h.flags.0 |= block.behavior.to_flags();
h.refresh_flags();
// Body: the raw histogram (20 x u16 le).
let mut body = Vec::with_capacity(40);
for c in &block.class_counts {
body.extend_from_slice(&c.to_le_bytes());
}
self.store.put_record(label, &h, &body);
label
}
/// Classify a single block.
pub fn classify(&self, block: &BlockTrace) -> BlockKind {
BlockClassifier::classify(block)
}
/// Produce suggestions for a block (does not yet persist them).
pub fn suggest(&self, block: &BlockTrace) -> Vec<Suggestion> {
let kind = BlockClassifier::classify(block);
let mut out = Vec::new();
match kind {
BlockKind::Computation => {
out.push(Suggestion {
rule: TranslationRule {
name: format!("compute-opt:{:?}", block.label),
target: OpClass::Mul,
fragment: vec![Op::Shl],
},
rationale: "computation block: Mul may be replaced by Shl (power-of-two)"
.into(),
});
}
BlockKind::Branch => {
out.push(Suggestion {
rule: TranslationRule {
name: format!("branch-opt:{:?}", block.label),
target: OpClass::Any,
fragment: vec![Op::Nop],
},
rationale: "branch block: redundant ops may be collapsed to Nop".into(),
});
}
BlockKind::IoSection => {
out.push(Suggestion {
rule: TranslationRule {
name: format!("io-batch:{:?}", block.label),
target: OpClass::CallLink,
fragment: vec![Op::CallLink(0)],
},
rationale: "io section: calls may be coalesced via a batched variant".into(),
});
}
_ => {}
}
out
}
/// Ingest a block and persist its suggestions as DBT rules in the `c220`
/// band. Returns the stored rule coordinates (empty if no suggestions).
pub fn ingest_and_suggest(&mut self, block: &BlockTrace) -> Vec<Czyx> {
self.ingest_trace(block);
let sugs = self.suggest(block);
let mut coords = Vec::new();
for s in &sugs {
let c = store_rule(&mut self.store, &s.rule, self.next_rule_x);
self.next_rule_x = self.next_rule_x.wrapping_add(1).max(1);
coords.push(c);
}
coords
}
/// Borrow the backing store (e.g. to hand to `cubedbt::DbRuntime`).
pub fn store(&self) -> &CubeStore<HashBackend> {
&self.store
}
}
impl Default for CubeAi {
fn default() -> Self {
Self::new()
}
}
#[cfg(test)]
mod tests {
use super::*;
use cubecode::opcode::Op;
use cubecoords::Czyx;
fn cell(coord: Czyx, code: Vec<Op>, beh: Behavior) -> CodeCell {
let mut h = CubeHeader::new();
h.title = Some("blk".into());
h.doc_type = Some(Kind::Fn.as_str().into());
h.flags.0 |= beh.to_flags();
h.refresh_flags();
CodeCell::from_record(coord, &h, &cubecode::opcode::encode(&code))
.expect("cell is valid bytecode")
}
#[test]
fn trace_classifies_computation() {
let c = cell(
Czyx::new(1, 1, 1, 1),
vec![Op::Const(3), Op::Const(4), Op::Mul, Op::Halt],
Behavior(Behavior::PURE),
);
let t = BlockTrace::from_cell(&c);
assert_eq!(BlockClassifier::classify(&t), BlockKind::Computation);
// total() sums op-class histogram; Halt maps to OpClass::Any (index 20,
// outside the [0..20) histogram), so 3 counted ops (Const, Const, Mul).
assert_eq!(t.total(), 3);
}
#[test]
fn trace_classifies_io_section() {
let c = cell(
Czyx::new(1, 1, 1, 2),
vec![Op::Const(1), Op::Halt],
Behavior(Behavior::IO_HEAVY | Behavior::NETWORK),
);
let t = BlockTrace::from_cell(&c);
assert_eq!(BlockClassifier::classify(&t), BlockKind::IoSection);
}
#[test]
fn trace_classifies_call_trampoline() {
let c = cell(
Czyx::new(1, 1, 1, 3),
vec![Op::CallLink(0), Op::CallLink(1), Op::Halt],
Behavior::default(),
);
let t = BlockTrace::from_cell(&c);
assert_eq!(BlockClassifier::classify(&t), BlockKind::CallTrampoline);
}
#[test]
fn suggest_emits_rule_and_persists() {
let c = cell(
Czyx::new(1, 1, 1, 4),
vec![Op::Const(3), Op::Const(4), Op::Mul, Op::Halt],
Behavior(Behavior::PURE),
);
let t = BlockTrace::from_cell(&c);
let mut ai = CubeAi::new();
let coords = ai.ingest_and_suggest(&t);
assert_eq!(coords.len(), 1, "computation block suggests one rule");
assert_eq!(coords[0].c, rule_band());
// The stored rule is discoverable by cubedbt.
let rt = cubedbt::DbRuntime::new(ai.store().clone());
assert_eq!(rt.discover_rules().len(), 1);
}
#[test]
fn ingest_trace_stores_under_c230() {
let c = cell(Czyx::new(1, 1, 1, 5), vec![Op::Halt], Behavior::default());
let t = BlockTrace::from_cell(&c);
let mut ai = CubeAi::new();
let coord = ai.ingest_trace(&t);
assert_eq!(coord.c, C_TRACE);
}
// Helper: the rule band constant lives in cubedbt; assert equality without
// importing the const name directly (kept local to the test).
fn rule_band() -> u8 {
cubedbt::C_DBT_RULE
}
}
+18
View File
@@ -0,0 +1,18 @@
[package]
name = "cubecli"
version = "0.1.0"
edition.workspace = true
license.workspace = true
description = "CUBELinux-2 local system CLI: in-process REPL + script runner over one shared CubeStore (cubefs + cubecode + cubecrypt)."
[[bin]]
name = "cube"
path = "src/main.rs"
[dependencies]
cubecoords = { path = "../cubecoords" }
cubestore = { path = "../cubestore" }
cubefs = { path = "../cubefs" }
cubecode = { path = "../cubecode" }
cubecrypt = { path = "../cubecrypt" }
cubesys = { path = "../cubesys" }
@@ -48,15 +48,49 @@ fn is_command_word(w: &str) -> bool {
| "seal"
| "open"
| "keyinit"
| "scan-word-flags"
| "show-flags"
| "trace-capture"
| "trace-replay"
| "golden-capture"
| "trace-verify"
| "trace-list"
| "trace-links"
| "query"
| "begin"
| "commit"
| "rollback"
| "stats"
| "audit"
| "note"
| "notes"
| "project"
| "projects"
)
}
/// Open (or create) the durable session-note store. Notes persist across CLI
/// invocations and sessions — the CUBE "message save path" — via the WAL +
/// checkpoint store. The directory is `$CUBE_NOTES_DIR` (`~/.cubelinux-notes`).
fn open_notes_store() -> cubesys::store::ConcurrentStore {
let dir = std::env::var("CUBE_NOTES_DIR").unwrap_or_else(|_| {
format!(
"{}/.cubelinux-notes",
std::env::var("HOME").unwrap_or_else(|_| "/tmp".to_string())
)
});
cubesys::store::ConcurrentStore::open(
&format!("{dir}/notes.store"),
&format!("{dir}/notes.wal"),
&format!("{dir}/recovery.log"),
cubesys::store::DurabilityConfig::default(),
)
.unwrap_or_else(|e| {
eprintln!("error: cannot open notes store {dir}: {e}");
std::process::exit(1);
})
}
fn main() {
let args: Vec<String> = std::env::args().collect();
match args.get(1).map(|s| s.as_str()) {
@@ -67,12 +101,33 @@ fn main() {
// the coordinate-addressed API a real CLI surface the OS can call.
Some(word) if is_command_word(word) => {
let line = args[1..].join(" ");
let mut session = Session::new();
match session.exec(&line) {
Ok(out) => println!("{out}"),
Err(e) => {
eprintln!("error: {e}");
std::process::exit(1);
// Session notes live in their own DURABLE store so a note written
// in one session is reviewable in the next (the message save path).
if word == "note" || word == "notes" || word == "project" || word == "projects" {
let store = open_notes_store();
let res = if word == "project" || word == "projects" {
cubesys::notes::project_command(&store, &line)
} else {
cubesys::notes::note_command(&store, &line)
};
// Force a durable flush/checkpoint before exiting so the note is
// persisted even though this process (the CLI) is short-lived.
store.checkpoint();
match res {
Ok(out) => println!("{out}"),
Err(e) => {
eprintln!("error: {e}");
std::process::exit(1);
}
}
} else {
let mut session = Session::new();
match session.exec(&line) {
Ok(out) => println!("{out}"),
Err(e) => {
eprintln!("error: {e}");
std::process::exit(1);
}
}
}
}
@@ -136,6 +191,11 @@ fn print_help() {
stat <path> getattr via cubefs\n \
seal <path> <K.Z.Y.X|auto> <tf> encrypt a record (tf: none|gcm|chacha|xts)\n \
open <path> <K.Z.Y.X|auto> <tf> decrypt + decode + run a sealed record\n \
keyinit ensure the OS Null-space keystore exists\n"
keyinit ensure the OS Null-space keystore exists\n \
note <text> add a session note (durable, WordFlags-tagged)\n \
note list [session] list notes (default: today's session)\n \
note search <text> search notes across sessions\n \
note show <C.Z.Y.X> show one note\n \
notes alias for `note list`\n"
);
}
+215
View File
@@ -0,0 +1,215 @@
//! Feature-vector ball / k-NN index: a metric VP-tree stored as CUBE records.
//! Each node is a feature-vector record (`doc_type = "vp-node"`) whose body holds
//! the pivot `(x,y,z)` + radial radius `tau`, and whose `linked_records` are its
//! two children. Ball recall walks the tree, pruning branches that cannot
//! intersect the query ball — exact, bounded node visits (whitepaper §4.1).
use cubecoords::{CubeHeader, Czyx};
use cubestore::{CubeBackend, CubeStore};
/// A 3-D feature vector (the `(fx,fy,fz)` MEM model).
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub struct Vec3 {
pub x: u64,
pub y: u64,
pub z: u64,
}
impl Vec3 {
pub fn new(x: u64, y: u64, z: u64) -> Self {
Vec3 { x, y, z }
}
pub fn d2(&self, o: &Vec3) -> u64 {
let dx = (self.x as i64 - o.x as i64).abs() as u64;
let dy = (self.y as i64 - o.y as i64).abs() as u64;
let dz = (self.z as i64 - o.z as i64).abs() as u64;
dx * dx + dy * dy + dz * dz
}
}
const VP_NODE: &str = "vp-node";
/// Namespace for index nodes (avoid collisions with user data spaces).
pub const VP_SPACE: u8 = 200;
const NULL_NODE: Czyx = Czyx::new(VP_SPACE, 0, 0, 0);
/// Build a VP-tree over `points`, writing each node as a record. Returns the root coord.
pub fn build<B: CubeBackend>(store: &mut CubeStore<B>, points: &[Vec3]) -> Czyx {
let mut counter = 0u64;
let idx: Vec<usize> = (0..points.len()).collect();
build_rec(store, points, &idx, &mut counter)
}
fn build_rec<B: CubeBackend>(
store: &mut CubeStore<B>,
points: &[Vec3],
box_idxs: &[usize],
counter: &mut u64,
) -> Czyx {
if box_idxs.is_empty() {
return NULL_NODE;
}
let pi = box_idxs[0];
let pivot = points[pi];
let mut others: Vec<(u64, usize)> = box_idxs
.iter()
.copied()
.filter(|&i| i != pi)
.map(|i| (pivot.d2(&points[i]), i))
.collect();
others.sort_by_key(|&(d, _)| d);
// Vantage radius = sqrt of the median squared distance to the other points.
let tau = if others.is_empty() {
0
} else {
(others[others.len() / 2].0 as f64).sqrt() as u64
};
let mid = others.len() / 2;
let left_idx: Vec<usize> = others.iter().take(mid).map(|&(_, i)| i).collect();
let right_idx: Vec<usize> = others.iter().skip(mid).map(|&(_, i)| i).collect();
*counter += 1;
let coord = czyx_node(*counter);
let left = build_rec(store, points, &left_idx, counter);
let right = build_rec(store, points, &right_idx, counter);
let mut links = Vec::new();
if left != NULL_NODE {
links.push(left);
}
if right != NULL_NODE {
links.push(right);
}
let mut h = CubeHeader::new();
h.doc_type = Some(VP_NODE.into());
h.title = Some("vp".into());
h.linked_records = links;
h.refresh_flags();
let mut body = Vec::with_capacity(32);
for v in [pivot.x, pivot.y, pivot.z, tau] {
body.extend_from_slice(&v.to_le_bytes());
}
store.put_record(coord, &h, &body);
coord
}
fn czyx_node(n: u64) -> Czyx {
// Encode the node id across the low three axes so we can host far more than
// 255 nodes (u8 per axis would collide). c = index namespace.
Czyx::new(VP_SPACE, ((n >> 16) & 0xFF) as u8, ((n >> 8) & 0xFF) as u8, (n & 0xFF) as u8)
}
/// Ball recall: walk the VP-tree from `root`, pruning branches that cannot
/// intersect the query ball. Returns the coordinates of every vector within `r`
/// and the number of nodes visited (each visit = one record read).
pub fn ball_stats<B: CubeBackend>(
store: &CubeStore<B>,
root: Czyx,
q: Vec3,
r: u64,
) -> (Vec<Czyx>, usize) {
let r2 = r * r;
let mut out = Vec::new();
let mut visits = 0usize;
ball_rec(store, root, &q, r2, r, &mut out, &mut visits);
(out, visits)
}
fn ball_rec<B: CubeBackend>(
store: &CubeStore<B>,
node: Czyx,
q: &Vec3,
r2: u64,
r: u64,
out: &mut Vec<Czyx>,
visits: &mut usize,
) {
if node == NULL_NODE {
return;
}
*visits += 1;
let (h, body) = match store.get_record(&node) {
Some(x) => x,
None => return,
};
if body.len() < 32 {
return;
}
let pivot = Vec3::new(
u64::from_le_bytes(body[0..8].try_into().unwrap()),
u64::from_le_bytes(body[8..16].try_into().unwrap()),
u64::from_le_bytes(body[16..24].try_into().unwrap()),
);
let tau = u64::from_le_bytes(body[24..32].try_into().unwrap());
let d2v = pivot.d2(q);
if d2v <= r2 {
out.push(node);
}
let links = h.linked_records;
let d = (d2v as f64).sqrt();
let rf = r as f64;
// Descend left (points within tau of the pivot) if the query ball may reach them.
if links.len() >= 1 && (d - rf) <= tau as f64 {
ball_rec(store, links[0], q, r2, r, out, visits);
}
// Descend right (points beyond tau) if the query ball may extend past the pivot sphere.
if links.len() >= 2 && (d + rf) > tau as f64 {
ball_rec(store, links[1], q, r2, r, out, visits);
}
}
#[cfg(test)]
mod tests {
use super::*;
use cubestore::HashBackend;
#[test]
fn vp_tree_exact_ball_recall() {
let n = 2000u64;
let mut points = Vec::with_capacity(n as usize);
for c in 0..16u64 {
let (bx, by, bz) = (c * 40, (c * 7) % 16 * 7, (c * 13) % 16 * 5);
for j in 0..(n / 16) {
let o = j as u64 % 12;
points.push(Vec3::new(bx + o, by + (o * 3) % 12, bz + (o * 7) % 12));
}
}
let mut store = CubeStore::new(HashBackend::new());
let root = build(&mut store, &points);
let r = 30u64;
let r2 = r * r;
let mut total_rec = 0usize;
let mut total_true = 0usize;
let mut visits_sum = 0usize;
for anchor in 0..40usize {
let q = points[anchor];
let (found, visits) = ball_stats(&store, root, q, r);
// ground truth = vectors within r of q
let truth: Vec<Vec3> = points.iter().copied().filter(|p| q.d2(p) <= r2).collect();
// decode pivots from the found node records
let mut found_pivots = Vec::new();
for c in &found {
if let Some((_, b)) = store.get_record(c) {
if b.len() >= 24 {
found_pivots.push(Vec3::new(
u64::from_le_bytes(b[0..8].try_into().unwrap()),
u64::from_le_bytes(b[8..16].try_into().unwrap()),
u64::from_le_bytes(b[16..24].try_into().unwrap()),
));
}
}
}
let rec = truth.iter().filter(|v| found_pivots.contains(v)).count();
total_rec += rec;
total_true += truth.len();
visits_sum += visits;
}
eprintln!(
"exact recall {:.1}% ({} of {}), avg {} node-visits per query (N={})",
100.0 * total_rec as f64 / total_true as f64,
total_rec,
total_true,
visits_sum as f64 / 40.0,
n
);
assert!(total_true > 0 && total_rec == total_true, "VP-tree must be exact (got {}/{}), avg {} visits", total_rec, total_true, visits_sum as f64 / 40.0);
assert!(visits_sum as f64 / 40.0 < (n as f64) / 3.0, "VP-tree must prune (avg {} visits)", visits_sum as f64 / 40.0);
}
}
+169
View File
@@ -0,0 +1,169 @@
//! CUBELinux-2 cubebook / kernel-convention constants.
//!
//! These are the conventions the *OS* layer uses to store its computational
//! behavior in CUBE. Per the PDF (Package 4, and the "behavior descriptors"
//! discussion at PDF §524525), functions/operators that live in CUBE are:
//! * addressed in a reserved `C` axis band (here `c210..=c219`),
//! * tagged with a `kind` (`fn`/`kernel`/`layer`/`checkpoint`/`variant`), and
//! * annotated with *behavior descriptor* flags carried in the record
//! header's out-of-band flag bits (tag 12, see cubestore encode/decode).
//!
//! The full spec-derived descriptor set (PDF §524525) is: "pure function,"
//! "I/O heavy," "allocates memory," "touches network," "hot path," and
//! "security sensitive." We store each directly as a dedicated header-flag bit
//! inside the spare 8..=15 range, deliberately leaving bit 12
//! (`cubecrypt::HEADER_FLAG_ENCRYPTED`) clear. `CubeHeader::refresh_flags()`
//! preserves spare bits 8..=15, so a header carrying descriptors survives an
//! encode→decode→refresh round-trip.
//!
//! This is deliberately NOT a store-IO model: the cubevm never reads or writes
//! OS state records directly. The OS's *decisions/computation* live as CUBE
//! kernels; the native OS layer is a thin effector that reads a kernel's
//! computed result and applies the effect (writes the record, runs the
//! command). See `cubesys/src/commands.rs` `tick` + `native-apply`.
/// First `C` axis value reserved for OS operator kernels (call-graph roots and
/// leaves). The OS keeps its behavioral substrate here, separate from the
/// `c200` operational-snapshot data band and the `c001/c002` doc examples.
pub const C_OS_KERNEL: u8 = 210;
/// `C` axis band for OS *data* records the native layer writes after a kernel
/// decides an effect (kept distinct from the compute-kernel band above).
pub const C_OS_EFFECT: u8 = 211;
/// Header-flag bits reserved for the behavior-descriptor field. Six spec
/// descriptors, each a dedicated bit in the spare 8..=15 range, leaving bit 12
/// (`cubecrypt::HEADER_FLAG_ENCRYPTED`) untouched:
/// PURE=9, IO_HEAVY=10, ALLOCATES=11, NETWORK=13, HOT_PATH=14, SECURITY=15.
pub const HEADER_FLAG_BEHAVIOR: u16 = 0b1110_1110_0000_0000; // bits 9,10,11,13,14,15 (bit12 reserved)
/// Behavior descriptors for an OS operator kernel (PDF §524525, full set).
///
/// Each variant is stored directly as its dedicated header-flag bit (subset of
/// `HEADER_FLAG_BEHAVIOR`), so `Behavior` carries the raw descriptor bits and
/// round-trips through the header codec without bit-shift collisions.
#[derive(Copy, Clone, Debug, Default, PartialEq, Eq)]
pub struct Behavior(pub u16);
impl Behavior {
/// Pure function: no side effects, deterministic on inputs.
pub const PURE: u16 = 1 << 9; // 0x0200
/// I/O heavy: performs significant store/device I/O.
pub const IO_HEAVY: u16 = 1 << 10; // 0x0400
/// Allocates memory: grows the heap / maps pages.
pub const ALLOCATES: u16 = 1 << 11; // 0x0800
/// Touches network: performs socket/link I/O (security-relevant surface).
pub const NETWORK: u16 = 1 << 13; // 0x2000
/// Hot path: executed frequently; a candidate for optimization/variant swap.
pub const HOT_PATH: u16 = 1 << 14; // 0x4000
/// Security sensitive: elevated privilege / trust boundary crossing.
pub const SECURITY_SENSITIVE: u16 = 1 << 15; // 0x8000
/// Combine descriptor bits (e.g. `Behavior::PURE | Behavior::HOT_PATH`).
pub fn new(bits: u16) -> Self {
Behavior(bits & HEADER_FLAG_BEHAVIOR)
}
/// Encode into the out-of-band header flag bits.
pub fn to_flags(self) -> u16 {
self.0 & HEADER_FLAG_BEHAVIOR
}
/// Decode from a raw 16-bit flag word (keeps only the behavior bits).
pub fn from_flags(flags: u16) -> Self {
Behavior(flags & HEADER_FLAG_BEHAVIOR)
}
/// True if any descriptor bit is set.
pub fn is_any(self) -> bool {
self.0 != 0
}
/// Human-readable descriptor tags (used by `ls`/`stat` and the effector).
pub fn tags(self) -> Vec<&'static str> {
let mut out = Vec::new();
if self.0 & Self::PURE != 0 {
out.push("pure");
}
if self.0 & Self::IO_HEAVY != 0 {
out.push("io");
}
if self.0 & Self::ALLOCATES != 0 {
out.push("alloc");
}
if self.0 & Self::NETWORK != 0 {
out.push("net");
}
if self.0 & Self::HOT_PATH != 0 {
out.push("hot");
}
if self.0 & Self::SECURITY_SENSITIVE != 0 {
out.push("sec");
}
out
}
}
#[cfg(test)]
mod tests {
use super::*;
use cubecoords::{CubeHeader, Czyx};
use cubestore::{CubeStore, HashBackend};
#[test]
fn behavior_round_trips_through_store() {
// All six spec descriptors must survive the real store round-trip
// (what native-apply reads back). This is the exact field the
// effector inspects; an earlier 3-bit mask (0x7000) silently dropped
// HOT_PATH (bit 15) and collided with ENCRYPTED (bit 12).
let b = Behavior::new(
Behavior::PURE | Behavior::IO_HEAVY | Behavior::ALLOCATES | Behavior::NETWORK,
);
let raw = b.to_flags();
assert_eq!(
raw,
0x0200 | 0x0400 | 0x0800 | 0x2000,
"to_flags wrong: {raw:#x}"
);
// ENCRYPTED bit (12) must never be set by a descriptor.
assert_eq!(raw & (1 << 12), 0, "descriptor must not set ENCRYPTED bit");
let mut store = CubeStore::new(HashBackend::new());
let coord = Czyx::new(210, 1, 1, 3);
let mut h = CubeHeader::new();
h.doc_type = Some("kernel".into());
h.title = Some("summarize-procs".into());
h.flags.0 |= raw;
h.refresh_flags();
store.put_record(coord, &h, &[1, 2, 3]);
let (read_h, _) = store.get_record(&coord).expect("record present");
let back = Behavior::from_flags(read_h.flags.bits());
assert_eq!(
back,
b,
"behavior dropped on store round-trip: stored {raw:#x}, got {:#x}",
read_h.flags.bits()
);
}
#[test]
fn security_sensitive_bit_used_and_round_trips() {
// Explicitly cover the 6th spec descriptor (security sensitive, bit15).
let b = Behavior::new(Behavior::SECURITY_SENSITIVE | Behavior::HOT_PATH);
let mut store = CubeStore::new(HashBackend::new());
let coord = Czyx::new(210, 1, 1, 5);
let mut h = CubeHeader::new();
h.doc_type = Some("kernel".into());
h.title = Some("privileged-op".into());
h.flags.0 |= b.to_flags();
h.refresh_flags();
store.put_record(coord, &h, &[]);
let (read_h, _) = store.get_record(&coord).expect("record present");
assert_eq!(
Behavior::from_flags(read_h.flags.bits()),
b,
"security-sensitive descriptor lost on round-trip"
);
}
}
+1
View File
@@ -71,6 +71,7 @@ impl Kind {
/// in the body. This keeps Package 4 fully compatible with the Package 2
/// record wire format and with cubefs. `code_kind`/`set_kind` bridge the
/// enum to the string.
#[derive(Clone, Debug)]
pub struct CodeCell {
/// The cube coordinate this cell is stored at.
pub label: Czyx,
+8 -1
View File
@@ -35,10 +35,17 @@
#![forbid(unsafe_code)]
#![warn(missing_docs)]
pub mod cb;
pub mod cell;
pub mod ballindex;
pub mod opcode;
pub mod trace;
pub mod vm;
pub use cb::{Behavior, C_OS_EFFECT, C_OS_KERNEL, HEADER_FLAG_BEHAVIOR};
pub use cell::{CodeCell, Kind};
pub use opcode::{decode, encode, CodeError, Op};
pub use vm::{Fault, RunResult, Vm, SYS_DEGREE, SYS_LINKED_EXISTS, SYS_NOP, SYS_TRACE};
pub use ballindex::{ball_stats, build, Vec3, VP_SPACE};
pub use trace::{capture_golden, lineage, replay_trace, store_trace, verify_golden, LineageHop};
pub use vm::{replay, Fault, RunResult, Vm, SYS_DEGREE, SYS_LINKED_EXISTS, SYS_NOP, SYS_TRACE};
+263
View File
@@ -0,0 +1,263 @@
//! Execution-trace artifacts: store a captured machine-code trace as a
//! first-class AI-artifact record (`Kind::Layer`) linked back to its source,
//! and replay it to reproduce the run — so an agent can persist & reproduce
//! interesting executions as ordinary CZYX records.
//!
//! A trace is the exact `Vec<Op>` a program actually executed ([`Vm::run_captured`]).
//! We persist it as a record body (via [`crate::opcode::encode`]) with
//! `doc_type = "layer"` (the AI-artifact kind) and a `linked_records` edge to
//! the source cell. Loading it back (via [`CodeCell::from_record`]) decodes the
//! ops and [`replay`] reproduces the observable behaviour with no access to the
//! source program.
use crate::{opcode, replay, Kind, Op, RunResult, Vm};
use cubecoords::{CubeHeader, Czyx};
use cubestore::{CubeBackend, CubeStore};
/// Store a captured execution trace as an AI-artifact record at `coord`,
/// linked to its `source` cell. `name` becomes the record title; the body is
/// the encoded trace; `doc_type` is `Kind::Layer`.
pub fn store_trace<B: CubeBackend>(
store: &mut CubeStore<B>,
coord: Czyx,
source: Czyx,
trace: &[Op],
name: &str,
) {
let mut h = CubeHeader::new();
h.title = Some(name.to_string());
h.doc_type = Some(Kind::Layer.as_str().to_string());
h.linked_records = vec![source];
h.refresh_flags();
store.put_record(coord, &h, &opcode::encode(trace));
}
/// Load a stored trace record at `coord` and replay it, reproducing the run's
/// observable behaviour (data-stack result + `SYS_TRACE` output) from the
/// stored record — no access to the original program. Returns `None` if the
/// record is missing or its body is not a valid trace.
pub fn replay_trace<B: CubeBackend>(
store: &CubeStore<B>,
coord: Czyx,
) -> Option<(RunResult, Vec<u8>)> {
let (_h, body) = store.get_record(&coord)?;
let ops = opcode::decode(&body).ok()?;
Some(replay(&ops))
}
/// Capture a run and persist it as a "golden": the executed trace stored as a
/// `Kind::Layer` record (linked to `source`) and the observed output stored as a
/// companion `golden` record (linked to the trace). Returns the observed output.
pub fn capture_golden<B: CubeBackend + Clone>(
store: &mut CubeStore<B>,
source: Czyx,
trace_coord: Czyx,
golden_coord: Czyx,
name: &str,
) -> Vec<u8> {
let mut vm = Vm::new((*store).clone());
let (_r, trace) = vm.run_captured(source);
let out = vm.output().to_vec();
store_trace(store, trace_coord, source, &trace, name);
let mut h = CubeHeader::new();
h.title = Some(format!("golden:{name}"));
h.doc_type = Some("golden".into());
h.linked_records = vec![trace_coord];
h.refresh_flags();
store.put_record(golden_coord, &h, &out);
out
}
/// Golden-trace regression check: re-run `source` fresh AND replay the stored
/// trace, and assert BOTH reproduce the recorded golden output.
/// * replay vs golden -> the stored trace is a faithful reproduction;
/// * fresh run vs golden -> the program has not drifted (or it would change
/// the observed output).
/// Returns `Ok(())` if the behavior is unchanged, else `Err(reason)`.
pub fn verify_golden<B: CubeBackend + Clone>(
store: &CubeStore<B>,
source: Czyx,
trace_coord: Czyx,
golden_coord: Czyx,
) -> Result<(), String> {
let golden = store
.get_record(&golden_coord)
.map(|(_, b)| b)
.ok_or("golden record missing")?;
let (_, replay_out) = replay_trace(store, trace_coord).ok_or("trace record missing/invalid")?;
if replay_out != golden {
return Err(format!(
"replay output differs from golden ({} vs {} bytes)",
replay_out.len(),
golden.len()
));
}
let mut vm = Vm::new((*store).clone());
let _ = vm.run(source);
let fresh = vm.output().to_vec();
if fresh != golden {
return Err(format!(
"fresh run differs from golden (program changed: {} vs {} bytes)",
fresh.len(),
golden.len()
));
}
Ok(())
}
/// A hop in the experiment-lineage graph: a record and the records it links to.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct LineageHop {
/// The record's coordinate.
pub coord: Czyx,
/// Its `doc_type` (kind: fn/kernel/layer/checkpoint/variant/golden/...).
pub doc_type: String,
/// Its human title, if any.
pub title: String,
/// The CZYX records this record links to (its edges).
pub links: Vec<Czyx>,
}
/// Walk the linked-record graph starting at `start`, following
/// `linked_records` edges (breadth-first) up to `depth` levels. Returns the
/// ordered list of reachable records — the experiment lineage chain
/// (source -> data -> weights -> trace -> golden, etc.). Each record is the
/// kind the CUBE associates; provenance is a graph walk, not a lookup.
pub fn lineage<B: CubeBackend>(
store: &CubeStore<B>,
start: Czyx,
depth: usize,
) -> Vec<LineageHop> {
let mut out = Vec::new();
let mut frontier = vec![start];
let mut seen: std::collections::HashSet<Czyx> = std::collections::HashSet::new();
for _ in 0..=depth {
if frontier.is_empty() {
break;
}
let mut next = Vec::new();
for c in frontier {
if !seen.insert(c) {
continue;
}
if let Some((h, _)) = store.get_record(&c) {
let links = h.linked_records.clone();
out.push(LineageHop {
coord: c,
doc_type: h.doc_type.clone().unwrap_or_default(),
title: h.title.clone().unwrap_or_default(),
links: links.clone(),
});
next.extend(links);
}
}
frontier = next;
}
out
}
#[cfg(test)]
mod tests {
use super::*;
use crate::{SYS_TRACE, Vm};
use cubestore::HashBackend;
#[test]
fn trace_stored_as_layer_and_replayed_without_source() {
let mut store = CubeStore::new(HashBackend::new());
let entry = Czyx::new(1, 1, 1, 7);
let ops = [
Op::Const(0), Op::Const(1), Op::Syscall(SYS_TRACE),
Op::Const(1), Op::Const(1), Op::Syscall(SYS_TRACE),
Op::Const(2), Op::Const(2), Op::Syscall(SYS_TRACE),
Op::Halt,
];
let mut h = CubeHeader::new();
h.title = Some("solitaire".into());
h.doc_type = Some("fn".into());
h.refresh_flags();
store.put_record(entry, &h, &opcode::encode(&ops));
// 1) Capture the run as a machine-code trace.
let mut vm = Vm::new(store.clone());
let (_r, trace) = vm.run_captured(entry);
let produced = vm.output().to_vec();
assert!(!trace.is_empty());
// 2) Store it as a Kind::Layer AI-artifact record linked to the source.
let trace_coord = Czyx::new(9, 0, 0, 1);
store_trace(&mut store, trace_coord, entry, &trace, "solitaire-run-1");
// 3) The record is a "layer" artifact with an edge back to the source.
let (th, _tb) = store.get_record(&trace_coord).unwrap();
assert_eq!(th.doc_type.as_deref(), Some("layer"));
assert_eq!(th.linked_records, vec![entry]);
// 4) Reproduce the run purely from the stored trace record.
let (result, replayed) = replay_trace(&store, trace_coord).unwrap();
assert_eq!(result, RunResult::Halted { top: None });
assert_eq!(replayed, produced, "stored trace must reproduce the run");
assert!(replayed.contains(&b'\n'));
}
#[test]
fn golden_regression_detects_drift() {
let mut store = CubeStore::new(HashBackend::new());
let entry = Czyx::new(1, 1, 1, 9);
let mk = |val: u8| {
vec![Op::Const(val), Op::Syscall(SYS_TRACE), Op::Halt]
};
let mut h = CubeHeader::new();
h.doc_type = Some("fn".into());
h.refresh_flags();
store.put_record(entry, &h, &opcode::encode(&mk(7)));
let trace_c = Czyx::new(9, 0, 0, 1);
let golden_c = Czyx::new(9, 0, 0, 2);
let out = capture_golden(&mut store, entry, trace_c, golden_c, "regress");
assert_eq!(out, b"\x07\n");
assert!(verify_golden(&store, entry, trace_c, golden_c).is_ok());
// Drift the program: change the emitted value -> fresh run differs.
store.put_record(entry, &h, &opcode::encode(&mk(9)));
let r = verify_golden(&store, entry, trace_c, golden_c);
assert!(r.is_err(), "program drift must be detected: {r:?}");
// The stored trace is immutable: it still replays the ORIGINAL golden.
let (_, rep) = replay_trace(&store, trace_c).unwrap();
assert_eq!(rep, b"\x07\n", "stored trace still reproduces the original run");
}
#[test]
fn lineage_walks_source_data_weights_trace() {
let mut store = CubeStore::new(HashBackend::new());
// Build experiment lineage: source -> data -> weights -> trace -> golden.
let mut mk = |coord: Czyx, doc: &str, title: &str, links: Vec<Czyx>| {
let mut h = CubeHeader::new();
h.title = Some(title.into());
h.doc_type = Some(doc.into());
h.linked_records = links;
h.refresh_flags();
store.put_record(coord, &h, &[]);
};
let source = Czyx::new(1, 1, 1, 1);
let data = Czyx::new(2, 0, 0, 1);
let weights = Czyx::new(3, 0, 0, 1);
let trace = Czyx::new(4, 0, 0, 1);
let golden = Czyx::new(5, 0, 0, 1);
mk(source, "fn", "train", vec![data, weights]);
mk(data, "data", "batch", vec![]);
mk(weights, "layer", "model-v1", vec![]);
mk(trace, "layer", "trace:train", vec![source]);
mk(golden, "golden", "golden:train", vec![trace]);
// Walk from the trace back to its lineage.
let hops = lineage(&store, trace, 4);
assert!(hops.iter().any(|h| h.coord == trace && h.doc_type == "layer"));
assert!(hops.iter().any(|h| h.coord == source && h.doc_type == "fn"));
// Depth-limited: from a leaf with no links, only itself.
let leaf = lineage(&store, data, 3);
assert_eq!(leaf.len(), 1);
assert_eq!(leaf[0].coord, data);
assert_eq!(leaf[0].links, vec![]);
}
}
+141
View File
@@ -90,6 +90,12 @@ pub struct Vm<B: CubeBackend> {
output: Vec<u8>,
/// Maximum nesting of `CALL_LINK` frames.
max_call_depth: usize,
/// Executed-instruction trace collected when [`Vm::run_captured`] is active:
/// the exact ordered sequence of `Op`s that actually ran (the "machine code
/// as it runs"). Used to reproduce behaviour without the source.
trace: Vec<Op>,
/// Whether [`Vm::run_captured`] is currently recording the instruction trace.
capturing: bool,
}
impl<B: CubeBackend> Vm<B> {
@@ -104,6 +110,8 @@ impl<B: CubeBackend> Vm<B> {
store,
output: Vec::new(),
max_call_depth: Self::MAX_CALL_DEPTH,
trace: Vec::new(),
capturing: false,
}
}
@@ -158,6 +166,9 @@ impl<B: CubeBackend> Vm<B> {
while pc < code.len() {
let op = code[pc];
if self.capturing {
self.trace.push(op);
}
match op {
Op::Nop => {}
Op::Halt => {
@@ -258,6 +269,21 @@ impl<B: CubeBackend> Vm<B> {
Ok(stack.pop())
}
/// Run `entry` AND capture the executed-instruction trace (the "machine
/// code as it runs"): the exact ordered sequence of `Op`s executed, across
/// all cells, regardless of source. Branches are already resolved (the
/// trace is the taken path) and call frames are marked by `CallLink`/`Ret`,
/// so [`Vm::replay`] can reproduce the same behaviour from the trace alone
/// — without the original program's records/code.
pub fn run_captured(&mut self, entry: Czyx) -> (RunResult, Vec<Op>) {
self.output.clear();
self.trace.clear();
self.capturing = true;
let res = self.run(entry);
self.capturing = false;
(res, std::mem::take(&mut self.trace))
}
/// Host syscall dispatch. `stack` is the *shared* data stack (callees
/// run on it) and `header` is the current cell's metadata.
fn syscall(&mut self, id: u8, header: &CubeHeader, stack: &mut Vec<u8>) -> Result<(), Fault> {
@@ -288,6 +314,87 @@ impl<B: CubeBackend> Vm<B> {
}
}
/// Replay-time host syscall dispatch (no store/header). Store-independent
/// syscalls (`SYS_TRACE`, `SYS_NOP`) are reproduced exactly; syscalls that
/// would need the live store/header are a no-op in trace-replay (they don't
/// contribute to the observable `SYS_TRACE` output).
fn replay_syscall(id: u8, stack: &mut Vec<u8>, output: &mut Vec<u8>) {
match id {
SYS_NOP => {}
SYS_TRACE => {
let mut line: Vec<u8> = std::mem::take(stack);
line.push(b'\n');
output.extend_from_slice(&line);
}
_ => {}
}
}
/// Replay a captured instruction trace, reproducing the program's actions
/// (data-stack result + `SYS_TRACE` output) WITHOUT accessing any of the
/// original program's records/bytecode. This is the "duplicate behaviour
/// from machine code as it ran" path. Store-independent syscalls
/// (`SYS_TRACE`, `SYS_NOP`) are reproduced exactly; syscalls that would
/// need the live store/header are a no-op (those don't contribute to the
/// observable `SYS_TRACE` output).
pub fn replay(trace: &[Op]) -> (RunResult, Vec<u8>) {
let mut stack: Vec<u8> = Vec::new();
let mut frames: Vec<[u8; LOCALS]> = vec![[0; LOCALS]];
let mut output: Vec<u8> = Vec::new();
let mut top = None;
for &op in trace {
match op {
Op::Nop => {}
Op::Halt => {
top = stack.pop();
break;
}
Op::Const(v) => stack.push(v),
Op::Load(i) => stack.push(frames.last_mut().unwrap()[(i as usize) % LOCALS]),
Op::Store(i) => {
let v = stack.pop().unwrap_or(0);
frames.last_mut().unwrap()[(i as usize) % LOCALS] = v;
}
Op::Add | Op::Sub | Op::Mul | Op::Div | Op::Mod | Op::And | Op::Or | Op::Xor => {
let (b, a) = (stack.pop(), stack.pop());
stack.push(bin_op(op, a.unwrap_or(0), b.unwrap_or(0)).unwrap_or(0));
}
Op::Shl | Op::Shr => {
let (b, a) = (stack.pop(), stack.pop());
stack.push(shift_op(op, a.unwrap_or(0), b.unwrap_or(0)));
}
Op::Eq | Op::Ne | Op::Lt | Op::Gt | Op::Le | Op::Ge => {
let (b, a) = (stack.pop(), stack.pop());
stack.push(cmp_op(op, a.unwrap_or(0), b.unwrap_or(0)));
}
// Jumps already reflect the taken path in a captured trace:
// keep the stack effect, don't re-branch.
Op::Jmp(_) => {}
Op::Jz(_) | Op::Jnz(_) => {
let _ = stack.pop();
}
Op::CallLink(_) => frames.push([0; LOCALS]),
Op::Ret => {
if frames.len() > 1 {
frames.pop();
} else {
top = stack.pop();
break;
}
}
Op::Syscall(id) => replay_syscall(id, &mut stack, &mut output),
Op::Dup => {
let v = *stack.last().unwrap_or(&0);
stack.push(v);
}
Op::Drop => {
stack.pop();
}
}
}
(RunResult::Halted { top }, output)
}
fn b2(v: Option<u8>, cell: Czyx, pc: usize) -> Result<u8, (Czyx, usize, Fault)> {
v.ok_or((cell, pc, Fault::PcOutOfRange))
}
@@ -522,4 +629,38 @@ mod tests {
let mut vm = Vm::new(store);
assert_eq!(vm.run(entry), RunResult::Halted { top: Some(3) });
}
#[test]
fn capture_and_replay_reproduces_behavior_without_source() {
let mut store = CubeStore::new(HashBackend::new());
let entry = Czyx::new(1, 1, 1, 7);
// A tiny "solitaire" state machine: draw cards; even -> foundation
// (score++), odd -> waste. Emit (score, waste) after each play.
let ops = [
Op::Const(0), Op::Const(1), Op::Syscall(SYS_TRACE), // draw odd -> waste
Op::Const(1), Op::Const(1), Op::Syscall(SYS_TRACE), // draw even -> score
Op::Const(1), Op::Const(2), Op::Syscall(SYS_TRACE), // draw odd -> waste
Op::Const(2), Op::Const(2), Op::Syscall(SYS_TRACE), // draw even -> score
Op::Halt,
];
let mut h = CubeHeader::new();
h.title = Some("solitaire".into());
h.doc_type = Some("fn".into());
h.refresh_flags();
store.put_record(entry, &h, &crate::opcode::encode(&ops));
let mut vm = Vm::new(store);
// Capture: run the program AND record the exact machine code executed.
let (result, trace) = vm.run_captured(entry);
assert_eq!(result, RunResult::Halted { top: None });
assert!(!trace.is_empty(), "capture must record the executed ops");
let produced = vm.output().to_vec();
assert_eq!(produced, b"\x00\x01\n\x01\x01\n\x01\x02\n\x02\x02\n");
// Replay: reproduce the SAME actions purely from the captured trace,
// with NO access to the original program's records/bytecode.
let (result2, replayed) = replay(&trace);
assert_eq!(result2, RunResult::Halted { top: None });
assert_eq!(replayed, produced, "replay must reproduce the game's behaviour");
}
}
+147 -1
View File
@@ -160,6 +160,84 @@ impl TriEnc {
}
(control, ascii)
}
/// Pack 6 ASCII bytes plus a full 16-bit per-word flag/metadata field.
///
/// Layout (high -> low): `[ WordFlags: u16 (bits 48-63) ][ 48 ascii bits ]`.
/// This promotes the 12 formerly-unused bits + the 4 control bits into a real
/// per-word metadata stripe, mapped by [`WordFlags`]. Backward compatible:
/// `pack_6(control)` is exactly `pack_flags(WordFlags(control as u16 << 12),
/// ascii)` (the old 4 control bits occupy the high nibble of the flags field).
#[inline]
pub fn pack_flags(flags: WordFlags, ascii: [u8; 6]) -> TriWord {
let mut v: u64 = ((flags.0 as u64) & 0xFFFF) << 48;
for (i, b) in ascii.iter().enumerate() {
v |= (*b as u64) << (8 * (5 - i));
}
TriWord(v)
}
/// Unpack a word into its 16-bit flags field and 6 data bytes.
#[inline]
pub fn unpack_flags(word: TriWord) -> (WordFlags, [u8; 6]) {
let v = word.0;
let flags = WordFlags(((v >> 48) & 0xFFFF) as u16);
let mut ascii = [0u8; 6];
for (i, b) in ascii.iter_mut().enumerate() {
*b = ((v >> (8 * (5 - i))) & 0xFF) as u8;
}
(flags, ascii)
}
}
/// The 16-bit **per-word** flag/metadata field co-packed with 6 data bytes in a
/// tri-channel [`TriWord`] (bits 48-63). Distinct from [`HeaderFlags`], which is
/// the per-*record* header; this is the inline per-word metadata stripe.
#[derive(Copy, Clone, Eq, PartialEq, Debug, Default)]
pub struct WordFlags(pub u16);
impl WordFlags {
/// Packing arrangement (00 tri6, 01 pair, 10 triad, 11 quad).
pub const ARR_MASK: u16 = 0b0000_0000_0000_0011;
/// Flag: first word of a record.
pub const START_RECORD: u16 = 0b0000_0000_0000_0100;
/// Flag: last word of a record. (Pair with the PDF's end-of-header marker.)
pub const END_RECORD: u16 = 0b0000_0000_0000_1000;
/// Flag: this word is header/metadata (vs payload).
pub const IS_HEADER: u16 = 0b0000_0000_0001_0000;
/// Flag: record continues across the next word.
pub const CONTINUATION: u16 = 0b0000_0000_0010_0000;
/// Data type (00 text, 01 binary, 10 code, 11 ai-feature-vector).
pub const TYPE_MASK: u16 = 0b0000_0000_1100_0000;
/// Permission: root-only.
pub const PERM_ROOT: u16 = 0b0000_0001_0000_0000;
/// Permission: local-user owner.
pub const PERM_USER: u16 = 0b0000_0010_0000_0000;
/// Flag: association edge present on this word.
pub const ASSOC_EDGE: u16 = 0b0000_0100_0000_0000;
/// Flag: this group is encrypted (cubecrypt key/tweak selector).
pub const ENCRYPTED: u16 = 0b0000_1000_0000_0000;
/// Flag: this group is compressed.
pub const COMPRESSED: u16 = 0b0001_0000_0000_0000;
/// Flag: this group uses the high-entropy/stego transform (obfuscation layer).
pub const STEGO: u16 = 0b0010_0000_0000_0000;
/// Flag: this group is a Null/cube lookup marker.
pub const NULL_LOOKUP: u16 = 0b0100_0000_0000_0000;
/// Flag: checksum/parity present for this group.
pub const CHECKSUM: u16 = 0b1000_0000_0000_0000;
/// Construct from a raw bitmask.
#[inline]
pub const fn from_bits(bits: u16) -> Self { WordFlags(bits) }
/// The raw bitmask.
#[inline]
pub const fn bits(&self) -> u16 { self.0 }
/// Set a flag bit.
#[inline]
pub fn set(&mut self, flag: u16) { self.0 |= flag; }
/// Test a flag bit.
#[inline]
pub fn has(&self, flag: u16) -> bool { self.0 & flag != 0 }
}
/// Header flag bits, mirroring the PDF's title/type/date/size/permission
@@ -185,6 +263,13 @@ impl HeaderFlags {
pub const PERM_REMOTE_USER: u16 = 1 << 6;
/// Flag 8: has outgoing association links.
pub const HAS_ASSOCIATIONS: u16 = 1 << 7;
/// Flag 9: carries a POSIX `path` metatag (the original filesystem path
/// the record was migrated from). Lets the associative query layer
/// reconstruct a directory hierarchy WITHOUT the filesystem needing to
/// support arbitrary recursive nesting — the path is data, addressed by
/// flag, exactly as the source PDF prescribes ("operate on CZYX records
/// and Null-space flags rather than paths and inodes").
pub const HAS_PATH: u16 = 1 << 8;
// Flag 255 (conceptual end-of-header) is represented out-of-band by the
// record serializer; there is no bit for it.
@@ -231,6 +316,11 @@ impl HeaderFlags {
pub struct CubeHeader {
/// Flag bits (derived; kept in sync by the accessors).
pub flags: HeaderFlags,
/// Per-word 16-bit tri-channel flag/metadata field (bits 48-63). Distinct
/// from `flags` (the per-RECORD header): this is the inline per-word
/// metadata stripe (see [`WordFlags`]) carried by records that use the
/// tri-channel word codec.
pub word_flags: WordFlags,
/// Flag 1: human title.
pub title: Option<String>,
/// Flag 2: document type (like a file extension).
@@ -245,6 +335,13 @@ pub struct CubeHeader {
pub owner_remote_user: Option<String>,
/// Association links to other records (flags 519).
pub linked_records: Vec<Czyx>,
/// POSIX path metatag: the original filesystem path this record was
/// migrated from (e.g. "/etc/network/interfaces"). Empty/absent means the
/// record has no path identity. When present, the `HAS_PATH` flag bit is
/// set so `scan_by_path`/`query_by_path` can recover the hierarchy from
/// metatags alone (see cubestore). This is the PDF's "path is a view over
/// coordinates + flags" made concrete.
pub path: Option<String>,
/// Total local accesses (from the PDF's association/permission flags).
pub total_accesses: u64,
/// Total remote accesses.
@@ -292,8 +389,14 @@ impl CubeHeader {
if !self.linked_records.is_empty() {
f |= HeaderFlags::HAS_ASSOCIATIONS;
}
if self.path.is_some() {
f |= HeaderFlags::HAS_PATH;
}
// Preserve spare/out-of-band flag bits (bits 8..=15) that are not
// derived from structured fields.
// derived from structured fields. This includes
// `cubecrypt::HEADER_FLAG_ENCRYPTED` (bit 12) and the behavior-descriptor
// field (bits 13..=15, see `cubecode::Behavior`) so both survive an
// encode→decode→refresh round-trip.
const DERIVED_BITS: u16 = 0x00FF;
f |= self.flags.bits() & !DERIVED_BITS;
self.flags = HeaderFlags::from_bits(f);
@@ -346,3 +449,46 @@ mod tests {
assert!(!h.flags.has(HeaderFlags::DOC_TYPE));
}
}
#[cfg(test)]
mod wordflags_tests {
use super::*;
#[test]
fn roundtrip_flags_and_data() {
let cases = [
(WordFlags(0), *b"abcdef"),
(WordFlags::from_bits(WordFlags::ENCRYPTED | WordFlags::START_RECORD), *b"secret"),
(WordFlags::from_bits(WordFlags::COMPRESSED | WordFlags::ASSOC_EDGE), *b"123456"),
(WordFlags::from_bits(WordFlags::ARR_MASK & 0b10 | WordFlags::NULL_LOOKUP), *b"AB\x00\xff\xee\x7f"),
];
for (flags, data) in cases {
let w = TriEnc::pack_flags(flags, data);
let (f, d) = TriEnc::unpack_flags(w);
assert_eq!(f.0, flags.0, "flags must round-trip");
assert_eq!(d, data, "data must round-trip");
}
}
#[test]
fn flags_word_is_16_bits_and_data_is_6_bytes() {
let w = TriEnc::pack_flags(WordFlags::from_bits(0xFFFF), [0xFF; 6]);
assert_eq!(w.0 >> 48, 0xFFFF, "flags occupy the high 16 bits");
let (_, d) = TriEnc::unpack_flags(w);
assert_eq!(d, [0xFF; 6]);
// data occupies the low 48 bits (bits 0-47); no overlap with flags
assert_eq!(TriEnc::pack_flags(WordFlags::from_bits(0x0001), [0x00; 6]).0 >> 63, 0);
}
#[test]
fn backward_compat_pack_6_equals_flags_high_nibble() {
let ctrl = 0xAu8;
let ascii = *b"abcdef";
let a = TriEnc::pack_6(ctrl, ascii);
let b = TriEnc::pack_flags(WordFlags::from_bits((ctrl as u16) << 12), ascii);
assert_eq!(a.0, b.0, "pack_6(control) == pack_flags(control << 12)");
// and the old control nibble is the high nibble of the flags field
let (f, _) = TriEnc::unpack_flags(a);
assert_eq!((f.0 >> 12) & 0x0F, ctrl as u16);
}
}
+11
View File
@@ -0,0 +1,11 @@
[package]
name = "cubedbt"
version = "0.1.0"
edition.workspace = true
license.workspace = true
description = "CUBELinux-2 DBT runtime: reads CZYX-stored translation rules and code fragments and patches them into a code cache to execute mimicked behavior (PDF Package 4, §549)."
[dependencies]
cubecoords = { path = "../cubecoords" }
cubestore = { path = "../cubestore" }
cubecode = { path = "../cubecode" }
+421
View File
@@ -0,0 +1,421 @@
//! CUBELinux-2 DBT (dynamic binary translation) runtime.
//!
//! Per the PDF (Package 4, §549): *"cubedbt: a runtime that reads CZYXstored
//! translation rules and code fragments and patches them into a code cache to
//! execute mimicked behavior."*
//!
//! This crate is the substrate for the PDF's "watch a binary, archive its
//! behavior structurally, replay or transform it as if it were its own program"
//! story (§554). It is deliberately dependency-free and operates on the real
//! `CubeStore` / `CodeCell` / `Vm` types from the rest of the workspace, so it
//! is exercisable today on `HashBackend` and slots into `ConcurrentStore`
//! later without an API change.
//!
//! Model
//! -----
//! * A **translation rule** is a CZYX record (`Kind::Variant`) whose body is a
//! serialized [`TranslationRule`]. It names an *op kind* it can replace and
//! carries a *replacement fragment* (a `Vec<Op>`). The rule record links to
//! the original code cell it may substitute (the association edge).
//! * A **code cache** is the patched working set: `patch()` rewrites an
//! original `CodeCell`'s bytecode under the eligible rules and writes the
//! result back as a new record (preserving the original header's kind/title
//! and behavior descriptors, so it runs through the same `Vm`).
//! * A **DBT runtime** discovers rules in a store and can `mimic` a target
//! coordinate: fetch original → apply rules → run the patched version in the
//! `Vm`, returning the `RunResult`.
use cubecode::{opcode::Op, CodeCell, Kind, Vm};
use cubecoords::{CubeHeader, Czyx};
use cubestore::{CubeBackend, CubeStore, HashBackend};
/// `C` axis band where DBT translation rules are stored, kept distinct from the
/// `c210` OS-kernel band (`cubecode::C_OS_KERNEL`) and the `c200` snapshot data.
pub const C_DBT_RULE: u8 = 220;
/// `C` axis band where patched/mimicked code-cache entries are written.
pub const C_DBT_CACHE: u8 = 221;
/// The `Op` kinds a translation rule can target.
#[derive(Copy, Clone, Debug, PartialEq, Eq)]
pub enum OpClass {
Const,
Add,
Sub,
Mul,
Div,
Mod,
And,
Or,
Xor,
Shl,
Shr,
Eq,
Ne,
Lt,
Gt,
Le,
Ge,
Load,
Store,
CallLink,
Any,
}
impl OpClass {
/// Classify a live [`Op`].
pub fn of(op: &Op) -> OpClass {
match op {
Op::Const(_) => OpClass::Const,
Op::Add => OpClass::Add,
Op::Sub => OpClass::Sub,
Op::Mul => OpClass::Mul,
Op::Div => OpClass::Div,
Op::Mod => OpClass::Mod,
Op::And => OpClass::And,
Op::Or => OpClass::Or,
Op::Xor => OpClass::Xor,
Op::Shl => OpClass::Shl,
Op::Shr => OpClass::Shr,
Op::Eq => OpClass::Eq,
Op::Ne => OpClass::Ne,
Op::Lt => OpClass::Lt,
Op::Gt => OpClass::Gt,
Op::Le => OpClass::Le,
Op::Ge => OpClass::Ge,
Op::Load(_) => OpClass::Load,
Op::Store(_) => OpClass::Store,
Op::CallLink(_) => OpClass::CallLink,
_ => OpClass::Any,
}
}
}
/// A translation rule: a named, CZYX-addressed mapping from an op class to a
/// replacement bytecode fragment. Serialized into a rule record's body.
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct TranslationRule {
/// Human name (also the rule record's title).
pub name: String,
/// Which op class this rule can substitute.
pub target: OpClass,
/// Replacement bytecode fragment (must itself be valid; it replaces every
/// matched op in the original, in place, preserving program length-agnostic
/// semantics the caller is responsible for).
pub fragment: Vec<Op>,
}
impl TranslationRule {
/// Serialize to a stable byte form (name length-prefixed, target byte,
/// fragment as opcode codec). No external deps.
pub fn to_bytes(&self) -> Vec<u8> {
let mut out = Vec::new();
let nb = self.name.as_bytes();
out.push(nb.len() as u8);
out.extend_from_slice(nb);
out.push(self.target as u8);
out.push(self.fragment.len() as u8);
for op in &self.fragment {
out.extend_from_slice(&cubecode::opcode::encode(&[*op]));
}
out
}
/// Inverse of [`to_bytes`]. Returns `None` on any malformed input.
pub fn from_bytes(b: &[u8]) -> Option<TranslationRule> {
let mut i = 0;
let nlen = *b.get(i)? as usize;
i += 1;
if i + nlen > b.len() {
return None;
}
let name = String::from_utf8(b[i..i + nlen].to_vec()).ok()?;
i += nlen;
let target = *b.get(i)?;
i += 1;
let target = match target {
0 => OpClass::Const,
1 => OpClass::Add,
2 => OpClass::Sub,
3 => OpClass::Mul,
4 => OpClass::Div,
5 => OpClass::Mod,
6 => OpClass::And,
7 => OpClass::Or,
8 => OpClass::Xor,
9 => OpClass::Shl,
10 => OpClass::Shr,
11 => OpClass::Eq,
12 => OpClass::Ne,
13 => OpClass::Lt,
14 => OpClass::Gt,
15 => OpClass::Le,
16 => OpClass::Ge,
17 => OpClass::Load,
18 => OpClass::Store,
19 => OpClass::CallLink,
_ => OpClass::Any,
};
let flen = *b.get(i)? as usize;
i += 1;
let mut fragment = Vec::new();
for _ in 0..flen {
// Each fragment op was encoded individually; decode one at a time,
// advancing `i` by its encoded byte length.
let one = decode_one(&b[i..])?;
i += one.len;
fragment.push(one.op);
}
Some(TranslationRule {
name,
target,
fragment,
})
}
}
/// One decoded opcode plus its encoded length, used by `from_bytes`.
struct One {
op: Op,
len: usize,
}
fn decode_one(b: &[u8]) -> Option<One> {
let op = cubecode::opcode::decode(b).ok()?;
let op = op.into_iter().next()?;
let len = cubecode::opcode::encode(&[op]).len();
Some(One { op, len })
}
/// A patched working set ("code cache") inside a [`CubeStore`].
///
/// `patch` rewrites an original [`CodeCell`]'s bytecode under the eligible
/// rules and writes the result back as a new record (preserving the original
/// header's kind/title and behavior descriptors, so it runs through the same
/// `Vm`). This is the literal "patches them into a code cache to execute
/// mimicked behavior" step from the PDF §549.
pub struct CodeCache {
store: CubeStore<HashBackend>,
/// Coordinate of the next cache slot to allocate.
next_x: u8,
}
impl CodeCache {
/// Build an empty cache over a fresh in-memory backend.
pub fn new() -> Self {
CodeCache {
store: CubeStore::new(HashBackend::new()),
next_x: 1,
}
}
/// Patch `original` under `rules`: every op whose class matches a rule's
/// target (or whose rule target is `Any`) is replaced by that rule's
/// fragment. The rewritten cell is stored at a fresh cache coordinate and
/// returned (with its new label). The original store is untouched.
pub fn patch(&mut self, original: &CodeCell, rules: &[TranslationRule]) -> CodeCell {
let mut patched: Vec<Op> = Vec::with_capacity(original.code.len());
for op in &original.code {
let class = OpClass::of(op);
let mut applied = false;
for r in rules {
if r.target == class || r.target == OpClass::Any {
patched.extend_from_slice(&r.fragment);
applied = true;
break;
}
}
if !applied {
patched.push(*op);
}
}
let label = Czyx::new(C_DBT_CACHE, 1, 1, self.next_x);
self.next_x = self.next_x.wrapping_add(1).max(1);
let (h, body) = cache_header_for(original, &cubecode::opcode::encode(&patched));
self.store.put_record(label, &h, &body);
CodeCell::from_record(label, &h, &body).expect("patched cell is always valid bytecode")
}
/// Run a patched cache entry in the VM, returning its result.
pub fn run(&self, cell: &CodeCell) -> cubecode::RunResult {
let mut vm = Vm::new(self.store.clone());
vm.run(cell.label)
}
/// Borrow the backing store (e.g. to persist or inspect).
pub fn store(&self) -> &CubeStore<HashBackend> {
&self.store
}
}
impl Default for CodeCache {
fn default() -> Self {
Self::new()
}
}
/// Build a cache record header that preserves the original's kind/title and
/// behavior descriptors, but marks it a `Variant` (a mimicked implementation).
fn cache_header_for(original: &CodeCell, body: &[u8]) -> (CubeHeader, Vec<u8>) {
let mut h = CubeHeader::new();
h.title = original.name().map(|s| format!("mimic:{}", s));
h.doc_type = Some(Kind::Variant.as_str().into());
h.size_bytes = Some(body.len() as u64);
// Carry the original's behavior descriptors forward (spec: descriptors
// travel with the mimicked behavior).
h.flags.0 |= original.header.flags.0 & cubecode::HEADER_FLAG_BEHAVIOR;
h.refresh_flags();
(h, body.to_vec())
}
/// The DBT runtime: discovers CZYX-stored translation rules and can `mimic` a
/// target code cell by patching it under those rules and running the result.
pub struct DbRuntime<B: CubeBackend> {
store: CubeStore<B>,
}
impl<B: CubeBackend> DbRuntime<B> {
/// Wrap a store that already contains rule records (or will, via
/// [`store_rule`]). The runtime is read-only over this store.
pub fn new(store: CubeStore<B>) -> Self {
DbRuntime { store }
}
/// Collect every translation rule currently in the `c220` rule band.
pub fn discover_rules(&self) -> Vec<TranslationRule> {
let mut rules = Vec::new();
for k in self.store.keys() {
if k.c != C_DBT_RULE {
continue;
}
if let Some((_, body)) = self.store.get_record(&k) {
if let Some(r) = TranslationRule::from_bytes(&body) {
rules.push(r);
}
}
}
rules
}
/// Fetch a code cell by coordinate from the wrapped store.
pub fn fetch(&self, label: Czyx) -> Option<CodeCell> {
let (h, b) = self.store.get_record(&label)?;
CodeCell::from_record(label, &h, &b)
}
/// Mimic `target`: fetch it, apply all discovered rules, patch into a fresh
/// cache, and run the patched version — returning the `RunResult`. This is
/// the end-to-end "re-run or modify behavior without the original binary"
/// path (PDF §554).
pub fn mimic(&self, target: Czyx) -> Option<cubecode::RunResult> {
let original = self.fetch(target)?;
let rules = self.discover_rules();
let mut cache = CodeCache::new();
let patched = cache.patch(&original, &rules);
Some(cache.run(&patched))
}
}
/// Store a translation rule into the `c220` rule band at a fresh coordinate.
/// Returns the coordinate it was written to.
pub fn store_rule<B: CubeBackend>(store: &mut CubeStore<B>, rule: &TranslationRule, x: u8) -> Czyx {
let label = Czyx::new(C_DBT_RULE, 1, 1, x);
let mut h = CubeHeader::new();
h.title = Some(rule.name.clone());
h.doc_type = Some(Kind::Variant.as_str().into());
h.size_bytes = Some(rule.to_bytes().len() as u64);
h.refresh_flags();
store.put_record(label, &h, &rule.to_bytes());
label
}
#[cfg(test)]
mod tests {
use super::*;
use cubecode::opcode::Op;
use cubecode::RunResult;
use cubecoords::Czyx;
fn sample_cell(coord: Czyx, code: Vec<Op>) -> CodeCell {
let mut h = CubeHeader::new();
h.title = Some("sample".into());
h.doc_type = Some(Kind::Fn.as_str().into());
h.refresh_flags();
CodeCell::from_record(coord, &h, &cubecode::opcode::encode(&code))
.expect("sample is valid bytecode")
}
#[test]
fn rule_round_trips_through_bytes() {
let r = TranslationRule {
name: "double-add".into(),
target: OpClass::Add,
fragment: vec![Op::Const(2), Op::Mul],
};
let bytes = r.to_bytes();
let back = TranslationRule::from_bytes(&bytes).expect("rule decodes");
assert_eq!(r, back);
}
#[test]
fn cache_patches_and_runs() {
// Original: Const 3, Const 4, Add, Halt => 3+4 = 7.
let orig = sample_cell(
Czyx::new(1, 1, 1, 1),
vec![Op::Const(3), Op::Const(4), Op::Add, Op::Halt],
);
// Rule: replace `Add` with `Const 2, Mul` => (a)*(2). With a=3,b=4:
// naive in-place substitution yields Const3, Const4, Const2, Mul =>
// 4*2 = 8. This proves the patched fragment is what runs.
let rule = TranslationRule {
name: "mul-by-2".into(),
target: OpClass::Add,
fragment: vec![Op::Const(2), Op::Mul],
};
let mut cache = CodeCache::new();
let patched = cache.patch(&orig, &[rule]);
assert_eq!(patched.kind(), Kind::Variant, "cache entry is a Variant");
match cache.run(&patched) {
RunResult::Halted { top: Some(v) } => assert_eq!(v, 8, "patched semantics differ"),
other => panic!("patched run failed: {:?}", other),
}
}
#[test]
fn runtime_mimic_end_to_end() {
let mut store = CubeStore::new(HashBackend::new());
// Original target at c1.
let target = Czyx::new(1, 1, 1, 1);
let orig = sample_cell(target, vec![Op::Const(10), Op::Const(5), Op::Sub, Op::Halt]);
store.put_record(orig.label, &orig.header, &orig.body());
// Rule in the c220 band: replace Sub with Add (10+5=15 instead of 5).
let rule = TranslationRule {
name: "sub->add".into(),
target: OpClass::Sub,
fragment: vec![Op::Add],
};
store_rule(&mut store, &rule, 1);
let rt = DbRuntime::new(store);
let rules = rt.discover_rules();
assert_eq!(rules.len(), 1, "rule discovered in c220 band");
match rt.mimic(target) {
Some(RunResult::Halted { top: Some(v) }) => assert_eq!(v, 15, "mimic applied rule"),
other => panic!("mimic failed: {:?}", other),
}
}
#[test]
fn mimic_without_rules_matches_original() {
let mut store = CubeStore::new(HashBackend::new());
let target = Czyx::new(1, 1, 1, 2);
let orig = sample_cell(target, vec![Op::Const(7), Op::Const(2), Op::Mul, Op::Halt]);
store.put_record(orig.label, &orig.header, &orig.body());
let rt = DbRuntime::new(store);
match rt.mimic(target) {
Some(RunResult::Halted { top: Some(v) }) => assert_eq!(v, 14),
other => panic!("mimic without rules should run original: {:?}", other),
}
}
}
+2
View File
@@ -448,6 +448,8 @@ impl<B: CubeBackend + 'static> Filesystem for CubeFuse<B> {
kind: Kind::Directory,
size: 0,
mode: (mode & 0o7777) as u16,
word_flags: 0,
header_flags: 0,
uid: req.uid(),
gid: req.gid(),
created_at: 0,
+27 -4
View File
@@ -106,6 +106,10 @@ pub struct Attr {
pub created_at: u64,
/// Number of hard links: 1 for files, 2 for directories (`.` and `..`).
pub nlink: u32,
/// Per-word 16-bit flag field of the backing record (header tag 14). 0 for dirs.
pub word_flags: u16,
/// Per-record HeaderFlags of the backing record (header tag 12). 0 for dirs.
pub header_flags: u16,
}
/// The filesystem.
@@ -234,14 +238,14 @@ impl<B: CubeBackend> CubeFs<B> {
let kind = self.kind_of(p)?;
let coord = parsed.prefix_coord();
let acl = self.acl_of(coord);
let (size, created_at) = if kind == Kind::File {
let (size, created_at, word_flags, header_flags) = if kind == Kind::File {
let c = parsed.czyx().expect("file implies full coord");
match self.store.get_record(&c) {
Some((h, body)) => (body.len() as u64, h.created_at.unwrap_or(0)),
None => (0, 0),
Some((h, body)) => (body.len() as u64, h.created_at.unwrap_or(0), h.word_flags.bits(), h.flags.bits()),
None => (0, 0, 0, 0),
}
} else {
(0, 0)
(0, 0, 0, 0)
};
// Decision: directories report 0o755 unless an explicit ACL exists,
// because the file default (0644) would make every directory
@@ -262,6 +266,8 @@ impl<B: CubeBackend> CubeFs<B> {
gid: acl.gid,
created_at,
nlink: if kind == Kind::Directory { 2 } else { 1 },
word_flags,
header_flags,
})
}
@@ -771,6 +777,23 @@ mod tests {
assert!(f.read(p, 0, 1, 2000, 2000).is_ok());
}
#[test]
fn getattr_surfaces_word_flags_and_header_flags() {
use cubecoords::WordFlags;
let mut f = fs();
let mut h = CubeHeader::new();
h.doc_type = Some("fn".into());
h.word_flags = WordFlags::from_bits(
WordFlags::START_RECORD | WordFlags::END_RECORD | WordFlags::IS_HEADER | (0b10 << 6),
);
h.refresh_flags();
f.store_mut().put_record(cubecoords::Czyx::new(1, 1, 1, 1), &h, b"hi");
let a = f.getattr("/c001/z001/y001/x001").unwrap();
assert_eq!(a.word_flags & WordFlags::START_RECORD, WordFlags::START_RECORD);
assert_eq!((a.word_flags & WordFlags::TYPE_MASK) >> 6, 0b10, "type=code");
assert_eq!(a.header_flags & cubecoords::HeaderFlags::DOC_TYPE, cubecoords::HeaderFlags::DOC_TYPE);
}
#[test]
fn directories_default_to_traversable() {
let mut f = fs();
+528 -12
View File
@@ -151,7 +151,6 @@ pub struct CubeStore<B: CubeBackend> {
/// encoding is used to avoid a serde dependency at Package 1.)
mod record_codec {
use cubecoords::{CubeHeader, Czyx};
use std::collections::HashMap;
// Compact, dependency-free encoding of the header.
// Fields are written in a fixed tag-length-value stream so unknown
@@ -170,6 +169,11 @@ mod record_codec {
// 9 total_remote_accesses (u64 le)
// 10 last_access (u64 le)
// 11 last_remote_access (u64 le)
// 14 word_flags (u16 le) — per-word 16-bit tri-channel WordFlags
// 13 path (utf8) — POSIX path metatag (the original filesystem path a
// record was migrated from). Carries the directory hierarchy as DATA
// so the associative query layer can reconstruct nesting from flags.
// (Tag 12, raw flag bits, is documented at its emit site below.)
pub fn encode_header(h: &CubeHeader) -> Vec<u8> {
let mut out = Vec::new();
@@ -213,6 +217,9 @@ mod record_codec {
if let Some(a) = h.last_remote_access {
put_u64(&mut out, 11, a);
}
if let Some(p) = &h.path {
put_utf8(&mut out, 13, p);
}
// Tag 12: raw flag bits. Serializes out-of-band/spare bits (e.g.
// `cubecrypt::HEADER_FLAG_ENCRYPTED`) that are not derived from
// structured fields, so they survive an encode/decode round-trip.
@@ -220,6 +227,11 @@ mod record_codec {
out.push(12);
out.extend_from_slice(&h.flags.bits().to_le_bytes());
}
// Tag 14: per-word 16-bit tri-channel WordFlags field (see cubecoords).
if h.word_flags.bits() != 0 {
out.push(14);
out.extend_from_slice(&h.word_flags.bits().to_le_bytes());
}
out
}
@@ -297,6 +309,11 @@ mod record_codec {
h.last_remote_access = Some(v);
b = rest;
}
13 => {
let (v, rest) = take_utf8(b)?;
h.path = Some(v);
b = rest;
}
12 => {
if b.len() < 2 {
return None;
@@ -305,6 +322,14 @@ mod record_codec {
h.flags = cubecoords::HeaderFlags::flags_from_bits(raw);
b = &b[2..];
}
14 => {
if b.len() < 2 {
return None;
}
let raw = u16::from_le_bytes([b[0], b[1]]);
h.word_flags = cubecoords::WordFlags::from_bits(raw);
b = &b[2..];
}
_ => return None, // unknown tag -> reject (strict at Package 1)
}
}
@@ -344,12 +369,6 @@ mod record_codec {
let s = String::from_utf8(rest[..n].to_vec()).ok()?;
Some((s, &rest[n..]))
}
/// Keep `HashMap` referenced so the dependency is explicit in this module
/// even though the codec itself is generic over bytes. (Prevents an
/// unused-import warning if the backend type changes.)
#[allow(dead_code)]
pub(crate) fn _assert_backend_assoc(_: &HashMap<u32, Vec<u8>>) {}
}
impl<B: CubeBackend> CubeStore<B> {
@@ -387,20 +406,55 @@ impl<B: CubeBackend> CubeStore<B> {
}
/// Fetch and split a record into `(header, body)`.
///
/// Records written through the record path carry a TLV envelope
/// (`u32 header-len | header | body`). Records written through the *raw*
/// path (`put_raw` / the daemon's `rawput` verb, used by OS-layer services)
/// carry no envelope at all.
///
/// Historically a non-enveloped payload made this return `None`, which
/// callers such as `cubefs`'s `getattr`/`read` and the daemon's `stat` verb
/// translate into "size 0 / empty file". A record holding real bytes was
/// therefore *silently invisible* through the filesystem while `rawget`
/// happily returned its contents — the OS-in-CUBE migration hit exactly
/// this (2026-08-13): every `rawput` OS record listed as a 0-byte file.
///
/// Losing data silently is never the right failure mode, so a payload that
/// is not a well-formed envelope is now surfaced as a raw body under a
/// synthesized header. `rawput` data becomes readable through the
/// filesystem, and no caller has to special-case the two write paths.
pub fn get_record(&self, label: &Czyx) -> Option<(CubeHeader, Vec<u8>)> {
let raw = self.backend.get(label)?;
// Fall back to treating the payload as a raw (un-enveloped) body when
// it cannot be parsed as `len | header | body`.
let raw_fallback = |bytes: &[u8]| {
let mut h = CubeHeader::new();
h.size_bytes = Some(bytes.len() as u64);
// Keep the synthesized header's flag bits consistent with its
// fields, so associative queries (`scan_by_flag(SIZE_BYTES)`) see
// raw records too. Only the derived low bits are recomputed; any
// out-of-band bits already on the payload-derived header (none
// here, since raw payloads carry no header) are preserved by
// `refresh_flags`.
h.refresh_flags();
Some((h, bytes.to_vec()))
};
if raw.len() < 4 {
return None;
return raw_fallback(&raw);
}
let mut len = [0u8; 4];
len.copy_from_slice(&raw[..4]);
let hlen = u32::from_le_bytes(len) as usize;
if raw.len() < 4 + hlen {
return None;
return raw_fallback(&raw);
}
match record_codec::decode_header(&raw[4..4 + hlen]) {
Some(hdr) => {
let body = raw[4 + hlen..].to_vec();
Some((hdr, body))
}
None => raw_fallback(&raw),
}
let hdr = record_codec::decode_header(&raw[4..4 + hlen])?;
let body = raw[4 + hlen..].to_vec();
Some((hdr, body))
}
/// Raw backend write (non-record payloads, e.g. ACL/xattr/volume buckets).
@@ -472,6 +526,167 @@ impl<B: CubeBackend> CubeStore<B> {
out.sort();
out
}
/// PDF Package 2 API: `scan_by_flag` — associative storage lookup.
///
/// Returns every coordinate whose decoded [`CubeHeader`] carries `flag`
/// set. This is the "query by metatag/flag, not by path" primitive the
/// source PDF describes (§"associative storage"): records are addressed
/// by *what they are* (a flag bit) rather than *where they live* (a
/// coordinate path). `cube-os-*` services that emit typed records
/// (e.g. a log line tagged `doc_type = "klog"`) become discoverable by
/// event type without knowing their CZYX address in advance.
///
/// Records written through the RAW path (`rawput`, used by the OS layers)
/// carry only a synthesized `size_bytes` header, so they will NOT match
/// a content flag unless the writer also set one. The associative query
/// is therefore most powerful when records are written through
/// [`CubeStore::put_record`] with a populated [`CubeHeader`].
pub fn scan_by_flag(&self, flag: u16) -> Vec<Czyx> {
let mut out: Vec<Czyx> = self
.backend
.keys()
.into_iter()
.filter(|k| {
self.get_record(k)
.map(|(h, _)| h.flags.has(flag))
.unwrap_or(false)
})
.collect();
out.sort();
out
}
/// Query records by per-word flag (the 16-bit WordFlags metadata stripe
/// carried in the header, tag 14). Parallel to [`scan_by_flag`] but on
/// `word_flags`, so per-word metadata (encrypted/compressed/stego/type/
/// arrangement/continuation) is queryable without decoding the body.
pub fn scan_by_word_flag(&self, flag: u16) -> Vec<Czyx> {
let mut out: Vec<Czyx> = self
.backend
.keys()
.into_iter()
.filter(|k| {
self.get_record(k)
.map(|(h, _)| h.word_flags.has(flag))
.unwrap_or(false)
})
.collect();
out.sort();
out
}
/// PDF Package 2 API: `scan_by_type` — query records by `doc_type`
/// (the "event type" / file-extension analogue the PDF calls Flag 2).
///
/// This is the concrete "log lookup by query" the OS layers want: instead
/// of addressing `/cubefs/c200/z004/...` directly, you ask "every record
/// whose doc_type is `klog`" and get back all matching coordinates. The
/// comparison is exact-match (case-sensitive) against the header field.
pub fn scan_by_type(&self, doc_type: &str) -> Vec<Czyx> {
let mut out: Vec<Czyx> = self
.backend
.keys()
.into_iter()
.filter(|k| {
self.get_record(k)
.map(|(h, _)| h.doc_type.as_deref() == Some(doc_type))
.unwrap_or(false)
})
.collect();
out.sort();
out
}
/// Return `(label, header, body)` for every record matching `flag`.
/// Convenience wrapper over [`CubeStore::scan_by_flag`] that also pulls
/// the decoded payload so a caller (e.g. a `cubelog` query tool) can
/// present the matching records directly.
pub fn query_by_flag(&self, flag: u16) -> Vec<(Czyx, CubeHeader, Vec<u8>)> {
self.scan_by_flag(flag)
.into_iter()
.filter_map(|k| self.get_record(&k).map(|(h, b)| (k, h, b)))
.collect()
}
/// Return `(label, header, body)` for every record whose `doc_type`
/// matches. See [`CubeStore::scan_by_type`] for the matching semantics.
pub fn query_by_type(&self, doc_type: &str) -> Vec<(Czyx, CubeHeader, Vec<u8>)> {
self.scan_by_type(doc_type)
.into_iter()
.filter_map(|k| self.get_record(&k).map(|(h, b)| (k, h, b)))
.collect()
}
/// PDF Package 2 / OS-in-CUBE: reconstruct a *directory hierarchy from
/// metatags*. Returns every coordinate whose header carries a `path`
/// metatag equal to `path` (exact) — i.e. "the file at this path".
///
/// Combined with [`CubeStore::scan_by_path_prefix`], this lets the cube
/// answer "everything under /etc" WITHOUT the filesystem supporting
/// recursive nesting: the path is stored as a flag-addressed field, not
/// as an inode tree. This is the source PDF's prescribed model ("operate
/// on CZYX records and Null-space flags rather than paths and inodes").
pub fn scan_by_path(&self, path: &str) -> Vec<Czyx> {
let mut out: Vec<Czyx> = self
.backend
.keys()
.into_iter()
.filter(|k| {
self.get_record(k)
.map(|(h, _)| h.path.as_deref() == Some(path))
.unwrap_or(false)
})
.collect();
out.sort();
out
}
/// "List a directory": every record whose `path` metatag is *under* the
/// given directory prefix (e.g. `scan_by_path_prefix("/etc")` returns
/// `/etc/passwd`, `/etc/network/interfaces`, ...). Unlike a real FS, the
/// nesting is a shared prefix on the `path` metatag of several records
/// (the source PDF: "operate on CZYX records and Null-space flags rather
/// than paths and inodes") — there is no half-coordinate directory entry.
///
/// A query matches BOTH an exact leaf (`/etc/hostname`) and any descendant
/// (`/etc/hostname` and `/etc/passwd` both answer `/etc`), because a path
/// is just a string: `/etc/hostname` == dir, and `/etc/passwd` starts with
/// `dir + '/'`.
pub fn scan_by_path_prefix(&self, dir: &str) -> Vec<Czyx> {
let sep = format!("{dir}/");
let mut out: Vec<Czyx> = self
.backend
.keys()
.into_iter()
.filter(|k| {
self.get_record(k)
.map(|(h, _)| match &h.path {
Some(p) => p == dir || p.starts_with(&sep),
None => false,
})
.unwrap_or(false)
})
.collect();
out.sort();
out
}
/// `(label, header, body)` for [`CubeStore::scan_by_path`].
pub fn query_by_path(&self, path: &str) -> Vec<(Czyx, CubeHeader, Vec<u8>)> {
self.scan_by_path(path)
.into_iter()
.filter_map(|k| self.get_record(&k).map(|(h, b)| (k, h, b)))
.collect()
}
/// `(label, header, body)` for [`CubeStore::scan_by_path_prefix`].
pub fn query_by_path_prefix(&self, dir: &str) -> Vec<(Czyx, CubeHeader, Vec<u8>)> {
self.scan_by_path_prefix(dir)
.into_iter()
.filter_map(|k| self.get_record(&k).map(|(h, b)| (k, h, b)))
.collect()
}
}
#[cfg(test)]
@@ -501,6 +716,25 @@ mod tests {
assert!(rh.flags.has(cubecoords::HeaderFlags::HAS_ASSOCIATIONS));
}
#[test]
fn word_flags_roundtrip() {
let mut h = CubeHeader::new();
h.word_flags = cubecoords::WordFlags::from_bits(
cubecoords::WordFlags::ENCRYPTED
| cubecoords::WordFlags::CONTINUATION
| (cubecoords::WordFlags::TYPE_MASK & 0b11),
);
let enc = super::record_codec::encode_header(&h);
let dec = super::record_codec::decode_header(&enc).unwrap();
assert_eq!(dec.word_flags.bits(), h.word_flags.bits(), "word_flags must round-trip");
// refresh_flags recomputes only the per-RECORD HeaderFlags; the separate
// per-word word_flags field must be preserved untouched.
let mut dec2 = dec;
dec2.refresh_flags();
assert_eq!(dec2.word_flags.bits(), h.word_flags.bits());
}
#[test]
fn missing_record_is_none() {
let store = CubeStore::new(HashBackend::new());
@@ -515,4 +749,286 @@ mod tests {
assert_eq!(store.get_raw(&Czyx::new(0, 0, 0, 0)), Some(vec![1]));
assert_eq!(store.get_raw(&Czyx::new(0, 0, 0, 1)), Some(vec![2]));
}
/// Regression (2026-08-13, OS-in-CUBE migration): a payload written via the
/// RAW path (`put_raw`, i.e. the daemon's `rawput` verb used by OS-layer
/// services) must still be READABLE through `get_record`, because that is
/// what `cubefs` getattr/read and the `stat` verb go through. Before the
/// fix these records reported size 0 and read back empty — real bytes were
/// silently invisible through the filesystem.
#[test]
fn get_record_surfaces_raw_unenveloped_payloads() {
let mut store = CubeStore::new(HashBackend::new());
let coord = Czyx::new(200, 70, 1, 1);
let payload = b"CUBELINUX OS PROCESS SNAPSHOT\nprocs: 118\n".to_vec();
store.put_raw(coord, payload.clone());
let (hdr, body) = store
.get_record(&coord)
.expect("raw payload must be visible as a record, not vanish");
assert_eq!(body, payload, "body must round-trip byte-for-byte");
assert_eq!(
hdr.size_bytes,
Some(payload.len() as u64),
"synthesized header must report the true size so getattr is correct"
);
// A short payload (< 4 bytes, cannot even hold a length prefix) is the
// other edge the old code dropped: counters like "118" land here.
let short = Czyx::new(200, 70, 2, 1);
store.put_raw(short, b"118".to_vec());
let (h2, b2) = store.get_record(&short).expect("short raw payload visible");
assert_eq!(b2, b"118");
assert_eq!(h2.size_bytes, Some(3));
}
/// The envelope path must be unaffected by the raw fallback: a properly
/// stored record still decodes its real header (not a synthesized one).
#[test]
fn get_record_still_prefers_the_real_envelope() {
let mut store = CubeStore::new(HashBackend::new());
let coord = Czyx::new(201, 5, 1, 1);
let mut hdr = CubeHeader::new();
hdr.title = Some("real-record".to_string());
store.put_record(coord, &hdr, b"payload");
let (got, body) = store.get_record(&coord).expect("enveloped record");
assert_eq!(body, b"payload");
assert_eq!(
got.title.as_deref(),
Some("real-record"),
"must decode the true header, not fall back to raw"
);
}
/// PDF Package 2: `scan_by_flag` finds records by flag bit, independent of
/// their coordinate address. A record tagged by `DOC_TYPE` must surface
/// when queried for that flag and be absent otherwise.
#[test]
fn scan_by_flag_finds_typed_records() {
let mut store = CubeStore::new(HashBackend::new());
// A "klog" typed event record.
let mut klog = CubeHeader::new();
klog.doc_type = Some("klog".into());
klog.refresh_flags();
store.put_record(Czyx::new(200, 4, 2, 1), &klog, b"kernel: eth0 up");
// An untyped raw record (the common OS-layer case).
store.put_raw(Czyx::new(200, 70, 1, 1), b"118".to_vec());
let by_doc_type = store.scan_by_flag(cubecoords::HeaderFlags::DOC_TYPE);
assert_eq!(by_doc_type, vec![Czyx::new(200, 4, 2, 1)]);
// The raw record carries only a synthesized size flag, so it must NOT
// match DOC_TYPE.
assert!(!by_doc_type.contains(&Czyx::new(200, 70, 1, 1)));
let by_size = store.scan_by_flag(cubecoords::HeaderFlags::SIZE_BYTES);
assert!(
!by_size.is_empty(),
"synthesized raw headers carry SIZE_BYTES"
);
}
#[test]
fn scan_by_word_flag_finds_records() {
let mut store = CubeStore::new(HashBackend::new());
let mut h = CubeHeader::new();
h.word_flags = cubecoords::WordFlags::from_bits(cubecoords::WordFlags::ENCRYPTED);
store.put_record(Czyx::new(200, 5, 1, 1), &h, b"secret body");
let mut h2 = CubeHeader::new();
h2.doc_type = Some("plain".into());
h2.refresh_flags();
store.put_record(Czyx::new(200, 5, 2, 2), &h2, b"plain body");
let enc = store.scan_by_word_flag(cubecoords::WordFlags::ENCRYPTED);
assert_eq!(enc, vec![Czyx::new(200, 5, 1, 1)]);
assert!(!enc.contains(&Czyx::new(200, 5, 2, 2)));
}
/// `scan_by_type` is the concrete "log lookup by query" — pull every record
/// of a given event type without knowing its coordinate.
#[test]
fn scan_by_type_event_lookup() {
let mut store = CubeStore::new(HashBackend::new());
let mut a = CubeHeader::new();
a.doc_type = Some("klog".into());
let mut b = CubeHeader::new();
b.doc_type = Some("klog".into());
let mut c = CubeHeader::new();
c.doc_type = Some("state".into());
store.put_record(Czyx::new(200, 4, 2, 1), &a, b"k1");
store.put_record(Czyx::new(200, 4, 2, 2), &b, b"k2");
store.put_record(Czyx::new(200, 1, 1, 1), &c, b"s1");
let klogs = store.scan_by_type("klog");
assert_eq!(klogs.len(), 2);
assert!(klogs.contains(&Czyx::new(200, 4, 2, 1)));
assert!(klogs.contains(&Czyx::new(200, 4, 2, 2)));
// query_by_type returns the decoded payloads too.
let klogs_full = store.query_by_type("klog");
assert_eq!(klogs_full.len(), 2);
assert!(klogs_full.iter().any(|(_, _, b)| b == b"k1"));
assert!(klogs_full.iter().any(|(_, _, b)| b == b"k2"));
assert_eq!(store.scan_by_type("state"), vec![Czyx::new(200, 1, 1, 1)]);
assert!(store.scan_by_type("nonexistent").is_empty());
}
/// Path metatags reconstruct a nested directory hierarchy WITHOUT the FS
/// needing recursive inode trees (source PDF: "operate on CZYX records and
/// Null-space flags rather than paths and inodes"). `/etc/passwd` and
/// `/etc/network/interfaces` live at unrelated coordinates but share the
/// `/etc` prefix, so `scan_by_path_prefix("/etc")` finds both.
#[test]
fn scan_by_path_prefix_lists_directory() {
let mut store = CubeStore::new(HashBackend::new());
let mut mk = |c: u8, path: &str, dt: &str| {
let mut h = CubeHeader::new();
h.path = Some(path.to_string());
h.doc_type = Some(dt.to_string());
h.refresh_flags();
store.put_record(Czyx::new(200, 1, c, 1), &h, b"body");
};
mk(1, "/etc/passwd", "file");
mk(2, "/etc/network/interfaces", "file");
mk(3, "/etc/hosts", "file");
mk(4, "/usr/bin/ls", "file");
// Exact match.
assert_eq!(
store.scan_by_path("/etc/passwd"),
vec![Czyx::new(200, 1, 1, 1)]
);
// Directory listing via metatag prefix. Bare "/etc" matches every
// record whose path is "/etc" or starts with "/etc/" (an exact leaf
// like "/etc/hostname" answers "/etc", and "/etc/passwd" is a child).
let etc = store.scan_by_path_prefix("/etc");
assert_eq!(etc.len(), 3);
assert!(etc.contains(&Czyx::new(200, 1, 1, 1)));
assert!(etc.contains(&Czyx::new(200, 1, 2, 1)));
assert!(etc.contains(&Czyx::new(200, 1, 3, 1)));
assert!(
!etc.contains(&Czyx::new(200, 1, 4, 1)),
"/usr must not appear under /etc"
);
// An exact leaf answers its own parent directory query.
assert_eq!(
store.scan_by_path_prefix("/etc/hosts"),
vec![Czyx::new(200, 1, 3, 1)]
);
// Empty prefix = all paths.
assert_eq!(store.scan_by_path_prefix("").len(), 4);
// The path flag bit is set so the record is also flag-discoverable.
let by_path_flag = store.scan_by_flag(cubecoords::HeaderFlags::HAS_PATH);
assert_eq!(by_path_flag.len(), 4);
}
// ---- HashBackend trait-level tests (PDF spec: HashMap<u32, Vec<u8>>) ----
#[test]
fn hash_backend_put_get_delete_roundtrip() {
let mut b = HashBackend::new();
let k = Czyx::new(10, 20, 30, 40);
b.put(k, b"payload".to_vec());
assert_eq!(b.get(&k), Some(b"payload".to_vec()));
b.delete(&k);
assert_eq!(b.get(&k), None);
}
#[test]
fn hash_backend_overwrite_replaces_value() {
let mut b = HashBackend::new();
let k = Czyx::new(0, 1, 2, 3);
b.put(k, b"old".to_vec());
b.put(k, b"new".to_vec());
assert_eq!(b.get(&k), Some(b"new".to_vec()));
}
#[test]
fn hash_backend_keys_sorted_order() {
let mut b = HashBackend::new();
for c in 1u8..=3 {
b.put(Czyx::new(50, c, 10, 1), vec![c]);
}
let keys = b.keys();
assert_eq!(keys.len(), 3);
// HashMap iteration is unordered; keys() must sort for stable output.
let mut sorted = keys.clone();
sorted.sort();
assert_eq!(keys, sorted);
for k in &keys {
assert_eq!(k.c, 50);
}
}
#[test]
fn hash_backend_scan_prefix_filters_correctly() {
let mut b = HashBackend::new();
b.put(Czyx::new(1, 2, 3, 4), b"a".to_vec());
b.put(Czyx::new(1, 9, 9, 9), b"b".to_vec());
b.put(Czyx::new(2, 0, 0, 0), b"c".to_vec());
b.put(Czyx::new(1, 2, 9, 9), b"d".to_vec());
// C=1 only.
let c1 = b.scan_prefix(1, None, None);
assert_eq!(c1.len(), 3);
for k in &c1 {
assert_eq!(k.c, 1);
}
// C=1, Z=2 only.
let c1z2 = b.scan_prefix(1, Some(2), None);
assert_eq!(c1z2.len(), 2);
for k in &c1z2 {
assert_eq!(k.c, 1);
assert_eq!(k.z, 2);
}
// C=2 only.
let c2 = b.scan_prefix(2, None, None);
assert_eq!(c2, vec![Czyx::new(2, 0, 0, 0)]);
// Non-existent C.
assert!(b.scan_prefix(99, None, None).is_empty());
}
#[test]
fn hash_backend_empty_keys_and_scan() {
let b = HashBackend::new();
assert!(b.keys().is_empty());
assert!(b.scan_prefix(0, None, None).is_empty());
}
#[test]
fn hash_backend_null_coord_is_valid_key() {
let mut b = HashBackend::new();
let null = Czyx::new(0, 0, 0, 0);
b.put(null, b"null-cube".to_vec());
assert_eq!(b.get(&null), Some(b"null-cube".to_vec()));
b.delete(&null);
assert_eq!(b.get(&null), None);
// Null cube (0,0,0,0) is distinct from a nearby coord.
let near = Czyx::new(0, 0, 0, 1);
b.put(near, b"near".to_vec());
assert_eq!(b.get(&near), Some(b"near".to_vec()));
assert_eq!(b.get(&null), None);
}
#[test]
fn hash_backend_multiple_backends_are_independent() {
let b1 = HashBackend::new();
let mut b2 = HashBackend::new();
b2.put(Czyx::new(1, 1, 1, 1), b"x".to_vec());
assert!(b1.get(&Czyx::new(1, 1, 1, 1)).is_none());
assert_eq!(b2.get(&Czyx::new(1, 1, 1, 1)), Some(b"x".to_vec()));
}
}
-16
View File
@@ -16,19 +16,3 @@ cubestore = { path = "../cubestore" }
cubefs = { path = "../cubefs" }
cubecode = { path = "../cubecode" }
cubecrypt = { path = "../cubecrypt" }
[[bin]]
name = "cube"
path = "src/bin/cube.rs"
[[bin]]
name = "cube-demo"
path = "src/bin/cube_demo.rs"
[[bin]]
name = "cube-server"
path = "src/bin/cube-server.rs"
[[bin]]
name = "cubec"
path = "src/bin/cubec.rs"
+3 -1
View File
@@ -76,9 +76,11 @@ the spec never defined.
## Binaries
* `cube-demo` — self-contained tour (write two linked cells, run; seal a
record, reopen + run). Prints evidence at each step.
record, reopen + run). Prints evidence at each step. (in `cubesys`)
* `cube` — CLI: `write`, `run`, `ls`, `stat`, `seal`, `open` over one store,
interactively (`repl`), from a file (`script`), or as the demo (`demo`).
Now a standalone crate (`cubecli/`); identical behavior to the former
`cubesys/src/bin/cube.rs`.
* `cube-server` — long-lived daemon: holds ONE `CubeStore` for its whole
lifetime and serves the same command language over a Unix-domain socket.
Snapshots the store to a JSON file on every request so the cube survives
+805 -18
View File
@@ -15,7 +15,7 @@ use crate::audit::{Audit, OP_DELETE, OP_GRANT, OP_OPEN, OP_READ, OP_REVOKE, OP_S
use crate::grants::{grant, grant_allows, perms_from_str, revoke, Owner, Perm};
use crate::store::ConcurrentStore;
use crate::tenant::{TenantIdentity, TenantSession};
use cubecode::{CodeCell, Kind, Op, Vm};
use cubecode::{CodeCell, Kind, Op, RunResult, Vm};
use cubecoords::{CubeHeader, Czyx};
use cubecrypt::{CubeEnv, KeySlot, Selector, TransformId};
use cubestore::{CubeStore, HashBackend};
@@ -448,10 +448,218 @@ impl Session {
))
}
}
"query-path" => {
// Reconstruct a directory listing from path metatags (no FS
// inode tree needed). `query-path /etc` returns every record
// whose `path` metatag is under /etc — the PDF's "operate on
// CZYX records and Null-space flags rather than paths/inodes".
let dir = it
.next()
.ok_or_else(|| "query-path needs <dir-prefix>".to_string())?;
let matches = store.query_by_path_prefix(dir);
if matches.is_empty() {
Ok(format!("query-path {dir} -> (no matches)"))
} else {
let mut names: Vec<String> = matches
.iter()
.filter_map(|c| {
store.get_record(c).map(|(h, _)| {
let p = h.path.as_deref().unwrap_or("");
format!("{}:{}", c.pack_u32(), p)
})
})
.collect();
names.sort();
Ok(format!(
"query-path {dir} -> {} matches:\n {}",
names.len(),
names.join("\n ")
))
}
}
"scan-word-flags" => {
let a = it
.next()
.ok_or_else(|| "scan-word-flags needs <flag> (name or 0xbits)".to_string())?;
let bits = parse_word_flag(a)?;
let coords = store.scan_by_word_flag(bits);
if coords.is_empty() {
Ok(format!("scan-word-flags {a} (0x{bits:04x}) -> (no matches)"))
} else {
let names: Vec<String> =
coords.iter().map(|c| c.pack_u32().to_string()).collect();
Ok(format!(
"scan-word-flags {a} (0x{bits:04x}) -> {} matches:\n {}",
coords.len(),
names.join("\n ")
))
}
}
"show-flags" => {
let cs = it.next().ok_or_else(|| "show-flags needs <C.Z.Y.X>".to_string())?;
let coord = parse_coord(cs).ok_or_else(|| format!("bad coordinate '{cs}' (want C.Z.Y.X)"))?;
match store.get_record(&coord) {
None => Ok(format!("show-flags {cs} -> (no record)")),
Some((h, body)) => Ok(format!(
"show-flags {cs}\n header_flags: 0x{:04x} {}\n word_flags: 0x{:04x} {}\n size: {} bytes",
h.flags.bits(), header_flag_names(h.flags.bits()),
h.word_flags.bits(), word_flag_names(h.word_flags.bits()),
body.len()))
}
}
// Session-message / note log: a searchable, per-session, per-day
// note area stored as CZYX records carrying WordFlags metadata
// (the structured replacement for the hand-maintained RESUME-*.md
// files and the Hermes session store).
"note" | "notes" => crate::notes::note_command(store, line),
// Project threads: list projects / walk a project's associated notes
// (the first-class-association model for tracking work between sessions).
"project" | "projects" => crate::notes::project_command(store, line),
"trace-capture" => {
let src_s = it.next().ok_or_else(|| "trace-capture needs <source C.Z.Y.X> <target C.Z.Y.X>".to_string())?;
let tgt_s = it.next().ok_or_else(|| "trace-capture needs <target C.Z.Y.X>".to_string())?;
let src = parse_coord(src_s).ok_or_else(|| format!("bad source coord '{src_s}'"))?;
let tgt = parse_coord(tgt_s).ok_or_else(|| format!("bad target coord '{tgt_s}'"))?;
let snap = store.read_snapshot();
let mut vm = cubecode::Vm::new(snap);
let (result, trace) = vm.run_captured(src);
let out = vm.output().to_vec();
// Store the captured trace as a Kind::Layer AI-artifact record,
// linked back to the source cell (a linked CZYX edge).
let mut h = CubeHeader::new();
h.title = Some(format!("trace:{}", src.pack_u32()));
h.doc_type = Some(cubecode::Kind::Layer.as_str().to_string());
h.linked_records = vec![src];
h.refresh_flags();
store.put_record(tgt, &h, &cubecode::encode(&trace));
Ok(format!(
"trace-capture {src_s} -> {tgt_s} (layer, {} ops, {:?}) output {} bytes",
trace.len(),
result,
out.len()
))
}
"trace-replay" => {
let tgt_s = it.next().ok_or_else(|| "trace-replay needs <C.Z.Y.X>".to_string())?;
let tgt = parse_coord(tgt_s).ok_or_else(|| format!("bad coord '{tgt_s}'"))?;
let snap = store.read_snapshot();
match cubecode::replay_trace(&snap, tgt) {
Some((result, out)) => Ok(format!(
"trace-replay {tgt_s}: {:?}\n output ({} bytes): {}",
result,
out.len(),
String::from_utf8_lossy(&out)
)),
None => Err(format!("trace-replay {tgt_s}: no valid trace record at that coord")),
}
}
"trace-verify" => {
let src_s = it.next().ok_or_else(|| "trace-verify needs <source> <trace> <golden>".to_string())?;
let tr_s = it.next().ok_or_else(|| "trace-verify needs <trace>".to_string())?;
let go_s = it.next().ok_or_else(|| "trace-verify needs <golden>".to_string())?;
let src = parse_coord(src_s).ok_or_else(|| format!("bad coord '{src_s}'"))?;
let tr = parse_coord(tr_s).ok_or_else(|| format!("bad coord '{tr_s}'"))?;
let go = parse_coord(go_s).ok_or_else(|| format!("bad coord '{go_s}'"))?;
let snap = store.read_snapshot();
match cubecode::verify_golden(&snap, src, tr, go) {
Ok(()) => Ok(format!(
"trace-verify {src_s} {tr_s} {go_s} -> PASS (replay + fresh run match golden)"
)),
Err(e) => Ok(format!("trace-verify {src_s} {tr_s} {go_s} -> FAIL: {e}")),
}
}
"trace-list" => {
let snap = store.read_snapshot();
let mut rows = Vec::new();
for k in snap.keys() {
if let Some((h, _)) = snap.get_record(&k) {
let dt = h.doc_type.clone().unwrap_or_default();
let is_artifact = matches!(dt.as_str(), "layer" | "golden" | "checkpoint" | "variant")
|| h.title.as_deref().map_or(false, |t| t.starts_with("trace:") || t.starts_with("golden:"));
if is_artifact {
let links = h.linked_records
.iter().map(|c| c.pack_u32().to_string()).collect::<Vec<_>>().join(",");
rows.push(format!("{} [{}] {} -> links: {links}",
k.pack_u32(), dt, h.title.as_deref().unwrap_or("")));
}
}
}
if rows.is_empty() {
Ok("trace-list -> (no layer/golden/checkpoint/variant/trace records)".to_string())
} else {
Ok(format!("trace-list -> {} records:\n {}", rows.len(), rows.join("\n ")))
}
}
"trace-links" => {
let cs = it.next().ok_or_else(|| "trace-links needs <C.Z.Y.X> [depth]".to_string())?;
let coord = parse_coord(cs).ok_or_else(|| format!("bad coord '{cs}'"))?;
let depth = it.next().and_then(|d| d.parse::<usize>().ok()).unwrap_or(3);
let snap = store.read_snapshot();
let hops = cubecode::lineage(&snap, coord, depth);
if hops.is_empty() {
return Ok(format!("trace-links {cs} -> (no record)"));
}
let mut lines = Vec::new();
for h in &hops {
let links = h.links.iter().map(|c| c.pack_u32().to_string()).collect::<Vec<_>>().join(" ");
lines.push(format!(" {} [{}] {} -> {links}", h.coord.pack_u32(), h.doc_type, h.title));
}
Ok(format!("trace-links {cs} (depth {depth}) -> {} hops:\n{}", hops.len(), lines.join("\n")))
}
"golden-capture" => {
let src_s = it.next().ok_or_else(|| "golden-capture needs <source> <trace> <golden>".to_string())?;
let tr_s = it.next().ok_or_else(|| "golden-capture needs <trace>".to_string())?;
let go_s = it.next().ok_or_else(|| "golden-capture needs <golden>".to_string())?;
let name = it.next().unwrap_or("golden").to_string();
let src = parse_coord(src_s).ok_or_else(|| format!("bad coord '{src_s}'"))?;
let tr = parse_coord(tr_s).ok_or_else(|| format!("bad coord '{tr_s}'"))?;
let go = parse_coord(go_s).ok_or_else(|| format!("bad coord '{go_s}'"))?;
let snap = store.read_snapshot();
let mut vm = cubecode::Vm::new(snap);
let (_r, trace) = vm.run_captured(src);
let out = vm.output().to_vec();
// Trace record: Kind::Layer, linked to the source.
let mut h = CubeHeader::new();
h.title = Some(format!("trace:{name}"));
h.doc_type = Some(cubecode::Kind::Layer.as_str().to_string());
h.linked_records = vec![src];
h.refresh_flags();
store.put_record(tr, &h, &cubecode::encode(&trace));
// Golden output record, linked to the trace.
let mut hg = CubeHeader::new();
hg.title = Some(format!("golden:{name}"));
hg.doc_type = Some("golden".into());
hg.linked_records = vec![tr];
hg.refresh_flags();
store.put_record(go, &hg, &out);
Ok(format!(
"golden-capture {src_s} -> {tr_s} (layer) + {go_s} (golden), output {} bytes",
out.len()
))
}
"prog" => {
let path = it.next().ok_or_else(|| "prog needs <path>".to_string())?;
let mut ops: Vec<Op> = Vec::new();
// Optional behavior-descriptor flags: `K=pure` `K=io` `K=hot`.
// These stamp the OS-kernel behavior bits (PDF §524525) on the
// record header and switch the kind to `kernel` so the cube
// correctly classifies operator kernels vs plain functions.
let mut descriptor: Option<cubecode::Behavior> = None;
while let Some(tok) = it.next() {
if let Some(flag) = tok.strip_prefix("K=") {
let bit = match flag {
"pure" => cubecode::Behavior::PURE,
"io" => cubecode::Behavior::IO_HEAVY,
"alloc" => cubecode::Behavior::ALLOCATES,
"net" => cubecode::Behavior::NETWORK,
"sec" => cubecode::Behavior::SECURITY_SENSITIVE,
"hot" => cubecode::Behavior::HOT_PATH,
other => return Err(format!("prog: unknown descriptor K={other}")),
};
let cur = descriptor.unwrap_or_default();
descriptor = Some(cubecode::Behavior(cur.0 | bit));
continue;
}
let arg = if takes_arg(tok) {
it.next()
.and_then(|a| a.parse::<u8>().ok())
@@ -464,11 +672,13 @@ impl Session {
if ops.is_empty() {
return Err("prog: no ops given".to_string());
}
let is_kernel = descriptor.is_some();
let kind = if is_kernel { Kind::Kernel } else { Kind::Fn };
let name = path.rsplit('/').next().unwrap_or(path);
// Compute the exact record bytes `put_record` would write, using
// a throwaway store so we can buffer (or apply) them without
// duplicating the record codec.
let coord = scratch_code_coord(path, Kind::Fn, name, &ops)?;
let coord = scratch_code_coord(path, kind, name, &ops)?;
// Task 6: reject overwriting a record owned by a different owner.
if let Some(msg) = self.admit_mutate(coord, Perm::Write) {
self.audit_now(OP_WRITE, coord, false);
@@ -478,28 +688,341 @@ impl Session {
let owner = self.identity.as_ref().map(|i| i.owner_local.as_str());
let value = {
let mut scratch = CubeStore::new(HashBackend::new());
crate::store_code_cell(&mut scratch, path, Kind::Fn, name, &[], &ops, owner)
.map_err(|e| e.to_string())?;
crate::store_code_cell(
&mut scratch,
path,
kind,
name,
&[],
&ops,
owner,
descriptor,
)
.map_err(|e| e.to_string())?;
scratch.get_raw(&coord).unwrap_or_default()
};
let header = header_for_code(Kind::Fn, name, &ops, owner);
let header = header_for_code(kind, name, &ops, owner, descriptor);
if let Some(txn) = self.txn.as_mut() {
txn.ops.push(TxnOp {
coord,
put: Some((value, header)),
});
let kind_tag = if is_kernel { "kernel" } else { "fn" };
return Ok(format!(
"buffered prog {path} ({kind_tag}, {} ops) — commit to apply",
ops.len()
));
}
let coord = store
.put_code_cell(path, kind, name, &[], &ops, owner, descriptor)
.map_err(|e| e.to_string())?;
let kind_tag = if is_kernel { "kernel" } else { "fn" };
Ok(format!(
"wrote program {path} -> coord {} ({kind_tag}, {} ops)",
coord.pack_u32(),
ops.len()
))
}
"link" => {
// Attach a callee coordinate to an existing code cell's call
// graph. `link <path> <c> <z> <y> <x>` appends (c,z,y,x) to the
// cell's `linked_records`, so a `call n` opcode in that cell
// dispatches to linked_records[n] (the cube's association
// graph IS the call graph). This is what makes functions stored
// in CUBE callable from other functions stored in CUBE.
let path = it.next().ok_or_else(|| "link needs <path>".to_string())?;
let c = parse_u8(it.next(), "link needs <c>")?;
let z = parse_u8(it.next(), "link needs <z>")?;
let y = parse_u8(it.next(), "link needs <y>")?;
let x = parse_u8(it.next(), "link needs <x>")?;
let target = Czyx::new(c, z, y, x);
let coord = crate::path_to_czyx(path).map_err(|e| e.to_string())?;
if let Some(msg) = self.admit_mutate(coord, Perm::Write) {
self.audit_now(OP_WRITE, coord, false);
return Err(msg);
}
self.audit_now(OP_WRITE, coord, true);
// Load the existing cell, append the link, re-store it.
let cell = crate::load_code_cell(&store.read_snapshot(), path)
.map_err(|e| format!("link: cannot load {path}: {e:?}"))?;
let mut links = cell.links().to_vec();
if links.contains(&target) {
return Ok(format!(
"link {path}: {target:?} already linked (degree {})",
links.len()
));
}
links.push(target);
let code = cubecode::decode(&cell.body())
.map_err(|e| format!("link: bad bytecode in {path}: {e:?}"))?;
let owner = self.identity.as_ref().map(|i| i.owner_local.as_str());
store
.put_code_cell(
path,
cell.kind(),
cell.name().unwrap_or(path),
&links,
&code,
owner,
None,
)
.map_err(|e| e.to_string())?;
Ok(format!(
"linked {path} -> {target:?} (call-graph degree now {})",
links.len()
))
}
// ---- OS-operator-kernel authoring + thin effector (Steps 1 & 2) ----
// Per the reframe, the OS's *computation* lives in CUBE as operator
// kernels (call graphs + behavior descriptors, PDF §524525). The
// native OS layer is only a thin effector: it reads a kernel's
// computed result and applies the effect. The cubevm never does
// store-IO itself. `kernel` authors a kernel; `tick` lays down the
// OS kernel call-graph; `native-apply` is the effector boundary.
"kernel" => {
// `kernel <path> [K=pure|io|hot ...] <op> <arg> ...`
// Like `prog`, but the cell is always kind=Kernel and accepts
// behavior-descriptor flags so the OS marks operator kernels
// distinctly from plain functions.
let path = it.next().ok_or_else(|| "kernel needs <path>".to_string())?;
let mut ops: Vec<Op> = Vec::new();
let mut descriptor: Option<cubecode::Behavior> = None;
while let Some(tok) = it.next() {
if let Some(flag) = tok.strip_prefix("K=") {
let bit = match flag {
"pure" => cubecode::Behavior::PURE,
"io" => cubecode::Behavior::IO_HEAVY,
"alloc" => cubecode::Behavior::ALLOCATES,
"net" => cubecode::Behavior::NETWORK,
"sec" => cubecode::Behavior::SECURITY_SENSITIVE,
"hot" => cubecode::Behavior::HOT_PATH,
other => return Err(format!("kernel: unknown descriptor K={other}")),
};
let cur = descriptor.unwrap_or_default();
descriptor = Some(cubecode::Behavior(cur.0 | bit));
continue;
}
let arg = if takes_arg(tok) {
it.next()
.and_then(|a| a.parse::<u8>().ok())
.ok_or_else(|| format!("kernel: {tok} needs a u8 argument"))?
} else {
0
};
ops.push(make_op(tok, arg)?);
}
if ops.is_empty() {
return Err("kernel: no ops given".to_string());
}
let name = path.rsplit('/').next().unwrap_or(path);
let coord = scratch_code_coord(path, Kind::Kernel, name, &ops)?;
if let Some(msg) = self.admit_mutate(coord, Perm::Write) {
self.audit_now(OP_WRITE, coord, false);
return Err(msg);
}
self.audit_now(OP_WRITE, coord, true);
let owner = self.identity.as_ref().map(|i| i.owner_local.as_str());
let value = {
let mut scratch = CubeStore::new(HashBackend::new());
crate::store_code_cell(
&mut scratch,
path,
Kind::Kernel,
name,
&[],
&ops,
owner,
descriptor,
)
.map_err(|e| e.to_string())?;
scratch.get_raw(&coord).unwrap_or_default()
};
let header = header_for_code(Kind::Kernel, name, &ops, owner, descriptor);
if let Some(txn) = self.txn.as_mut() {
txn.ops.push(TxnOp {
coord,
put: Some((value, header)),
});
return Ok(format!(
"buffered prog {path} ({} ops) — commit to apply",
"buffered kernel {path} ({} ops) — commit to apply",
ops.len()
));
}
let coord = store
.put_code_cell(path, Kind::Fn, name, &[], &ops, owner)
.put_code_cell(path, Kind::Kernel, name, &[], &ops, owner, descriptor)
.map_err(|e| e.to_string())?;
Ok(format!(
"wrote program {path} -> coord {} ({} ops)",
"wrote kernel {path} -> coord {} ({} ops, descriptors={:?})",
coord.pack_u32(),
ops.len()
ops.len(),
descriptor.map(|b| b.tags()).unwrap_or_default()
))
}
"tick" => {
// Lay down the OS operator-kernel call graph (Step 1). Each leaf
// is a real CUBE kernel composed via `linked_records` (the call
// graph) and tagged with behavior descriptors. `cube-os-tick`
// calls the three leaves in order. Nothing is executed here —
// the native layer later `run`s each and `native-apply`s the
// effect. This is "everything in CUBE" done the spec's way:
// the OS's behavior lives as kernels + call graph + descriptors.
let c = cubecode::C_OS_KERNEL;
let cfg = format!("/c{c}/z001/y001/x001"); // normalize-config: pure
let decide = format!("/c{c}/z001/y001/x002"); // decide-snapshot: io
let summarize = format!("/c{c}/z001/y001/x003"); // summarize-procs: pure+hot
let tick = format!("/c{c}/z001/y001/x004"); // cube-os-tick: hot (calls 0..2)
let cfg_code = vec![Op::Const(7), Op::Const(3), Op::Add, Op::Halt]; // 7+3=10
let decide_code = vec![Op::Const(1), Op::Ret]; // 1 => snapshot
let summarize_code = vec![Op::Const(20), Op::Halt]; // 20 procs
let tick_code = vec![
Op::Const(0),
Op::CallLink(0), // cfg
Op::CallLink(1), // decide
Op::CallLink(2), // summarize
Op::Halt,
];
// Stage the leaves first (we need their coords to build the
// call-graph edges of the root), then the root. `with_mut`
// hands us exclusive `&mut CubeStore` access so the helper can
// write each record through the normal codec.
let cfg_c = store
.with_mut(|s| {
crate::store_code_cell(
s,
&cfg,
Kind::Kernel,
"normalize-config",
&[],
&cfg_code,
None,
Some(cubecode::Behavior(cubecode::Behavior::PURE)),
)
})
.map_err(|e: crate::SysError| e.to_string())?;
let dec_c = store
.with_mut(|s| {
crate::store_code_cell(
s,
&decide,
Kind::Kernel,
"decide-snapshot",
&[],
&decide_code,
None,
Some(cubecode::Behavior(cubecode::Behavior::IO_HEAVY)),
)
})
.map_err(|e: crate::SysError| e.to_string())?;
let sum_c = store
.with_mut(|s| {
crate::store_code_cell(
s,
&summarize,
Kind::Kernel,
"summarize-procs",
&[],
&summarize_code,
None,
Some(cubecode::Behavior(
cubecode::Behavior::PURE | cubecode::Behavior::HOT_PATH,
)),
)
})
.map_err(|e: crate::SysError| e.to_string())?;
let tick_c = store
.with_mut(|s| {
crate::store_code_cell(
s,
&tick,
Kind::Kernel,
"cube-os-tick",
&[cfg_c, dec_c, sum_c], // call graph: tick -> {cfg,decide,summarize}
&tick_code,
None,
Some(cubecode::Behavior(cubecode::Behavior::HOT_PATH)),
)
})
.map_err(|e: crate::SysError| e.to_string())?;
Ok(format!(
"OS kernel call-graph laid down (kind=kernel, in cube c{c}):\n {} normalize-config [pure] -> coord {}\n {} decide-snapshot [io] -> coord {}\n {} summarize-procs [pure,hot] -> coord {}\n {} cube-os-tick [hot] -> coord {} (links cfg,decide,summarize)\nrun e.g.: run {}\nthen effector: native-apply {}",
cfg, cfg_c.pack_u32(), decide, dec_c.pack_u32(), summarize,
sum_c.pack_u32(), tick, tick_c.pack_u32(), tick_c.pack_u32(),
tick
))
}
"native-apply" => {
// Step 2: the thin effector. The decision/computation already
let path = it
.next()
.ok_or_else(|| "native-apply needs <kernel-path>".to_string())?;
let cell = crate::load_code_cell(&store.read_snapshot(), path)
.map_err(|e| format!("native-apply: cannot load {path}: {e}"))?;
if cell.kind() != Kind::Kernel {
return Err(format!(
"native-apply: {path} is kind={:?}, expected a kernel",
cell.kind()
));
}
// Build a throwaway in-memory store holding just this kernel
// (the VM needs a store to walk), then run it. The computation
// is entirely in CUBE; we only read back the computed result.
let coord = crate::path_to_czyx(path).map_err(|e| e.to_string())?;
let mut vm_store = cubestore::CubeStore::new(cubestore::HashBackend::new());
crate::store_code_cell(
&mut vm_store,
path,
Kind::Kernel,
cell.name().unwrap_or(path),
cell.links(),
&cell.code,
None,
None,
)
.map_err(|e| format!("native-apply: stage failed: {e}"))?;
let mut vm = Vm::new(vm_store);
let result = match vm.run(coord) {
RunResult::Halted { top } => top,
other => {
return Err(format!(
"native-apply: kernel did not halt cleanly: {other:?}"
))
}
};
let r = result.unwrap_or(0);
let effect = match cell.name() {
Some("decide-snapshot") => {
if r != 0 {
format!(
"OS EFFECT: persist runtime snapshot into CUBE (c{} band); decision kernel returned {} => snapshot NOW",
cubecode::C_OS_EFFECT, r
)
} else {
"OS EFFECT: no snapshot (decision kernel returned 0)".to_string()
}
}
_ => format!(
"OS EFFECT: apply computed result {} from kernel {}@{}",
r,
cell.name().unwrap_or("?"),
path
),
};
Ok(format!(
"native-apply {} (kind=kernel, descriptors={:?}):\n computed result = {}\n {}",
path,
cubecode::Behavior::from_flags(
store
.read_snapshot()
.get_record(&coord)
.map(|(h, _)| h.flags.bits())
.unwrap_or(0)
)
.tags(),
r,
effect
))
}
"write" => {
@@ -520,11 +1043,20 @@ impl Session {
let owner = self.identity.as_ref().map(|i| i.owner_local.as_str());
let value = {
let mut scratch = CubeStore::new(HashBackend::new());
crate::store_code_cell(&mut scratch, path, Kind::Fn, name, &[], &code, owner)
.map_err(|e| e.to_string())?;
crate::store_code_cell(
&mut scratch,
path,
Kind::Fn,
name,
&[],
&code,
owner,
None,
)
.map_err(|e| e.to_string())?;
scratch.get_raw(&coord).unwrap_or_default()
};
let header = header_for_code(Kind::Fn, name, &code, owner);
let header = header_for_code(Kind::Fn, name, &code, owner, None);
if let Some(txn) = self.txn.as_mut() {
txn.ops.push(TxnOp {
coord,
@@ -536,7 +1068,7 @@ impl Session {
));
}
let coord = store
.put_code_cell(path, Kind::Fn, name, &[], &code, owner)
.put_code_cell(path, Kind::Fn, name, &[], &code, owner, None)
.map_err(|e| e.to_string())?;
Ok(format!("wrote {path} -> coord {}", coord.pack_u32()))
}
@@ -596,6 +1128,66 @@ impl Session {
store.delete_raw(&coord);
Ok(format!("ok: deleted {}", coord.pack_u32()))
}
// --- Enveloped coordinate API (PDF Package 2: `put(C,Z,Y,X,
// bytes, flags)`). Unlike `rawput`, this writes a full record
// with a `CubeHeader`, so the record carries metatags
// (doc_type / title) that the associative `query` verb can
// target by event type. `rawput` stays for bulk/append payloads
// (e.g. the rolling kernel-log window) that do not need header
// metadata; `put` is for addressed, discoverable records. ---
"put" => {
let c = parse_u8(it.next(), "put needs <c>")?;
let z = parse_u8(it.next(), "put needs <z>")?;
let y = parse_u8(it.next(), "put needs <y>")?;
let x = parse_u8(it.next(), "put needs <x>")?;
let hex = it
.next()
.ok_or_else(|| "put needs <hex-bytes>".to_string())?;
let val = hex_decode(hex).ok_or_else(|| "put: value must be hex".to_string())?;
let coord = Czyx::new(c, z, y, x);
// Optional metatags: `doc_type=<t>` and/or `title=<t>` may
// follow. These are what `query <doc_type>` matches on.
let mut header = CubeHeader::new();
for tok in it {
if let Some(v) = tok.strip_prefix("doc_type=") {
header.doc_type = Some(v.to_string());
} else if let Some(v) = tok.strip_prefix("title=") {
header.title = Some(v.to_string());
} else if let Some(v) = tok.strip_prefix("path=") {
// POSIX path metatag: the original filesystem path.
// Lets scan_by_path_prefix reconstruct nesting from flags.
header.path = Some(v.to_string());
} else {
return Err(format!("put: unknown option {tok}"));
}
}
header.created_at = Some(
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_secs())
.unwrap_or(0),
);
header.refresh_flags();
let doc_type_dbg = header.doc_type.clone();
if let Some(txn) = self.txn.as_mut() {
txn.ops.push(TxnOp {
coord,
put: Some((val, header)),
});
Ok(format!(
"buffered put {} (doc_type={:?}) — commit to apply",
coord.pack_u32(),
doc_type_dbg
))
} else {
store.put_record(coord, &header, &val);
Ok(format!(
"ok: wrote {} (doc_type={:?})",
coord.pack_u32(),
doc_type_dbg
))
}
}
"rawkeys" => {
let ks: Vec<String> = store
.keys()
@@ -752,7 +1344,20 @@ impl Session {
let dir = it.next().ok_or_else(|| "ls needs <dir>".to_string())?;
// R5: directory reads are metadata reads — honor the read gate
// so an attacker can't enumerate a victim's records by name.
let dir_coord = crate::path_to_czyx(dir).map_err(|e| e.to_string())?;
// Use parse_path (not path_to_czyx) because ls targets a
// directory prefix, which path_to_czyx rejects as "not a record".
let parsed = cubefs::path::parse_path(dir)
.map_err(|e| format!("ls: bad path {dir}: {e}"))?;
// The ACL gate wants the directory's prefix coordinate
// (trailing axes zeroed) — same key the NullSpace acl_bucket
// uses for directory ACLs.
let dir_coord = match parsed.axes.len() {
0 => Czyx::new(0, 0, 0, 0), // root
1 => Czyx::new(parsed.axes[0], 0, 0, 0),
2 => Czyx::new(parsed.axes[0], parsed.axes[1], 0, 0),
3 => Czyx::new(parsed.axes[0], parsed.axes[1], parsed.axes[2], 0),
_ => return Err("ls: path too deep".to_string()),
};
if let Some(msg) = self.admit_read(dir_coord) {
self.audit_now(OP_READ, dir_coord, false);
return Err(msg);
@@ -783,8 +1388,15 @@ impl Session {
.getattr(path)
.map_err(|e| format!("stat {path}: {e:?}"))?;
Ok(format!(
"stat {path} -> ino={} kind={:?} size={} mode={:o}",
a.ino, a.kind, a.size, a.mode
"stat {path} -> ino={} kind={:?} size={} mode={:o} word_flags=0x{:04x} {} header_flags=0x{:04x} {}",
a.ino,
a.kind,
a.size,
a.mode,
a.word_flags,
word_flag_names(a.word_flags),
a.header_flags,
header_flag_names(a.header_flags)
))
}
"keyinit" => {
@@ -952,19 +1564,31 @@ pub fn txn_snapshot(s: &Session) -> CubeStore<HashBackend> {
/// writing — used to buffer `prog`/`write` mutations during a transaction.
fn scratch_code_coord(path: &str, kind: Kind, name: &str, code: &[Op]) -> Result<Czyx, String> {
let mut scratch = CubeStore::new(HashBackend::new());
crate::store_code_cell(&mut scratch, path, kind, name, &[], code, None)
crate::store_code_cell(&mut scratch, path, kind, name, &[], code, None, None)
.map_err(|e| e.to_string())
}
/// Build the `CubeHeader` a `store_code_cell` call would attach (mirrors
/// `crate::store_code_cell`), so a buffered txn put carries the same header.
/// `owner` (when set) is stamped on `owner_local_user` for Task 6 enforcement.
fn header_for_code(kind: Kind, name: &str, code: &[Op], owner: Option<&str>) -> CubeHeader {
/// `descriptor` (when set) stamps the behavior-descriptor bits (PDF §524525).
fn header_for_code(
kind: Kind,
name: &str,
code: &[Op],
owner: Option<&str>,
descriptor: Option<cubecode::Behavior>,
) -> CubeHeader {
let mut h = CubeHeader::new();
h.title = Some(name.to_string());
h.doc_type = Some(kind.as_str().to_string());
h.linked_records = Vec::new();
h.owner_local_user = owner.map(|o| o.to_string());
// OS-created records carry per-word flags (tag 14).
h.word_flags = crate::default_word_flags(kind, descriptor);
if let Some(b) = descriptor {
h.flags.0 |= b.to_flags();
}
if h.doc_type.as_deref() == Some("fn") {
h.size_bytes = Some(cubecode::encode(code).len() as u64);
}
@@ -992,6 +1616,72 @@ pub fn parse_coord(s: &str) -> Option<cubecoords::Czyx> {
}
/// Parse a single `u8` axis token: decimal (`7`) or hex (`0x07`).
fn header_flag_names(bits: u16) -> String {
let mut v: Vec<&str> = Vec::new();
if bits & cubecoords::HeaderFlags::TITLE != 0 { v.push("title"); }
if bits & cubecoords::HeaderFlags::DOC_TYPE != 0 { v.push("doc_type"); }
if bits & cubecoords::HeaderFlags::CREATED_AT != 0 { v.push("created_at"); }
if bits & cubecoords::HeaderFlags::SIZE_BYTES != 0 { v.push("size_bytes"); }
if bits & cubecoords::HeaderFlags::PERM_ROOT_ONLY != 0 { v.push("root_only"); }
if bits & cubecoords::HeaderFlags::PERM_LOCAL_USER != 0 { v.push("local_user"); }
if bits & cubecoords::HeaderFlags::PERM_REMOTE_USER != 0 { v.push("remote_user"); }
if bits & cubecoords::HeaderFlags::HAS_ASSOCIATIONS != 0 { v.push("associations"); }
if bits & cubecoords::HeaderFlags::HAS_PATH != 0 { v.push("path"); }
if v.is_empty() { String::new() } else { format!("({})", v.join("|")) }
}
fn word_flag_names(bits: u16) -> String {
let mut v: Vec<&str> = Vec::new();
let arr = bits & cubecoords::WordFlags::ARR_MASK;
if arr == 0 { v.push("tri6"); } else if arr == 1 { v.push("pair"); }
else if arr == 2 { v.push("triad"); } else { v.push("quad"); }
if bits & cubecoords::WordFlags::START_RECORD != 0 { v.push("start"); }
if bits & cubecoords::WordFlags::END_RECORD != 0 { v.push("end"); }
if bits & cubecoords::WordFlags::IS_HEADER != 0 { v.push("header"); }
if bits & cubecoords::WordFlags::CONTINUATION != 0 { v.push("cont"); }
let ty = (bits & cubecoords::WordFlags::TYPE_MASK) >> 6;
match ty { 0 => v.push("type=text"), 1 => v.push("type=binary"), 2 => v.push("type=code"), _ => v.push("type=ai") }
if bits & cubecoords::WordFlags::PERM_ROOT != 0 { v.push("root"); }
if bits & cubecoords::WordFlags::PERM_USER != 0 { v.push("user"); }
if bits & cubecoords::WordFlags::ASSOC_EDGE != 0 { v.push("assoc"); }
if bits & cubecoords::WordFlags::ENCRYPTED != 0 { v.push("encrypted"); }
if bits & cubecoords::WordFlags::COMPRESSED != 0 { v.push("compressed"); }
if bits & cubecoords::WordFlags::STEGO != 0 { v.push("stego"); }
if bits & cubecoords::WordFlags::NULL_LOOKUP != 0 { v.push("null"); }
if bits & cubecoords::WordFlags::CHECKSUM != 0 { v.push("checksum"); }
format!("({})", v.join("|"))
}
fn parse_word_flag(a: &str) -> Result<u16, String> {
let lower = a.to_lowercase();
let bits = match lower.as_str() {
"start" => cubecoords::WordFlags::START_RECORD,
"end" => cubecoords::WordFlags::END_RECORD,
"header" => cubecoords::WordFlags::IS_HEADER,
"cont" | "continuation" => cubecoords::WordFlags::CONTINUATION,
"text" | "type-text" => 0 << 6,
"binary" | "type-binary" => 1 << 6,
"code" | "type-code" => 2 << 6,
"ai" | "type-ai" => 3 << 6,
"root" => cubecoords::WordFlags::PERM_ROOT,
"user" => cubecoords::WordFlags::PERM_USER,
"assoc" | "assoc-edge" => cubecoords::WordFlags::ASSOC_EDGE,
"encrypted" | "enc" => cubecoords::WordFlags::ENCRYPTED,
"compressed" | "comp" => cubecoords::WordFlags::COMPRESSED,
"stego" => cubecoords::WordFlags::STEGO,
"null" | "null-lookup" => cubecoords::WordFlags::NULL_LOOKUP,
"checksum" => cubecoords::WordFlags::CHECKSUM,
other => match other.strip_prefix("0x") {
Some(hex) => return u16::from_str_radix(hex, 16)
.map_err(|_| format!("invalid hex word flag '{a}'")),
None => return other.parse::<u16>()
.map_err(|_| format!("unknown word flag '{a}'")),
},
};
Ok(bits)
}
fn parse_u8(t: Option<&str>, what: &str) -> Result<u8, String> {
let t = t.ok_or_else(|| what.to_string())?;
t.parse::<u8>()
@@ -1101,6 +1791,82 @@ mod tests {
Session::with_store(Arc::new(ConcurrentStore::memory()))
}
#[test]
fn os_kernels_live_in_cube_with_call_graph_and_descriptors() {
// Step 1: OS operator behavior lives in CUBE as kernels, composed via
// a call graph (linked_records) and tagged with behavior descriptors.
// Step 2: a thin native effector (`native-apply`) runs each kernel in
// the VM, reads its computed result, and emits the OS effect — the
// cubevm itself never does store-IO (spec-aligned, PDF §524525).
let mut s = session();
// Lay down the OS kernel call graph.
let out = s.exec("tick").expect("tick should lay down kernels");
assert!(
out.contains("normalize-config"),
"cfg kernel missing: {out}"
);
assert!(
out.contains("decide-snapshot"),
"decide kernel missing: {out}"
);
assert!(
out.contains("summarize-procs"),
"summarize kernel missing: {out}"
);
assert!(out.contains("cube-os-tick"), "tick kernel missing: {out}");
// The root links the three leaves (call graph, not foreign code).
assert!(
out.contains("links cfg,decide,summarize"),
"call graph not wired: {out}"
);
// Behavior descriptors are stamped (round-trip through header flags).
assert!(
out.contains("[pure]") && out.contains("[io]") && out.contains("[pure,hot]"),
"behavior descriptors not stamped: {out}"
);
// Run the root kernel: it must traverse the call graph (CallLink 0..2)
// and return, proving the OS's behavior lives as addressable kernels.
let run_out = s
.exec("run /c210/z001/y001/x004")
.expect("tick kernel must run");
assert!(
run_out.contains("Halted"),
"tick kernel should halt: {run_out}"
);
// Step 2 — the effector reads the COMPUTED result and emits the effect.
let eff = s
.exec("native-apply /c210/z001/y001/x002")
.expect("effector must run decide-snapshot");
assert!(
eff.contains("computed result = 1"),
"decide kernel result wrong: {eff}"
);
assert!(
eff.contains("OS EFFECT"),
"effector must emit OS EFFECT: {eff}"
);
assert!(
eff.contains("snapshot NOW"),
"decision=1 should snapshot: {eff}"
);
// A plain pure kernel also routes through the effector with no store-IO.
let eff2 = s
.exec("native-apply /c210/z001/y001/x003")
.expect("effector must run summarize-procs");
assert!(
eff2.contains("computed result = 20"),
"summarize result wrong: {eff2}"
);
assert!(
eff2.contains("descriptors=[\"pure\", \"hot\"]"),
"descriptor readback wrong: {eff2}"
);
}
#[test]
fn begin_commit_applies_buffered_writes() {
let mut s = session();
@@ -1803,4 +2569,25 @@ mod tests {
);
let _ = std::fs::remove_dir_all(&dir);
}
#[test]
fn header_for_code_sets_word_flags() {
// Fn cell with a security-sensitive descriptor -> per-word flags set.
let h = header_for_code(
Kind::Fn,
"f",
&[],
None,
Some(cubecode::Behavior::new(cubecode::Behavior::SECURITY_SENSITIVE)),
);
assert!(h.word_flags.has(cubecoords::WordFlags::START_RECORD));
assert!(h.word_flags.has(cubecoords::WordFlags::END_RECORD));
assert!(h.word_flags.has(cubecoords::WordFlags::IS_HEADER));
assert!(h.word_flags.has(cubecoords::WordFlags::ENCRYPTED), "sec descriptor -> ENCRYPTED");
assert_eq!((h.word_flags.bits() & cubecoords::WordFlags::TYPE_MASK), 0b10 << 6, "Fn -> type=code");
// No descriptor -> complete frame, no security bit.
let h2 = header_for_code(Kind::Fn, "g", &[], None, None);
assert!(h2.word_flags.has(cubecoords::WordFlags::START_RECORD));
assert!(!h2.word_flags.has(cubecoords::WordFlags::ENCRYPTED));
}
}
+2
View File
@@ -276,6 +276,7 @@ fn write_bucket(store: &ConcurrentStore, grants: &[Grant]) {
pub fn grant_header() -> CubeHeader {
CubeHeader {
flags: cubecoords::HeaderFlags(cubecoords::HeaderFlags::DOC_TYPE),
word_flags: cubecoords::WordFlags::default(),
title: None,
doc_type: Some("grant-table".to_string()),
created_at: None,
@@ -283,6 +284,7 @@ pub fn grant_header() -> CubeHeader {
owner_local_user: None,
owner_remote_user: None,
linked_records: Vec::new(),
path: None,
total_accesses: 0,
total_remote_accesses: 0,
last_access: None,
+45
View File
@@ -54,6 +54,11 @@ pub mod commands;
pub mod grants;
/// Length-framed Unix-domain-socket transport shared by client and server.
pub mod net;
/// Session-message / note log: a searchable, per-session, per-day note area
/// stored as CZYX records carrying WordFlags metadata. This is the structured
/// replacement for the hand-maintained `RESUME-*.md` files so a new session
/// can review prior sessions' messages over the cube store (see [`notes`]).
pub mod notes;
/// Dependency-free JSON snapshot load/dump for daemon store persistence.
pub mod persist;
/// Concurrent, durable store: mutex-wrapped [`CubeStore`] + NDJSON write-ahead
@@ -133,6 +138,35 @@ pub fn load_code_cell<B: CubeBackend>(
/// Build a [`CodeCell`] from parts and write it back as a record at `path`,
/// so code written by the VM (or tooling) is visible to cubefs at that path.
#[allow(clippy::too_many_arguments)]
/// Per-word flags (tag 14) stamped on OS-created records: a complete single
/// record frame + a type note (code cells = type=code) + a security note, so the
/// store actually carries WordFlags metadata (not just stores it).
pub(crate) fn default_word_flags(
kind: Kind,
descriptor: Option<cubecode::Behavior>,
) -> cubecoords::WordFlags {
let mut wf = cubecoords::WordFlags::from_bits(
cubecoords::WordFlags::START_RECORD
| cubecoords::WordFlags::END_RECORD
| cubecoords::WordFlags::IS_HEADER,
);
if matches!(kind, Kind::Fn | Kind::Kernel) {
wf.0 |= 0b10 << 6; // type=code
}
if let Some(b) = descriptor {
if b.0 & cubecode::Behavior::SECURITY_SENSITIVE != 0 {
wf.0 |= cubecoords::WordFlags::ENCRYPTED;
}
}
wf
}
/// Store a CUBEVM code cell at the coordinate derived from `path`. Builds the
/// record header (title/doc_type/owner) and stamps per-word flags (tag 14) via
/// [`default_word_flags`], so OS-created code cells carry WordFlags metadata.
/// Returns the resolved coordinate; used by the `prog`/`write`/`kernel`/`fn`
/// OS commands and the `cubefs` socket backend.
pub fn store_code_cell<B: CubeBackend>(
store: &mut CubeStore<B>,
path: &str,
@@ -141,6 +175,10 @@ pub fn store_code_cell<B: CubeBackend>(
links: &[Czyx],
code: &[cubecode::Op],
owner: Option<&str>,
// Behavior-descriptor flags (PDF §524525: pure / I/O heavy / hot path) to
// stamp on the record header's out-of-band bits. `None` leaves the field
// at zero. See `cubecode::Behavior`.
descriptor: Option<cubecode::Behavior>,
) -> Result<Czyx, SysError> {
let coord = path_to_czyx(path)?;
let mut h = CubeHeader::new();
@@ -148,6 +186,10 @@ pub fn store_code_cell<B: CubeBackend>(
h.doc_type = Some(kind.as_str().to_string());
h.linked_records = links.to_vec();
h.owner_local_user = owner.map(|o| o.to_string());
h.word_flags = default_word_flags(kind, descriptor);
if let Some(b) = descriptor {
h.flags.0 |= b.to_flags();
}
if h.doc_type.as_deref() == Some("fn") {
h.size_bytes = Some(cubecode::encode(code).len() as u64);
}
@@ -187,6 +229,7 @@ mod tests {
&[],
&[Op::Const(2), Op::Const(3), Op::Add, Op::Halt],
None,
None,
)
.unwrap();
@@ -300,6 +343,7 @@ pub mod demo {
&[],
&double_code,
None,
None,
)
.expect("store double");
let entry_coord = super::store_code_cell(
@@ -310,6 +354,7 @@ pub mod demo {
&[double_coord],
&entry_code,
None,
None,
)
.expect("store entry");
println!(" wrote {double} -> coord {}", double_coord.pack_u32());
+754
View File
@@ -0,0 +1,754 @@
//! CUBELinux session-message / note log with **categories** and **projects**.
//!
//! The "message save path" helper. Each note is a CZYX record with a text-type
//! [`WordFlags`] stamp, a `created_at` day, and the session label as the owner.
//! Two extra, searchable dimensions layer on top:
//! * **category** — encoded in `doc_type` as `note:<category>` (so
//! [`scan_by_type`] / filtering slices notes by kind: design, impl, cube, ...).
//! * **project** — notes are *associated* (via `linked_records`) to a
//! `project` record (`doc_type = "project"`), so `cube project walk`
//! follows the links into a project's timeline/thread. This is the
//! CUBELinux first-class-association model (EDG) applied to project work.
//!
//! Cross-session: `created_at` (day) + session label + category + project make
//! any note recoverable as `note search <text> [--project p] [--cat c]`, and a
//! new harness session can `note list --project p` / `project walk` to see where
//! a project left off.
use std::time::{SystemTime, UNIX_EPOCH};
use cubecoords::{CubeHeader, Czyx, WordFlags};
use crate::store::ConcurrentStore;
/// Base `doc_type` for a note without a category.
pub const NOTE_DOC_TYPE: &str = "note";
/// `doc_type` for a project record.
pub const PROJECT_DOC_TYPE: &str = "project";
/// The default session label for CLI-created notes (today's calendar date).
pub fn default_session() -> String {
date_string(now())
}
/// The WordFlags stamped on a note: a complete single-record frame with the
/// text data type (type bits 00). Other metadata bits (encrypted, assoc, ...)
/// are left unset for the caller to raise as needed.
pub fn note_word_flags() -> WordFlags {
WordFlags::from_bits(
WordFlags::START_RECORD | WordFlags::END_RECORD | WordFlags::IS_HEADER,
)
}
/// Current unix time (seconds).
pub fn now() -> u64 {
SystemTime::now()
.duration_since(UNIX_EPOCH)
.map(|d| d.as_secs())
.unwrap_or(0)
}
/// A returned note (coordinate + header + body bytes).
#[derive(Clone, Debug)]
pub struct Note {
/// The note's coordinate (`C.Z.Y.X`).
pub coord: Czyx,
/// The note's record header (title, session label, WordFlags, category).
pub header: CubeHeader,
/// The note's body text as raw bytes.
pub body: Vec<u8>,
}
/// Query dimensions for listing/searching notes. `since`/`until` are
/// day-of-epoch (see [`date_to_day`]); the rest are exact-match labels.
#[derive(Clone, Default, Debug)]
pub struct NoteFilter {
/// Session label (owner_local_user).
pub session: Option<String>,
/// Project name (note must be associated to that project record).
pub project: Option<String>,
/// Category (doc_type `note:<category>`).
pub category: Option<String>,
/// Earliest day-of-epoch, inclusive.
pub since: Option<u64>,
/// Latest day-of-epoch, inclusive.
pub until: Option<u64>,
}
/// The category portion of a note's `doc_type`, if any (`note:<category>`).
pub fn category_of(doc_type: &str) -> Option<&str> {
doc_type.strip_prefix(&format!("{NOTE_DOC_TYPE}:"))
}
fn fnv1a(bytes: &[u8]) -> u64 {
let mut h: u64 = 0xcbf2_9ce4_8422_2325;
for &b in bytes {
h ^= b as u64;
h = h.wrapping_mul(0x0000_0100_0000_01b3);
}
h
}
fn non_zero(b: u8) -> u8 {
if b == 0 {
1
} else {
b
}
}
/// The single-byte notes class for a session label, so `scan_prefix(class,
/// None, None)` returns exactly that session's notes.
fn session_class(session: &str) -> u8 {
let h = fnv1a(session.as_bytes());
1 + ((h >> 24) as u8 & 0x7f) // 1..=128
}
/// Project records live in the upper class range (128..=255) so they never
/// collide with note session classes (1..=128).
fn project_class(name: &str) -> u8 {
let h = fnv1a(name.as_bytes());
128 + ((h >> 24) as u8 & 0x7f)
}
/// Derive a deterministic, collision-light coordinate for a note: the C axis
/// is the session class, Z/Y/X come from the note content + a salt.
fn note_coord(session: &str, created: u64, subject: &str, body: &str, salt: u64) -> Czyx {
let mut mix = fnv1a(session.as_bytes());
mix ^= created.wrapping_mul(0x9e37_79b9);
mix ^= fnv1a(subject.as_bytes());
mix ^= fnv1a(body.as_bytes());
mix ^= salt.wrapping_mul(0x85eb_ca6b);
Czyx::new(
session_class(session),
non_zero((mix >> 16) as u8),
non_zero((mix >> 8) as u8),
non_zero(mix as u8 ^ (salt as u8)),
)
}
/// Derive the deterministic coordinate of a project record.
fn project_coord(name: &str, salt: u64) -> Czyx {
let mut mix = fnv1a(name.as_bytes());
mix ^= salt.wrapping_mul(0x8549_b2a1);
Czyx::new(
project_class(name),
non_zero((mix >> 16) as u8),
non_zero((mix >> 8) as u8),
non_zero(mix as u8 ^ (salt as u8)),
)
}
/// Find an existing project record by name, without creating it.
pub fn find_project(store: &ConcurrentStore, name: &str) -> Option<Czyx> {
store
.read_snapshot()
.query_by_type(PROJECT_DOC_TYPE)
.into_iter()
.find(|(_, h, _)| {
h.title.as_deref() == Some(name)
|| h.doc_type.as_deref() == Some(PROJECT_DOC_TYPE) && h.path.as_deref() == Some(name)
})
.map(|(c, _, _)| c)
}
/// Find-or-create a project record named `name`, returning its coordinate.
/// Notes tagged with a project are associated to it via `linked_records`.
pub fn ensure_project(store: &ConcurrentStore, name: &str, session: &str) -> Result<Czyx, String> {
if let Some(c) = find_project(store, name) {
return Ok(c);
}
let mut salt = 0u64;
let mut coord = project_coord(name, salt);
while store.get_record(&coord).is_some() && salt < 4096 {
salt += 1;
coord = project_coord(name, salt);
}
if salt >= 4096 {
return Err("project store: could not find a free coordinate".into());
}
let mut h = CubeHeader::new();
h.title = Some(name.to_string());
h.path = Some(name.to_string());
h.doc_type = Some(PROJECT_DOC_TYPE.to_string());
h.created_at = Some(now());
h.owner_local_user = Some(session.to_string());
h.refresh_flags();
store.put_record(coord, &h, &[]);
Ok(coord)
}
/// Store a note record under `session`, with a `subject` title and `body`
/// text. `project` (if given) associates the note to a project record; `category`
/// (if given) sets the note's `doc_type` to `note:<category>`. Returns the note's
/// coordinate.
pub fn store_note(
store: &ConcurrentStore,
session: &str,
subject: &str,
body: &str,
project: Option<&str>,
category: Option<&str>,
) -> Result<Czyx, String> {
let created = now();
let proj_coord = match project {
Some(p) => Some(ensure_project(store, p, session)?),
None => None,
};
let mut salt = 0u64;
let mut coord = note_coord(session, created, subject, body, salt);
while store.get_record(&coord).is_some() && salt < 4096 {
salt += 1;
coord = note_coord(session, created, subject, body, salt);
}
if salt >= 4096 {
return Err("note store: could not find a free coordinate (4096 tries)".into());
}
let mut h = CubeHeader::new();
h.title = Some(subject.to_string());
h.doc_type = Some(match category {
Some(c) => format!("{NOTE_DOC_TYPE}:{c}"),
None => NOTE_DOC_TYPE.to_string(),
});
h.created_at = Some(created);
h.owner_local_user = Some(session.to_string());
if let Some(pc) = proj_coord {
h.linked_records = vec![pc];
}
h.word_flags = note_word_flags();
h.refresh_flags();
store.put_record(coord, &h, body.as_bytes());
Ok(coord)
}
/// All note records (`doc_type == "note"` or `"note:<cat>"`), regardless of
/// category, so a category filter is applied in Rust (the store's exact
/// `scan_by_type("note")` would miss categorized notes).
fn all_notes(snap: &cubestore::CubeStore<cubestore::HashBackend>) -> Vec<Note> {
snap.keys()
.into_iter()
.filter_map(|coord| {
let (header, body) = snap.get_record(&coord)?;
let dt = header.doc_type.as_deref().unwrap_or("");
if dt == NOTE_DOC_TYPE || dt.starts_with(&format!("{NOTE_DOC_TYPE}:")) {
Some(Note {
coord,
header,
body,
})
} else {
None
}
})
.collect()
}
/// List notes, newest first, filtered by [`NoteFilter`].
pub fn list_notes(store: &ConcurrentStore, filter: Option<&NoteFilter>) -> Vec<Note> {
let snap = store.read_snapshot();
let mut out: Vec<Note> = all_notes(&snap)
.into_iter()
.filter(|n| note_matches_filtered(store, n, filter))
.collect();
out.sort_by(|a, b| b.header.created_at.cmp(&a.header.created_at));
out
}
fn note_matches_filtered(store: &ConcurrentStore, n: &Note, filter: Option<&NoteFilter>) -> bool {
let Some(f) = filter else {
return true;
};
if let Some(s) = &f.session {
if n.header.owner_local_user.as_deref() != Some(s.as_str()) {
return false;
}
}
if let Some(cat) = &f.category {
let expected = format!("{NOTE_DOC_TYPE}:{cat}");
if n.header.doc_type.as_deref() != Some(expected.as_str()) {
return false;
}
}
if let Some(sd) = f.since {
if n.header.created_at.map(|c| c / 86_400).unwrap_or(0) < sd {
return false;
}
}
if let Some(ud) = f.until {
if n.header.created_at.map(|c| c / 86_400).unwrap_or(u64::MAX) > ud {
return false;
}
}
if let Some(p) = &f.project {
if let Some(pc) = find_project(store, p) {
if !n.header.linked_records.contains(&pc) && n.header.path.as_deref() != Some(p.as_str()) {
return false;
}
} else {
return false;
}
}
true
}
/// Search all notes (session + body) for `needle`, filtered, newest first.
pub fn search_notes(store: &ConcurrentStore, needle: &str, filter: Option<&NoteFilter>) -> Vec<Note> {
let needle = needle.to_lowercase();
let snap = store.read_snapshot();
let mut out: Vec<Note> = all_notes(&snap)
.into_iter()
.filter(|n| note_matches_filtered(store, n, filter))
.filter(|n| {
let subject = n.header.title.as_deref().unwrap_or("").to_lowercase();
let b = String::from_utf8_lossy(&n.body).to_lowercase();
subject.contains(&needle) || b.contains(&needle)
})
.collect();
out.sort_by(|a, b| b.header.created_at.cmp(&a.header.created_at));
out
}
/// Fetch a single note by coordinate.
pub fn show_note(store: &ConcurrentStore, coord: Czyx) -> Option<Note> {
store.get_record(&coord).map(|(header, body)| Note {
coord,
header,
body,
})
}
/// List all project records, newest first.
pub fn project_list(store: &ConcurrentStore) -> Vec<(Czyx, CubeHeader, Vec<u8>)> {
let mut v = store.read_snapshot().query_by_type(PROJECT_DOC_TYPE);
v.sort_by(|a, b| b.1.created_at.cmp(&a.1.created_at));
v
}
/// Return the notes associated to a project record (its thread/timeline),
/// newest first.
pub fn project_walk(store: &ConcurrentStore, coord: Czyx) -> Vec<Note> {
let snap = store.read_snapshot();
let mut out: Vec<Note> = all_notes(&snap)
.into_iter()
.filter(|n| n.header.linked_records.contains(&coord))
.collect();
out.sort_by(|a, b| b.header.created_at.cmp(&a.header.created_at));
out
}
/// Format a unix timestamp as `YYYY-MM-DD` (UTC).
pub fn date_string(ts: u64) -> String {
let days = (ts / 86_400) as i64;
let (y, m, d) = civil_from_days(days);
format!("{y:04}-{m:02}-{d:02}")
}
/// Parse `YYYY-MM-DD` into a day-of-epoch (days since 1970-01-01).
pub fn date_to_day(s: &str) -> Option<u64> {
let mut it = s.split('-');
let y: i64 = it.next()?.parse().ok()?;
let m: u32 = it.next()?.parse().ok()?;
let d: u32 = it.next()?.parse().ok()?;
if it.next().is_some() || !(1..=12).contains(&m) || !(1..=31).contains(&d) {
return None;
}
Some(days_from_civil(y, m, d))
}
/// Days since 1970-01-01 for a civil date (Howard Hinnant's algorithm).
fn days_from_civil(y: i64, m: u32, d: u32) -> u64 {
let yy = if m <= 2 { y - 1 } else { y };
let era = if yy >= 0 { yy } else { yy - 399 } / 400;
let yoe = yy - era * 400;
let mp = (m as i64 + 9) % 12;
let doy = (153 * mp + 2) / 5 + d as i64 - 1;
let doe = yoe * 365 + yoe / 4 - yoe / 100 + doy;
(era * 146_097 + doe - 719_468).max(0) as u64
}
fn civil_from_days(days: i64) -> (i64, u32, u32) {
let z = days + 719_468;
let era = if z >= 0 { z } else { z - 146_096 } / 146_097;
let doe = z - era * 146_097;
let yoe = (doe - doe / 1460 + doe / 36_524 - doe / 146_096) / 365;
let y = yoe + era * 400;
let doy = doe - (365 * yoe + yoe / 4 - yoe / 100);
let mp = (5 * doy + 2) / 153;
let d = (doy - (153 * mp + 2) / 5 + 1) as u32;
let m = if mp < 10 { mp + 3 } else { mp - 9 } as u32;
(if m <= 2 { y + 1 } else { y }, m, d)
}
/// Render a coordinate as `C.Z.Y.X` (the [`crate::commands::parse_coord`]
/// form), so `note show <coord>` round-trips.
fn coord_str(c: Czyx) -> String {
format!("{}.{}.{}.{}", c.c, c.z, c.y, c.x)
}
/// Subject = the first line-ish portion of the note text (first 60 chars).
fn subject_of(text: &str) -> String {
let s = text.trim();
let s: String = s.chars().take(60).collect();
if s.len() < text.trim().len() {
format!("{s}")
} else {
s
}
}
fn format_notes(notes: &[Note]) -> String {
if notes.is_empty() {
return "(no notes)".to_string();
}
let lines: Vec<String> = notes
.iter()
.map(|n| {
let subj = n.header.title.as_deref().unwrap_or("");
let when = n
.header
.created_at
.map(|t| date_string(t))
.unwrap_or_else(|| "?".into());
let cat = n
.header
.doc_type
.as_deref()
.and_then(category_of)
.map(|c| format!(" [{c}]"))
.unwrap_or_default();
format!(" {} [{}]{} {}", coord_str(n.coord), when, cat, subj)
})
.collect();
format!("{} note(s):\n{}", notes.len(), lines.join("\n"))
}
/// Parse `--key value` flags out of a command string; returns `(flags,
/// remainingPositionalJoined)`. A flag with no following token gets `"true"`.
fn split_flags(s: &str) -> (Vec<(String, String)>, String) {
let tokens: Vec<&str> = s.split_whitespace().collect();
let mut flags = Vec::new();
let mut pos: Vec<&str> = Vec::new();
let mut i = 0;
while i < tokens.len() {
let t = tokens[i];
if let Some(k) = t.strip_prefix("--") {
let (key, val) = if i + 1 < tokens.len() && !tokens[i + 1].starts_with("--") {
i += 1;
(k.to_string(), tokens[i].to_string())
} else {
(k.to_string(), "true".to_string())
};
flags.push((key, val));
} else {
pos.push(t);
}
i += 1;
}
(flags, pos.join(" "))
}
/// Process one `note`/`notes` command line against `store`.
///
/// Syntax (REPL/CLI):
/// `note <text>` quick note
/// `note add [--project P] [--cat C] <text>` add a note to a project/category
/// `note list [session] [--project P] [--cat C] [--since D] [--until D]`
/// `note search <text> [--project P] [--cat C]`
/// `note show <C.Z.Y.X>`
/// `project list` / `project show <C.Z.Y.X>` (see [`project_command`])
pub fn note_command(store: &ConcurrentStore, line: &str) -> Result<String, String> {
let body = line
.splitn(2, char::is_whitespace)
.nth(1)
.unwrap_or("")
.trim()
.to_string();
let Some((head, rest)) = split_head(&body) else {
let session = default_session();
let notes = list_notes(store, Some(&NoteFilter { session: Some(session), ..Default::default() }));
return Ok(format_notes(&notes));
};
match head {
"list" => {
let (flags, pos) = split_flags(rest);
let f = filter_from_flags(&flags);
let session = if pos.is_empty() { None } else { Some(pos.clone()) };
let f = NoteFilter { session, ..f };
let notes = list_notes(store, Some(&f));
Ok(format_notes(&notes))
}
"search" => {
let (flags, pos) = split_flags(rest);
let f = filter_from_flags(&flags);
if pos.is_empty() {
return Err("note search needs <text>".to_string());
}
let notes = search_notes(store, &pos, Some(&f));
Ok(format!("search '{pos}':\n{}", format_notes(&notes)))
}
"show" => {
let cs = rest.trim();
let coord = crate::commands::parse_coord(cs)
.ok_or_else(|| format!("bad coordinate '{cs}' (want C.Z.Y.X)"))?;
match show_note(store, coord) {
Some(n) => {
let subj = n.header.title.as_deref().unwrap_or("");
let when = n
.header
.created_at
.map(|t| date_string(t))
.unwrap_or_else(|| "?".into());
Ok(format!(
"note {cs} [{}] {}\n{}",
when,
subj,
String::from_utf8_lossy(&n.body)
))
}
None => Ok(format!("note {cs} -> (no note)")),
}
}
"add" | _ => {
let (flags, pos) = split_flags(rest);
let mut project = None;
let mut category = None;
for (k, v) in &flags {
match k.as_str() {
"project" | "p" => project = Some(v.clone()),
"cat" | "category" | "c" => category = Some(v.clone()),
_ => {}
}
}
let text = pos.trim();
if text.is_empty() {
return Err("note needs <text>".to_string());
}
let session = default_session();
let subject = subject_of(text);
let coord = store_note(store, &session, &subject, text, project.as_deref(), category.as_deref())?;
let proj = project.map(|p| format!(" (project {p})")).unwrap_or_default();
Ok(format!("note saved at {} (session {}){proj}", coord_str(coord), session))
}
}
}
fn filter_from_flags(flags: &[(String, String)]) -> NoteFilter {
let mut f = NoteFilter::default();
for (k, v) in flags {
match k.as_str() {
"project" | "p" => f.project = Some(v.clone()),
"cat" | "category" | "c" => f.category = Some(v.clone()),
"since" => f.since = date_to_day(v),
"until" => f.until = date_to_day(v),
_ => {}
}
}
f
}
/// Set a "resume" marker for a project. Stored as a note tagged
/// `category = "resume"` and associated to the project, so a new session can
/// ask "where did this project leave off?" and get it back.
pub fn project_resume(
store: &ConcurrentStore,
name: &str,
text: &str,
session: &str,
) -> Result<Czyx, String> {
let subject = format!("resume: {name}");
store_note(store, session, &subject, text, Some(name), Some("resume"))
}
/// Build a compact "project state" summary for context injection: the project
/// name, its note count, the resume marker (if any), and the latest notes with
/// date/category. A new session reads this to know where a project left off.
pub fn project_context(store: &ConcurrentStore, name: &str) -> String {
let Some(coord) = find_project(store, name) else {
return format!("(no project '{name}')");
};
let notes = project_walk(store, coord);
let mut out = String::new();
out.push_str(&format!("Project: {name}\n"));
out.push_str(&format!(" {} note(s); thread {}.\n", notes.len(), coord_str(coord)));
if let Some(r) = notes
.iter()
.find(|n| category_of(n.header.doc_type.as_deref().unwrap_or("")) == Some("resume"))
{
out.push_str(&format!(
" RESUME: {}\n",
String::from_utf8_lossy(&r.body).trim()
));
}
out.push_str(" Latest:\n");
for n in notes.iter().take(10) {
let subj = n.header.title.as_deref().unwrap_or("");
let when = n.header.created_at.map(|t| date_string(t)).unwrap_or_else(|| "?".into());
let cat = category_of(n.header.doc_type.as_deref().unwrap_or(""))
.map(|c| format!("[{c}] "))
.unwrap_or_default();
out.push_str(&format!(" {when} {cat}{subj}\n"));
}
out
}
/// Process a `project` command line: `project list`, `project show <C.Z.Y.X>`,
/// `project resume <name> <text>`, `project context <name>`.
pub fn project_command(store: &ConcurrentStore, line: &str) -> Result<String, String> {
let body = line
.splitn(2, char::is_whitespace)
.nth(1)
.unwrap_or("")
.trim()
.to_string();
let Some((head, rest)) = split_head(&body) else {
return Ok(format_projects(project_list(store)));
};
match head {
"list" => Ok(format_projects(project_list(store))),
"show" | "walk" => {
let cs = rest.trim();
let coord = crate::commands::parse_coord(cs)
.ok_or_else(|| format!("bad coordinate '{cs}' (want C.Z.Y.X)"))?;
match show_note(store, coord) {
Some(p) => {
let name = p.header.title.as_deref().unwrap_or("?");
let notes = project_walk(store, coord);
Ok(format!(
"project {cs} \"{name}\" — {} note(s):\n{}",
notes.len(),
format_notes(&notes)
))
}
None => Ok(format!("project {cs} -> (no project record)")),
}
}
"resume" => {
let (_, pos) = split_flags(rest);
let mut words = pos.splitn(2, char::is_whitespace);
let name = words.next().unwrap_or("").to_string();
let text = words.next().unwrap_or("").trim().to_string();
if name.is_empty() || text.is_empty() {
return Err("project resume needs <name> <text>".to_string());
}
let session = default_session();
let coord = project_resume(store, &name, &text, &session)?;
Ok(format!("resume set for '{name}' at {} (session {})", coord_str(coord), session))
}
"context" => {
let name = rest.trim();
if name.is_empty() {
return Err("project context needs <name>".to_string());
}
Ok(project_context(store, name))
}
_ => Err(format!("project: unknown subcommand '{head}' (want list|show|resume|context)")),
}
}
fn format_projects(projects: Vec<(Czyx, CubeHeader, Vec<u8>)>) -> String {
if projects.is_empty() {
return "(no projects)".to_string();
}
let lines: Vec<String> = projects
.iter()
.map(|(c, h, _)| {
let name = h.title.as_deref().unwrap_or("");
let when = h.created_at.map(|t| date_string(t)).unwrap_or_else(|| "?".into());
format!(" {} [{}] {}", coord_str(*c), when, name)
})
.collect();
format!("{} project(s):\n{}", projects.len(), lines.join("\n"))
}
/// Split the first whitespace-delimited token from the rest of a string.
fn split_head(s: &str) -> Option<(&str, &str)> {
let idx = s.find(char::is_whitespace)?;
Some((&s[..idx], &s[idx..].trim()))
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn store_and_list_a_note() {
let store = ConcurrentStore::memory();
let coord = store_note(&store, "test-session", "hello", "world note", None, None).unwrap();
assert!(store.get_record(&coord).is_some());
let all = list_notes(&store, None);
assert_eq!(all.len(), 1);
assert_eq!(all[0].header.doc_type.as_deref(), Some(NOTE_DOC_TYPE));
assert_eq!(all[0].header.owner_local_user.as_deref(), Some("test-session"));
assert_eq!(all[0].body, b"world note");
assert!(all[0].header.word_flags.has(WordFlags::START_RECORD));
assert!(!all[0].header.word_flags.has(WordFlags::ENCRYPTED));
}
#[test]
fn session_and_day_filters() {
let store = ConcurrentStore::memory();
store_note(&store, "a", "one", "first", None, None).unwrap();
store_note(&store, "b", "two", "second", None, None).unwrap();
assert_eq!(
list_notes(&store, Some(&NoteFilter { session: Some("a".into()), ..Default::default() })).len(),
1
);
assert_eq!(list_notes(&store, None).len(), 2);
let today = now() / 86_400;
assert_eq!(
list_notes(&store, Some(&NoteFilter { since: Some(today), ..Default::default() })).len(),
2
);
}
#[test]
fn category_and_project_filters() {
let store = ConcurrentStore::memory();
let a = store_note(&store, "s", "design", "widget design", Some("cubelinux"), Some("design")).unwrap();
let b = store_note(&store, "s", "impl", "widget impl", None, Some("impl")).unwrap();
let c = store_note(&store, "s", "cube", "cube notes", None, None).unwrap();
// category filter
assert_eq!(
list_notes(&store, Some(&NoteFilter { category: Some("design".into()), ..Default::default() })).len(),
1
);
// all categorized + base notes are still found by an unfiltered list
assert_eq!(
list_notes(&store, Some(&NoteFilter { session: Some("s".into()), ..Default::default() })).len(),
3
);
// project filter (a is linked to project cubelinux)
let pc = ensure_project(&store, "cubelinux", "s").unwrap();
assert_eq!(store.get_record(&a).unwrap().0.linked_records, vec![pc]);
assert_eq!(store.get_record(&b).unwrap().0.linked_records, vec![]);
assert_eq!(store.get_record(&c).unwrap().0.linked_records, vec![]);
let proj_notes = list_notes(&store, Some(&NoteFilter { project: Some("cubelinux".into()), ..Default::default() }));
assert_eq!(proj_notes.len(), 1);
assert_eq!(proj_notes[0].header.title.as_deref(), Some("design"));
// project walk returns the thread
assert_eq!(project_walk(&store, pc).len(), 1);
}
#[test]
fn full_text_search_and_dates() {
let store = ConcurrentStore::memory();
store_note(&store, "s", "kernel", "the ipu4 camera driver linked", None, None).unwrap();
store_note(&store, "s", "mail", "opendkim signing works", None, None).unwrap();
assert_eq!(search_notes(&store, "ipu4", None).len(), 1);
assert_eq!(search_notes(&store, "opendkim", None).len(), 1);
assert_eq!(search_notes(&store, "nomatchxyz", None).len(), 0);
assert_eq!(date_to_day("2026-09-08"), Some(20_704));
assert_eq!(date_string(20_704 * 86_400), "2026-09-08");
}
#[test]
fn project_resume_and_context() {
let store = ConcurrentStore::memory();
store_note(&store, "s", "design", "widget design", Some("p"), Some("design")).unwrap();
project_resume(&store, "p", "at the design review", "s").unwrap();
let ctx = project_context(&store, "p");
assert!(ctx.contains("Project: p"));
assert!(ctx.contains("RESUME: at the design review"));
assert!(ctx.contains("[design]"));
}
}
+22 -1
View File
@@ -574,6 +574,25 @@ impl ConcurrentStore {
out
}
/// Reconstruct a directory listing from `path` metatags: every record
/// whose `path` header field is under `dir` (the PDF's "operate on CZYX
/// records and Null-space flags rather than paths and inodes"). Delegates
/// to [`CubeStore::scan_by_path_prefix`].
pub fn query_by_path_prefix(&self, dir: &str) -> Vec<Czyx> {
self.inner.read().unwrap().scan_by_path_prefix(dir)
}
/// Exact-path metatag lookup. Delegates to [`CubeStore::scan_by_path`].
pub fn query_by_path(&self, path: &str) -> Vec<Czyx> {
self.inner.read().unwrap().scan_by_path(path)
}
/// Query records by per-word flag (the 16-bit WordFlags metadata stripe in
/// the header, tag 14). Delegates to [`CubeStore::scan_by_word_flag`].
pub fn scan_by_word_flag(&self, flag: u16) -> Vec<Czyx> {
self.inner.read().unwrap().scan_by_word_flag(flag)
}
/// The `owner_local_user` stamped on the record at `key`, if it has one.
/// Used by owner enforcement (Task 6): a mutating command may only
/// overwrite a record whose owner matches the session's identity owner.
@@ -683,6 +702,7 @@ impl ConcurrentStore {
/// Store a code cell at `path` (the path->code bridge), durability-logged.
/// `owner` (when set) is stamped on the record's `owner_local_user` field
/// so owner enforcement (Task 6) can later reject cross-owner overwrites.
#[allow(clippy::too_many_arguments)]
pub fn put_code_cell(
&self,
path: &str,
@@ -691,9 +711,10 @@ impl ConcurrentStore {
links: &[Czyx],
code: &[Op],
owner: Option<&str>,
descriptor: Option<cubecode::Behavior>,
) -> Result<Czyx, crate::SysError> {
let coord = self.with_mut(|store| {
crate::store_code_cell(store, path, kind, name, links, code, owner)
crate::store_code_cell(store, path, kind, name, links, code, owner, descriptor)
})?;
if let Some(v) = self.get_raw(&coord) {
self.log_put(coord, v);
+11
View File
@@ -0,0 +1,11 @@
[package]
name = "cubetrace"
version = "0.1.0"
edition.workspace = true
license.workspace = true
description = "CUBELinux-2 tracing package: wraps a DBI engine (via FFI seam) and streams trace events (basic blocks, syscalls) into cubestore, tagging each with CZYX coordinates and header flags (PDF Package 4, §547)."
[dependencies]
cubecoords = { path = "../cubecoords" }
cubestore = { path = "../cubestore" }
cubecode = { path = "../cubecode" }
+306
View File
@@ -0,0 +1,306 @@
//! CUBELinux-2 tracing package.
//!
//! Per the PDF (Package 4, §547 / §1113): *"cubetrace: wraps a DBI engine
//! (via FFI) and streams trace events (basic blocks, syscalls) into
//! cubestore, tagging each with CZYX coordinates and header flags."*
//!
//! This crate is the *head* of the Package4 pipeline (cubetrace → cubeai →
//! cubedbt). It is dependency-free and exercises today on a `Null` (offline)
//! source so it builds and is testable without an external DBI engine; the
//! real engine is reached through the documented `DbiEngine` FFI seam.
//!
//! Model
//! -----
//! * A [`TraceEvent`] is a basic block / syscall / metadata observation with a
//! timestamp, a target coordinate, and policy/behavior header flags.
//! * A [`Tracer`] streams events into a [`CubeStore`] keyed by CZYX (the
//! `c240` trace band), tagging each record's header with time, environment,
//! and policy flags (§976).
//! * `replay_all` deterministically replays the stored stream in timestamp
//! order (§977/§1118: "deterministic replay first").
use cubecode::{Behavior, Kind};
use cubecoords::{CubeHeader, Czyx};
use cubestore::{CubeBackend, CubeStore, HashBackend};
/// `C` axis band where trace events are stored (keyed CZYX, §975).
pub const C_TRACE: u8 = 240;
/// Kind of traced observation.
#[derive(Copy, Clone, Debug, PartialEq, Eq)]
pub enum EventKind {
/// A basic block executed (carries the block's code/length).
BasicBlock,
/// A syscall entered/exited (carries the syscall number).
Syscall,
/// Free-form metadata about the run (policy, environment).
Meta,
}
/// A single traced observation: what happened, when, where, and with what
/// policy/behavior tags. Serialized into a trace record's body.
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct TraceEvent {
pub kind: EventKind,
/// Monotonic timestamp (ns from trace start).
pub ts: u64,
/// Target CZYX the event is about (the block/syscall coordinate).
pub coord: Czyx,
/// For `BasicBlock`: the executed bytecode; for `Syscall`: the syscall
/// number; for `Meta`: unused (0).
pub payload: Vec<u8>,
/// Behavior/policy descriptors to stamp onto the record header.
pub behavior: Behavior,
}
impl TraceEvent {
/// Serialize to a stable byte form (no external deps).
pub fn to_bytes(&self) -> Vec<u8> {
let mut out = Vec::new();
out.push(self.kind as u8);
out.extend_from_slice(&self.ts.to_le_bytes());
out.extend_from_slice(&CZYX_BYTES);
out.extend_from_slice(&self.coord.pack_u32().to_le_bytes());
out.extend_from_slice(&self.behavior.to_flags().to_le_bytes());
out.push(self.payload.len() as u8);
out.extend_from_slice(&self.payload);
out
}
/// Inverse of [`to_bytes`]. Returns `None` on malformed input.
pub fn from_bytes(b: &[u8]) -> Option<TraceEvent> {
let mut i = 0;
let kind = match *b.get(i)? {
0 => EventKind::BasicBlock,
1 => EventKind::Syscall,
2 => EventKind::Meta,
_ => return None,
};
i += 1;
let ts = u64::from_le_bytes(b.get(i..i + 8)?.try_into().ok()?);
i += 8;
i += 4; // skip CZYX_BYTES sentinel
let packed = u32::from_le_bytes(b.get(i..i + 4)?.try_into().ok()?);
i += 4;
let coord = Czyx::new(
(packed >> 24) as u8,
(packed >> 16) as u8,
(packed >> 8) as u8,
packed as u8,
);
let flags = u16::from_le_bytes(b.get(i..i + 2)?.try_into().ok()?);
i += 2;
let plen = *b.get(i)? as usize;
i += 1;
let payload = b.get(i..i + plen)?.to_vec();
Some(TraceEvent {
kind,
ts,
coord,
payload,
behavior: Behavior::from_flags(flags),
})
}
}
/// Sentinel to make the on-wire format self-identifying as a CZYX record.
const CZYX_BYTES: [u8; 4] = *b"CZYX";
/// Source of trace events. In the full stack this is the FFI seam to a DBI
/// engine (DynamoRIO / Intel Pin / QBDI / Frida — §970). Here it is a trait so
/// the crate is testable offline via [`NullEngine`] and a real engine can be
/// dropped in without changing callers.
pub trait DbiEngine {
/// Attach to a target (pid or path). Returns `Err` if the engine can't.
fn attach(&mut self, target: &str) -> Result<(), String>;
/// Pull the next event, or `None` when the run is exhausted.
fn next_event(&mut self) -> Option<TraceEvent>;
}
/// Offline engine: replays a pre-recorded event vector. Stands in for a live
/// DBI backend in tests and headless environments.
pub struct NullEngine {
events: Vec<TraceEvent>,
idx: usize,
}
impl NullEngine {
pub fn new(events: Vec<TraceEvent>) -> Self {
NullEngine { events, idx: 0 }
}
}
impl DbiEngine for NullEngine {
fn attach(&mut self, _target: &str) -> Result<(), String> {
Ok(())
}
fn next_event(&mut self) -> Option<TraceEvent> {
if self.idx < self.events.len() {
let e = self.events[self.idx].clone();
self.idx += 1;
Some(e)
} else {
None
}
}
}
/// Streams trace events into a [`CubeStore`] (the `c240` band), tagging each
/// record with time, environment, and policy header flags (PDF §976).
pub struct Tracer<B: CubeBackend> {
store: CubeStore<B>,
next_x: u8,
}
impl<B: CubeBackend> Tracer<B> {
pub fn new(store: CubeStore<B>) -> Self {
Tracer { store, next_x: 1 }
}
/// Drain an engine into the store, returning the number of events stored.
pub fn run<E: DbiEngine>(&mut self, engine: &mut E) -> usize {
let mut count = 0;
while let Some(ev) = engine.next_event() {
self.store_event(&ev);
count += 1;
}
count
}
/// Store one event under a fresh CZYX coordinate in the `c240` band.
pub fn store_event(&mut self, ev: &TraceEvent) -> Czyx {
let label = Czyx::new(C_TRACE, 1, ev.kind as u8, self.next_x);
self.next_x = self.next_x.wrapping_add(1).max(1);
let mut h = CubeHeader::new();
h.title = Some(format!("{:?}:{:?}", ev.kind, ev.coord));
h.doc_type = Some(Kind::Other.as_str().into());
h.size_bytes = Some(ev.to_bytes().len() as u64);
h.flags.0 |= ev.behavior.to_flags();
h.refresh_flags();
self.store.put_record(label, &h, &ev.to_bytes());
label
}
/// Borrow the backing store (e.g. to hand to `cubeai`/others).
pub fn store(&self) -> &CubeStore<B> {
&self.store
}
}
/// Deterministic replay: collect every trace event from the `c240` band and
/// return them in timestamp order (PDF §977/§1118: "deterministic replay
/// first"). The trace is the immutable cube record; replay never mutates it.
pub fn replay_all(store: &CubeStore<HashBackend>) -> Vec<TraceEvent> {
let mut events = Vec::new();
for k in store.keys() {
if k.c != C_TRACE {
continue;
}
if let Some((_, body)) = store.get_record(&k) {
if let Some(ev) = TraceEvent::from_bytes(&body) {
events.push(ev);
}
}
}
events.sort_by_key(|e| e.ts);
events
}
#[cfg(test)]
mod tests {
use super::*;
fn ev(kind: EventKind, ts: u64, coord: Czyx, beh: Behavior) -> TraceEvent {
TraceEvent {
kind,
ts,
coord,
payload: vec![kind as u8],
behavior: beh,
}
}
fn behavior(flag: u16) -> Behavior {
Behavior(flag)
}
#[test]
fn event_round_trips_through_bytes() {
let e = ev(
EventKind::Syscall,
42,
Czyx::new(1, 2, 3, 4),
behavior(Behavior::HOT_PATH),
);
let back = TraceEvent::from_bytes(&e.to_bytes()).expect("decodes");
assert_eq!(e, back);
}
#[test]
fn tracer_streams_to_c240() {
let mut engine = NullEngine::new(vec![
ev(
EventKind::BasicBlock,
10,
Czyx::new(1, 0, 0, 1),
behavior(Behavior::PURE),
),
ev(
EventKind::Syscall,
20,
Czyx::new(1, 0, 0, 2),
behavior(Behavior::IO_HEAVY),
),
ev(
EventKind::Meta,
5,
Czyx::new(1, 0, 0, 3),
Behavior::default(),
),
]);
let mut tracer = Tracer::new(CubeStore::new(HashBackend::new()));
assert_eq!(tracer.run(&mut engine), 3);
// All records landed in the c240 band.
let stored: usize = tracer
.store()
.keys()
.into_iter()
.filter(|k| k.c == C_TRACE)
.count();
assert_eq!(stored, 3);
}
#[test]
fn replay_is_deterministic_and_ordered() {
let mut engine = NullEngine::new(vec![
ev(
EventKind::BasicBlock,
30,
Czyx::new(1, 0, 0, 1),
behavior(Behavior::PURE),
),
ev(
EventKind::Syscall,
10,
Czyx::new(1, 0, 0, 2),
behavior(Behavior::IO_HEAVY),
),
ev(
EventKind::Meta,
20,
Czyx::new(1, 0, 0, 3),
Behavior::default(),
),
]);
let mut tracer = Tracer::new(CubeStore::new(HashBackend::new()));
tracer.run(&mut engine);
let replayed = replay_all(tracer.store());
assert_eq!(replayed.len(), 3);
// Sorted by ts: 10 (Syscall), 20 (Meta), 30 (BasicBlock).
assert_eq!(replayed[0].kind, EventKind::Syscall);
assert_eq!(replayed[1].kind, EventKind::Meta);
assert_eq!(replayed[2].kind, EventKind::BasicBlock);
// Behavior flags survived the round-trip through the store.
assert_eq!(replayed[0].behavior, Behavior(Behavior::IO_HEAVY));
}
}