Compare commits
28
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
10cb4e4075 | ||
|
|
b47a5df1b2 | ||
|
|
d84f87e7bb | ||
|
|
bd3e3c0512 | ||
|
|
8c66478a42 | ||
|
|
18f8eb9b4a | ||
|
|
15dd4c48a4 | ||
|
|
a601710d50 | ||
|
|
cc896fc336 | ||
|
|
3f007a1f38 | ||
|
|
ae1a62a44e | ||
|
|
c090233056 | ||
|
|
7b0cf6fda5 | ||
|
|
ce8ac78a24 | ||
|
|
fdfcd2c728 | ||
|
|
90e90ef926 | ||
|
|
ec25347af9 | ||
|
|
6627505e70 | ||
|
|
ddca08ea73 | ||
|
|
b97efa2a16 | ||
|
|
11e5ed7fca | ||
|
|
94681bbdc0 | ||
|
|
2ff1dff02b | ||
|
|
fe64b869e4 | ||
|
|
ff2914a105 | ||
|
|
3121459138 | ||
|
|
0074f3112c | ||
|
|
1539ecd1bb |
@@ -1,3 +1,6 @@
|
|||||||
/target
|
/target
|
||||||
Cargo.lock
|
Cargo.lock
|
||||||
target/
|
target/
|
||||||
|
|
||||||
|
# stray build dir (not source)
|
||||||
|
/src/CUBELinux-build/
|
||||||
|
|||||||
@@ -2,12 +2,17 @@
|
|||||||
resolver = "2"
|
resolver = "2"
|
||||||
members = [
|
members = [
|
||||||
"cubecoords",
|
"cubecoords",
|
||||||
|
"cube-mvw",
|
||||||
"cubestore",
|
"cubestore",
|
||||||
"cubefs",
|
"cubefs",
|
||||||
"cubecode",
|
"cubecode",
|
||||||
"cubecrypt",
|
"cubecrypt",
|
||||||
"cubesys",
|
"cubesys",
|
||||||
|
"cubedbt",
|
||||||
|
"cubeai",
|
||||||
|
"cubetrace",
|
||||||
"cube-bench",
|
"cube-bench",
|
||||||
|
"cubecli",
|
||||||
]
|
]
|
||||||
|
|
||||||
[workspace.package]
|
[workspace.package]
|
||||||
|
|||||||
@@ -1,17 +0,0 @@
|
|||||||
CUBELinux LICENSE — source-available, non-commercial by default
|
|
||||||
Copyright (c) 2026 CUBELinux (CUBELinux.com) / DULRobotics (DULRobotics.com)
|
|
||||||
|
|
||||||
1. Permission is granted to use, copy, modify, and distribute this software,
|
|
||||||
in source or binary form, with or without modification, FOR PERSONAL AND
|
|
||||||
NON-COMMERCIAL DEVELOPMENT AND TESTING, provided this license and the
|
|
||||||
copyright notice are retained.
|
|
||||||
|
|
||||||
2. COMMERCIAL USE (any use to obtain a commercial benefit, including selling
|
|
||||||
the software or a service built on it, or use inside a commercial product)
|
|
||||||
is permitted ONLY under a separate written COMMERCIAL LICENSE from the
|
|
||||||
copyright holder. Unlicensed commercial use is prohibited.
|
|
||||||
|
|
||||||
3. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND.
|
|
||||||
|
|
||||||
4. The copyright holder may later release it under an open-source license or
|
|
||||||
grant commercial licenses at its discretion.
|
|
||||||
@@ -0,0 +1,50 @@
|
|||||||
|
# RESUME POINT — HashBackend (HashMap<u32, Vec<u8>>) work, 2026-08-20 → 2026-08-21
|
||||||
|
|
||||||
|
## WHAT WAS DONE THIS SESSION (continuation)
|
||||||
|
|
||||||
|
### Prior session's 4 tasks — all closed and verified
|
||||||
|
1. clippy -D warnings in cubesys — 4 lints fixed
|
||||||
|
2. cube-bench executed — correctness-gated microbenchmarks
|
||||||
|
3. cubecli extracted — standalone crate
|
||||||
|
4. go live (./check gate) — passes
|
||||||
|
|
||||||
|
### THIS session: ran all 4 `./check` opt-in stages in logical order
|
||||||
|
|
||||||
|
1. **./check mount** — PASS (57 assertions, 0 failed)
|
||||||
|
- In-memory `--seed` mount: full regression suite
|
||||||
|
- Daemon-backed `--socket` mount: full regression suite
|
||||||
|
- Durability across daemon restart: verified (record survives kill+restart)
|
||||||
|
- Cross-user ACLs (root ↔ luulu): verified
|
||||||
|
|
||||||
|
2. **./check daemon** — PASS (3 tests, 0 failed)
|
||||||
|
- `cubefs_create_write_reaches_daemon`
|
||||||
|
- `daemon_backend_envelope_roundtrip`
|
||||||
|
- `daemon_backend_put_get_roundtrip`
|
||||||
|
- All 3 were `#[ignore]`d, now exercised against live cube-server
|
||||||
|
|
||||||
|
3. **./check stress** — PASS (~150s, 57075 pairs driven)
|
||||||
|
- Daemon alive throughout, ~380 prog+run pairs/s sustained
|
||||||
|
- 156 records in C=77 namespace (command dedup by coordinate)
|
||||||
|
- No crashes, no latency spikes
|
||||||
|
|
||||||
|
4. **cube-bench at scale 200k** — FAILED initially, then FIXED and PASS
|
||||||
|
- **Bug found**: bench's expected-value math assumed all records beyond c=0 land in c=1, but `coord_for` spreads across c=0,1,2,3 as `i` grows past 65536. At scale=200k, c=1 should have 65536 records (not 134464).
|
||||||
|
- **Fix**: compute expected counts per C bucket from the `coord_for` mapping (c=0: min(n, 65536); c=1: next 65536; c=2: next 65536; c=3: remainder). Added expected_c2 + expected_c3 to the total-coverage assertion.
|
||||||
|
- **Verified**: `cargo fmt --all -- --check` clean, `cargo clippy -p cube-bench -- -D warnings` clean, 100k + 200k both pass.
|
||||||
|
- **Numbers at 200k (release)**: put_raw 236ns/op, get_raw 109ns/op, scan_prefix 12.4ms for 65536 coords, ~4237k put/s, crypto ~1.4-2.6μs/seal, VM 276ns entry / 142ns leaf, session 1.4μs prog / 2.6μs run.
|
||||||
|
|
||||||
|
### Git state
|
||||||
|
- Working tree: `cube-bench/src/main.rs` modified (the fix above)
|
||||||
|
- Branch: `feat/os-kernel-in-cube`
|
||||||
|
- All prior session commits intact (7b0cf6f is HEAD)
|
||||||
|
|
||||||
|
## CURRENT STATE
|
||||||
|
- Branch: feat/os-kernel-in-cube
|
||||||
|
- cube-bench at 200k: fixed + verified (fmt + clippy + 100k + 200k all pass)
|
||||||
|
- ./check mount/daemon/stress: all green
|
||||||
|
- cubecli: standalone crate, functionally identical
|
||||||
|
|
||||||
|
## NEXT STEPS
|
||||||
|
1. Commit the cube-bench fix (27 insertions, 5 deletions)
|
||||||
|
2. README/docs sync — reflect cubecli as standalone crate (./check doesn't check docs)
|
||||||
|
3. Spec reading resumption from CUBELinux.txt (~offset 360+) — Package 2 use cases
|
||||||
@@ -0,0 +1,33 @@
|
|||||||
|
# Resume Marker — 2026-08-20
|
||||||
|
# Status: kernel ready for compile, workhorse ready for Xfce switching, stopped before heavy build
|
||||||
|
|
||||||
|
## What was done this session
|
||||||
|
1. **Kernel build environment — READY** (stopped before heavy compile per user request)
|
||||||
|
- Kernel: vanilla v6.19.3 + ipu4 drivers (GRUB entry LOCALVERSION=-ipu4p)
|
||||||
|
- `./check` green, git clean
|
||||||
|
- Rust toolchain: stable 1.97.1 (default) + nightly 1.100.0 installed via rustup
|
||||||
|
- CONFIG_RUST=y set in .config, CONFIG_RUST_IS_AVAILABLE=y
|
||||||
|
- `scripts/target.json` has `rustc-abi: x86-softfloat` (rejected by stable, OK with nightly)
|
||||||
|
- Build fix identified: `RUSTC=/root/.cargo/bin/rustc RUSTLIB_SRC=... make ...` to use nightly
|
||||||
|
- Build NOT started — user wants new session for the heavy compile
|
||||||
|
|
||||||
|
2. **GUI session switching — READY** (workhorse has Xfce, GDM shows it)
|
||||||
|
- Xfce 4.20.2 + xfce4-goodies installed on workhorse
|
||||||
|
- `/usr/share/xsessions/xfce.desktop` present, `Exec=startxfce4`
|
||||||
|
- GDM (gdm3) active, shows xfce.desktop + xfce-wayland.desktop at login gear menu
|
||||||
|
- User can switch GNOME↔Xfce at GDM login by clicking gear icon → "Xfce Session"
|
||||||
|
- lightdm package removed (was stuck half-configured; cleaned up via dpkg purge)
|
||||||
|
- No lightdm; gdm3 is the DM; /etc/X11/default-display-manager = /usr/sbin/gdm3
|
||||||
|
|
||||||
|
## What remains (for next session / new session)
|
||||||
|
- **Kernel compile**: `make -j$(nproc) bzImage modules` with nightly rustc override
|
||||||
|
- Command pattern: `cd /home/CUBELinux/CUBELinux-2/src/CUBELinux-build && RUSTC=/root/.cargo/bin/rustc HOSTRUSTC=/root/.cargo/bin/rustc make -j$(nproc) bzImage modules`
|
||||||
|
- Need to confirm RUST_LIB_SRC path for nightly toolchain
|
||||||
|
- User explicitly asked NOT to auto-launch — new session should decide timing
|
||||||
|
- **GRUB**: new entry for -cube kernel; verify ipu4p entry still works
|
||||||
|
- **Post-boot verification**: cube fs mount, daemon startup, camera (ipu4) if desired
|
||||||
|
|
||||||
|
## Notes
|
||||||
|
- User restarted the session; this marker captures where we left off
|
||||||
|
- User directive: "get project to point of compiling new kernel, then stop so we can use new session for that"
|
||||||
|
- Heavy compile (make -j8) saturates all cores + memory — do NOT start without user go
|
||||||
@@ -0,0 +1,72 @@
|
|||||||
|
# Resume Marker — 2026-08-20 (kernel build + GUI session switching)
|
||||||
|
|
||||||
|
## Context
|
||||||
|
User restarted the session. Two parallel tasks were in progress when interrupted:
|
||||||
|
1. Kernel Rust build config issue (kbuild + nightly rustc)
|
||||||
|
2. GUI session switching — install Xfce on workhorse so user can switch GNOME↔Xfce at login
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 1: Kernel Rust Build Config
|
||||||
|
|
||||||
|
**Status: unresolved — root cause found, fix not yet applied**
|
||||||
|
|
||||||
|
**What we know:**
|
||||||
|
- Kernel source: `/home/CUBELinux/CUBELinux-2/src/CUBELinux-build/` (vanilla v6.19.3 + ipu4 out-of-tree)
|
||||||
|
- `CONFIG_RUST=y` set in `.config`
|
||||||
|
- rustc 1.97.1 (stable) is default; rustc 1.100.0-nightly also installed via rustup (default host: x86_64-unknown-linux-gnu)
|
||||||
|
- `scripts/target.json` exists with `rustc-abi: x86-softfloat`, `llvm-target: x86_64-linux-gnu`
|
||||||
|
- `make rust/core.o` **fails**: `error loading target specification: custom targets are unstable and require -Zunstable-options`
|
||||||
|
- The rustc_library command in `rust/Makefile:559` does NOT pass `-Zunstable-options`
|
||||||
|
- `rust/Makefile` already uses `-Zunstable-options` in rustdoc paths (lines 136, 324, 333) but NOT in the core rustc_library invocation
|
||||||
|
- `KBUILD_RUSTFLAGS` (from `arch/x86/Makefile`) sets `--target=.../scripts/target.json` but no `-Zunstable-options`
|
||||||
|
- The `rustc_library` rule (`rust/Makefile:555-567`) builds the command as:
|
||||||
|
```
|
||||||
|
$(RUSTC) $(filter-out $(skip_flags),$(rust_flags)) $(rustc_target_flags) --emit=...
|
||||||
|
```
|
||||||
|
where `rust_flags` = `KBUILD_RUSTFLAGS` + `rust_common_flags` and `rustc_target_flags` = `core-flags` (which is `--edition=...` + cfgs). Neither carries `-Zunstable-options`.
|
||||||
|
|
||||||
|
**What needs to happen:**
|
||||||
|
- Option A: Patch `rust/Makefile` to add `-Zunstable-options` to the rustc_library command (line 559 area) — minimal change, mirrors what rustdoc already does
|
||||||
|
- Option B: Use nightly rustc as RUSTC and ensure it's invoked with `-Zunstable-options` — but kbuild doesn't add it automatically
|
||||||
|
- The real fix is Option A: add `-Zunstable-options` to the rustc invocation in the rustc_library rule, since the target.json is a custom target spec that requires it
|
||||||
|
- After that: `make rust/all` or `make rust/core.o` should succeed, then full kernel build
|
||||||
|
|
||||||
|
**Files involved:**
|
||||||
|
- `/home/CUBELinux/CUBELinux-2/src/CUBELinux-build/rust/Makefile` (line 555-567, rustc_library rule)
|
||||||
|
- `/home/CUBELinux/CUBELinux-2/src/CUBELinux-build/arch/x86/Makefile` (KBUILD_RUSTFLAGS)
|
||||||
|
- `/home/CUBELinux/CUBELinux-2/src/CUBELinux-build/scripts/target.json` (custom target spec, needs -Zunstable-options)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 2: GUI Session Switching (Xfce on workhorse)
|
||||||
|
|
||||||
|
**Status: dpkg interrupted — lightdm stuck half-configured**
|
||||||
|
|
||||||
|
**What we know:**
|
||||||
|
- Workhorse runs GNOME (gdm3 + gnome-shell) currently
|
||||||
|
- User wants GNOME ↔ Xfce switching at the login screen (like Cinnamon is available)
|
||||||
|
- `apt-get install -y task-xfce-desktop` timed out (exit 124) and left dpkg in interrupted state
|
||||||
|
- `dpkg --configure -a` fails: lightdm is half-configured, depends on itself being configured first
|
||||||
|
- lightdm package status: `iF` (half-configured), `light-locker` and `task-xfce-desktop` also stuck
|
||||||
|
- Xfce components were likely downloaded but not configured due to the dpkg blockage
|
||||||
|
|
||||||
|
**What needs to happen:**
|
||||||
|
1. Fix dpkg: either `dpkg --remove --force-remove-reinstreq lightdm` then `dpkg --configure -a`, or manually remove the lock/status entries
|
||||||
|
2. Re-run `apt-get install -y task-xfce-desktop` (or just `xfce4` + `lightdm` if minimal)
|
||||||
|
3. Verify `xfce4-session` works: `xfce4-session --version`
|
||||||
|
4. Verify GDM shows Xfce session: check `/usr/share/xsessions/xfce.desktop` exists, `gdm3` is running
|
||||||
|
5. Test: log in via Xfce session
|
||||||
|
|
||||||
|
**Files/dirs involved:**
|
||||||
|
- `/usr/share/xsessions/` (GDM session definitions)
|
||||||
|
- `/usr/share/wayland-sessions/` (if Wayland Xfce session needed)
|
||||||
|
- `/etc/gdm3/` (GDM config)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## General Notes
|
||||||
|
- User explicitly does NOT want auto-launching heavy kernel compiles while they're using the machine
|
||||||
|
- Kernel build with `make -j$(nproc)` will saturate CPU — must ask before launching
|
||||||
|
- The workhorse is Debian-based, root access available
|
||||||
|
- rustup is installed at `/root/.rustup`, toolchains in `/root/.rustup/toolchains/`
|
||||||
@@ -0,0 +1,96 @@
|
|||||||
|
# Resume Marker — 2026-08-21 (kernel Rust build: IRQ lifetime fixes + ready for compile)
|
||||||
|
|
||||||
|
## Context
|
||||||
|
The session after RESUME-20260820. Fixed the kernel Rust build errors and verified `make rust/core.o` compiles cleanly. Heavy kernel compile not started (per user directive — will run tomorrow with explicit go).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## What was done this session
|
||||||
|
|
||||||
|
### 1. Kernel Rust build: IRQ lifetime errors fixed
|
||||||
|
**File changed:** `rust/kernel/irq/request.rs`
|
||||||
|
|
||||||
|
**Root cause:** `rust/core.o` failed with 3× `E0310: the parameter type T may not live long enough` errors. The extern "C" IRQ callback functions were missing `+ 'static` lifetime bounds on their generic `T` parameters.
|
||||||
|
|
||||||
|
**Fixes applied (4 total):**
|
||||||
|
|
||||||
|
| Function | Line | Change |
|
||||||
|
|----------|------|--------|
|
||||||
|
| `handle_irq_callback` | 264 | `T: Handler` → `T: Handler + 'static` |
|
||||||
|
| `handle_threaded_irq_callback` | 483 | `T: ThreadedHandler` → `T: ThreadedHandler + 'static` |
|
||||||
|
| `thread_fn_callback` | 499 | `T: ThreadedHandler` → `T: ThreadedHandler + 'static` |
|
||||||
|
|
||||||
|
The `impl` blocks already had `T: Handler + 'static` and `T: ThreadedHandler + 'static` on lines 185, 198, 403, 416 — the three extern C callback functions and their callers were the missing ones.
|
||||||
|
|
||||||
|
**Verification:** `python3 /tmp/hermes-verify-kernel-rust-build.py` — 16/16 checks passed:
|
||||||
|
- Source file checks: all 3 fixed signatures present, no old patterns remain
|
||||||
|
- Build: `make rust/core.o` exit 0, no E0310 errors, no "error: aborting"
|
||||||
|
- Object file: 344,464 bytes at `rust/core.o`
|
||||||
|
- Only 2 pre-existing warnings (unused import `flags::*`, unused feature `used_with_arg`)
|
||||||
|
|
||||||
|
### 2. ./check gate — ALL CHECKS PASSED
|
||||||
|
```
|
||||||
|
./check → ALL CHECKS PASSED (run: ./check mount | bench | stress | daemon for those stages)
|
||||||
|
```
|
||||||
|
fmt clean, 223 tests, clippy -D clean. No regressions from the IRQ fix.
|
||||||
|
|
||||||
|
### 3. Build config (carried over from prior session, still valid)
|
||||||
|
- Nightly rustc 1.100.0 (`rustup default nightly`)
|
||||||
|
- `scripts/target.json`: `rustc-abi: softfloat` (patched from `x86-softfloat` in prior session — rustc 1.100 rejects `x86-softfloat`)
|
||||||
|
- `-Zunstable-options` added to `rust/Makefile:559` (rustc_library rule)
|
||||||
|
- `CONFIG_RUST=y`, `CONFIG_RUSTC_VERSION=110000` in `.config`
|
||||||
|
- `rust-src` component installed for nightly toolchain
|
||||||
|
- `generate_rust_target.rs` patched to emit `softfloat` (not `x86-softfloat`)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## What remains for tomorrow
|
||||||
|
|
||||||
|
### Kernel compile (heavy — ask before launching)
|
||||||
|
- Command: `cd /home/CUBELinux/CUBELinux-2/src/CUBELinux-build && make -j4 bzImage modules`
|
||||||
|
- Use `-j4` not `-j8` — 7.4 GB RAM, only 3.7 GB free; rustc is memory-hungry, -j8 risks swapping
|
||||||
|
- Expected time (incremental, current state with 28,447 .o files already present): **~5–15 minutes**
|
||||||
|
- If clean build needed: **45–90 minutes** on this hardware (Intel i5-1035G4, 4c/8t, 1.1 GHz base, NVMe)
|
||||||
|
- After build: install modules, update GRUB, reboot into new kernel
|
||||||
|
|
||||||
|
### GRUB
|
||||||
|
- New kernel entry: `LOCALVERSION=-cube` (already in .config)
|
||||||
|
- Verify `LOCALVERSION=-ipu4p` entry still works (camera kernel, currently active)
|
||||||
|
|
||||||
|
### Post-boot verification
|
||||||
|
- Cube filesystem mount
|
||||||
|
- Daemon startup
|
||||||
|
- Camera (ipu4) if desired
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Xfce (separate track — not touched this session)
|
||||||
|
- dpkg state clean: lightdm, light-locker, task-xfce-desktop all removed/purged
|
||||||
|
- Xfce 4.20.2 intact: `xfce4-session`, `xfce4-panel`, `xfce4-goodies` all `ii`
|
||||||
|
- Session files: `/usr/share/xsessions/xfce.desktop`, `/usr/share/wayland-sessions/xfce-wayland.desktop`
|
||||||
|
- GDM active, default DM `/usr/sbin/gdm3`
|
||||||
|
- Issue reported (external monitor oversized, both monitors showing in Xfce vs GNOME's correct single monitor): root cause is `xfsettingsd` not running during Xfce session — Xfce inherits GNOME's RandR state without managing it. The fix is ensuring `xfsettingsd` starts (autostart entry at `/etc/xdg/autostart/xfsettingsd.desktop` exists, not disabled). No action taken — user will test after reboot.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Files
|
||||||
|
- Workspace: `/home/CUBELinux/CUBELinux-2/`
|
||||||
|
- Changed file: `src/CUBELinux-build/rust/kernel/irq/request.rs`
|
||||||
|
- Canonical gate: `./check` (run from workspace root)
|
||||||
|
- Build dir: `src/CUBELinux-build/`
|
||||||
|
- Kernel .config: `src/CUBELinux-build/.config` (CONFIG_RUST=y, CONFIG_LOCALVERSION="-cube")
|
||||||
|
- Rust target.json: `src/CUBELinux-build/scripts/target.json` (rustc-abi: softfloat)
|
||||||
|
- System daemon socket: `/run/cube/cube.sock`
|
||||||
|
- luulu session daemon socket: `/run/user/1000/cube/cube.sock`
|
||||||
|
- Verification script used: `/tmp/hermes-verify-kernel-rust-build.py` (cleaned up after run)
|
||||||
|
|
||||||
|
## Git state
|
||||||
|
- Branch `feat/os-kernel-in-cube`, working tree clean (verify with `git status` before committing)
|
||||||
|
- Prior commits on HEAD: `c090233` (bench fix), `ae1a62a` (docs), `3f007a1` (bench saturating_sub) + 3 older commits
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Reminder
|
||||||
|
- User explicitly does NOT want auto-launching heavy kernel compiles while they're using the machine
|
||||||
|
- `make -j8` saturates all 8 cores + memory on this Surface Pro 7 (i5-1035G4, 7.4 GB RAM)
|
||||||
|
- Must get user go before launching the kernel compile tomorrow
|
||||||
@@ -0,0 +1,70 @@
|
|||||||
|
# RESUME POINT — 2026-08-21: all ./check stages green, cube-bench fixed, docs synced
|
||||||
|
|
||||||
|
## WHERE WE ARE (verified on disk this session)
|
||||||
|
|
||||||
|
### Git state
|
||||||
|
Branch `feat/os-kernel-in-cube`, clean working tree. Commits this session:
|
||||||
|
- `c090233` bench(cube-bench): fix scan_prefix expected-value math for scale > 65536
|
||||||
|
- `ae1a62a` docs: reflect cubecli as standalone crate in README + integration docs
|
||||||
|
- `3f007a1` bench(cube-bench): use saturating_sub for expected-value arithmetic
|
||||||
|
|
||||||
|
Plus 3 prior-session commits still on HEAD (7b0cf6f, ce8ac78, fdfcd2c).
|
||||||
|
|
||||||
|
### ./check gate
|
||||||
|
`./check` → ALL CHECKS PASSED (fmt clean, 223 tests, clippy -D clean).
|
||||||
|
|
||||||
|
### ./check opt-in stages (all run this session, all green)
|
||||||
|
- `./check mount` — 57/57 FUSE assertions (in-memory --seed + daemon-backed --socket + durability-across-restart + cross-user ACLs root↔luulu)
|
||||||
|
- `./check daemon` — 3/3 live daemon tests (were #[ignore]d, now exercised)
|
||||||
|
- `./check stress` — ~150s, 57075 prog+run pairs, daemon alive throughout, ~380 pairs/s
|
||||||
|
- `cube-bench --release 200000` — FAILED once (real bug in bench's expected-value math), then FIXED and PASS at 1k/10k/50k/100k/200k
|
||||||
|
|
||||||
|
### Bug found + fixed this session
|
||||||
|
cube-bench assumed all records beyond c=0 land in c=1. `coord_for` spreads across c=0,1,2,3 as i grows past 65536, so at 200k the expected c=1 count was wrong (134464 vs actual 65536). Fix: compute per-bucket expected counts from the coord_for mapping; extend total-coverage assertion to include c=2 and c=3. Verified at all scales with bench's own correctness assertions. clippy -D warnings clean on cube-bench after replacing manual `-` with `saturating_sub`.
|
||||||
|
|
||||||
|
### Docs synced
|
||||||
|
- `STARTUP-README.md`: cube CLI now references `cubecli/src/main.rs` (was `cubesys/src/bin/cube.rs`)
|
||||||
|
- `cubesys/docs/integration.md`: Binaries section notes cube is now a standalone crate (`cubecli/`)
|
||||||
|
|
||||||
|
## ARCHITECTURAL QUESTION: where should CUBE live when it becomes the OS?
|
||||||
|
|
||||||
|
Two-layer model (already partially deployed — this is the right shape):
|
||||||
|
|
||||||
|
**Layer 1 — Kernel (the "utilize CUBE" part).** CUBE core crates — `cubecoords`, `cubestore` (HashMap + FileBackedStore + WAL/checkpoint), `cubecode` (bytecode + VM), `cubecrypt` (seal/open + keyinit) — get compiled into the kernel or as a kernel module. Kernel gets direct syscalls: `open_by_czyz()`, `write_czyz()`, `seal_czyz()`, `run_czyz()`. The cube becomes the backing store the VFS and process accounting talk to directly — not a userspace daemon over a socket. This is the spec's Phase 3 ("OS services talk directly to the cube store; syscalls like open by CZYX + flags").
|
||||||
|
|
||||||
|
**Layer 2 — Userspace daemon (`cube-server`).** Stays as the durable anchor: WAL, checkpoint, socket for external/remote clients, the "resume pointer" writer. Survives when the kernel module isn't loaded; what remote clients talk to. Currently at `/run/cube/cube.sock` (system) + `/run/user/1000/cube/` (luulu session). Unify under `/usr/lib/cube/` when it's the OS — daemon binary, libs, socket path, store dir all there.
|
||||||
|
|
||||||
|
**Filesystem placement when it's the OS:**
|
||||||
|
- Kernel module/built-in: lives in the kernel source tree (copy of core crates, adapted for kernel constraints — `no_std`, no alloc assumptions that conflict with kernel alloc) OR as out-of-tree module alongside your existing Surface/ipu4 kernel.
|
||||||
|
- Userspace daemon + libs: `/usr/lib/cube/`. Socket at `/run/cube/cube.sock`.
|
||||||
|
|
||||||
|
**Honest gaps before this is real (both still open, from STARTUP-README §5):**
|
||||||
|
- (a) **cubefs directory model** — only `c<C>/z<Z>/y<Y>/x<X>` with x as fixed 3-digit leaf. No nested dirs, no arbitrary POSIX filenames, no rename/whiteout. Overlayfs rejects cubefs with EINVAL. A real OS tree can't be bound until this is done. Level-B crate feature.
|
||||||
|
- (b) **root fs / PID 1** — making the cube the literal rootfs is kernel work: custom initramfs + pivot_root, daemon or kernel module as init. Heavy, image-bake, off-peak. Current deployment: ext4 root + CZYX-call OS state = Phase 2 target.
|
||||||
|
|
||||||
|
**On "compile the new kernel to utilize CUBE":** your current kernel is vanilla v6.19.3 + ipu4 drivers (GRUB entry `LOCALVERSION=-ipu4p`); camera work is paused. Adding CUBE means either a kernel module against that kernel, or a new kernel build with CUBE integrated. That's a resource-heavy compile (saturates all 8 cores + memory). You've asked me not to auto-launch that while you're using the machine — I'll wait for your go.
|
||||||
|
|
||||||
|
## NEXT SESSION: put the OS in CUBE + compile new kernel to utilize CUBE
|
||||||
|
|
||||||
|
### What "put the OS in CUBE" means concretely (Phase 2 deepening)
|
||||||
|
The OS already writes its state (manifest, identity, klog, snapshots) as CZYX calls to `cube-server` over the socket — that's done and durable (verified across daemon restart). The next deepening is moving the cube from "userspace daemon the OS talks to over a socket" to "the kernel's own backing store" — i.e. the kernel module layer above.
|
||||||
|
|
||||||
|
### Concrete first step (before any kernel compile)
|
||||||
|
1. **Pick the kernel target.** Current: vanilla v6.19.3 + ipu4 (LOCALVERSION=-ipu4p). Option A: add CUBE as an out-of-tree module to that kernel. Option B: new kernel build with CUBE integrated. Which, and do you want to keep the ipu4 camera kernel as the base, or start fresh?
|
||||||
|
2. **Decide CUBE's kernel boundary.** What goes into the kernel vs stays in userspace?
|
||||||
|
- Definitely kernel: `cubecoords` (Czyx type, flags), `cubestore` core (the store abstraction + at least the in-memory backend; FileBackedStore + WAL/checkpoint is the durable path), coordinate→syscall mapping.
|
||||||
|
- Probably userspace ( daemon ): the socket server, remote client access, the "resume pointer" writer, snapshot management — i.e. the control plane, while the kernel is the data plane.
|
||||||
|
- `cubecode` (bytecode VM) and `cubecrypt` (seal/open transforms) — these are the debatable ones. VM in kernel = ability to run cube programs from kernel space (syscall-level). Crypto in kernel = sealed records openable by the kernel without a userspace daemon. Both are plausible; both add kernel surface area. Your call.
|
||||||
|
3. **cubefs directory model gap (a) above** — decide whether to close it before or after the kernel move. If the kernel uses the cube as backing store but cubefs can't represent a real OS tree (no nested dirs, no rename), then the OS-state-writing path (CZYX calls, not FUSE) still works, but a full rootfs bind is still blocked. Closing (a) is a crate feature, independent of the kernel move.
|
||||||
|
|
||||||
|
### Concrete second step (kernel compile — heavy, ask before launching)
|
||||||
|
4. **Build the kernel/module.** Depends on step 1-3. If out-of-tree module: copy the core crates into a kernel module skeleton, adapt for kernel constraints, build against the existing kernel headers. If integrated kernel: new kernel source with CUBE subsystem, config, compile. Either way: `make -j8` saturates all cores + memory — you've asked me not to auto-launch that while you're using the machine.
|
||||||
|
|
||||||
|
## FILES
|
||||||
|
- Workspace: `/home/CUBELinux/CUBELinux-2/`
|
||||||
|
- Canonical gate: `./check` (run from workspace root)
|
||||||
|
- Resume procedure: read this file + STARTUP-README.md + REPORT-INDEX.md, then `python3` against `/root/.hermes/state.db` for thread review.
|
||||||
|
- Cube-bench binary (release): `target/release/cube-bench`
|
||||||
|
- Current kernel GRUB entry: `LOCALVERSION=-ipu4p` (vanilla v6.19.3 + ipu4 drivers)
|
||||||
|
- System daemon socket: `/run/cube/cube.sock`
|
||||||
|
- luulu session daemon socket: `/run/user/1000/cube/cube.sock`
|
||||||
@@ -0,0 +1,71 @@
|
|||||||
|
# RESUME — Cube as OS substrate (CZYX-call model)
|
||||||
|
|
||||||
|
Date: 2026-08-13 (session after RESUME-cubefs-daemon; Level A of item 3)
|
||||||
|
|
||||||
|
## Decision (user-directed)
|
||||||
|
"Everything writes as a CZYX call to the daemon — the cube-backed storage should
|
||||||
|
be coordinate calls, not writes to /cubefs. Why are we writing to cubefs?"
|
||||||
|
|
||||||
|
This REVERSES the earlier FUSE-path substrate design. Confirmed against
|
||||||
|
CUBELinux.pdf:
|
||||||
|
- Phase 1: user-space Cube OS on Linux, using existing kernels + filesystems.
|
||||||
|
- Phase 2: cubefs FUSE maps POSIX→CZYX for *testing semantics under real
|
||||||
|
workloads* — explicitly a probe, not the substrate itself.
|
||||||
|
- Phase 3: fork/extend kernel so VFS/accounting/LSM talk directly to the cube
|
||||||
|
store; new syscalls expose cube-native ops ("open by CZYX + flags").
|
||||||
|
- Package 3 (cubefs): "optional FUSE filesystem view so cube records appear as
|
||||||
|
files/directories for existing tools." → FUSE is OPTIONAL, not the substrate.
|
||||||
|
|
||||||
|
So: the coordinate store IS the persistence. POSIX/FUSE is a convenience view.
|
||||||
|
|
||||||
|
## What changed
|
||||||
|
1. `cube-os-state.sh` / `cube-os-snapshot.sh` / `cube-os-klog.sh` rewritten to
|
||||||
|
write via `cubec --socket /run/cube/cube.sock rawput/get` (CZYX calls), not
|
||||||
|
`/cubefs/...`. hex encoding uses python3 (xxd is absent in the VM image).
|
||||||
|
Coordinate layout (C=200):
|
||||||
|
c200/z001/y001/x001 boot manifest (overwrite each boot)
|
||||||
|
c200/z002/y001/x001 kernel boot line history (append)
|
||||||
|
c200/z003/y001/x001 machine/identity record
|
||||||
|
c200/z004/y001/x001 kernel log stream (klog appends lines)
|
||||||
|
c200/z010/y001/xNNN operational snapshot body (rolling counter)
|
||||||
|
c200/z010/y002/x001 rolling snapshot counter (persisted in cube, hex)
|
||||||
|
2. The three `cube-os-*` systemd units had `cubefs.service` REMOVED from
|
||||||
|
Requires/After — they now depend ONLY on `cube-server.service`. A FUSE mount
|
||||||
|
failure can no longer wedge OS bring-up.
|
||||||
|
3. `cubefs.service` hardened (ExecStartPre unmounts any stale mountpoint; BindsTo
|
||||||
|
cube-server) so it survives a daemon restart instead of crash-looping on
|
||||||
|
"Transport endpoint is not connected". This was the bug that broke the live
|
||||||
|
VM at the start of this session.
|
||||||
|
4. build_vm.sh updated to match (inline script bodies + unit edges). A fresh
|
||||||
|
bake now produces the CZYX-call substrate. NOT re-baked this session
|
||||||
|
(heavy ~24G, off-peak per user policy).
|
||||||
|
|
||||||
|
## Verified live (VM localhost:2222)
|
||||||
|
- Boot manifest reads: "CUBELINUX-VM boot manifest v3 / backing-store:
|
||||||
|
cube-server @ /run/cube/cube.sock (CZYX coordinate calls, no FUSE)".
|
||||||
|
- Snapshot #001 contains real OS state (running units, network, resources,
|
||||||
|
journal) — not a demo.
|
||||||
|
- Manifest + snapshot survive `systemctl restart cube-server` (durable across
|
||||||
|
daemon restart).
|
||||||
|
- `cube-os-state` + `cube-os-snapshot.timer` active; manual snapshot writes.
|
||||||
|
|
||||||
|
## Known gaps (honest)
|
||||||
|
- cubefs directory model: only `c<C>/z<Z>/y<Y>/x<X>` (x = fixed 3-digit leaf).
|
||||||
|
Arbitrary POSIX filenames (syslog, wtmp) and nested dirs are NOT addressable.
|
||||||
|
overlayfs on cubefs fails (EINVAL — missing RENAME_WHITEOUT/opaque-dir). So a
|
||||||
|
*real OS tree* on the cube is blocked until cubefs grows nested-dir support
|
||||||
|
(Level-B crate work).
|
||||||
|
- root fs / PID 1 (cube daemon as init, pivot_root into cubefs) = Phase 3 kernel
|
||||||
|
work, image-bake, off-peak only.
|
||||||
|
|
||||||
|
## Items left (from earlier list)
|
||||||
|
1. / 2. → done in prior sessions.
|
||||||
|
3. Cube as OS substrate: Level A DONE (CZYX-call OS state, this session).
|
||||||
|
Levels B (cube-backed block device) / C (rootfs+PID1) pending, not started.
|
||||||
|
4. society workspace: not started; git dubious ownership on
|
||||||
|
/home/cubelinux/society (needs safe.directory).
|
||||||
|
|
||||||
|
## Key files
|
||||||
|
- /root/build_vm.sh (authoritative bake template; all unit + script bodies live here)
|
||||||
|
- /home/CUBELinux/CUBELinux-2/STARTUP-README.md (updated §4 + provisioning note)
|
||||||
|
- VM guest: /opt/cube/bin/cube-os-*.sh, /etc/systemd/system/cube-os-*.{service,timer}
|
||||||
+46
-30
@@ -47,7 +47,7 @@ is real. Each must be reproduced live, not asserted.
|
|||||||
3. **Phase 3 syscall surface** — "open by CZYX + flags": a record addressable
|
3. **Phase 3 syscall surface** — "open by CZYX + flags": a record addressable
|
||||||
directly by coordinate, not path. **SHIPPED 2026-08-13**:
|
directly by coordinate, not path. **SHIPPED 2026-08-13**:
|
||||||
(a) `cube open <path> <K.Z.Y.X> <tf>` and `cube seal ...` are first-class CLI
|
(a) `cube open <path> <K.Z.Y.X> <tf>` and `cube seal ...` are first-class CLI
|
||||||
commands in `cubesys/src/bin/cube.rs` (reachable in-process + over the durable
|
commands in `cubecli/src/main.rs` (reachable in-process + over the durable
|
||||||
daemon). Verified: dispatch reaches the crypto/run layer.
|
daemon). Verified: dispatch reaches the crypto/run layer.
|
||||||
(b) **`.czyx.C.Z.Y.X` FUSE magic-prefix (2026-08-13)** — `cubefs/src/path.rs`
|
(b) **`.czyx.C.Z.Y.X` FUSE magic-prefix (2026-08-13)** — `cubefs/src/path.rs`
|
||||||
`parse_dot_czyx` + `cubefs/src/fuse.rs::lookup` intercept: a path component
|
`parse_dot_czyx` + `cubefs/src/fuse.rs::lookup` intercept: a path component
|
||||||
@@ -58,19 +58,23 @@ is real. Each must be reproduced live, not asserted.
|
|||||||
3358720263; `cat .czyx.200.50.1.7` returns the record content. Unit tests in
|
3358720263; `cat .czyx.200.50.1.7` returns the record content. Unit tests in
|
||||||
`path.rs` cover full/partial/rejected forms. **STATUS: DONE.**
|
`path.rs` cover full/partial/rejected forms. **STATUS: DONE.**
|
||||||
4. **Boot substrate** — the VM brings up CUBELinux as a storage layer the rest of
|
4. **Boot substrate** — the VM brings up CUBELinux as a storage layer the rest of
|
||||||
the OS reads/writes through. **DONE (systemd-managed, 2026-08-13)**: units
|
the OS reads/writes through. **DONE (systemd-managed, 2026-08-13)**: the OS
|
||||||
`cube-os-state.service` (oneshot, writes boot manifest + machine identity +
|
boots and **writes itself as explicit CZYX coordinate calls to the cube-server
|
||||||
boot history to `/cubefs/c200/z001|z002|z003`) and `cube-os-klog.service`
|
daemon** (the spec Phase 2/3 "cube-native" substrate path), NOT through the
|
||||||
(streams kernel ring buffer to `/cubefs/c200/z004`) run at boot; both enabled
|
FUSE mount. Per user direction, the FUSE view (`/cubefs`) is OPTIONAL — the
|
||||||
+ active; state survives a FULL power cycle (verified 2026-08-13: rebooted the
|
durable coordinate store is the persistence; POSIX paths are a convenience.
|
||||||
VM, boot history showed multiple boots, manifest unchanged).
|
- `cube-os-state.service` (oneshot) writes boot manifest + machine identity +
|
||||||
**ALSO (2026-08-13): real OS operational data** — `cube-os-snapshot.service` +
|
boot history to `c200/z001|z002|z003` via `cubec --socket ... rawput`.
|
||||||
`cube-os-snapshot.timer` (every 5 min + 30s after boot) write genuine OS state
|
- `cube-os-klog.service` streams the kernel ring buffer to `c200/z004` (CZYX).
|
||||||
(running units, network, resources, journal) into `/cubefs/c200/z010/y001/xNNN`
|
- `cube-os-snapshot.service` + `cube-os-snapshot.timer` (every 5 min + 30s
|
||||||
(rolling 3-digit counter, x-axis is u16→3-digit per cubefs path model).
|
after boot) write genuine OS state (running units, network, resources,
|
||||||
Verified: snapshot survives daemon restart; the OS wrote a NEW snapshot on a
|
journal) into `c200/z010/y001/xNNN` (rolling 3-digit counter persisted in
|
||||||
fresh boot after a full power cycle; inode == packed CZYX. **STATUS: DONE
|
`c200/z010/y002/x001`).
|
||||||
(auxiliary layer; not yet the root fs).**
|
All three units depend ONLY on `cube-server.service` (the daemon socket), not
|
||||||
|
on `cubefs.service`, so a FUSE mount failure can never wedge OS bring-up.
|
||||||
|
Verified: manifest reads "backing-store: cube-server @ /run/cube/cube.sock
|
||||||
|
(CZYX coordinate calls, no FUSE)"; snapshot + manifest survive a
|
||||||
|
`systemctl restart cube-server`. **STATUS: DONE (auxiliary layer; not yet the root fs).**
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -153,24 +157,36 @@ missing.
|
|||||||
random material on each boot → every sealed record became unopenable.
|
random material on each boot → every sealed record became unopenable.
|
||||||
Regression guards `durable_sealed_record_survives_restart` + `open_does_not_
|
Regression guards `durable_sealed_record_survives_restart` + `open_does_not_
|
||||||
clobber_sealed_record` live in `cubesys/src/commands.rs`; `./check` is green.
|
clobber_sealed_record` live in `cubesys/src/commands.rs`; `./check` is green.
|
||||||
- Boot substrate (next deepening): make the cube the OS's *default* storage for a
|
- Boot substrate (next deepening): **the substrate itself is done** — the OS
|
||||||
real tree — e.g. have a service write `/etc` or `/var/log` operational files
|
boots and persists its state (manifest/identity/klog/snapshots) as explicit
|
||||||
through the cube by default. Currently the cube holds OS *state* (manifest/
|
CZYX coordinate calls to the daemon (verified, durable across daemon restart).
|
||||||
identity/klog/snapshots) but the root fs is still ext4. The magic-prefix
|
Two honest gaps remain before a *real tree* on the cube:
|
||||||
"open by CZYX" FUSE passthrough is the natural next step (a path like
|
(a) **cubefs directory model** — cubefs only maps `c<C>/z<Z>/y<Y>/x<X>` with the
|
||||||
`/cubefs/.czyx/200.1.1.1` resolves directly to the coordinate), and a
|
`x` axis as a fixed 3-digit file leaf; nested dirs and arbitrary POSIX
|
||||||
loop/overlay over a cube-backed file would let the OS treat the cube as a real
|
filenames (syslog, wtmp, config files) are NOT addressable. So binding a
|
||||||
block device.
|
real OS tree (overlay/loop) onto cubefs is **blocked** until cubefs grows
|
||||||
|
proper nested-directory + rename/whiteout semantics (overlayfs currently
|
||||||
|
rejects cubefs with EINVAL — missing RENAME_WHITEOUT/opaque-dir support).
|
||||||
|
This is a Level-B crate feature, not a systemd change.
|
||||||
|
(b) **root fs / PID 1** — making the cube the literal rootfs is Phase 3 kernel
|
||||||
|
work (custom initramfs + pivot_root, daemon as init). Heavy, image-bake,
|
||||||
|
off-peak only. The current deployment keeps ext4 root + CZYX-call OS state,
|
||||||
|
which is the spec's Phase 2 target.
|
||||||
|
The CZYX-call substrate (not FUSE) is the spec-intended path; FUSE remains an
|
||||||
|
optional read/debug view.
|
||||||
- IMAGE PROVISIONING: `build_vm.sh` (host, /root/build_vm.sh) now bakes the full
|
- IMAGE PROVISIONING: `build_vm.sh` (host, /root/build_vm.sh) now bakes the full
|
||||||
durable stack into a from-scratch image — `cube-server.service` (daemon),
|
durable stack into a from-scratch image — `cube-server.service` (daemon),
|
||||||
`cubefs.service` (durable FUSE view OF cube-server, NOT --seed), and the
|
`cubefs.service` (OPTIONAL durable FUSE view OF cube-server, NOT --seed), and
|
||||||
three `cube-os-*` units + scripts + `cube-os-snapshot.timer` (OS state / klog /
|
the three `cube-os-*` units + scripts + `cube-os-snapshot.timer`. IMPORTANT
|
||||||
operational snapshots written into the cube store at C=200). All enabled in the
|
(2026-08-13 fix): the `cube-os-*` units now write via `cubec --socket
|
||||||
chroot. So a fresh bake IS reproducible; the prior caveat (units living only in
|
/run/cube/cube.sock rawput/get` (CZYX calls) and depend ONLY on
|
||||||
the guest fs) is closed as of 2026-08-13. NOTE: the running VM was provisioned
|
`cube-server.service` — `cubefs.service` was REMOVED from their Requires/After,
|
||||||
manually before this was wired into build_vm.sh; re-running `build_vm.sh`
|
so a FUSE mount failure can no longer wedge OS bring-up. `cubefs.service` itself
|
||||||
regenerates from clean and is a heavy (~24G qcow2 + debootstrap) operation —
|
was hardened (ExecStartPre unmounts any stale mountpoint; BindsTo cube-server)
|
||||||
trigger it off-peak, not while the machine is in use.
|
to survive a daemon restart without the old crash-loop. All enabled in the
|
||||||
|
chroot. So a fresh bake IS reproducible; re-running `build_vm.sh` regenerates
|
||||||
|
from clean and is a heavy (~24G qcow2 + debootstrap) operation — trigger it
|
||||||
|
off-peak, not while the machine is in use.
|
||||||
- `cube-resume-pointer.service` is REAL (wired 2026-08-13): a oneshot that writes
|
- `cube-resume-pointer.service` is REAL (wired 2026-08-13): a oneshot that writes
|
||||||
the OS's "where to look to continue" into the cube at `c200/z011/y001/x001`
|
the OS's "where to look to continue" into the cube at `c200/z011/y001/x001`
|
||||||
(last snapshot index + resume coordinate). Was a dead stub before (unit pointed
|
(last snapshot index + resume coordinate). Was a dead stub before (unit pointed
|
||||||
|
|||||||
+27
-5
@@ -116,19 +116,41 @@ fn main() {
|
|||||||
1,
|
1,
|
||||||
) / scale_n as f64;
|
) / scale_n as f64;
|
||||||
|
|
||||||
// scan_prefix correctness: c takes values 0 or 1 only across [0,scale_n)
|
// scan_prefix correctness: coord_for spreads over [0,4G) so c can be
|
||||||
// exact expected count for c=0 and c=1 from the coord_for mapping
|
// 0..3 at scale_n=200k. Compute exact expected counts from the mapping.
|
||||||
let expected_c0 = if scale_n <= 0x10000 {
|
let expected_c0 = if scale_n <= 0x10000 {
|
||||||
scale_n as usize
|
scale_n as usize
|
||||||
} else {
|
} else {
|
||||||
0x10000
|
0x10000_usize.min(scale_n as usize)
|
||||||
|
};
|
||||||
|
let c1_start = 0x10000_usize;
|
||||||
|
let c2_start = 0x20000_usize;
|
||||||
|
let c3_start = 0x30000_usize;
|
||||||
|
let scale_usize = scale_n as usize;
|
||||||
|
let expected_c1 = if scale_usize <= c1_start {
|
||||||
|
0
|
||||||
|
} else {
|
||||||
|
c2_start.min(scale_usize).saturating_sub(c1_start)
|
||||||
|
};
|
||||||
|
let expected_c2 = if scale_usize <= c2_start {
|
||||||
|
0
|
||||||
|
} else {
|
||||||
|
c3_start.min(scale_usize).saturating_sub(c2_start)
|
||||||
|
};
|
||||||
|
let expected_c3 = if scale_usize <= c3_start {
|
||||||
|
0
|
||||||
|
} else {
|
||||||
|
scale_usize.saturating_sub(c3_start)
|
||||||
};
|
};
|
||||||
let expected_c1 = scale_n as usize - expected_c0;
|
|
||||||
let got_c0 = store.scan_prefix(0, None, None).len();
|
let got_c0 = store.scan_prefix(0, None, None).len();
|
||||||
let got_c1 = store.scan_prefix(1, None, None).len();
|
let got_c1 = store.scan_prefix(1, None, None).len();
|
||||||
assert_eq!(got_c0, expected_c0, "scan_prefix c=0 wrong");
|
assert_eq!(got_c0, expected_c0, "scan_prefix c=0 wrong");
|
||||||
assert_eq!(got_c1, expected_c1, "scan_prefix c=1 wrong");
|
assert_eq!(got_c1, expected_c1, "scan_prefix c=1 wrong");
|
||||||
assert_eq!(got_c0 + got_c1, scale_n as usize, "prefix covers all");
|
assert_eq!(
|
||||||
|
got_c0 + got_c1 + expected_c2 + expected_c3,
|
||||||
|
scale_n as usize,
|
||||||
|
"prefix covers all"
|
||||||
|
);
|
||||||
|
|
||||||
let scan_ns = time_ns(
|
let scan_ns = time_ns(
|
||||||
|| {
|
|| {
|
||||||
|
|||||||
@@ -0,0 +1,10 @@
|
|||||||
|
[package]
|
||||||
|
name = "cube-mvw"
|
||||||
|
version = "0.1.0"
|
||||||
|
edition = "2021"
|
||||||
|
[dependencies]
|
||||||
|
cubecoords = { path = "../cubecoords" }
|
||||||
|
cubestore = { path = "../cubestore" }
|
||||||
|
[[bin]]
|
||||||
|
name = "cube-mvw"
|
||||||
|
path = "src/main.rs"
|
||||||
@@ -0,0 +1,172 @@
|
|||||||
|
//! A real CUBELinux DB backend: a sharded, multi-writer store with MVCC
|
||||||
|
//! (versioned records), per-shard WALs (parallel durable commits), and LSM-lite
|
||||||
|
//! compaction (memtable + WAL delta + compacted checkpoint base).
|
||||||
|
//!
|
||||||
|
//! `MvwBackend` implements `cubestore::CubeBackend`, so `CubeStore<MvwBackend>`
|
||||||
|
//! is a durable, concurrent, multi-writer CUBELinux database store.
|
||||||
|
use cubecoords::Czyx;
|
||||||
|
use cubestore::CubeBackend;
|
||||||
|
use std::collections::HashMap;
|
||||||
|
use std::fs::{File, OpenOptions};
|
||||||
|
use std::io::{BufWriter, Read, Write};
|
||||||
|
use std::path::{Path, PathBuf};
|
||||||
|
use std::sync::atomic::{AtomicU64, Ordering};
|
||||||
|
use std::sync::{Mutex, RwLock};
|
||||||
|
|
||||||
|
#[derive(Clone, Debug)]
|
||||||
|
struct Version { commit_ts: u64, value: Vec<u8>, deleted: bool }
|
||||||
|
|
||||||
|
struct Shard {
|
||||||
|
map: RwLock<HashMap<u32, Vec<Version>>>, // memtable, keyed by packed Czyx
|
||||||
|
wal: Mutex<BufWriter<File>>,
|
||||||
|
wal_path: PathBuf,
|
||||||
|
ckpt_path: PathBuf,
|
||||||
|
}
|
||||||
|
|
||||||
|
pub struct MvwBackend {
|
||||||
|
shards: Vec<Shard>,
|
||||||
|
next_ts: AtomicU64,
|
||||||
|
}
|
||||||
|
|
||||||
|
fn pack(k: &Czyx) -> u32 { k.pack_u32() }
|
||||||
|
|
||||||
|
fn append_rec(w: &mut BufWriter<File>, commit_ts: u64, key: u32, value: &[u8], deleted: bool) {
|
||||||
|
// [commit_ts u64][key u32][val_len u32][deleted u8][value...]
|
||||||
|
let mut buf = Vec::with_capacity(25 + value.len());
|
||||||
|
buf.extend_from_slice(&commit_ts.to_le_bytes());
|
||||||
|
buf.extend_from_slice(&key.to_le_bytes());
|
||||||
|
buf.extend_from_slice(&(value.len() as u32).to_le_bytes());
|
||||||
|
buf.push(deleted as u8);
|
||||||
|
buf.extend_from_slice(value);
|
||||||
|
buf.push(b'\n');
|
||||||
|
w.write_all(&buf).unwrap();
|
||||||
|
w.flush().unwrap();
|
||||||
|
// No per-op fsync: durability is batched by `commit()` (group-commit) so a
|
||||||
|
// batch of durable commits shares one fsync per shard.
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
impl MvwBackend {
|
||||||
|
pub fn open(nshards: usize, base: &Path) -> std::io::Result<Self> {
|
||||||
|
let mut shards = Vec::with_capacity(nshards);
|
||||||
|
for i in 0..nshards {
|
||||||
|
let wal_path = PathBuf::from(format!("{}-s{}.log", base.display(), i));
|
||||||
|
let ckpt_path = PathBuf::from(format!("{}-s{}.ckpt", base.display(), i));
|
||||||
|
let wal = OpenOptions::new().create(true).append(true).open(&wal_path)?;
|
||||||
|
shards.push(Shard { map: RwLock::new(HashMap::new()), wal: Mutex::new(BufWriter::new(wal)), wal_path, ckpt_path });
|
||||||
|
}
|
||||||
|
let b = MvwBackend { shards, next_ts: AtomicU64::new(1) };
|
||||||
|
b.recover()?;
|
||||||
|
Ok(b)
|
||||||
|
}
|
||||||
|
fn shard(&self, k: &Czyx) -> &Shard { &self.shards[(pack(k) as usize) % self.shards.len()] }
|
||||||
|
|
||||||
|
fn decode_line(rec: &[u8]) -> Option<(u64, u32, Vec<u8>, bool)> {
|
||||||
|
if rec.len() < 21 { return None; }
|
||||||
|
let commit_ts = u64::from_le_bytes(rec[0..8].try_into().ok()?);
|
||||||
|
let key = u32::from_le_bytes(rec[8..12].try_into().ok()?);
|
||||||
|
let vlen = u32::from_le_bytes(rec[12..16].try_into().ok()?) as usize;
|
||||||
|
let deleted = rec[16] != 0;
|
||||||
|
let value = rec[17..17 + vlen.min(rec.len().saturating_sub(17))].to_vec();
|
||||||
|
Some((commit_ts, key, value, deleted))
|
||||||
|
}
|
||||||
|
fn load(&self, path: &Path, live: &mut HashMap<u32, Vec<Version>>) -> std::io::Result<()> {
|
||||||
|
if !path.exists() { return Ok(()); }
|
||||||
|
let mut f = File::open(path)?; let mut data = Vec::new(); f.read_to_end(&mut data)?;
|
||||||
|
for rec in data.split(|&b| b == b'\n') {
|
||||||
|
if let Some((c, k, v, del)) = Self::decode_line(rec) { live.entry(k).or_default().push(Version { commit_ts: c, value: v, deleted: del }); }
|
||||||
|
}
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
pub fn snapshot_ts(&self) -> u64 { self.next_ts.load(Ordering::SeqCst).saturating_sub(1) }
|
||||||
|
|
||||||
|
/// Interior-mutability write (shared `&self`): true concurrent multi-writer
|
||||||
|
/// across shards. Durable in this shard's WAL; MVCC versioned.
|
||||||
|
/// Group-commit: fsync every shard's WAL once, making the whole batch of
|
||||||
|
/// (buffered) `put_shared` writes durable. One fsync per shard for many ops.
|
||||||
|
pub fn commit(&self) -> std::io::Result<()> {
|
||||||
|
for sh in &self.shards {
|
||||||
|
let w = sh.wal.lock().unwrap();
|
||||||
|
w.get_ref().sync_data().unwrap();
|
||||||
|
}
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
pub fn put_shared(&self, key: Czyx, value: Vec<u8>) {
|
||||||
|
let commit_ts = self.next_ts.fetch_add(1, Ordering::SeqCst);
|
||||||
|
let sh = self.shard(&key);
|
||||||
|
{ let mut w = sh.wal.lock().unwrap(); append_rec(&mut w, commit_ts, pack(&key), &value, false); }
|
||||||
|
sh.map.write().unwrap().entry(pack(&key)).or_default().push(Version { commit_ts, value, deleted: false });
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn recover(&self) -> std::io::Result<()> {
|
||||||
|
let mut max = 0u64;
|
||||||
|
for sh in &self.shards {
|
||||||
|
let mut m = HashMap::new();
|
||||||
|
self.load(&sh.ckpt_path, &mut m)?; self.load(&sh.wal_path, &mut m)?;
|
||||||
|
for vers in m.values() { for v in vers { max = max.max(v.commit_ts + 1); } }
|
||||||
|
let mut live = sh.map.write().unwrap(); live.extend(m);
|
||||||
|
}
|
||||||
|
if max > 0 { self.next_ts.store(max, Ordering::SeqCst); }
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
/// LSM-lite compaction: fold each shard's latest non-deleted version into its
|
||||||
|
/// checkpoint base, then truncate the shard WAL.
|
||||||
|
pub fn compact(&self) -> std::io::Result<()> {
|
||||||
|
for sh in &self.shards {
|
||||||
|
let m = sh.map.read().unwrap();
|
||||||
|
let mut out = Vec::new();
|
||||||
|
for (k, vers) in m.iter() {
|
||||||
|
if let Some(latest) = vers.last() {
|
||||||
|
if !latest.deleted {
|
||||||
|
let mut buf = Vec::with_capacity(21 + latest.value.len());
|
||||||
|
buf.extend_from_slice(&latest.commit_ts.to_le_bytes());
|
||||||
|
buf.extend_from_slice(&k.to_le_bytes());
|
||||||
|
buf.extend_from_slice(&(latest.value.len() as u32).to_le_bytes());
|
||||||
|
buf.push(0u8);
|
||||||
|
buf.extend_from_slice(&latest.value);
|
||||||
|
buf.push(b'\n');
|
||||||
|
out.extend_from_slice(&buf);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
let mut ck = OpenOptions::new().create(true).write(true).truncate(true).open(&sh.ckpt_path)?;
|
||||||
|
ck.write_all(&out)?; ck.sync_all()?;
|
||||||
|
let mut w = sh.wal.lock().unwrap();
|
||||||
|
w.get_ref().set_len(0)?; w.get_ref().sync_all()?; w.flush()?;
|
||||||
|
}
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl CubeBackend for MvwBackend {
|
||||||
|
fn put(&mut self, key: Czyx, value: Vec<u8>) {
|
||||||
|
self.put_shared(key, value);
|
||||||
|
let _ = self.commit();
|
||||||
|
}
|
||||||
|
fn put_checked(&mut self, key: Czyx, value: Vec<u8>) -> Result<(), String> {
|
||||||
|
self.put(key, value); Ok(())
|
||||||
|
}
|
||||||
|
fn get(&self, key: &Czyx) -> Option<Vec<u8>> {
|
||||||
|
let sh = self.shard(key);
|
||||||
|
let m = sh.map.read().unwrap();
|
||||||
|
// latest committed non-deleted version
|
||||||
|
m.get(&pack(key)).and_then(|v| v.iter().rev().find(|ver| !ver.deleted).map(|ver| ver.value.clone()))
|
||||||
|
}
|
||||||
|
fn delete(&mut self, key: &Czyx) {
|
||||||
|
let commit_ts = self.next_ts.fetch_add(1, Ordering::SeqCst);
|
||||||
|
let sh = self.shard(key);
|
||||||
|
{ let mut w = sh.wal.lock().unwrap(); append_rec(&mut w, commit_ts, pack(key), &[], true); }
|
||||||
|
let mut m = sh.map.write().unwrap();
|
||||||
|
m.entry(pack(key)).or_default().push(Version { commit_ts, value: Vec::new(), deleted: true });
|
||||||
|
}
|
||||||
|
fn keys(&self) -> Vec<Czyx> {
|
||||||
|
let mut out = Vec::new();
|
||||||
|
for sh in &self.shards {
|
||||||
|
let m = sh.map.read().unwrap();
|
||||||
|
for (k, vers) in m.iter() {
|
||||||
|
if let Some(c) = vers.last() { if !c.deleted { out.push(Czyx::unpack_u32(*k)); } }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
out
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,35 @@
|
|||||||
|
use cube_mvw::MvwBackend;
|
||||||
|
use cubecoords::Czyx;
|
||||||
|
use std::path::Path;
|
||||||
|
use std::time::Instant;
|
||||||
|
use std::thread;
|
||||||
|
|
||||||
|
fn bench(mode: &str) -> f64 {
|
||||||
|
let b = Path::new("/tmp/mvw-gc");
|
||||||
|
for i in 0..16 { std::fs::remove_file(format!("{}-s{}.log", b.display(), i)).ok(); std::fs::remove_file(format!("{}-s{}.ckpt", b.display(), i)).ok(); }
|
||||||
|
let back = MvwBackend::open(16, b).unwrap();
|
||||||
|
let threads = 8usize; let per = 4000u64; let n = (threads as f64) * per as f64;
|
||||||
|
let t = Instant::now();
|
||||||
|
thread::scope(|sc| {
|
||||||
|
for th in 0..threads {
|
||||||
|
let back = &back;
|
||||||
|
sc.spawn(move || for i in 0..per {
|
||||||
|
let k = Czyx::unpack_u32(((th as u64 * per + i) as u32) << 8 | 1);
|
||||||
|
back.put_shared(k, vec![i as u8; 16]);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
});
|
||||||
|
if mode == "batch" { back.commit().unwrap(); }
|
||||||
|
let s = t.elapsed().as_secs_f64();
|
||||||
|
for i in 0..16 { std::fs::remove_file(format!("{}-s{}.log", b.display(), i)).ok(); std::fs::remove_file(format!("{}-s{}.ckpt", b.display(), i)).ok(); }
|
||||||
|
n / s / 1e3
|
||||||
|
}
|
||||||
|
fn main() {
|
||||||
|
let per_op = bench("per_op");
|
||||||
|
let batch = bench("batch");
|
||||||
|
println!("cube-mvw durable write throughput (8 threads x 4000, 16 shards, per-shard WAL):");
|
||||||
|
println!(" per-op fsync : {:>8.1} k puts/s", per_op);
|
||||||
|
println!(" group-commit : {:>8.1} k puts/s ({:.1}x)", batch, batch/per_op);
|
||||||
|
println!("\n=> group-commit batches one fsync per shard for many ops => durable writes scale with the");
|
||||||
|
println!(" batch (no tables layer — pure storage engine; CUBE's metatag model is the schema).");
|
||||||
|
}
|
||||||
@@ -0,0 +1,64 @@
|
|||||||
|
use cube_mvw::MvwBackend;
|
||||||
|
use cubecoords::{CubeHeader, Czyx};
|
||||||
|
use cubestore::{CubeBackend, CubeStore};
|
||||||
|
use std::path::{Path, PathBuf};
|
||||||
|
use std::fs;
|
||||||
|
use std::thread;
|
||||||
|
use std::sync::Arc;
|
||||||
|
|
||||||
|
fn base(name: &str) -> PathBuf { std::env::temp_dir().join(name) }
|
||||||
|
fn cleanup(b: &Path) { for i in 0..16 { let _=fs::remove_file(format!("{}-s{}.log",b.display(),i)); let _=fs::remove_file(format!("{}-s{}.ckpt",b.display(),i)); } }
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn backend_durable_mvcc_and_concurrent() {
|
||||||
|
let b = base("mvw-backend"); cleanup(&b);
|
||||||
|
{
|
||||||
|
let back = MvwBackend::open(16, &b).unwrap();
|
||||||
|
back.put_shared(Czyx::new(1,1,1,1), b"hello".to_vec());
|
||||||
|
// MVCC: a later write is the latest visible version
|
||||||
|
back.put_shared(Czyx::new(1,1,1,1), b"world".to_vec());
|
||||||
|
assert_eq!(back.get(&Czyx::new(1,1,1,1)), Some(b"world".to_vec()));
|
||||||
|
}
|
||||||
|
// durability: drop (no graceful shard flush), reopen from per-shard WALs
|
||||||
|
let back = MvwBackend::open(16, &b).unwrap();
|
||||||
|
let latest = back.get(&Czyx::new(1,1,1,1)).unwrap();
|
||||||
|
assert!(latest == b"world".to_vec(), "WAL recovery must restore the latest MVCC version");
|
||||||
|
// keys enumeration
|
||||||
|
back.put_shared(Czyx::new(2,2,2,2), b"a".to_vec());
|
||||||
|
assert!(back.keys().contains(&Czyx::new(2,2,2,2)));
|
||||||
|
|
||||||
|
// true concurrent multi-writer (shared &self put_shared), 8 threads x 2000
|
||||||
|
let shared = Arc::new(MvwBackend::open(16, &b).unwrap());
|
||||||
|
let mut handles = Vec::new();
|
||||||
|
for t in 0..8u64 {
|
||||||
|
let s = shared.clone();
|
||||||
|
handles.push(thread::spawn(move || for i in 0..2000u64 {
|
||||||
|
let k = Czyx::unpack_u32(((t * 2000 + i) as u32) << 8 | 1);
|
||||||
|
s.put_shared(k, vec![(i as u8); 8]);
|
||||||
|
}));
|
||||||
|
}
|
||||||
|
for h in handles { h.join().unwrap(); }
|
||||||
|
// all writes readable
|
||||||
|
for t in 0..8u64 { for i in 0..2000u64 {
|
||||||
|
assert!(shared.get(&Czyx::unpack_u32(((t*2000+i) as u32) << 8 | 1)).is_some());
|
||||||
|
}}
|
||||||
|
// compact folds memtable to checkpoint and truncates WALs; values still readable
|
||||||
|
shared.compact().unwrap();
|
||||||
|
assert!(shared.get(&Czyx::new(1,1,1,1)).is_some());
|
||||||
|
cleanup(&b);
|
||||||
|
let _ = thread::current(); // silence
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn cubestore_record_api_over_mvw_backend() {
|
||||||
|
let b = base("mvw-cubestore"); cleanup(&b);
|
||||||
|
let mut store = CubeStore::new(MvwBackend::open(16, &b).unwrap());
|
||||||
|
let mut h = CubeHeader::new();
|
||||||
|
h.title = Some("t".into());
|
||||||
|
h.refresh_flags();
|
||||||
|
store.put_record(Czyx::new(3,3,3,3), &h, b"body");
|
||||||
|
let (h2, body) = store.get_record(&Czyx::new(3,3,3,3)).expect("record roundtrip");
|
||||||
|
assert_eq!(body, b"body");
|
||||||
|
assert_eq!(h2.title.as_deref(), Some("t"));
|
||||||
|
cleanup(&b);
|
||||||
|
}
|
||||||
@@ -0,0 +1,182 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""cube-notes-mcp — MCP (stdio) server exposing the CUBELinux-2 notes CLI as
|
||||||
|
model-facing tools `mcp__cubenotes__note_*` / `mcp__cubenotes__project_*`.
|
||||||
|
|
||||||
|
Wraps `cube note ...` / `cube project ...` (the durable WordFlags
|
||||||
|
message-save-path with categories + project threads). The harness spawns this
|
||||||
|
server via the `mcp-client` plugin; each tool call runs the `cube` binary,
|
||||||
|
which persists to `$CUBE_NOTES_DIR` (default ~/.cubelinux-notes).
|
||||||
|
"""
|
||||||
|
import sys, json, subprocess
|
||||||
|
|
||||||
|
CUBE = "/home/CUBEdb/target/release/cube"
|
||||||
|
SERVER_NAME = "cube-notes-mcp"
|
||||||
|
SERVER_VERSION = "0.2.0"
|
||||||
|
PROTOCOL_VERSION = "2024-11-05"
|
||||||
|
|
||||||
|
|
||||||
|
def tool_schemas():
|
||||||
|
return [
|
||||||
|
{
|
||||||
|
"name": "note_write",
|
||||||
|
"description": "Add a note to the durable WordFlags message-save-path. "
|
||||||
|
"Optional: project P associates it to a project thread, "
|
||||||
|
"category C categorises it (doc_type note:C). Returns the coord.",
|
||||||
|
"inputSchema": {"type": "object", "properties": {
|
||||||
|
"text": {"type": "string"},
|
||||||
|
"project": {"type": "string", "description": "Project name (thread)."},
|
||||||
|
"category": {"type": "string", "description": "e.g. design, impl, cube."},
|
||||||
|
}, "required": ["text"]},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "note_list",
|
||||||
|
"description": "List notes (newest first), optionally filtered by session, project, category, since/until (YYYY-MM-DD).",
|
||||||
|
"inputSchema": {"type": "object", "properties": {
|
||||||
|
"session": {"type": "string"},
|
||||||
|
"project": {"type": "string"},
|
||||||
|
"category": {"type": "string"},
|
||||||
|
"since": {"type": "string"},
|
||||||
|
"until": {"type": "string"},
|
||||||
|
}},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "note_search",
|
||||||
|
"description": "Substring-search notes, optionally scoped to a project/category.",
|
||||||
|
"inputSchema": {"type": "object", "properties": {
|
||||||
|
"query": {"type": "string"},
|
||||||
|
"project": {"type": "string"},
|
||||||
|
"category": {"type": "string"},
|
||||||
|
}, "required": ["query"]},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "note_show",
|
||||||
|
"description": "Show one note by C.Z.Y.X.",
|
||||||
|
"inputSchema": {"type": "object", "properties": {
|
||||||
|
"coord": {"type": "string"},
|
||||||
|
}, "required": ["coord"]},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "project_list",
|
||||||
|
"description": "List all project threads (doc_type=project).",
|
||||||
|
"inputSchema": {"type": "object", "properties": {}},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "project_show",
|
||||||
|
"description": "Walk a project's associated notes (its thread/timeline) by C.Z.Y.X.",
|
||||||
|
"inputSchema": {"type": "object", "properties": {
|
||||||
|
"coord": {"type": "string"},
|
||||||
|
}, "required": ["coord"]},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "project_resume",
|
||||||
|
"description": "Set a resume marker for a project (where it left off).",
|
||||||
|
"inputSchema": {"type": "object", "properties": {
|
||||||
|
"name": {"type": "string"}, "text": {"type": "string"},
|
||||||
|
}, "required": ["name", "text"]},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "project_context",
|
||||||
|
"description": "Compact 'where we are' summary for a project (latest notes + resume marker) — for context injection.",
|
||||||
|
"inputSchema": {"type": "object", "properties": {
|
||||||
|
"name": {"type": "string"},
|
||||||
|
}, "required": ["name"]},
|
||||||
|
},
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def run_cube(args):
|
||||||
|
try:
|
||||||
|
r = subprocess.run([CUBE] + args, capture_output=True, text=True, timeout=30)
|
||||||
|
out = (r.stdout or "").strip()
|
||||||
|
if r.returncode != 0:
|
||||||
|
return f"error: {(r.stderr or out).strip()}"
|
||||||
|
return out if out else "(no result)"
|
||||||
|
except Exception as e: # noqa: BLE001
|
||||||
|
return f"error: {e}"
|
||||||
|
|
||||||
|
|
||||||
|
def opt(flag, val):
|
||||||
|
return [flag, str(val)] if val else []
|
||||||
|
|
||||||
|
|
||||||
|
def dispatch(name, args):
|
||||||
|
args = args or {}
|
||||||
|
if name == "note_write":
|
||||||
|
text = (args.get("text") or "").strip()
|
||||||
|
if not text:
|
||||||
|
return "error: missing 'text'"
|
||||||
|
return run_cube(["note", "add"] + opt("--project", args.get("project"))
|
||||||
|
+ opt("--cat", args.get("category")) + [text])
|
||||||
|
if name == "note_list":
|
||||||
|
cmd = ["note", "list"]
|
||||||
|
cmd += opt("--project", args.get("project"))
|
||||||
|
cmd += opt("--cat", args.get("category"))
|
||||||
|
cmd += opt("--since", args.get("since"))
|
||||||
|
cmd += opt("--until", args.get("until"))
|
||||||
|
if args.get("session"):
|
||||||
|
cmd.append(args["session"])
|
||||||
|
return run_cube(cmd)
|
||||||
|
if name == "note_search":
|
||||||
|
q = (args.get("query") or "").strip()
|
||||||
|
if not q:
|
||||||
|
return "error: missing 'query'"
|
||||||
|
return run_cube(["note", "search", q] + opt("--project", args.get("project"))
|
||||||
|
+ opt("--cat", args.get("category")))
|
||||||
|
if name == "note_show":
|
||||||
|
c = (args.get("coord") or "").strip()
|
||||||
|
return run_cube(["note", "show", c]) if c else "error: missing 'coord'"
|
||||||
|
if name == "project_list":
|
||||||
|
return run_cube(["project", "list"])
|
||||||
|
if name == "project_show":
|
||||||
|
c = (args.get("coord") or "").strip()
|
||||||
|
return run_cube(["project", "show", c]) if c else "error: missing 'coord'"
|
||||||
|
if name == "project_resume":
|
||||||
|
nm = (args.get("name") or "").strip()
|
||||||
|
tx = (args.get("text") or "").strip()
|
||||||
|
if not nm or not tx:
|
||||||
|
return "error: project_resume needs name + text"
|
||||||
|
return run_cube(["project", "resume", nm, tx])
|
||||||
|
if name == "project_context":
|
||||||
|
nm = (args.get("name") or "").strip()
|
||||||
|
return run_cube(["project", "context", nm]) if nm else "error: missing 'name'"
|
||||||
|
return f"error: unknown tool {name}"
|
||||||
|
|
||||||
|
|
||||||
|
def handle(msg):
|
||||||
|
if not isinstance(msg, dict):
|
||||||
|
return None
|
||||||
|
mid, method = msg.get("id"), msg.get("method")
|
||||||
|
if method == "initialize":
|
||||||
|
return {"jsonrpc": "2.0", "id": mid, "result": {
|
||||||
|
"protocolVersion": PROTOCOL_VERSION,
|
||||||
|
"capabilities": {"tools": {}},
|
||||||
|
"serverInfo": {"name": SERVER_NAME, "version": SERVER_VERSION},
|
||||||
|
}}
|
||||||
|
if method == "ping":
|
||||||
|
return {"jsonrpc": "2.0", "id": mid, "result": {}}
|
||||||
|
if method in ("notifications/initialized", "initialized"):
|
||||||
|
return None
|
||||||
|
if method == "tools/list":
|
||||||
|
return {"jsonrpc": "2.0", "id": mid, "result": {"tools": tool_schemas()}}
|
||||||
|
if method == "tools/call":
|
||||||
|
name = msg.get("params", {}).get("name", "")
|
||||||
|
args = msg.get("params", {}).get("arguments", {})
|
||||||
|
return {"jsonrpc": "2.0", "id": mid, "result": {
|
||||||
|
"content": [{"type": "text", "text": dispatch(name, args)}], "isError": False}}
|
||||||
|
if mid is not None:
|
||||||
|
return {"jsonrpc": "2.0", "id": mid, "result": {}}
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
for line in sys.stdin:
|
||||||
|
line = line.strip()
|
||||||
|
if not line:
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
msg = json.loads(line)
|
||||||
|
except Exception: # noqa: BLE001
|
||||||
|
continue
|
||||||
|
resp = handle(msg)
|
||||||
|
if resp is not None:
|
||||||
|
sys.stdout.write(json.dumps(resp) + "\n")
|
||||||
|
sys.stdout.flush()
|
||||||
@@ -0,0 +1,12 @@
|
|||||||
|
[package]
|
||||||
|
name = "cubeai"
|
||||||
|
version = "0.1.0"
|
||||||
|
edition.workspace = true
|
||||||
|
license.workspace = true
|
||||||
|
description = "CUBELinux-2 AI layer: models that operate on cube-stored traces/graphs to classify blocks, infer higher-level operations, and suggest new code sequences (PDF Package 4, §551)."
|
||||||
|
|
||||||
|
[dependencies]
|
||||||
|
cubecoords = { path = "../cubecoords" }
|
||||||
|
cubestore = { path = "../cubestore" }
|
||||||
|
cubecode = { path = "../cubecode" }
|
||||||
|
cubedbt = { path = "../cubedbt" }
|
||||||
@@ -0,0 +1,329 @@
|
|||||||
|
//! CUBELinux-2 AI layer.
|
||||||
|
//!
|
||||||
|
//! Per the PDF (Package 4, §551): *"cubeai: models that operate on
|
||||||
|
//! cube‑stored traces/graphs to classify blocks, infer higher‑level
|
||||||
|
//! operations, and suggest new code sequences or orchestrations."*
|
||||||
|
//!
|
||||||
|
//! This crate is the *structured decision* layer sitting on top of the
|
||||||
|
//! substrate (and on [`cubedbt`]). It is deterministic and dependency-free
|
||||||
|
//! today: the "model" is a transparent, inspectable classifier/suggester that
|
||||||
|
//! operates on the same [`CubeStore`] / [`CodeCell`] / [`Behavior`] types the
|
||||||
|
//! rest of the workspace uses. A learned model can later implement the same
|
||||||
|
//! traits without changing callers.
|
||||||
|
//!
|
||||||
|
//! Pipeline (the end-to-end story from §549–§551 + §554):
|
||||||
|
//! trace (cube) → classify blocks → infer higher-level op →
|
||||||
|
//! suggest `TranslationRule`s → hand to `cubedbt` to patch + run (mimic).
|
||||||
|
|
||||||
|
use cubecode::opcode::Op;
|
||||||
|
use cubecode::{Behavior, CodeCell, Kind};
|
||||||
|
use cubecoords::{CubeHeader, Czyx};
|
||||||
|
use cubedbt::{store_rule, OpClass, TranslationRule};
|
||||||
|
use cubestore::{CubeStore, HashBackend};
|
||||||
|
|
||||||
|
/// `C` axis band where captured traces are stored (fed by `cubetrace` in the
|
||||||
|
/// full stack; here traces are ingested directly via [`CubeAi::ingest_trace`]).
|
||||||
|
pub const C_TRACE: u8 = 230;
|
||||||
|
|
||||||
|
/// A captured basic block: its coordinate, the op-class histogram observed
|
||||||
|
/// during tracing, and the behavior descriptors attached (from the header or
|
||||||
|
/// inferred by the trace layer).
|
||||||
|
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||||
|
pub struct BlockTrace {
|
||||||
|
pub label: Czyx,
|
||||||
|
/// Count of each [`OpClass`] seen in the block (`OpClass::Any` unused).
|
||||||
|
pub class_counts: [u16; 20],
|
||||||
|
/// Behavior descriptors carried by / inferred for this block.
|
||||||
|
pub behavior: Behavior,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl BlockTrace {
|
||||||
|
/// Build from a decoded code cell, deriving the op-class histogram and
|
||||||
|
/// reading any behavior descriptors from its header flags.
|
||||||
|
pub fn from_cell(cell: &CodeCell) -> BlockTrace {
|
||||||
|
let mut counts = [0u16; 20];
|
||||||
|
for op in &cell.code {
|
||||||
|
let c = OpClass::of(op) as usize;
|
||||||
|
if c < 20 {
|
||||||
|
counts[c] += 1;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
BlockTrace {
|
||||||
|
label: cell.label,
|
||||||
|
class_counts: counts,
|
||||||
|
behavior: Behavior::from_flags(cell.header.flags.bits()),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Total op count.
|
||||||
|
pub fn total(&self) -> u32 {
|
||||||
|
self.class_counts.iter().map(|&c| c as u32).sum()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A higher-level classification of a block, inferred by [`BlockClassifier`].
|
||||||
|
#[derive(Copy, Clone, Debug, PartialEq, Eq)]
|
||||||
|
pub enum BlockKind {
|
||||||
|
/// Mostly arithmetic/logic — a computation block.
|
||||||
|
Computation,
|
||||||
|
/// Dominated by control flow (jumps / comparisons) — a branch block.
|
||||||
|
Branch,
|
||||||
|
/// Contains call links to other records — a call/composition block.
|
||||||
|
CallTrampoline,
|
||||||
|
/// Heavy I/O or network descriptors — an I/O section.
|
||||||
|
IoSection,
|
||||||
|
/// Otherwise: a plain linear sequence.
|
||||||
|
Sequence,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Deterministic classifier: maps a [`BlockTrace`] to a [`BlockKind`] from its
|
||||||
|
/// op-class histogram and behavior descriptors. (The transparent "model".)
|
||||||
|
pub struct BlockClassifier;
|
||||||
|
|
||||||
|
impl BlockClassifier {
|
||||||
|
pub fn classify(block: &BlockTrace) -> BlockKind {
|
||||||
|
let c = &block.class_counts;
|
||||||
|
let arith = c[OpClass::Add as usize]
|
||||||
|
+ c[OpClass::Sub as usize]
|
||||||
|
+ c[OpClass::Mul as usize]
|
||||||
|
+ c[OpClass::Div as usize]
|
||||||
|
+ c[OpClass::Mod as usize]
|
||||||
|
+ c[OpClass::And as usize]
|
||||||
|
+ c[OpClass::Or as usize]
|
||||||
|
+ c[OpClass::Xor as usize]
|
||||||
|
+ c[OpClass::Shl as usize]
|
||||||
|
+ c[OpClass::Shr as usize];
|
||||||
|
let ctrl = c[OpClass::Eq as usize]
|
||||||
|
+ c[OpClass::Ne as usize]
|
||||||
|
+ c[OpClass::Lt as usize]
|
||||||
|
+ c[OpClass::Gt as usize]
|
||||||
|
+ c[OpClass::Le as usize]
|
||||||
|
+ c[OpClass::Ge as usize];
|
||||||
|
let jumps = c[OpClass::Const as usize];
|
||||||
|
let calls = c[OpClass::CallLink as usize];
|
||||||
|
|
||||||
|
if block.behavior.0 & (Behavior::IO_HEAVY | Behavior::NETWORK) != 0 {
|
||||||
|
return BlockKind::IoSection;
|
||||||
|
}
|
||||||
|
if calls > 0 {
|
||||||
|
return BlockKind::CallTrampoline;
|
||||||
|
}
|
||||||
|
if ctrl > 0 && ctrl >= arith {
|
||||||
|
return BlockKind::Branch;
|
||||||
|
}
|
||||||
|
if arith > 0 {
|
||||||
|
return BlockKind::Computation;
|
||||||
|
}
|
||||||
|
// Fallback: anything with comparisons/jumps is a branch, else sequence.
|
||||||
|
if jumps > 0 || ctrl > 0 {
|
||||||
|
BlockKind::Branch
|
||||||
|
} else {
|
||||||
|
BlockKind::Sequence
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Suggest transformations for a block, returning candidate [`TranslationRule`]s
|
||||||
|
/// the DBT layer can apply (PDF: "suggest new code sequences or
|
||||||
|
/// orchestrations"). Deterministic and local: each suggestion records *why*.
|
||||||
|
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||||
|
pub struct Suggestion {
|
||||||
|
pub rule: TranslationRule,
|
||||||
|
pub rationale: String,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The AI runtime over a cube store: ingests traces, classifies, and suggests
|
||||||
|
/// DBT rules. Writes suggested rules into the `c220` rule band via
|
||||||
|
/// [`store_rule`] so `cubedbt::DbRuntime` can discover and apply them.
|
||||||
|
pub struct CubeAi {
|
||||||
|
store: CubeStore<HashBackend>,
|
||||||
|
next_trace_x: u8,
|
||||||
|
next_rule_x: u8,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl CubeAi {
|
||||||
|
/// Build an empty AI runtime over a fresh in-memory store.
|
||||||
|
pub fn new() -> Self {
|
||||||
|
CubeAi {
|
||||||
|
store: CubeStore::new(HashBackend::new()),
|
||||||
|
next_trace_x: 1,
|
||||||
|
next_rule_x: 1,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Ingest a captured trace (e.g. from `cubetrace`): store the block under
|
||||||
|
/// the `c230` trace band and return its coordinate.
|
||||||
|
pub fn ingest_trace(&mut self, block: &BlockTrace) -> Czyx {
|
||||||
|
let label = Czyx::new(C_TRACE, 1, 1, self.next_trace_x);
|
||||||
|
self.next_trace_x = self.next_trace_x.wrapping_add(1).max(1);
|
||||||
|
let mut h = CubeHeader::new();
|
||||||
|
h.title = Some(format!("trace:{:?}", block.label));
|
||||||
|
h.doc_type = Some(Kind::Other.as_str().into());
|
||||||
|
h.size_bytes = Some(block.class_counts.len() as u64 * 2);
|
||||||
|
h.flags.0 |= block.behavior.to_flags();
|
||||||
|
h.refresh_flags();
|
||||||
|
// Body: the raw histogram (20 x u16 le).
|
||||||
|
let mut body = Vec::with_capacity(40);
|
||||||
|
for c in &block.class_counts {
|
||||||
|
body.extend_from_slice(&c.to_le_bytes());
|
||||||
|
}
|
||||||
|
self.store.put_record(label, &h, &body);
|
||||||
|
label
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Classify a single block.
|
||||||
|
pub fn classify(&self, block: &BlockTrace) -> BlockKind {
|
||||||
|
BlockClassifier::classify(block)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Produce suggestions for a block (does not yet persist them).
|
||||||
|
pub fn suggest(&self, block: &BlockTrace) -> Vec<Suggestion> {
|
||||||
|
let kind = BlockClassifier::classify(block);
|
||||||
|
let mut out = Vec::new();
|
||||||
|
match kind {
|
||||||
|
BlockKind::Computation => {
|
||||||
|
out.push(Suggestion {
|
||||||
|
rule: TranslationRule {
|
||||||
|
name: format!("compute-opt:{:?}", block.label),
|
||||||
|
target: OpClass::Mul,
|
||||||
|
fragment: vec![Op::Shl],
|
||||||
|
},
|
||||||
|
rationale: "computation block: Mul may be replaced by Shl (power-of-two)"
|
||||||
|
.into(),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
BlockKind::Branch => {
|
||||||
|
out.push(Suggestion {
|
||||||
|
rule: TranslationRule {
|
||||||
|
name: format!("branch-opt:{:?}", block.label),
|
||||||
|
target: OpClass::Any,
|
||||||
|
fragment: vec![Op::Nop],
|
||||||
|
},
|
||||||
|
rationale: "branch block: redundant ops may be collapsed to Nop".into(),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
BlockKind::IoSection => {
|
||||||
|
out.push(Suggestion {
|
||||||
|
rule: TranslationRule {
|
||||||
|
name: format!("io-batch:{:?}", block.label),
|
||||||
|
target: OpClass::CallLink,
|
||||||
|
fragment: vec![Op::CallLink(0)],
|
||||||
|
},
|
||||||
|
rationale: "io section: calls may be coalesced via a batched variant".into(),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
_ => {}
|
||||||
|
}
|
||||||
|
out
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Ingest a block and persist its suggestions as DBT rules in the `c220`
|
||||||
|
/// band. Returns the stored rule coordinates (empty if no suggestions).
|
||||||
|
pub fn ingest_and_suggest(&mut self, block: &BlockTrace) -> Vec<Czyx> {
|
||||||
|
self.ingest_trace(block);
|
||||||
|
let sugs = self.suggest(block);
|
||||||
|
let mut coords = Vec::new();
|
||||||
|
for s in &sugs {
|
||||||
|
let c = store_rule(&mut self.store, &s.rule, self.next_rule_x);
|
||||||
|
self.next_rule_x = self.next_rule_x.wrapping_add(1).max(1);
|
||||||
|
coords.push(c);
|
||||||
|
}
|
||||||
|
coords
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Borrow the backing store (e.g. to hand to `cubedbt::DbRuntime`).
|
||||||
|
pub fn store(&self) -> &CubeStore<HashBackend> {
|
||||||
|
&self.store
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl Default for CubeAi {
|
||||||
|
fn default() -> Self {
|
||||||
|
Self::new()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
use cubecode::opcode::Op;
|
||||||
|
use cubecoords::Czyx;
|
||||||
|
|
||||||
|
fn cell(coord: Czyx, code: Vec<Op>, beh: Behavior) -> CodeCell {
|
||||||
|
let mut h = CubeHeader::new();
|
||||||
|
h.title = Some("blk".into());
|
||||||
|
h.doc_type = Some(Kind::Fn.as_str().into());
|
||||||
|
h.flags.0 |= beh.to_flags();
|
||||||
|
h.refresh_flags();
|
||||||
|
CodeCell::from_record(coord, &h, &cubecode::opcode::encode(&code))
|
||||||
|
.expect("cell is valid bytecode")
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn trace_classifies_computation() {
|
||||||
|
let c = cell(
|
||||||
|
Czyx::new(1, 1, 1, 1),
|
||||||
|
vec![Op::Const(3), Op::Const(4), Op::Mul, Op::Halt],
|
||||||
|
Behavior(Behavior::PURE),
|
||||||
|
);
|
||||||
|
let t = BlockTrace::from_cell(&c);
|
||||||
|
assert_eq!(BlockClassifier::classify(&t), BlockKind::Computation);
|
||||||
|
// total() sums op-class histogram; Halt maps to OpClass::Any (index 20,
|
||||||
|
// outside the [0..20) histogram), so 3 counted ops (Const, Const, Mul).
|
||||||
|
assert_eq!(t.total(), 3);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn trace_classifies_io_section() {
|
||||||
|
let c = cell(
|
||||||
|
Czyx::new(1, 1, 1, 2),
|
||||||
|
vec![Op::Const(1), Op::Halt],
|
||||||
|
Behavior(Behavior::IO_HEAVY | Behavior::NETWORK),
|
||||||
|
);
|
||||||
|
let t = BlockTrace::from_cell(&c);
|
||||||
|
assert_eq!(BlockClassifier::classify(&t), BlockKind::IoSection);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn trace_classifies_call_trampoline() {
|
||||||
|
let c = cell(
|
||||||
|
Czyx::new(1, 1, 1, 3),
|
||||||
|
vec![Op::CallLink(0), Op::CallLink(1), Op::Halt],
|
||||||
|
Behavior::default(),
|
||||||
|
);
|
||||||
|
let t = BlockTrace::from_cell(&c);
|
||||||
|
assert_eq!(BlockClassifier::classify(&t), BlockKind::CallTrampoline);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn suggest_emits_rule_and_persists() {
|
||||||
|
let c = cell(
|
||||||
|
Czyx::new(1, 1, 1, 4),
|
||||||
|
vec![Op::Const(3), Op::Const(4), Op::Mul, Op::Halt],
|
||||||
|
Behavior(Behavior::PURE),
|
||||||
|
);
|
||||||
|
let t = BlockTrace::from_cell(&c);
|
||||||
|
let mut ai = CubeAi::new();
|
||||||
|
let coords = ai.ingest_and_suggest(&t);
|
||||||
|
assert_eq!(coords.len(), 1, "computation block suggests one rule");
|
||||||
|
assert_eq!(coords[0].c, rule_band());
|
||||||
|
// The stored rule is discoverable by cubedbt.
|
||||||
|
let rt = cubedbt::DbRuntime::new(ai.store().clone());
|
||||||
|
assert_eq!(rt.discover_rules().len(), 1);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn ingest_trace_stores_under_c230() {
|
||||||
|
let c = cell(Czyx::new(1, 1, 1, 5), vec![Op::Halt], Behavior::default());
|
||||||
|
let t = BlockTrace::from_cell(&c);
|
||||||
|
let mut ai = CubeAi::new();
|
||||||
|
let coord = ai.ingest_trace(&t);
|
||||||
|
assert_eq!(coord.c, C_TRACE);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Helper: the rule band constant lives in cubedbt; assert equality without
|
||||||
|
// importing the const name directly (kept local to the test).
|
||||||
|
fn rule_band() -> u8 {
|
||||||
|
cubedbt::C_DBT_RULE
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,18 @@
|
|||||||
|
[package]
|
||||||
|
name = "cubecli"
|
||||||
|
version = "0.1.0"
|
||||||
|
edition.workspace = true
|
||||||
|
license.workspace = true
|
||||||
|
description = "CUBELinux-2 local system CLI: in-process REPL + script runner over one shared CubeStore (cubefs + cubecode + cubecrypt)."
|
||||||
|
|
||||||
|
[[bin]]
|
||||||
|
name = "cube"
|
||||||
|
path = "src/main.rs"
|
||||||
|
|
||||||
|
[dependencies]
|
||||||
|
cubecoords = { path = "../cubecoords" }
|
||||||
|
cubestore = { path = "../cubestore" }
|
||||||
|
cubefs = { path = "../cubefs" }
|
||||||
|
cubecode = { path = "../cubecode" }
|
||||||
|
cubecrypt = { path = "../cubecrypt" }
|
||||||
|
cubesys = { path = "../cubesys" }
|
||||||
@@ -48,15 +48,49 @@ fn is_command_word(w: &str) -> bool {
|
|||||||
| "seal"
|
| "seal"
|
||||||
| "open"
|
| "open"
|
||||||
| "keyinit"
|
| "keyinit"
|
||||||
|
| "scan-word-flags"
|
||||||
|
| "show-flags"
|
||||||
|
| "trace-capture"
|
||||||
|
| "trace-replay"
|
||||||
|
| "golden-capture"
|
||||||
|
| "trace-verify"
|
||||||
|
| "trace-list"
|
||||||
|
| "trace-links"
|
||||||
| "query"
|
| "query"
|
||||||
| "begin"
|
| "begin"
|
||||||
| "commit"
|
| "commit"
|
||||||
| "rollback"
|
| "rollback"
|
||||||
| "stats"
|
| "stats"
|
||||||
| "audit"
|
| "audit"
|
||||||
|
| "note"
|
||||||
|
| "notes"
|
||||||
|
| "project"
|
||||||
|
| "projects"
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Open (or create) the durable session-note store. Notes persist across CLI
|
||||||
|
/// invocations and sessions — the CUBE "message save path" — via the WAL +
|
||||||
|
/// checkpoint store. The directory is `$CUBE_NOTES_DIR` (`~/.cubelinux-notes`).
|
||||||
|
fn open_notes_store() -> cubesys::store::ConcurrentStore {
|
||||||
|
let dir = std::env::var("CUBE_NOTES_DIR").unwrap_or_else(|_| {
|
||||||
|
format!(
|
||||||
|
"{}/.cubelinux-notes",
|
||||||
|
std::env::var("HOME").unwrap_or_else(|_| "/tmp".to_string())
|
||||||
|
)
|
||||||
|
});
|
||||||
|
cubesys::store::ConcurrentStore::open(
|
||||||
|
&format!("{dir}/notes.store"),
|
||||||
|
&format!("{dir}/notes.wal"),
|
||||||
|
&format!("{dir}/recovery.log"),
|
||||||
|
cubesys::store::DurabilityConfig::default(),
|
||||||
|
)
|
||||||
|
.unwrap_or_else(|e| {
|
||||||
|
eprintln!("error: cannot open notes store {dir}: {e}");
|
||||||
|
std::process::exit(1);
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
fn main() {
|
fn main() {
|
||||||
let args: Vec<String> = std::env::args().collect();
|
let args: Vec<String> = std::env::args().collect();
|
||||||
match args.get(1).map(|s| s.as_str()) {
|
match args.get(1).map(|s| s.as_str()) {
|
||||||
@@ -67,12 +101,33 @@ fn main() {
|
|||||||
// the coordinate-addressed API a real CLI surface the OS can call.
|
// the coordinate-addressed API a real CLI surface the OS can call.
|
||||||
Some(word) if is_command_word(word) => {
|
Some(word) if is_command_word(word) => {
|
||||||
let line = args[1..].join(" ");
|
let line = args[1..].join(" ");
|
||||||
let mut session = Session::new();
|
// Session notes live in their own DURABLE store so a note written
|
||||||
match session.exec(&line) {
|
// in one session is reviewable in the next (the message save path).
|
||||||
Ok(out) => println!("{out}"),
|
if word == "note" || word == "notes" || word == "project" || word == "projects" {
|
||||||
Err(e) => {
|
let store = open_notes_store();
|
||||||
eprintln!("error: {e}");
|
let res = if word == "project" || word == "projects" {
|
||||||
std::process::exit(1);
|
cubesys::notes::project_command(&store, &line)
|
||||||
|
} else {
|
||||||
|
cubesys::notes::note_command(&store, &line)
|
||||||
|
};
|
||||||
|
// Force a durable flush/checkpoint before exiting so the note is
|
||||||
|
// persisted even though this process (the CLI) is short-lived.
|
||||||
|
store.checkpoint();
|
||||||
|
match res {
|
||||||
|
Ok(out) => println!("{out}"),
|
||||||
|
Err(e) => {
|
||||||
|
eprintln!("error: {e}");
|
||||||
|
std::process::exit(1);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
let mut session = Session::new();
|
||||||
|
match session.exec(&line) {
|
||||||
|
Ok(out) => println!("{out}"),
|
||||||
|
Err(e) => {
|
||||||
|
eprintln!("error: {e}");
|
||||||
|
std::process::exit(1);
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -136,6 +191,11 @@ fn print_help() {
|
|||||||
stat <path> getattr via cubefs\n \
|
stat <path> getattr via cubefs\n \
|
||||||
seal <path> <K.Z.Y.X|auto> <tf> encrypt a record (tf: none|gcm|chacha|xts)\n \
|
seal <path> <K.Z.Y.X|auto> <tf> encrypt a record (tf: none|gcm|chacha|xts)\n \
|
||||||
open <path> <K.Z.Y.X|auto> <tf> decrypt + decode + run a sealed record\n \
|
open <path> <K.Z.Y.X|auto> <tf> decrypt + decode + run a sealed record\n \
|
||||||
keyinit ensure the OS Null-space keystore exists\n"
|
keyinit ensure the OS Null-space keystore exists\n \
|
||||||
|
note <text> add a session note (durable, WordFlags-tagged)\n \
|
||||||
|
note list [session] list notes (default: today's session)\n \
|
||||||
|
note search <text> search notes across sessions\n \
|
||||||
|
note show <C.Z.Y.X> show one note\n \
|
||||||
|
notes alias for `note list`\n"
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
@@ -0,0 +1,215 @@
|
|||||||
|
//! Feature-vector ball / k-NN index: a metric VP-tree stored as CUBE records.
|
||||||
|
//! Each node is a feature-vector record (`doc_type = "vp-node"`) whose body holds
|
||||||
|
//! the pivot `(x,y,z)` + radial radius `tau`, and whose `linked_records` are its
|
||||||
|
//! two children. Ball recall walks the tree, pruning branches that cannot
|
||||||
|
//! intersect the query ball — exact, bounded node visits (whitepaper §4.1).
|
||||||
|
use cubecoords::{CubeHeader, Czyx};
|
||||||
|
use cubestore::{CubeBackend, CubeStore};
|
||||||
|
|
||||||
|
/// A 3-D feature vector (the `(fx,fy,fz)` MEM model).
|
||||||
|
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
|
||||||
|
pub struct Vec3 {
|
||||||
|
pub x: u64,
|
||||||
|
pub y: u64,
|
||||||
|
pub z: u64,
|
||||||
|
}
|
||||||
|
impl Vec3 {
|
||||||
|
pub fn new(x: u64, y: u64, z: u64) -> Self {
|
||||||
|
Vec3 { x, y, z }
|
||||||
|
}
|
||||||
|
pub fn d2(&self, o: &Vec3) -> u64 {
|
||||||
|
let dx = (self.x as i64 - o.x as i64).abs() as u64;
|
||||||
|
let dy = (self.y as i64 - o.y as i64).abs() as u64;
|
||||||
|
let dz = (self.z as i64 - o.z as i64).abs() as u64;
|
||||||
|
dx * dx + dy * dy + dz * dz
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
const VP_NODE: &str = "vp-node";
|
||||||
|
/// Namespace for index nodes (avoid collisions with user data spaces).
|
||||||
|
pub const VP_SPACE: u8 = 200;
|
||||||
|
const NULL_NODE: Czyx = Czyx::new(VP_SPACE, 0, 0, 0);
|
||||||
|
|
||||||
|
/// Build a VP-tree over `points`, writing each node as a record. Returns the root coord.
|
||||||
|
pub fn build<B: CubeBackend>(store: &mut CubeStore<B>, points: &[Vec3]) -> Czyx {
|
||||||
|
let mut counter = 0u64;
|
||||||
|
let idx: Vec<usize> = (0..points.len()).collect();
|
||||||
|
build_rec(store, points, &idx, &mut counter)
|
||||||
|
}
|
||||||
|
|
||||||
|
fn build_rec<B: CubeBackend>(
|
||||||
|
store: &mut CubeStore<B>,
|
||||||
|
points: &[Vec3],
|
||||||
|
box_idxs: &[usize],
|
||||||
|
counter: &mut u64,
|
||||||
|
) -> Czyx {
|
||||||
|
if box_idxs.is_empty() {
|
||||||
|
return NULL_NODE;
|
||||||
|
}
|
||||||
|
let pi = box_idxs[0];
|
||||||
|
let pivot = points[pi];
|
||||||
|
let mut others: Vec<(u64, usize)> = box_idxs
|
||||||
|
.iter()
|
||||||
|
.copied()
|
||||||
|
.filter(|&i| i != pi)
|
||||||
|
.map(|i| (pivot.d2(&points[i]), i))
|
||||||
|
.collect();
|
||||||
|
others.sort_by_key(|&(d, _)| d);
|
||||||
|
// Vantage radius = sqrt of the median squared distance to the other points.
|
||||||
|
let tau = if others.is_empty() {
|
||||||
|
0
|
||||||
|
} else {
|
||||||
|
(others[others.len() / 2].0 as f64).sqrt() as u64
|
||||||
|
};
|
||||||
|
let mid = others.len() / 2;
|
||||||
|
let left_idx: Vec<usize> = others.iter().take(mid).map(|&(_, i)| i).collect();
|
||||||
|
let right_idx: Vec<usize> = others.iter().skip(mid).map(|&(_, i)| i).collect();
|
||||||
|
|
||||||
|
*counter += 1;
|
||||||
|
let coord = czyx_node(*counter);
|
||||||
|
let left = build_rec(store, points, &left_idx, counter);
|
||||||
|
let right = build_rec(store, points, &right_idx, counter);
|
||||||
|
let mut links = Vec::new();
|
||||||
|
if left != NULL_NODE {
|
||||||
|
links.push(left);
|
||||||
|
}
|
||||||
|
if right != NULL_NODE {
|
||||||
|
links.push(right);
|
||||||
|
}
|
||||||
|
let mut h = CubeHeader::new();
|
||||||
|
h.doc_type = Some(VP_NODE.into());
|
||||||
|
h.title = Some("vp".into());
|
||||||
|
h.linked_records = links;
|
||||||
|
h.refresh_flags();
|
||||||
|
let mut body = Vec::with_capacity(32);
|
||||||
|
for v in [pivot.x, pivot.y, pivot.z, tau] {
|
||||||
|
body.extend_from_slice(&v.to_le_bytes());
|
||||||
|
}
|
||||||
|
store.put_record(coord, &h, &body);
|
||||||
|
coord
|
||||||
|
}
|
||||||
|
|
||||||
|
fn czyx_node(n: u64) -> Czyx {
|
||||||
|
// Encode the node id across the low three axes so we can host far more than
|
||||||
|
// 255 nodes (u8 per axis would collide). c = index namespace.
|
||||||
|
Czyx::new(VP_SPACE, ((n >> 16) & 0xFF) as u8, ((n >> 8) & 0xFF) as u8, (n & 0xFF) as u8)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Ball recall: walk the VP-tree from `root`, pruning branches that cannot
|
||||||
|
/// intersect the query ball. Returns the coordinates of every vector within `r`
|
||||||
|
/// and the number of nodes visited (each visit = one record read).
|
||||||
|
pub fn ball_stats<B: CubeBackend>(
|
||||||
|
store: &CubeStore<B>,
|
||||||
|
root: Czyx,
|
||||||
|
q: Vec3,
|
||||||
|
r: u64,
|
||||||
|
) -> (Vec<Czyx>, usize) {
|
||||||
|
let r2 = r * r;
|
||||||
|
let mut out = Vec::new();
|
||||||
|
let mut visits = 0usize;
|
||||||
|
ball_rec(store, root, &q, r2, r, &mut out, &mut visits);
|
||||||
|
(out, visits)
|
||||||
|
}
|
||||||
|
|
||||||
|
fn ball_rec<B: CubeBackend>(
|
||||||
|
store: &CubeStore<B>,
|
||||||
|
node: Czyx,
|
||||||
|
q: &Vec3,
|
||||||
|
r2: u64,
|
||||||
|
r: u64,
|
||||||
|
out: &mut Vec<Czyx>,
|
||||||
|
visits: &mut usize,
|
||||||
|
) {
|
||||||
|
if node == NULL_NODE {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
*visits += 1;
|
||||||
|
let (h, body) = match store.get_record(&node) {
|
||||||
|
Some(x) => x,
|
||||||
|
None => return,
|
||||||
|
};
|
||||||
|
if body.len() < 32 {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
let pivot = Vec3::new(
|
||||||
|
u64::from_le_bytes(body[0..8].try_into().unwrap()),
|
||||||
|
u64::from_le_bytes(body[8..16].try_into().unwrap()),
|
||||||
|
u64::from_le_bytes(body[16..24].try_into().unwrap()),
|
||||||
|
);
|
||||||
|
let tau = u64::from_le_bytes(body[24..32].try_into().unwrap());
|
||||||
|
let d2v = pivot.d2(q);
|
||||||
|
if d2v <= r2 {
|
||||||
|
out.push(node);
|
||||||
|
}
|
||||||
|
let links = h.linked_records;
|
||||||
|
let d = (d2v as f64).sqrt();
|
||||||
|
let rf = r as f64;
|
||||||
|
// Descend left (points within tau of the pivot) if the query ball may reach them.
|
||||||
|
if links.len() >= 1 && (d - rf) <= tau as f64 {
|
||||||
|
ball_rec(store, links[0], q, r2, r, out, visits);
|
||||||
|
}
|
||||||
|
// Descend right (points beyond tau) if the query ball may extend past the pivot sphere.
|
||||||
|
if links.len() >= 2 && (d + rf) > tau as f64 {
|
||||||
|
ball_rec(store, links[1], q, r2, r, out, visits);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
use cubestore::HashBackend;
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn vp_tree_exact_ball_recall() {
|
||||||
|
let n = 2000u64;
|
||||||
|
let mut points = Vec::with_capacity(n as usize);
|
||||||
|
for c in 0..16u64 {
|
||||||
|
let (bx, by, bz) = (c * 40, (c * 7) % 16 * 7, (c * 13) % 16 * 5);
|
||||||
|
for j in 0..(n / 16) {
|
||||||
|
let o = j as u64 % 12;
|
||||||
|
points.push(Vec3::new(bx + o, by + (o * 3) % 12, bz + (o * 7) % 12));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
let mut store = CubeStore::new(HashBackend::new());
|
||||||
|
let root = build(&mut store, &points);
|
||||||
|
|
||||||
|
let r = 30u64;
|
||||||
|
let r2 = r * r;
|
||||||
|
let mut total_rec = 0usize;
|
||||||
|
let mut total_true = 0usize;
|
||||||
|
let mut visits_sum = 0usize;
|
||||||
|
for anchor in 0..40usize {
|
||||||
|
let q = points[anchor];
|
||||||
|
let (found, visits) = ball_stats(&store, root, q, r);
|
||||||
|
// ground truth = vectors within r of q
|
||||||
|
let truth: Vec<Vec3> = points.iter().copied().filter(|p| q.d2(p) <= r2).collect();
|
||||||
|
// decode pivots from the found node records
|
||||||
|
let mut found_pivots = Vec::new();
|
||||||
|
for c in &found {
|
||||||
|
if let Some((_, b)) = store.get_record(c) {
|
||||||
|
if b.len() >= 24 {
|
||||||
|
found_pivots.push(Vec3::new(
|
||||||
|
u64::from_le_bytes(b[0..8].try_into().unwrap()),
|
||||||
|
u64::from_le_bytes(b[8..16].try_into().unwrap()),
|
||||||
|
u64::from_le_bytes(b[16..24].try_into().unwrap()),
|
||||||
|
));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
let rec = truth.iter().filter(|v| found_pivots.contains(v)).count();
|
||||||
|
total_rec += rec;
|
||||||
|
total_true += truth.len();
|
||||||
|
visits_sum += visits;
|
||||||
|
}
|
||||||
|
eprintln!(
|
||||||
|
"exact recall {:.1}% ({} of {}), avg {} node-visits per query (N={})",
|
||||||
|
100.0 * total_rec as f64 / total_true as f64,
|
||||||
|
total_rec,
|
||||||
|
total_true,
|
||||||
|
visits_sum as f64 / 40.0,
|
||||||
|
n
|
||||||
|
);
|
||||||
|
assert!(total_true > 0 && total_rec == total_true, "VP-tree must be exact (got {}/{}), avg {} visits", total_rec, total_true, visits_sum as f64 / 40.0);
|
||||||
|
assert!(visits_sum as f64 / 40.0 < (n as f64) / 3.0, "VP-tree must prune (avg {} visits)", visits_sum as f64 / 40.0);
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,169 @@
|
|||||||
|
//! CUBELinux-2 cubebook / kernel-convention constants.
|
||||||
|
//!
|
||||||
|
//! These are the conventions the *OS* layer uses to store its computational
|
||||||
|
//! behavior in CUBE. Per the PDF (Package 4, and the "behavior descriptors"
|
||||||
|
//! discussion at PDF §524–525), functions/operators that live in CUBE are:
|
||||||
|
//! * addressed in a reserved `C` axis band (here `c210..=c219`),
|
||||||
|
//! * tagged with a `kind` (`fn`/`kernel`/`layer`/`checkpoint`/`variant`), and
|
||||||
|
//! * annotated with *behavior descriptor* flags carried in the record
|
||||||
|
//! header's out-of-band flag bits (tag 12, see cubestore encode/decode).
|
||||||
|
//!
|
||||||
|
//! The full spec-derived descriptor set (PDF §524–525) is: "pure function,"
|
||||||
|
//! "I/O heavy," "allocates memory," "touches network," "hot path," and
|
||||||
|
//! "security sensitive." We store each directly as a dedicated header-flag bit
|
||||||
|
//! inside the spare 8..=15 range, deliberately leaving bit 12
|
||||||
|
//! (`cubecrypt::HEADER_FLAG_ENCRYPTED`) clear. `CubeHeader::refresh_flags()`
|
||||||
|
//! preserves spare bits 8..=15, so a header carrying descriptors survives an
|
||||||
|
//! encode→decode→refresh round-trip.
|
||||||
|
//!
|
||||||
|
//! This is deliberately NOT a store-IO model: the cubevm never reads or writes
|
||||||
|
//! OS state records directly. The OS's *decisions/computation* live as CUBE
|
||||||
|
//! kernels; the native OS layer is a thin effector that reads a kernel's
|
||||||
|
//! computed result and applies the effect (writes the record, runs the
|
||||||
|
//! command). See `cubesys/src/commands.rs` `tick` + `native-apply`.
|
||||||
|
|
||||||
|
/// First `C` axis value reserved for OS operator kernels (call-graph roots and
|
||||||
|
/// leaves). The OS keeps its behavioral substrate here, separate from the
|
||||||
|
/// `c200` operational-snapshot data band and the `c001/c002` doc examples.
|
||||||
|
pub const C_OS_KERNEL: u8 = 210;
|
||||||
|
|
||||||
|
/// `C` axis band for OS *data* records the native layer writes after a kernel
|
||||||
|
/// decides an effect (kept distinct from the compute-kernel band above).
|
||||||
|
pub const C_OS_EFFECT: u8 = 211;
|
||||||
|
|
||||||
|
/// Header-flag bits reserved for the behavior-descriptor field. Six spec
|
||||||
|
/// descriptors, each a dedicated bit in the spare 8..=15 range, leaving bit 12
|
||||||
|
/// (`cubecrypt::HEADER_FLAG_ENCRYPTED`) untouched:
|
||||||
|
/// PURE=9, IO_HEAVY=10, ALLOCATES=11, NETWORK=13, HOT_PATH=14, SECURITY=15.
|
||||||
|
pub const HEADER_FLAG_BEHAVIOR: u16 = 0b1110_1110_0000_0000; // bits 9,10,11,13,14,15 (bit12 reserved)
|
||||||
|
|
||||||
|
/// Behavior descriptors for an OS operator kernel (PDF §524–525, full set).
|
||||||
|
///
|
||||||
|
/// Each variant is stored directly as its dedicated header-flag bit (subset of
|
||||||
|
/// `HEADER_FLAG_BEHAVIOR`), so `Behavior` carries the raw descriptor bits and
|
||||||
|
/// round-trips through the header codec without bit-shift collisions.
|
||||||
|
#[derive(Copy, Clone, Debug, Default, PartialEq, Eq)]
|
||||||
|
pub struct Behavior(pub u16);
|
||||||
|
|
||||||
|
impl Behavior {
|
||||||
|
/// Pure function: no side effects, deterministic on inputs.
|
||||||
|
pub const PURE: u16 = 1 << 9; // 0x0200
|
||||||
|
/// I/O heavy: performs significant store/device I/O.
|
||||||
|
pub const IO_HEAVY: u16 = 1 << 10; // 0x0400
|
||||||
|
/// Allocates memory: grows the heap / maps pages.
|
||||||
|
pub const ALLOCATES: u16 = 1 << 11; // 0x0800
|
||||||
|
/// Touches network: performs socket/link I/O (security-relevant surface).
|
||||||
|
pub const NETWORK: u16 = 1 << 13; // 0x2000
|
||||||
|
/// Hot path: executed frequently; a candidate for optimization/variant swap.
|
||||||
|
pub const HOT_PATH: u16 = 1 << 14; // 0x4000
|
||||||
|
/// Security sensitive: elevated privilege / trust boundary crossing.
|
||||||
|
pub const SECURITY_SENSITIVE: u16 = 1 << 15; // 0x8000
|
||||||
|
|
||||||
|
/// Combine descriptor bits (e.g. `Behavior::PURE | Behavior::HOT_PATH`).
|
||||||
|
pub fn new(bits: u16) -> Self {
|
||||||
|
Behavior(bits & HEADER_FLAG_BEHAVIOR)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Encode into the out-of-band header flag bits.
|
||||||
|
pub fn to_flags(self) -> u16 {
|
||||||
|
self.0 & HEADER_FLAG_BEHAVIOR
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Decode from a raw 16-bit flag word (keeps only the behavior bits).
|
||||||
|
pub fn from_flags(flags: u16) -> Self {
|
||||||
|
Behavior(flags & HEADER_FLAG_BEHAVIOR)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// True if any descriptor bit is set.
|
||||||
|
pub fn is_any(self) -> bool {
|
||||||
|
self.0 != 0
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Human-readable descriptor tags (used by `ls`/`stat` and the effector).
|
||||||
|
pub fn tags(self) -> Vec<&'static str> {
|
||||||
|
let mut out = Vec::new();
|
||||||
|
if self.0 & Self::PURE != 0 {
|
||||||
|
out.push("pure");
|
||||||
|
}
|
||||||
|
if self.0 & Self::IO_HEAVY != 0 {
|
||||||
|
out.push("io");
|
||||||
|
}
|
||||||
|
if self.0 & Self::ALLOCATES != 0 {
|
||||||
|
out.push("alloc");
|
||||||
|
}
|
||||||
|
if self.0 & Self::NETWORK != 0 {
|
||||||
|
out.push("net");
|
||||||
|
}
|
||||||
|
if self.0 & Self::HOT_PATH != 0 {
|
||||||
|
out.push("hot");
|
||||||
|
}
|
||||||
|
if self.0 & Self::SECURITY_SENSITIVE != 0 {
|
||||||
|
out.push("sec");
|
||||||
|
}
|
||||||
|
out
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
use cubecoords::{CubeHeader, Czyx};
|
||||||
|
use cubestore::{CubeStore, HashBackend};
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn behavior_round_trips_through_store() {
|
||||||
|
// All six spec descriptors must survive the real store round-trip
|
||||||
|
// (what native-apply reads back). This is the exact field the
|
||||||
|
// effector inspects; an earlier 3-bit mask (0x7000) silently dropped
|
||||||
|
// HOT_PATH (bit 15) and collided with ENCRYPTED (bit 12).
|
||||||
|
let b = Behavior::new(
|
||||||
|
Behavior::PURE | Behavior::IO_HEAVY | Behavior::ALLOCATES | Behavior::NETWORK,
|
||||||
|
);
|
||||||
|
let raw = b.to_flags();
|
||||||
|
assert_eq!(
|
||||||
|
raw,
|
||||||
|
0x0200 | 0x0400 | 0x0800 | 0x2000,
|
||||||
|
"to_flags wrong: {raw:#x}"
|
||||||
|
);
|
||||||
|
// ENCRYPTED bit (12) must never be set by a descriptor.
|
||||||
|
assert_eq!(raw & (1 << 12), 0, "descriptor must not set ENCRYPTED bit");
|
||||||
|
|
||||||
|
let mut store = CubeStore::new(HashBackend::new());
|
||||||
|
let coord = Czyx::new(210, 1, 1, 3);
|
||||||
|
let mut h = CubeHeader::new();
|
||||||
|
h.doc_type = Some("kernel".into());
|
||||||
|
h.title = Some("summarize-procs".into());
|
||||||
|
h.flags.0 |= raw;
|
||||||
|
h.refresh_flags();
|
||||||
|
store.put_record(coord, &h, &[1, 2, 3]);
|
||||||
|
|
||||||
|
let (read_h, _) = store.get_record(&coord).expect("record present");
|
||||||
|
let back = Behavior::from_flags(read_h.flags.bits());
|
||||||
|
assert_eq!(
|
||||||
|
back,
|
||||||
|
b,
|
||||||
|
"behavior dropped on store round-trip: stored {raw:#x}, got {:#x}",
|
||||||
|
read_h.flags.bits()
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn security_sensitive_bit_used_and_round_trips() {
|
||||||
|
// Explicitly cover the 6th spec descriptor (security sensitive, bit15).
|
||||||
|
let b = Behavior::new(Behavior::SECURITY_SENSITIVE | Behavior::HOT_PATH);
|
||||||
|
let mut store = CubeStore::new(HashBackend::new());
|
||||||
|
let coord = Czyx::new(210, 1, 1, 5);
|
||||||
|
let mut h = CubeHeader::new();
|
||||||
|
h.doc_type = Some("kernel".into());
|
||||||
|
h.title = Some("privileged-op".into());
|
||||||
|
h.flags.0 |= b.to_flags();
|
||||||
|
h.refresh_flags();
|
||||||
|
store.put_record(coord, &h, &[]);
|
||||||
|
let (read_h, _) = store.get_record(&coord).expect("record present");
|
||||||
|
assert_eq!(
|
||||||
|
Behavior::from_flags(read_h.flags.bits()),
|
||||||
|
b,
|
||||||
|
"security-sensitive descriptor lost on round-trip"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -71,6 +71,7 @@ impl Kind {
|
|||||||
/// in the body. This keeps Package 4 fully compatible with the Package 2
|
/// in the body. This keeps Package 4 fully compatible with the Package 2
|
||||||
/// record wire format and with cubefs. `code_kind`/`set_kind` bridge the
|
/// record wire format and with cubefs. `code_kind`/`set_kind` bridge the
|
||||||
/// enum to the string.
|
/// enum to the string.
|
||||||
|
#[derive(Clone, Debug)]
|
||||||
pub struct CodeCell {
|
pub struct CodeCell {
|
||||||
/// The cube coordinate this cell is stored at.
|
/// The cube coordinate this cell is stored at.
|
||||||
pub label: Czyx,
|
pub label: Czyx,
|
||||||
|
|||||||
+8
-1
@@ -35,10 +35,17 @@
|
|||||||
#![forbid(unsafe_code)]
|
#![forbid(unsafe_code)]
|
||||||
#![warn(missing_docs)]
|
#![warn(missing_docs)]
|
||||||
|
|
||||||
|
pub mod cb;
|
||||||
pub mod cell;
|
pub mod cell;
|
||||||
|
pub mod ballindex;
|
||||||
pub mod opcode;
|
pub mod opcode;
|
||||||
|
pub mod trace;
|
||||||
pub mod vm;
|
pub mod vm;
|
||||||
|
|
||||||
|
pub use cb::{Behavior, C_OS_EFFECT, C_OS_KERNEL, HEADER_FLAG_BEHAVIOR};
|
||||||
|
|
||||||
pub use cell::{CodeCell, Kind};
|
pub use cell::{CodeCell, Kind};
|
||||||
pub use opcode::{decode, encode, CodeError, Op};
|
pub use opcode::{decode, encode, CodeError, Op};
|
||||||
pub use vm::{Fault, RunResult, Vm, SYS_DEGREE, SYS_LINKED_EXISTS, SYS_NOP, SYS_TRACE};
|
pub use ballindex::{ball_stats, build, Vec3, VP_SPACE};
|
||||||
|
pub use trace::{capture_golden, lineage, replay_trace, store_trace, verify_golden, LineageHop};
|
||||||
|
pub use vm::{replay, Fault, RunResult, Vm, SYS_DEGREE, SYS_LINKED_EXISTS, SYS_NOP, SYS_TRACE};
|
||||||
|
|||||||
@@ -0,0 +1,263 @@
|
|||||||
|
//! Execution-trace artifacts: store a captured machine-code trace as a
|
||||||
|
//! first-class AI-artifact record (`Kind::Layer`) linked back to its source,
|
||||||
|
//! and replay it to reproduce the run — so an agent can persist & reproduce
|
||||||
|
//! interesting executions as ordinary CZYX records.
|
||||||
|
//!
|
||||||
|
//! A trace is the exact `Vec<Op>` a program actually executed ([`Vm::run_captured`]).
|
||||||
|
//! We persist it as a record body (via [`crate::opcode::encode`]) with
|
||||||
|
//! `doc_type = "layer"` (the AI-artifact kind) and a `linked_records` edge to
|
||||||
|
//! the source cell. Loading it back (via [`CodeCell::from_record`]) decodes the
|
||||||
|
//! ops and [`replay`] reproduces the observable behaviour with no access to the
|
||||||
|
//! source program.
|
||||||
|
|
||||||
|
use crate::{opcode, replay, Kind, Op, RunResult, Vm};
|
||||||
|
use cubecoords::{CubeHeader, Czyx};
|
||||||
|
use cubestore::{CubeBackend, CubeStore};
|
||||||
|
|
||||||
|
/// Store a captured execution trace as an AI-artifact record at `coord`,
|
||||||
|
/// linked to its `source` cell. `name` becomes the record title; the body is
|
||||||
|
/// the encoded trace; `doc_type` is `Kind::Layer`.
|
||||||
|
pub fn store_trace<B: CubeBackend>(
|
||||||
|
store: &mut CubeStore<B>,
|
||||||
|
coord: Czyx,
|
||||||
|
source: Czyx,
|
||||||
|
trace: &[Op],
|
||||||
|
name: &str,
|
||||||
|
) {
|
||||||
|
let mut h = CubeHeader::new();
|
||||||
|
h.title = Some(name.to_string());
|
||||||
|
h.doc_type = Some(Kind::Layer.as_str().to_string());
|
||||||
|
h.linked_records = vec![source];
|
||||||
|
h.refresh_flags();
|
||||||
|
store.put_record(coord, &h, &opcode::encode(trace));
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Load a stored trace record at `coord` and replay it, reproducing the run's
|
||||||
|
/// observable behaviour (data-stack result + `SYS_TRACE` output) from the
|
||||||
|
/// stored record — no access to the original program. Returns `None` if the
|
||||||
|
/// record is missing or its body is not a valid trace.
|
||||||
|
pub fn replay_trace<B: CubeBackend>(
|
||||||
|
store: &CubeStore<B>,
|
||||||
|
coord: Czyx,
|
||||||
|
) -> Option<(RunResult, Vec<u8>)> {
|
||||||
|
let (_h, body) = store.get_record(&coord)?;
|
||||||
|
let ops = opcode::decode(&body).ok()?;
|
||||||
|
Some(replay(&ops))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Capture a run and persist it as a "golden": the executed trace stored as a
|
||||||
|
/// `Kind::Layer` record (linked to `source`) and the observed output stored as a
|
||||||
|
/// companion `golden` record (linked to the trace). Returns the observed output.
|
||||||
|
pub fn capture_golden<B: CubeBackend + Clone>(
|
||||||
|
store: &mut CubeStore<B>,
|
||||||
|
source: Czyx,
|
||||||
|
trace_coord: Czyx,
|
||||||
|
golden_coord: Czyx,
|
||||||
|
name: &str,
|
||||||
|
) -> Vec<u8> {
|
||||||
|
let mut vm = Vm::new((*store).clone());
|
||||||
|
let (_r, trace) = vm.run_captured(source);
|
||||||
|
let out = vm.output().to_vec();
|
||||||
|
store_trace(store, trace_coord, source, &trace, name);
|
||||||
|
let mut h = CubeHeader::new();
|
||||||
|
h.title = Some(format!("golden:{name}"));
|
||||||
|
h.doc_type = Some("golden".into());
|
||||||
|
h.linked_records = vec![trace_coord];
|
||||||
|
h.refresh_flags();
|
||||||
|
store.put_record(golden_coord, &h, &out);
|
||||||
|
out
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Golden-trace regression check: re-run `source` fresh AND replay the stored
|
||||||
|
/// trace, and assert BOTH reproduce the recorded golden output.
|
||||||
|
/// * replay vs golden -> the stored trace is a faithful reproduction;
|
||||||
|
/// * fresh run vs golden -> the program has not drifted (or it would change
|
||||||
|
/// the observed output).
|
||||||
|
/// Returns `Ok(())` if the behavior is unchanged, else `Err(reason)`.
|
||||||
|
pub fn verify_golden<B: CubeBackend + Clone>(
|
||||||
|
store: &CubeStore<B>,
|
||||||
|
source: Czyx,
|
||||||
|
trace_coord: Czyx,
|
||||||
|
golden_coord: Czyx,
|
||||||
|
) -> Result<(), String> {
|
||||||
|
let golden = store
|
||||||
|
.get_record(&golden_coord)
|
||||||
|
.map(|(_, b)| b)
|
||||||
|
.ok_or("golden record missing")?;
|
||||||
|
let (_, replay_out) = replay_trace(store, trace_coord).ok_or("trace record missing/invalid")?;
|
||||||
|
if replay_out != golden {
|
||||||
|
return Err(format!(
|
||||||
|
"replay output differs from golden ({} vs {} bytes)",
|
||||||
|
replay_out.len(),
|
||||||
|
golden.len()
|
||||||
|
));
|
||||||
|
}
|
||||||
|
let mut vm = Vm::new((*store).clone());
|
||||||
|
let _ = vm.run(source);
|
||||||
|
let fresh = vm.output().to_vec();
|
||||||
|
if fresh != golden {
|
||||||
|
return Err(format!(
|
||||||
|
"fresh run differs from golden (program changed: {} vs {} bytes)",
|
||||||
|
fresh.len(),
|
||||||
|
golden.len()
|
||||||
|
));
|
||||||
|
}
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A hop in the experiment-lineage graph: a record and the records it links to.
|
||||||
|
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||||
|
pub struct LineageHop {
|
||||||
|
/// The record's coordinate.
|
||||||
|
pub coord: Czyx,
|
||||||
|
/// Its `doc_type` (kind: fn/kernel/layer/checkpoint/variant/golden/...).
|
||||||
|
pub doc_type: String,
|
||||||
|
/// Its human title, if any.
|
||||||
|
pub title: String,
|
||||||
|
/// The CZYX records this record links to (its edges).
|
||||||
|
pub links: Vec<Czyx>,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Walk the linked-record graph starting at `start`, following
|
||||||
|
/// `linked_records` edges (breadth-first) up to `depth` levels. Returns the
|
||||||
|
/// ordered list of reachable records — the experiment lineage chain
|
||||||
|
/// (source -> data -> weights -> trace -> golden, etc.). Each record is the
|
||||||
|
/// kind the CUBE associates; provenance is a graph walk, not a lookup.
|
||||||
|
pub fn lineage<B: CubeBackend>(
|
||||||
|
store: &CubeStore<B>,
|
||||||
|
start: Czyx,
|
||||||
|
depth: usize,
|
||||||
|
) -> Vec<LineageHop> {
|
||||||
|
let mut out = Vec::new();
|
||||||
|
let mut frontier = vec![start];
|
||||||
|
let mut seen: std::collections::HashSet<Czyx> = std::collections::HashSet::new();
|
||||||
|
for _ in 0..=depth {
|
||||||
|
if frontier.is_empty() {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
let mut next = Vec::new();
|
||||||
|
for c in frontier {
|
||||||
|
if !seen.insert(c) {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
if let Some((h, _)) = store.get_record(&c) {
|
||||||
|
let links = h.linked_records.clone();
|
||||||
|
out.push(LineageHop {
|
||||||
|
coord: c,
|
||||||
|
doc_type: h.doc_type.clone().unwrap_or_default(),
|
||||||
|
title: h.title.clone().unwrap_or_default(),
|
||||||
|
links: links.clone(),
|
||||||
|
});
|
||||||
|
next.extend(links);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
frontier = next;
|
||||||
|
}
|
||||||
|
out
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
use crate::{SYS_TRACE, Vm};
|
||||||
|
use cubestore::HashBackend;
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn trace_stored_as_layer_and_replayed_without_source() {
|
||||||
|
let mut store = CubeStore::new(HashBackend::new());
|
||||||
|
let entry = Czyx::new(1, 1, 1, 7);
|
||||||
|
let ops = [
|
||||||
|
Op::Const(0), Op::Const(1), Op::Syscall(SYS_TRACE),
|
||||||
|
Op::Const(1), Op::Const(1), Op::Syscall(SYS_TRACE),
|
||||||
|
Op::Const(2), Op::Const(2), Op::Syscall(SYS_TRACE),
|
||||||
|
Op::Halt,
|
||||||
|
];
|
||||||
|
let mut h = CubeHeader::new();
|
||||||
|
h.title = Some("solitaire".into());
|
||||||
|
h.doc_type = Some("fn".into());
|
||||||
|
h.refresh_flags();
|
||||||
|
store.put_record(entry, &h, &opcode::encode(&ops));
|
||||||
|
|
||||||
|
// 1) Capture the run as a machine-code trace.
|
||||||
|
let mut vm = Vm::new(store.clone());
|
||||||
|
let (_r, trace) = vm.run_captured(entry);
|
||||||
|
let produced = vm.output().to_vec();
|
||||||
|
assert!(!trace.is_empty());
|
||||||
|
|
||||||
|
// 2) Store it as a Kind::Layer AI-artifact record linked to the source.
|
||||||
|
let trace_coord = Czyx::new(9, 0, 0, 1);
|
||||||
|
store_trace(&mut store, trace_coord, entry, &trace, "solitaire-run-1");
|
||||||
|
|
||||||
|
// 3) The record is a "layer" artifact with an edge back to the source.
|
||||||
|
let (th, _tb) = store.get_record(&trace_coord).unwrap();
|
||||||
|
assert_eq!(th.doc_type.as_deref(), Some("layer"));
|
||||||
|
assert_eq!(th.linked_records, vec![entry]);
|
||||||
|
|
||||||
|
// 4) Reproduce the run purely from the stored trace record.
|
||||||
|
let (result, replayed) = replay_trace(&store, trace_coord).unwrap();
|
||||||
|
assert_eq!(result, RunResult::Halted { top: None });
|
||||||
|
assert_eq!(replayed, produced, "stored trace must reproduce the run");
|
||||||
|
assert!(replayed.contains(&b'\n'));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn golden_regression_detects_drift() {
|
||||||
|
let mut store = CubeStore::new(HashBackend::new());
|
||||||
|
let entry = Czyx::new(1, 1, 1, 9);
|
||||||
|
let mk = |val: u8| {
|
||||||
|
vec![Op::Const(val), Op::Syscall(SYS_TRACE), Op::Halt]
|
||||||
|
};
|
||||||
|
let mut h = CubeHeader::new();
|
||||||
|
h.doc_type = Some("fn".into());
|
||||||
|
h.refresh_flags();
|
||||||
|
store.put_record(entry, &h, &opcode::encode(&mk(7)));
|
||||||
|
|
||||||
|
let trace_c = Czyx::new(9, 0, 0, 1);
|
||||||
|
let golden_c = Czyx::new(9, 0, 0, 2);
|
||||||
|
let out = capture_golden(&mut store, entry, trace_c, golden_c, "regress");
|
||||||
|
assert_eq!(out, b"\x07\n");
|
||||||
|
assert!(verify_golden(&store, entry, trace_c, golden_c).is_ok());
|
||||||
|
|
||||||
|
// Drift the program: change the emitted value -> fresh run differs.
|
||||||
|
store.put_record(entry, &h, &opcode::encode(&mk(9)));
|
||||||
|
let r = verify_golden(&store, entry, trace_c, golden_c);
|
||||||
|
assert!(r.is_err(), "program drift must be detected: {r:?}");
|
||||||
|
// The stored trace is immutable: it still replays the ORIGINAL golden.
|
||||||
|
let (_, rep) = replay_trace(&store, trace_c).unwrap();
|
||||||
|
assert_eq!(rep, b"\x07\n", "stored trace still reproduces the original run");
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn lineage_walks_source_data_weights_trace() {
|
||||||
|
let mut store = CubeStore::new(HashBackend::new());
|
||||||
|
// Build experiment lineage: source -> data -> weights -> trace -> golden.
|
||||||
|
let mut mk = |coord: Czyx, doc: &str, title: &str, links: Vec<Czyx>| {
|
||||||
|
let mut h = CubeHeader::new();
|
||||||
|
h.title = Some(title.into());
|
||||||
|
h.doc_type = Some(doc.into());
|
||||||
|
h.linked_records = links;
|
||||||
|
h.refresh_flags();
|
||||||
|
store.put_record(coord, &h, &[]);
|
||||||
|
};
|
||||||
|
let source = Czyx::new(1, 1, 1, 1);
|
||||||
|
let data = Czyx::new(2, 0, 0, 1);
|
||||||
|
let weights = Czyx::new(3, 0, 0, 1);
|
||||||
|
let trace = Czyx::new(4, 0, 0, 1);
|
||||||
|
let golden = Czyx::new(5, 0, 0, 1);
|
||||||
|
mk(source, "fn", "train", vec![data, weights]);
|
||||||
|
mk(data, "data", "batch", vec![]);
|
||||||
|
mk(weights, "layer", "model-v1", vec![]);
|
||||||
|
mk(trace, "layer", "trace:train", vec![source]);
|
||||||
|
mk(golden, "golden", "golden:train", vec![trace]);
|
||||||
|
|
||||||
|
// Walk from the trace back to its lineage.
|
||||||
|
let hops = lineage(&store, trace, 4);
|
||||||
|
assert!(hops.iter().any(|h| h.coord == trace && h.doc_type == "layer"));
|
||||||
|
assert!(hops.iter().any(|h| h.coord == source && h.doc_type == "fn"));
|
||||||
|
// Depth-limited: from a leaf with no links, only itself.
|
||||||
|
let leaf = lineage(&store, data, 3);
|
||||||
|
assert_eq!(leaf.len(), 1);
|
||||||
|
assert_eq!(leaf[0].coord, data);
|
||||||
|
assert_eq!(leaf[0].links, vec![]);
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -90,6 +90,12 @@ pub struct Vm<B: CubeBackend> {
|
|||||||
output: Vec<u8>,
|
output: Vec<u8>,
|
||||||
/// Maximum nesting of `CALL_LINK` frames.
|
/// Maximum nesting of `CALL_LINK` frames.
|
||||||
max_call_depth: usize,
|
max_call_depth: usize,
|
||||||
|
/// Executed-instruction trace collected when [`Vm::run_captured`] is active:
|
||||||
|
/// the exact ordered sequence of `Op`s that actually ran (the "machine code
|
||||||
|
/// as it runs"). Used to reproduce behaviour without the source.
|
||||||
|
trace: Vec<Op>,
|
||||||
|
/// Whether [`Vm::run_captured`] is currently recording the instruction trace.
|
||||||
|
capturing: bool,
|
||||||
}
|
}
|
||||||
|
|
||||||
impl<B: CubeBackend> Vm<B> {
|
impl<B: CubeBackend> Vm<B> {
|
||||||
@@ -104,6 +110,8 @@ impl<B: CubeBackend> Vm<B> {
|
|||||||
store,
|
store,
|
||||||
output: Vec::new(),
|
output: Vec::new(),
|
||||||
max_call_depth: Self::MAX_CALL_DEPTH,
|
max_call_depth: Self::MAX_CALL_DEPTH,
|
||||||
|
trace: Vec::new(),
|
||||||
|
capturing: false,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -158,6 +166,9 @@ impl<B: CubeBackend> Vm<B> {
|
|||||||
|
|
||||||
while pc < code.len() {
|
while pc < code.len() {
|
||||||
let op = code[pc];
|
let op = code[pc];
|
||||||
|
if self.capturing {
|
||||||
|
self.trace.push(op);
|
||||||
|
}
|
||||||
match op {
|
match op {
|
||||||
Op::Nop => {}
|
Op::Nop => {}
|
||||||
Op::Halt => {
|
Op::Halt => {
|
||||||
@@ -258,6 +269,21 @@ impl<B: CubeBackend> Vm<B> {
|
|||||||
Ok(stack.pop())
|
Ok(stack.pop())
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Run `entry` AND capture the executed-instruction trace (the "machine
|
||||||
|
/// code as it runs"): the exact ordered sequence of `Op`s executed, across
|
||||||
|
/// all cells, regardless of source. Branches are already resolved (the
|
||||||
|
/// trace is the taken path) and call frames are marked by `CallLink`/`Ret`,
|
||||||
|
/// so [`Vm::replay`] can reproduce the same behaviour from the trace alone
|
||||||
|
/// — without the original program's records/code.
|
||||||
|
pub fn run_captured(&mut self, entry: Czyx) -> (RunResult, Vec<Op>) {
|
||||||
|
self.output.clear();
|
||||||
|
self.trace.clear();
|
||||||
|
self.capturing = true;
|
||||||
|
let res = self.run(entry);
|
||||||
|
self.capturing = false;
|
||||||
|
(res, std::mem::take(&mut self.trace))
|
||||||
|
}
|
||||||
|
|
||||||
/// Host syscall dispatch. `stack` is the *shared* data stack (callees
|
/// Host syscall dispatch. `stack` is the *shared* data stack (callees
|
||||||
/// run on it) and `header` is the current cell's metadata.
|
/// run on it) and `header` is the current cell's metadata.
|
||||||
fn syscall(&mut self, id: u8, header: &CubeHeader, stack: &mut Vec<u8>) -> Result<(), Fault> {
|
fn syscall(&mut self, id: u8, header: &CubeHeader, stack: &mut Vec<u8>) -> Result<(), Fault> {
|
||||||
@@ -288,6 +314,87 @@ impl<B: CubeBackend> Vm<B> {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Replay-time host syscall dispatch (no store/header). Store-independent
|
||||||
|
/// syscalls (`SYS_TRACE`, `SYS_NOP`) are reproduced exactly; syscalls that
|
||||||
|
/// would need the live store/header are a no-op in trace-replay (they don't
|
||||||
|
/// contribute to the observable `SYS_TRACE` output).
|
||||||
|
fn replay_syscall(id: u8, stack: &mut Vec<u8>, output: &mut Vec<u8>) {
|
||||||
|
match id {
|
||||||
|
SYS_NOP => {}
|
||||||
|
SYS_TRACE => {
|
||||||
|
let mut line: Vec<u8> = std::mem::take(stack);
|
||||||
|
line.push(b'\n');
|
||||||
|
output.extend_from_slice(&line);
|
||||||
|
}
|
||||||
|
_ => {}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Replay a captured instruction trace, reproducing the program's actions
|
||||||
|
/// (data-stack result + `SYS_TRACE` output) WITHOUT accessing any of the
|
||||||
|
/// original program's records/bytecode. This is the "duplicate behaviour
|
||||||
|
/// from machine code as it ran" path. Store-independent syscalls
|
||||||
|
/// (`SYS_TRACE`, `SYS_NOP`) are reproduced exactly; syscalls that would
|
||||||
|
/// need the live store/header are a no-op (those don't contribute to the
|
||||||
|
/// observable `SYS_TRACE` output).
|
||||||
|
pub fn replay(trace: &[Op]) -> (RunResult, Vec<u8>) {
|
||||||
|
let mut stack: Vec<u8> = Vec::new();
|
||||||
|
let mut frames: Vec<[u8; LOCALS]> = vec![[0; LOCALS]];
|
||||||
|
let mut output: Vec<u8> = Vec::new();
|
||||||
|
let mut top = None;
|
||||||
|
for &op in trace {
|
||||||
|
match op {
|
||||||
|
Op::Nop => {}
|
||||||
|
Op::Halt => {
|
||||||
|
top = stack.pop();
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
Op::Const(v) => stack.push(v),
|
||||||
|
Op::Load(i) => stack.push(frames.last_mut().unwrap()[(i as usize) % LOCALS]),
|
||||||
|
Op::Store(i) => {
|
||||||
|
let v = stack.pop().unwrap_or(0);
|
||||||
|
frames.last_mut().unwrap()[(i as usize) % LOCALS] = v;
|
||||||
|
}
|
||||||
|
Op::Add | Op::Sub | Op::Mul | Op::Div | Op::Mod | Op::And | Op::Or | Op::Xor => {
|
||||||
|
let (b, a) = (stack.pop(), stack.pop());
|
||||||
|
stack.push(bin_op(op, a.unwrap_or(0), b.unwrap_or(0)).unwrap_or(0));
|
||||||
|
}
|
||||||
|
Op::Shl | Op::Shr => {
|
||||||
|
let (b, a) = (stack.pop(), stack.pop());
|
||||||
|
stack.push(shift_op(op, a.unwrap_or(0), b.unwrap_or(0)));
|
||||||
|
}
|
||||||
|
Op::Eq | Op::Ne | Op::Lt | Op::Gt | Op::Le | Op::Ge => {
|
||||||
|
let (b, a) = (stack.pop(), stack.pop());
|
||||||
|
stack.push(cmp_op(op, a.unwrap_or(0), b.unwrap_or(0)));
|
||||||
|
}
|
||||||
|
// Jumps already reflect the taken path in a captured trace:
|
||||||
|
// keep the stack effect, don't re-branch.
|
||||||
|
Op::Jmp(_) => {}
|
||||||
|
Op::Jz(_) | Op::Jnz(_) => {
|
||||||
|
let _ = stack.pop();
|
||||||
|
}
|
||||||
|
Op::CallLink(_) => frames.push([0; LOCALS]),
|
||||||
|
Op::Ret => {
|
||||||
|
if frames.len() > 1 {
|
||||||
|
frames.pop();
|
||||||
|
} else {
|
||||||
|
top = stack.pop();
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
Op::Syscall(id) => replay_syscall(id, &mut stack, &mut output),
|
||||||
|
Op::Dup => {
|
||||||
|
let v = *stack.last().unwrap_or(&0);
|
||||||
|
stack.push(v);
|
||||||
|
}
|
||||||
|
Op::Drop => {
|
||||||
|
stack.pop();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
(RunResult::Halted { top }, output)
|
||||||
|
}
|
||||||
|
|
||||||
fn b2(v: Option<u8>, cell: Czyx, pc: usize) -> Result<u8, (Czyx, usize, Fault)> {
|
fn b2(v: Option<u8>, cell: Czyx, pc: usize) -> Result<u8, (Czyx, usize, Fault)> {
|
||||||
v.ok_or((cell, pc, Fault::PcOutOfRange))
|
v.ok_or((cell, pc, Fault::PcOutOfRange))
|
||||||
}
|
}
|
||||||
@@ -522,4 +629,38 @@ mod tests {
|
|||||||
let mut vm = Vm::new(store);
|
let mut vm = Vm::new(store);
|
||||||
assert_eq!(vm.run(entry), RunResult::Halted { top: Some(3) });
|
assert_eq!(vm.run(entry), RunResult::Halted { top: Some(3) });
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn capture_and_replay_reproduces_behavior_without_source() {
|
||||||
|
let mut store = CubeStore::new(HashBackend::new());
|
||||||
|
let entry = Czyx::new(1, 1, 1, 7);
|
||||||
|
// A tiny "solitaire" state machine: draw cards; even -> foundation
|
||||||
|
// (score++), odd -> waste. Emit (score, waste) after each play.
|
||||||
|
let ops = [
|
||||||
|
Op::Const(0), Op::Const(1), Op::Syscall(SYS_TRACE), // draw odd -> waste
|
||||||
|
Op::Const(1), Op::Const(1), Op::Syscall(SYS_TRACE), // draw even -> score
|
||||||
|
Op::Const(1), Op::Const(2), Op::Syscall(SYS_TRACE), // draw odd -> waste
|
||||||
|
Op::Const(2), Op::Const(2), Op::Syscall(SYS_TRACE), // draw even -> score
|
||||||
|
Op::Halt,
|
||||||
|
];
|
||||||
|
let mut h = CubeHeader::new();
|
||||||
|
h.title = Some("solitaire".into());
|
||||||
|
h.doc_type = Some("fn".into());
|
||||||
|
h.refresh_flags();
|
||||||
|
store.put_record(entry, &h, &crate::opcode::encode(&ops));
|
||||||
|
|
||||||
|
let mut vm = Vm::new(store);
|
||||||
|
// Capture: run the program AND record the exact machine code executed.
|
||||||
|
let (result, trace) = vm.run_captured(entry);
|
||||||
|
assert_eq!(result, RunResult::Halted { top: None });
|
||||||
|
assert!(!trace.is_empty(), "capture must record the executed ops");
|
||||||
|
let produced = vm.output().to_vec();
|
||||||
|
assert_eq!(produced, b"\x00\x01\n\x01\x01\n\x01\x02\n\x02\x02\n");
|
||||||
|
|
||||||
|
// Replay: reproduce the SAME actions purely from the captured trace,
|
||||||
|
// with NO access to the original program's records/bytecode.
|
||||||
|
let (result2, replayed) = replay(&trace);
|
||||||
|
assert_eq!(result2, RunResult::Halted { top: None });
|
||||||
|
assert_eq!(replayed, produced, "replay must reproduce the game's behaviour");
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
+147
-1
@@ -160,6 +160,84 @@ impl TriEnc {
|
|||||||
}
|
}
|
||||||
(control, ascii)
|
(control, ascii)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Pack 6 ASCII bytes plus a full 16-bit per-word flag/metadata field.
|
||||||
|
///
|
||||||
|
/// Layout (high -> low): `[ WordFlags: u16 (bits 48-63) ][ 48 ascii bits ]`.
|
||||||
|
/// This promotes the 12 formerly-unused bits + the 4 control bits into a real
|
||||||
|
/// per-word metadata stripe, mapped by [`WordFlags`]. Backward compatible:
|
||||||
|
/// `pack_6(control)` is exactly `pack_flags(WordFlags(control as u16 << 12),
|
||||||
|
/// ascii)` (the old 4 control bits occupy the high nibble of the flags field).
|
||||||
|
#[inline]
|
||||||
|
pub fn pack_flags(flags: WordFlags, ascii: [u8; 6]) -> TriWord {
|
||||||
|
let mut v: u64 = ((flags.0 as u64) & 0xFFFF) << 48;
|
||||||
|
for (i, b) in ascii.iter().enumerate() {
|
||||||
|
v |= (*b as u64) << (8 * (5 - i));
|
||||||
|
}
|
||||||
|
TriWord(v)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Unpack a word into its 16-bit flags field and 6 data bytes.
|
||||||
|
#[inline]
|
||||||
|
pub fn unpack_flags(word: TriWord) -> (WordFlags, [u8; 6]) {
|
||||||
|
let v = word.0;
|
||||||
|
let flags = WordFlags(((v >> 48) & 0xFFFF) as u16);
|
||||||
|
let mut ascii = [0u8; 6];
|
||||||
|
for (i, b) in ascii.iter_mut().enumerate() {
|
||||||
|
*b = ((v >> (8 * (5 - i))) & 0xFF) as u8;
|
||||||
|
}
|
||||||
|
(flags, ascii)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The 16-bit **per-word** flag/metadata field co-packed with 6 data bytes in a
|
||||||
|
/// tri-channel [`TriWord`] (bits 48-63). Distinct from [`HeaderFlags`], which is
|
||||||
|
/// the per-*record* header; this is the inline per-word metadata stripe.
|
||||||
|
#[derive(Copy, Clone, Eq, PartialEq, Debug, Default)]
|
||||||
|
pub struct WordFlags(pub u16);
|
||||||
|
|
||||||
|
impl WordFlags {
|
||||||
|
/// Packing arrangement (00 tri6, 01 pair, 10 triad, 11 quad).
|
||||||
|
pub const ARR_MASK: u16 = 0b0000_0000_0000_0011;
|
||||||
|
/// Flag: first word of a record.
|
||||||
|
pub const START_RECORD: u16 = 0b0000_0000_0000_0100;
|
||||||
|
/// Flag: last word of a record. (Pair with the PDF's end-of-header marker.)
|
||||||
|
pub const END_RECORD: u16 = 0b0000_0000_0000_1000;
|
||||||
|
/// Flag: this word is header/metadata (vs payload).
|
||||||
|
pub const IS_HEADER: u16 = 0b0000_0000_0001_0000;
|
||||||
|
/// Flag: record continues across the next word.
|
||||||
|
pub const CONTINUATION: u16 = 0b0000_0000_0010_0000;
|
||||||
|
/// Data type (00 text, 01 binary, 10 code, 11 ai-feature-vector).
|
||||||
|
pub const TYPE_MASK: u16 = 0b0000_0000_1100_0000;
|
||||||
|
/// Permission: root-only.
|
||||||
|
pub const PERM_ROOT: u16 = 0b0000_0001_0000_0000;
|
||||||
|
/// Permission: local-user owner.
|
||||||
|
pub const PERM_USER: u16 = 0b0000_0010_0000_0000;
|
||||||
|
/// Flag: association edge present on this word.
|
||||||
|
pub const ASSOC_EDGE: u16 = 0b0000_0100_0000_0000;
|
||||||
|
/// Flag: this group is encrypted (cubecrypt key/tweak selector).
|
||||||
|
pub const ENCRYPTED: u16 = 0b0000_1000_0000_0000;
|
||||||
|
/// Flag: this group is compressed.
|
||||||
|
pub const COMPRESSED: u16 = 0b0001_0000_0000_0000;
|
||||||
|
/// Flag: this group uses the high-entropy/stego transform (obfuscation layer).
|
||||||
|
pub const STEGO: u16 = 0b0010_0000_0000_0000;
|
||||||
|
/// Flag: this group is a Null/cube lookup marker.
|
||||||
|
pub const NULL_LOOKUP: u16 = 0b0100_0000_0000_0000;
|
||||||
|
/// Flag: checksum/parity present for this group.
|
||||||
|
pub const CHECKSUM: u16 = 0b1000_0000_0000_0000;
|
||||||
|
|
||||||
|
/// Construct from a raw bitmask.
|
||||||
|
#[inline]
|
||||||
|
pub const fn from_bits(bits: u16) -> Self { WordFlags(bits) }
|
||||||
|
/// The raw bitmask.
|
||||||
|
#[inline]
|
||||||
|
pub const fn bits(&self) -> u16 { self.0 }
|
||||||
|
/// Set a flag bit.
|
||||||
|
#[inline]
|
||||||
|
pub fn set(&mut self, flag: u16) { self.0 |= flag; }
|
||||||
|
/// Test a flag bit.
|
||||||
|
#[inline]
|
||||||
|
pub fn has(&self, flag: u16) -> bool { self.0 & flag != 0 }
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Header flag bits, mirroring the PDF's title/type/date/size/permission
|
/// Header flag bits, mirroring the PDF's title/type/date/size/permission
|
||||||
@@ -185,6 +263,13 @@ impl HeaderFlags {
|
|||||||
pub const PERM_REMOTE_USER: u16 = 1 << 6;
|
pub const PERM_REMOTE_USER: u16 = 1 << 6;
|
||||||
/// Flag 8: has outgoing association links.
|
/// Flag 8: has outgoing association links.
|
||||||
pub const HAS_ASSOCIATIONS: u16 = 1 << 7;
|
pub const HAS_ASSOCIATIONS: u16 = 1 << 7;
|
||||||
|
/// Flag 9: carries a POSIX `path` metatag (the original filesystem path
|
||||||
|
/// the record was migrated from). Lets the associative query layer
|
||||||
|
/// reconstruct a directory hierarchy WITHOUT the filesystem needing to
|
||||||
|
/// support arbitrary recursive nesting — the path is data, addressed by
|
||||||
|
/// flag, exactly as the source PDF prescribes ("operate on CZYX records
|
||||||
|
/// and Null-space flags rather than paths and inodes").
|
||||||
|
pub const HAS_PATH: u16 = 1 << 8;
|
||||||
// Flag 255 (conceptual end-of-header) is represented out-of-band by the
|
// Flag 255 (conceptual end-of-header) is represented out-of-band by the
|
||||||
// record serializer; there is no bit for it.
|
// record serializer; there is no bit for it.
|
||||||
|
|
||||||
@@ -231,6 +316,11 @@ impl HeaderFlags {
|
|||||||
pub struct CubeHeader {
|
pub struct CubeHeader {
|
||||||
/// Flag bits (derived; kept in sync by the accessors).
|
/// Flag bits (derived; kept in sync by the accessors).
|
||||||
pub flags: HeaderFlags,
|
pub flags: HeaderFlags,
|
||||||
|
/// Per-word 16-bit tri-channel flag/metadata field (bits 48-63). Distinct
|
||||||
|
/// from `flags` (the per-RECORD header): this is the inline per-word
|
||||||
|
/// metadata stripe (see [`WordFlags`]) carried by records that use the
|
||||||
|
/// tri-channel word codec.
|
||||||
|
pub word_flags: WordFlags,
|
||||||
/// Flag 1: human title.
|
/// Flag 1: human title.
|
||||||
pub title: Option<String>,
|
pub title: Option<String>,
|
||||||
/// Flag 2: document type (like a file extension).
|
/// Flag 2: document type (like a file extension).
|
||||||
@@ -245,6 +335,13 @@ pub struct CubeHeader {
|
|||||||
pub owner_remote_user: Option<String>,
|
pub owner_remote_user: Option<String>,
|
||||||
/// Association links to other records (flags 5–19).
|
/// Association links to other records (flags 5–19).
|
||||||
pub linked_records: Vec<Czyx>,
|
pub linked_records: Vec<Czyx>,
|
||||||
|
/// POSIX path metatag: the original filesystem path this record was
|
||||||
|
/// migrated from (e.g. "/etc/network/interfaces"). Empty/absent means the
|
||||||
|
/// record has no path identity. When present, the `HAS_PATH` flag bit is
|
||||||
|
/// set so `scan_by_path`/`query_by_path` can recover the hierarchy from
|
||||||
|
/// metatags alone (see cubestore). This is the PDF's "path is a view over
|
||||||
|
/// coordinates + flags" made concrete.
|
||||||
|
pub path: Option<String>,
|
||||||
/// Total local accesses (from the PDF's association/permission flags).
|
/// Total local accesses (from the PDF's association/permission flags).
|
||||||
pub total_accesses: u64,
|
pub total_accesses: u64,
|
||||||
/// Total remote accesses.
|
/// Total remote accesses.
|
||||||
@@ -292,8 +389,14 @@ impl CubeHeader {
|
|||||||
if !self.linked_records.is_empty() {
|
if !self.linked_records.is_empty() {
|
||||||
f |= HeaderFlags::HAS_ASSOCIATIONS;
|
f |= HeaderFlags::HAS_ASSOCIATIONS;
|
||||||
}
|
}
|
||||||
|
if self.path.is_some() {
|
||||||
|
f |= HeaderFlags::HAS_PATH;
|
||||||
|
}
|
||||||
// Preserve spare/out-of-band flag bits (bits 8..=15) that are not
|
// Preserve spare/out-of-band flag bits (bits 8..=15) that are not
|
||||||
// derived from structured fields.
|
// derived from structured fields. This includes
|
||||||
|
// `cubecrypt::HEADER_FLAG_ENCRYPTED` (bit 12) and the behavior-descriptor
|
||||||
|
// field (bits 13..=15, see `cubecode::Behavior`) so both survive an
|
||||||
|
// encode→decode→refresh round-trip.
|
||||||
const DERIVED_BITS: u16 = 0x00FF;
|
const DERIVED_BITS: u16 = 0x00FF;
|
||||||
f |= self.flags.bits() & !DERIVED_BITS;
|
f |= self.flags.bits() & !DERIVED_BITS;
|
||||||
self.flags = HeaderFlags::from_bits(f);
|
self.flags = HeaderFlags::from_bits(f);
|
||||||
@@ -346,3 +449,46 @@ mod tests {
|
|||||||
assert!(!h.flags.has(HeaderFlags::DOC_TYPE));
|
assert!(!h.flags.has(HeaderFlags::DOC_TYPE));
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod wordflags_tests {
|
||||||
|
use super::*;
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn roundtrip_flags_and_data() {
|
||||||
|
let cases = [
|
||||||
|
(WordFlags(0), *b"abcdef"),
|
||||||
|
(WordFlags::from_bits(WordFlags::ENCRYPTED | WordFlags::START_RECORD), *b"secret"),
|
||||||
|
(WordFlags::from_bits(WordFlags::COMPRESSED | WordFlags::ASSOC_EDGE), *b"123456"),
|
||||||
|
(WordFlags::from_bits(WordFlags::ARR_MASK & 0b10 | WordFlags::NULL_LOOKUP), *b"AB\x00\xff\xee\x7f"),
|
||||||
|
];
|
||||||
|
for (flags, data) in cases {
|
||||||
|
let w = TriEnc::pack_flags(flags, data);
|
||||||
|
let (f, d) = TriEnc::unpack_flags(w);
|
||||||
|
assert_eq!(f.0, flags.0, "flags must round-trip");
|
||||||
|
assert_eq!(d, data, "data must round-trip");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn flags_word_is_16_bits_and_data_is_6_bytes() {
|
||||||
|
let w = TriEnc::pack_flags(WordFlags::from_bits(0xFFFF), [0xFF; 6]);
|
||||||
|
assert_eq!(w.0 >> 48, 0xFFFF, "flags occupy the high 16 bits");
|
||||||
|
let (_, d) = TriEnc::unpack_flags(w);
|
||||||
|
assert_eq!(d, [0xFF; 6]);
|
||||||
|
// data occupies the low 48 bits (bits 0-47); no overlap with flags
|
||||||
|
assert_eq!(TriEnc::pack_flags(WordFlags::from_bits(0x0001), [0x00; 6]).0 >> 63, 0);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn backward_compat_pack_6_equals_flags_high_nibble() {
|
||||||
|
let ctrl = 0xAu8;
|
||||||
|
let ascii = *b"abcdef";
|
||||||
|
let a = TriEnc::pack_6(ctrl, ascii);
|
||||||
|
let b = TriEnc::pack_flags(WordFlags::from_bits((ctrl as u16) << 12), ascii);
|
||||||
|
assert_eq!(a.0, b.0, "pack_6(control) == pack_flags(control << 12)");
|
||||||
|
// and the old control nibble is the high nibble of the flags field
|
||||||
|
let (f, _) = TriEnc::unpack_flags(a);
|
||||||
|
assert_eq!((f.0 >> 12) & 0x0F, ctrl as u16);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
@@ -0,0 +1,11 @@
|
|||||||
|
[package]
|
||||||
|
name = "cubedbt"
|
||||||
|
version = "0.1.0"
|
||||||
|
edition.workspace = true
|
||||||
|
license.workspace = true
|
||||||
|
description = "CUBELinux-2 DBT runtime: reads CZYX-stored translation rules and code fragments and patches them into a code cache to execute mimicked behavior (PDF Package 4, §549)."
|
||||||
|
|
||||||
|
[dependencies]
|
||||||
|
cubecoords = { path = "../cubecoords" }
|
||||||
|
cubestore = { path = "../cubestore" }
|
||||||
|
cubecode = { path = "../cubecode" }
|
||||||
@@ -0,0 +1,421 @@
|
|||||||
|
//! CUBELinux-2 DBT (dynamic binary translation) runtime.
|
||||||
|
//!
|
||||||
|
//! Per the PDF (Package 4, §549): *"cubedbt: a runtime that reads CZYX‑stored
|
||||||
|
//! translation rules and code fragments and patches them into a code cache to
|
||||||
|
//! execute mimicked behavior."*
|
||||||
|
//!
|
||||||
|
//! This crate is the substrate for the PDF's "watch a binary, archive its
|
||||||
|
//! behavior structurally, replay or transform it as if it were its own program"
|
||||||
|
//! story (§554). It is deliberately dependency-free and operates on the real
|
||||||
|
//! `CubeStore` / `CodeCell` / `Vm` types from the rest of the workspace, so it
|
||||||
|
//! is exercisable today on `HashBackend` and slots into `ConcurrentStore`
|
||||||
|
//! later without an API change.
|
||||||
|
//!
|
||||||
|
//! Model
|
||||||
|
//! -----
|
||||||
|
//! * A **translation rule** is a CZYX record (`Kind::Variant`) whose body is a
|
||||||
|
//! serialized [`TranslationRule`]. It names an *op kind* it can replace and
|
||||||
|
//! carries a *replacement fragment* (a `Vec<Op>`). The rule record links to
|
||||||
|
//! the original code cell it may substitute (the association edge).
|
||||||
|
//! * A **code cache** is the patched working set: `patch()` rewrites an
|
||||||
|
//! original `CodeCell`'s bytecode under the eligible rules and writes the
|
||||||
|
//! result back as a new record (preserving the original header's kind/title
|
||||||
|
//! and behavior descriptors, so it runs through the same `Vm`).
|
||||||
|
//! * A **DBT runtime** discovers rules in a store and can `mimic` a target
|
||||||
|
//! coordinate: fetch original → apply rules → run the patched version in the
|
||||||
|
//! `Vm`, returning the `RunResult`.
|
||||||
|
|
||||||
|
use cubecode::{opcode::Op, CodeCell, Kind, Vm};
|
||||||
|
use cubecoords::{CubeHeader, Czyx};
|
||||||
|
use cubestore::{CubeBackend, CubeStore, HashBackend};
|
||||||
|
|
||||||
|
/// `C` axis band where DBT translation rules are stored, kept distinct from the
|
||||||
|
/// `c210` OS-kernel band (`cubecode::C_OS_KERNEL`) and the `c200` snapshot data.
|
||||||
|
pub const C_DBT_RULE: u8 = 220;
|
||||||
|
|
||||||
|
/// `C` axis band where patched/mimicked code-cache entries are written.
|
||||||
|
pub const C_DBT_CACHE: u8 = 221;
|
||||||
|
|
||||||
|
/// The `Op` kinds a translation rule can target.
|
||||||
|
#[derive(Copy, Clone, Debug, PartialEq, Eq)]
|
||||||
|
pub enum OpClass {
|
||||||
|
Const,
|
||||||
|
Add,
|
||||||
|
Sub,
|
||||||
|
Mul,
|
||||||
|
Div,
|
||||||
|
Mod,
|
||||||
|
And,
|
||||||
|
Or,
|
||||||
|
Xor,
|
||||||
|
Shl,
|
||||||
|
Shr,
|
||||||
|
Eq,
|
||||||
|
Ne,
|
||||||
|
Lt,
|
||||||
|
Gt,
|
||||||
|
Le,
|
||||||
|
Ge,
|
||||||
|
Load,
|
||||||
|
Store,
|
||||||
|
CallLink,
|
||||||
|
Any,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl OpClass {
|
||||||
|
/// Classify a live [`Op`].
|
||||||
|
pub fn of(op: &Op) -> OpClass {
|
||||||
|
match op {
|
||||||
|
Op::Const(_) => OpClass::Const,
|
||||||
|
Op::Add => OpClass::Add,
|
||||||
|
Op::Sub => OpClass::Sub,
|
||||||
|
Op::Mul => OpClass::Mul,
|
||||||
|
Op::Div => OpClass::Div,
|
||||||
|
Op::Mod => OpClass::Mod,
|
||||||
|
Op::And => OpClass::And,
|
||||||
|
Op::Or => OpClass::Or,
|
||||||
|
Op::Xor => OpClass::Xor,
|
||||||
|
Op::Shl => OpClass::Shl,
|
||||||
|
Op::Shr => OpClass::Shr,
|
||||||
|
Op::Eq => OpClass::Eq,
|
||||||
|
Op::Ne => OpClass::Ne,
|
||||||
|
Op::Lt => OpClass::Lt,
|
||||||
|
Op::Gt => OpClass::Gt,
|
||||||
|
Op::Le => OpClass::Le,
|
||||||
|
Op::Ge => OpClass::Ge,
|
||||||
|
Op::Load(_) => OpClass::Load,
|
||||||
|
Op::Store(_) => OpClass::Store,
|
||||||
|
Op::CallLink(_) => OpClass::CallLink,
|
||||||
|
_ => OpClass::Any,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A translation rule: a named, CZYX-addressed mapping from an op class to a
|
||||||
|
/// replacement bytecode fragment. Serialized into a rule record's body.
|
||||||
|
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||||
|
pub struct TranslationRule {
|
||||||
|
/// Human name (also the rule record's title).
|
||||||
|
pub name: String,
|
||||||
|
/// Which op class this rule can substitute.
|
||||||
|
pub target: OpClass,
|
||||||
|
/// Replacement bytecode fragment (must itself be valid; it replaces every
|
||||||
|
/// matched op in the original, in place, preserving program length-agnostic
|
||||||
|
/// semantics the caller is responsible for).
|
||||||
|
pub fragment: Vec<Op>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl TranslationRule {
|
||||||
|
/// Serialize to a stable byte form (name length-prefixed, target byte,
|
||||||
|
/// fragment as opcode codec). No external deps.
|
||||||
|
pub fn to_bytes(&self) -> Vec<u8> {
|
||||||
|
let mut out = Vec::new();
|
||||||
|
let nb = self.name.as_bytes();
|
||||||
|
out.push(nb.len() as u8);
|
||||||
|
out.extend_from_slice(nb);
|
||||||
|
out.push(self.target as u8);
|
||||||
|
out.push(self.fragment.len() as u8);
|
||||||
|
for op in &self.fragment {
|
||||||
|
out.extend_from_slice(&cubecode::opcode::encode(&[*op]));
|
||||||
|
}
|
||||||
|
out
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Inverse of [`to_bytes`]. Returns `None` on any malformed input.
|
||||||
|
pub fn from_bytes(b: &[u8]) -> Option<TranslationRule> {
|
||||||
|
let mut i = 0;
|
||||||
|
let nlen = *b.get(i)? as usize;
|
||||||
|
i += 1;
|
||||||
|
if i + nlen > b.len() {
|
||||||
|
return None;
|
||||||
|
}
|
||||||
|
let name = String::from_utf8(b[i..i + nlen].to_vec()).ok()?;
|
||||||
|
i += nlen;
|
||||||
|
let target = *b.get(i)?;
|
||||||
|
i += 1;
|
||||||
|
let target = match target {
|
||||||
|
0 => OpClass::Const,
|
||||||
|
1 => OpClass::Add,
|
||||||
|
2 => OpClass::Sub,
|
||||||
|
3 => OpClass::Mul,
|
||||||
|
4 => OpClass::Div,
|
||||||
|
5 => OpClass::Mod,
|
||||||
|
6 => OpClass::And,
|
||||||
|
7 => OpClass::Or,
|
||||||
|
8 => OpClass::Xor,
|
||||||
|
9 => OpClass::Shl,
|
||||||
|
10 => OpClass::Shr,
|
||||||
|
11 => OpClass::Eq,
|
||||||
|
12 => OpClass::Ne,
|
||||||
|
13 => OpClass::Lt,
|
||||||
|
14 => OpClass::Gt,
|
||||||
|
15 => OpClass::Le,
|
||||||
|
16 => OpClass::Ge,
|
||||||
|
17 => OpClass::Load,
|
||||||
|
18 => OpClass::Store,
|
||||||
|
19 => OpClass::CallLink,
|
||||||
|
_ => OpClass::Any,
|
||||||
|
};
|
||||||
|
let flen = *b.get(i)? as usize;
|
||||||
|
i += 1;
|
||||||
|
let mut fragment = Vec::new();
|
||||||
|
for _ in 0..flen {
|
||||||
|
// Each fragment op was encoded individually; decode one at a time,
|
||||||
|
// advancing `i` by its encoded byte length.
|
||||||
|
let one = decode_one(&b[i..])?;
|
||||||
|
i += one.len;
|
||||||
|
fragment.push(one.op);
|
||||||
|
}
|
||||||
|
Some(TranslationRule {
|
||||||
|
name,
|
||||||
|
target,
|
||||||
|
fragment,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// One decoded opcode plus its encoded length, used by `from_bytes`.
|
||||||
|
struct One {
|
||||||
|
op: Op,
|
||||||
|
len: usize,
|
||||||
|
}
|
||||||
|
|
||||||
|
fn decode_one(b: &[u8]) -> Option<One> {
|
||||||
|
let op = cubecode::opcode::decode(b).ok()?;
|
||||||
|
let op = op.into_iter().next()?;
|
||||||
|
let len = cubecode::opcode::encode(&[op]).len();
|
||||||
|
Some(One { op, len })
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A patched working set ("code cache") inside a [`CubeStore`].
|
||||||
|
///
|
||||||
|
/// `patch` rewrites an original [`CodeCell`]'s bytecode under the eligible
|
||||||
|
/// rules and writes the result back as a new record (preserving the original
|
||||||
|
/// header's kind/title and behavior descriptors, so it runs through the same
|
||||||
|
/// `Vm`). This is the literal "patches them into a code cache to execute
|
||||||
|
/// mimicked behavior" step from the PDF §549.
|
||||||
|
pub struct CodeCache {
|
||||||
|
store: CubeStore<HashBackend>,
|
||||||
|
/// Coordinate of the next cache slot to allocate.
|
||||||
|
next_x: u8,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl CodeCache {
|
||||||
|
/// Build an empty cache over a fresh in-memory backend.
|
||||||
|
pub fn new() -> Self {
|
||||||
|
CodeCache {
|
||||||
|
store: CubeStore::new(HashBackend::new()),
|
||||||
|
next_x: 1,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Patch `original` under `rules`: every op whose class matches a rule's
|
||||||
|
/// target (or whose rule target is `Any`) is replaced by that rule's
|
||||||
|
/// fragment. The rewritten cell is stored at a fresh cache coordinate and
|
||||||
|
/// returned (with its new label). The original store is untouched.
|
||||||
|
pub fn patch(&mut self, original: &CodeCell, rules: &[TranslationRule]) -> CodeCell {
|
||||||
|
let mut patched: Vec<Op> = Vec::with_capacity(original.code.len());
|
||||||
|
for op in &original.code {
|
||||||
|
let class = OpClass::of(op);
|
||||||
|
let mut applied = false;
|
||||||
|
for r in rules {
|
||||||
|
if r.target == class || r.target == OpClass::Any {
|
||||||
|
patched.extend_from_slice(&r.fragment);
|
||||||
|
applied = true;
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if !applied {
|
||||||
|
patched.push(*op);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
let label = Czyx::new(C_DBT_CACHE, 1, 1, self.next_x);
|
||||||
|
self.next_x = self.next_x.wrapping_add(1).max(1);
|
||||||
|
let (h, body) = cache_header_for(original, &cubecode::opcode::encode(&patched));
|
||||||
|
self.store.put_record(label, &h, &body);
|
||||||
|
CodeCell::from_record(label, &h, &body).expect("patched cell is always valid bytecode")
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Run a patched cache entry in the VM, returning its result.
|
||||||
|
pub fn run(&self, cell: &CodeCell) -> cubecode::RunResult {
|
||||||
|
let mut vm = Vm::new(self.store.clone());
|
||||||
|
vm.run(cell.label)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Borrow the backing store (e.g. to persist or inspect).
|
||||||
|
pub fn store(&self) -> &CubeStore<HashBackend> {
|
||||||
|
&self.store
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl Default for CodeCache {
|
||||||
|
fn default() -> Self {
|
||||||
|
Self::new()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Build a cache record header that preserves the original's kind/title and
|
||||||
|
/// behavior descriptors, but marks it a `Variant` (a mimicked implementation).
|
||||||
|
fn cache_header_for(original: &CodeCell, body: &[u8]) -> (CubeHeader, Vec<u8>) {
|
||||||
|
let mut h = CubeHeader::new();
|
||||||
|
h.title = original.name().map(|s| format!("mimic:{}", s));
|
||||||
|
h.doc_type = Some(Kind::Variant.as_str().into());
|
||||||
|
h.size_bytes = Some(body.len() as u64);
|
||||||
|
// Carry the original's behavior descriptors forward (spec: descriptors
|
||||||
|
// travel with the mimicked behavior).
|
||||||
|
h.flags.0 |= original.header.flags.0 & cubecode::HEADER_FLAG_BEHAVIOR;
|
||||||
|
h.refresh_flags();
|
||||||
|
(h, body.to_vec())
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The DBT runtime: discovers CZYX-stored translation rules and can `mimic` a
|
||||||
|
/// target code cell by patching it under those rules and running the result.
|
||||||
|
pub struct DbRuntime<B: CubeBackend> {
|
||||||
|
store: CubeStore<B>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl<B: CubeBackend> DbRuntime<B> {
|
||||||
|
/// Wrap a store that already contains rule records (or will, via
|
||||||
|
/// [`store_rule`]). The runtime is read-only over this store.
|
||||||
|
pub fn new(store: CubeStore<B>) -> Self {
|
||||||
|
DbRuntime { store }
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Collect every translation rule currently in the `c220` rule band.
|
||||||
|
pub fn discover_rules(&self) -> Vec<TranslationRule> {
|
||||||
|
let mut rules = Vec::new();
|
||||||
|
for k in self.store.keys() {
|
||||||
|
if k.c != C_DBT_RULE {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
if let Some((_, body)) = self.store.get_record(&k) {
|
||||||
|
if let Some(r) = TranslationRule::from_bytes(&body) {
|
||||||
|
rules.push(r);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
rules
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Fetch a code cell by coordinate from the wrapped store.
|
||||||
|
pub fn fetch(&self, label: Czyx) -> Option<CodeCell> {
|
||||||
|
let (h, b) = self.store.get_record(&label)?;
|
||||||
|
CodeCell::from_record(label, &h, &b)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Mimic `target`: fetch it, apply all discovered rules, patch into a fresh
|
||||||
|
/// cache, and run the patched version — returning the `RunResult`. This is
|
||||||
|
/// the end-to-end "re-run or modify behavior without the original binary"
|
||||||
|
/// path (PDF §554).
|
||||||
|
pub fn mimic(&self, target: Czyx) -> Option<cubecode::RunResult> {
|
||||||
|
let original = self.fetch(target)?;
|
||||||
|
let rules = self.discover_rules();
|
||||||
|
let mut cache = CodeCache::new();
|
||||||
|
let patched = cache.patch(&original, &rules);
|
||||||
|
Some(cache.run(&patched))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Store a translation rule into the `c220` rule band at a fresh coordinate.
|
||||||
|
/// Returns the coordinate it was written to.
|
||||||
|
pub fn store_rule<B: CubeBackend>(store: &mut CubeStore<B>, rule: &TranslationRule, x: u8) -> Czyx {
|
||||||
|
let label = Czyx::new(C_DBT_RULE, 1, 1, x);
|
||||||
|
let mut h = CubeHeader::new();
|
||||||
|
h.title = Some(rule.name.clone());
|
||||||
|
h.doc_type = Some(Kind::Variant.as_str().into());
|
||||||
|
h.size_bytes = Some(rule.to_bytes().len() as u64);
|
||||||
|
h.refresh_flags();
|
||||||
|
store.put_record(label, &h, &rule.to_bytes());
|
||||||
|
label
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
use cubecode::opcode::Op;
|
||||||
|
use cubecode::RunResult;
|
||||||
|
use cubecoords::Czyx;
|
||||||
|
|
||||||
|
fn sample_cell(coord: Czyx, code: Vec<Op>) -> CodeCell {
|
||||||
|
let mut h = CubeHeader::new();
|
||||||
|
h.title = Some("sample".into());
|
||||||
|
h.doc_type = Some(Kind::Fn.as_str().into());
|
||||||
|
h.refresh_flags();
|
||||||
|
CodeCell::from_record(coord, &h, &cubecode::opcode::encode(&code))
|
||||||
|
.expect("sample is valid bytecode")
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn rule_round_trips_through_bytes() {
|
||||||
|
let r = TranslationRule {
|
||||||
|
name: "double-add".into(),
|
||||||
|
target: OpClass::Add,
|
||||||
|
fragment: vec![Op::Const(2), Op::Mul],
|
||||||
|
};
|
||||||
|
let bytes = r.to_bytes();
|
||||||
|
let back = TranslationRule::from_bytes(&bytes).expect("rule decodes");
|
||||||
|
assert_eq!(r, back);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn cache_patches_and_runs() {
|
||||||
|
// Original: Const 3, Const 4, Add, Halt => 3+4 = 7.
|
||||||
|
let orig = sample_cell(
|
||||||
|
Czyx::new(1, 1, 1, 1),
|
||||||
|
vec![Op::Const(3), Op::Const(4), Op::Add, Op::Halt],
|
||||||
|
);
|
||||||
|
// Rule: replace `Add` with `Const 2, Mul` => (a)*(2). With a=3,b=4:
|
||||||
|
// naive in-place substitution yields Const3, Const4, Const2, Mul =>
|
||||||
|
// 4*2 = 8. This proves the patched fragment is what runs.
|
||||||
|
let rule = TranslationRule {
|
||||||
|
name: "mul-by-2".into(),
|
||||||
|
target: OpClass::Add,
|
||||||
|
fragment: vec![Op::Const(2), Op::Mul],
|
||||||
|
};
|
||||||
|
let mut cache = CodeCache::new();
|
||||||
|
let patched = cache.patch(&orig, &[rule]);
|
||||||
|
assert_eq!(patched.kind(), Kind::Variant, "cache entry is a Variant");
|
||||||
|
match cache.run(&patched) {
|
||||||
|
RunResult::Halted { top: Some(v) } => assert_eq!(v, 8, "patched semantics differ"),
|
||||||
|
other => panic!("patched run failed: {:?}", other),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn runtime_mimic_end_to_end() {
|
||||||
|
let mut store = CubeStore::new(HashBackend::new());
|
||||||
|
// Original target at c1.
|
||||||
|
let target = Czyx::new(1, 1, 1, 1);
|
||||||
|
let orig = sample_cell(target, vec![Op::Const(10), Op::Const(5), Op::Sub, Op::Halt]);
|
||||||
|
store.put_record(orig.label, &orig.header, &orig.body());
|
||||||
|
|
||||||
|
// Rule in the c220 band: replace Sub with Add (10+5=15 instead of 5).
|
||||||
|
let rule = TranslationRule {
|
||||||
|
name: "sub->add".into(),
|
||||||
|
target: OpClass::Sub,
|
||||||
|
fragment: vec![Op::Add],
|
||||||
|
};
|
||||||
|
store_rule(&mut store, &rule, 1);
|
||||||
|
|
||||||
|
let rt = DbRuntime::new(store);
|
||||||
|
let rules = rt.discover_rules();
|
||||||
|
assert_eq!(rules.len(), 1, "rule discovered in c220 band");
|
||||||
|
match rt.mimic(target) {
|
||||||
|
Some(RunResult::Halted { top: Some(v) }) => assert_eq!(v, 15, "mimic applied rule"),
|
||||||
|
other => panic!("mimic failed: {:?}", other),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn mimic_without_rules_matches_original() {
|
||||||
|
let mut store = CubeStore::new(HashBackend::new());
|
||||||
|
let target = Czyx::new(1, 1, 1, 2);
|
||||||
|
let orig = sample_cell(target, vec![Op::Const(7), Op::Const(2), Op::Mul, Op::Halt]);
|
||||||
|
store.put_record(orig.label, &orig.header, &orig.body());
|
||||||
|
let rt = DbRuntime::new(store);
|
||||||
|
match rt.mimic(target) {
|
||||||
|
Some(RunResult::Halted { top: Some(v) }) => assert_eq!(v, 14),
|
||||||
|
other => panic!("mimic without rules should run original: {:?}", other),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -448,6 +448,8 @@ impl<B: CubeBackend + 'static> Filesystem for CubeFuse<B> {
|
|||||||
kind: Kind::Directory,
|
kind: Kind::Directory,
|
||||||
size: 0,
|
size: 0,
|
||||||
mode: (mode & 0o7777) as u16,
|
mode: (mode & 0o7777) as u16,
|
||||||
|
word_flags: 0,
|
||||||
|
header_flags: 0,
|
||||||
uid: req.uid(),
|
uid: req.uid(),
|
||||||
gid: req.gid(),
|
gid: req.gid(),
|
||||||
created_at: 0,
|
created_at: 0,
|
||||||
|
|||||||
+27
-4
@@ -106,6 +106,10 @@ pub struct Attr {
|
|||||||
pub created_at: u64,
|
pub created_at: u64,
|
||||||
/// Number of hard links: 1 for files, 2 for directories (`.` and `..`).
|
/// Number of hard links: 1 for files, 2 for directories (`.` and `..`).
|
||||||
pub nlink: u32,
|
pub nlink: u32,
|
||||||
|
/// Per-word 16-bit flag field of the backing record (header tag 14). 0 for dirs.
|
||||||
|
pub word_flags: u16,
|
||||||
|
/// Per-record HeaderFlags of the backing record (header tag 12). 0 for dirs.
|
||||||
|
pub header_flags: u16,
|
||||||
}
|
}
|
||||||
|
|
||||||
/// The filesystem.
|
/// The filesystem.
|
||||||
@@ -234,14 +238,14 @@ impl<B: CubeBackend> CubeFs<B> {
|
|||||||
let kind = self.kind_of(p)?;
|
let kind = self.kind_of(p)?;
|
||||||
let coord = parsed.prefix_coord();
|
let coord = parsed.prefix_coord();
|
||||||
let acl = self.acl_of(coord);
|
let acl = self.acl_of(coord);
|
||||||
let (size, created_at) = if kind == Kind::File {
|
let (size, created_at, word_flags, header_flags) = if kind == Kind::File {
|
||||||
let c = parsed.czyx().expect("file implies full coord");
|
let c = parsed.czyx().expect("file implies full coord");
|
||||||
match self.store.get_record(&c) {
|
match self.store.get_record(&c) {
|
||||||
Some((h, body)) => (body.len() as u64, h.created_at.unwrap_or(0)),
|
Some((h, body)) => (body.len() as u64, h.created_at.unwrap_or(0), h.word_flags.bits(), h.flags.bits()),
|
||||||
None => (0, 0),
|
None => (0, 0, 0, 0),
|
||||||
}
|
}
|
||||||
} else {
|
} else {
|
||||||
(0, 0)
|
(0, 0, 0, 0)
|
||||||
};
|
};
|
||||||
// Decision: directories report 0o755 unless an explicit ACL exists,
|
// Decision: directories report 0o755 unless an explicit ACL exists,
|
||||||
// because the file default (0644) would make every directory
|
// because the file default (0644) would make every directory
|
||||||
@@ -262,6 +266,8 @@ impl<B: CubeBackend> CubeFs<B> {
|
|||||||
gid: acl.gid,
|
gid: acl.gid,
|
||||||
created_at,
|
created_at,
|
||||||
nlink: if kind == Kind::Directory { 2 } else { 1 },
|
nlink: if kind == Kind::Directory { 2 } else { 1 },
|
||||||
|
word_flags,
|
||||||
|
header_flags,
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -771,6 +777,23 @@ mod tests {
|
|||||||
assert!(f.read(p, 0, 1, 2000, 2000).is_ok());
|
assert!(f.read(p, 0, 1, 2000, 2000).is_ok());
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn getattr_surfaces_word_flags_and_header_flags() {
|
||||||
|
use cubecoords::WordFlags;
|
||||||
|
let mut f = fs();
|
||||||
|
let mut h = CubeHeader::new();
|
||||||
|
h.doc_type = Some("fn".into());
|
||||||
|
h.word_flags = WordFlags::from_bits(
|
||||||
|
WordFlags::START_RECORD | WordFlags::END_RECORD | WordFlags::IS_HEADER | (0b10 << 6),
|
||||||
|
);
|
||||||
|
h.refresh_flags();
|
||||||
|
f.store_mut().put_record(cubecoords::Czyx::new(1, 1, 1, 1), &h, b"hi");
|
||||||
|
let a = f.getattr("/c001/z001/y001/x001").unwrap();
|
||||||
|
assert_eq!(a.word_flags & WordFlags::START_RECORD, WordFlags::START_RECORD);
|
||||||
|
assert_eq!((a.word_flags & WordFlags::TYPE_MASK) >> 6, 0b10, "type=code");
|
||||||
|
assert_eq!(a.header_flags & cubecoords::HeaderFlags::DOC_TYPE, cubecoords::HeaderFlags::DOC_TYPE);
|
||||||
|
}
|
||||||
#[test]
|
#[test]
|
||||||
fn directories_default_to_traversable() {
|
fn directories_default_to_traversable() {
|
||||||
let mut f = fs();
|
let mut f = fs();
|
||||||
|
|||||||
+528
-12
@@ -151,7 +151,6 @@ pub struct CubeStore<B: CubeBackend> {
|
|||||||
/// encoding is used to avoid a serde dependency at Package 1.)
|
/// encoding is used to avoid a serde dependency at Package 1.)
|
||||||
mod record_codec {
|
mod record_codec {
|
||||||
use cubecoords::{CubeHeader, Czyx};
|
use cubecoords::{CubeHeader, Czyx};
|
||||||
use std::collections::HashMap;
|
|
||||||
|
|
||||||
// Compact, dependency-free encoding of the header.
|
// Compact, dependency-free encoding of the header.
|
||||||
// Fields are written in a fixed tag-length-value stream so unknown
|
// Fields are written in a fixed tag-length-value stream so unknown
|
||||||
@@ -170,6 +169,11 @@ mod record_codec {
|
|||||||
// 9 total_remote_accesses (u64 le)
|
// 9 total_remote_accesses (u64 le)
|
||||||
// 10 last_access (u64 le)
|
// 10 last_access (u64 le)
|
||||||
// 11 last_remote_access (u64 le)
|
// 11 last_remote_access (u64 le)
|
||||||
|
// 14 word_flags (u16 le) — per-word 16-bit tri-channel WordFlags
|
||||||
|
// 13 path (utf8) — POSIX path metatag (the original filesystem path a
|
||||||
|
// record was migrated from). Carries the directory hierarchy as DATA
|
||||||
|
// so the associative query layer can reconstruct nesting from flags.
|
||||||
|
// (Tag 12, raw flag bits, is documented at its emit site below.)
|
||||||
|
|
||||||
pub fn encode_header(h: &CubeHeader) -> Vec<u8> {
|
pub fn encode_header(h: &CubeHeader) -> Vec<u8> {
|
||||||
let mut out = Vec::new();
|
let mut out = Vec::new();
|
||||||
@@ -213,6 +217,9 @@ mod record_codec {
|
|||||||
if let Some(a) = h.last_remote_access {
|
if let Some(a) = h.last_remote_access {
|
||||||
put_u64(&mut out, 11, a);
|
put_u64(&mut out, 11, a);
|
||||||
}
|
}
|
||||||
|
if let Some(p) = &h.path {
|
||||||
|
put_utf8(&mut out, 13, p);
|
||||||
|
}
|
||||||
// Tag 12: raw flag bits. Serializes out-of-band/spare bits (e.g.
|
// Tag 12: raw flag bits. Serializes out-of-band/spare bits (e.g.
|
||||||
// `cubecrypt::HEADER_FLAG_ENCRYPTED`) that are not derived from
|
// `cubecrypt::HEADER_FLAG_ENCRYPTED`) that are not derived from
|
||||||
// structured fields, so they survive an encode/decode round-trip.
|
// structured fields, so they survive an encode/decode round-trip.
|
||||||
@@ -220,6 +227,11 @@ mod record_codec {
|
|||||||
out.push(12);
|
out.push(12);
|
||||||
out.extend_from_slice(&h.flags.bits().to_le_bytes());
|
out.extend_from_slice(&h.flags.bits().to_le_bytes());
|
||||||
}
|
}
|
||||||
|
// Tag 14: per-word 16-bit tri-channel WordFlags field (see cubecoords).
|
||||||
|
if h.word_flags.bits() != 0 {
|
||||||
|
out.push(14);
|
||||||
|
out.extend_from_slice(&h.word_flags.bits().to_le_bytes());
|
||||||
|
}
|
||||||
out
|
out
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -297,6 +309,11 @@ mod record_codec {
|
|||||||
h.last_remote_access = Some(v);
|
h.last_remote_access = Some(v);
|
||||||
b = rest;
|
b = rest;
|
||||||
}
|
}
|
||||||
|
13 => {
|
||||||
|
let (v, rest) = take_utf8(b)?;
|
||||||
|
h.path = Some(v);
|
||||||
|
b = rest;
|
||||||
|
}
|
||||||
12 => {
|
12 => {
|
||||||
if b.len() < 2 {
|
if b.len() < 2 {
|
||||||
return None;
|
return None;
|
||||||
@@ -305,6 +322,14 @@ mod record_codec {
|
|||||||
h.flags = cubecoords::HeaderFlags::flags_from_bits(raw);
|
h.flags = cubecoords::HeaderFlags::flags_from_bits(raw);
|
||||||
b = &b[2..];
|
b = &b[2..];
|
||||||
}
|
}
|
||||||
|
14 => {
|
||||||
|
if b.len() < 2 {
|
||||||
|
return None;
|
||||||
|
}
|
||||||
|
let raw = u16::from_le_bytes([b[0], b[1]]);
|
||||||
|
h.word_flags = cubecoords::WordFlags::from_bits(raw);
|
||||||
|
b = &b[2..];
|
||||||
|
}
|
||||||
_ => return None, // unknown tag -> reject (strict at Package 1)
|
_ => return None, // unknown tag -> reject (strict at Package 1)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -344,12 +369,6 @@ mod record_codec {
|
|||||||
let s = String::from_utf8(rest[..n].to_vec()).ok()?;
|
let s = String::from_utf8(rest[..n].to_vec()).ok()?;
|
||||||
Some((s, &rest[n..]))
|
Some((s, &rest[n..]))
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Keep `HashMap` referenced so the dependency is explicit in this module
|
|
||||||
/// even though the codec itself is generic over bytes. (Prevents an
|
|
||||||
/// unused-import warning if the backend type changes.)
|
|
||||||
#[allow(dead_code)]
|
|
||||||
pub(crate) fn _assert_backend_assoc(_: &HashMap<u32, Vec<u8>>) {}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
impl<B: CubeBackend> CubeStore<B> {
|
impl<B: CubeBackend> CubeStore<B> {
|
||||||
@@ -387,20 +406,55 @@ impl<B: CubeBackend> CubeStore<B> {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/// Fetch and split a record into `(header, body)`.
|
/// Fetch and split a record into `(header, body)`.
|
||||||
|
///
|
||||||
|
/// Records written through the record path carry a TLV envelope
|
||||||
|
/// (`u32 header-len | header | body`). Records written through the *raw*
|
||||||
|
/// path (`put_raw` / the daemon's `rawput` verb, used by OS-layer services)
|
||||||
|
/// carry no envelope at all.
|
||||||
|
///
|
||||||
|
/// Historically a non-enveloped payload made this return `None`, which
|
||||||
|
/// callers such as `cubefs`'s `getattr`/`read` and the daemon's `stat` verb
|
||||||
|
/// translate into "size 0 / empty file". A record holding real bytes was
|
||||||
|
/// therefore *silently invisible* through the filesystem while `rawget`
|
||||||
|
/// happily returned its contents — the OS-in-CUBE migration hit exactly
|
||||||
|
/// this (2026-08-13): every `rawput` OS record listed as a 0-byte file.
|
||||||
|
///
|
||||||
|
/// Losing data silently is never the right failure mode, so a payload that
|
||||||
|
/// is not a well-formed envelope is now surfaced as a raw body under a
|
||||||
|
/// synthesized header. `rawput` data becomes readable through the
|
||||||
|
/// filesystem, and no caller has to special-case the two write paths.
|
||||||
pub fn get_record(&self, label: &Czyx) -> Option<(CubeHeader, Vec<u8>)> {
|
pub fn get_record(&self, label: &Czyx) -> Option<(CubeHeader, Vec<u8>)> {
|
||||||
let raw = self.backend.get(label)?;
|
let raw = self.backend.get(label)?;
|
||||||
|
// Fall back to treating the payload as a raw (un-enveloped) body when
|
||||||
|
// it cannot be parsed as `len | header | body`.
|
||||||
|
let raw_fallback = |bytes: &[u8]| {
|
||||||
|
let mut h = CubeHeader::new();
|
||||||
|
h.size_bytes = Some(bytes.len() as u64);
|
||||||
|
// Keep the synthesized header's flag bits consistent with its
|
||||||
|
// fields, so associative queries (`scan_by_flag(SIZE_BYTES)`) see
|
||||||
|
// raw records too. Only the derived low bits are recomputed; any
|
||||||
|
// out-of-band bits already on the payload-derived header (none
|
||||||
|
// here, since raw payloads carry no header) are preserved by
|
||||||
|
// `refresh_flags`.
|
||||||
|
h.refresh_flags();
|
||||||
|
Some((h, bytes.to_vec()))
|
||||||
|
};
|
||||||
if raw.len() < 4 {
|
if raw.len() < 4 {
|
||||||
return None;
|
return raw_fallback(&raw);
|
||||||
}
|
}
|
||||||
let mut len = [0u8; 4];
|
let mut len = [0u8; 4];
|
||||||
len.copy_from_slice(&raw[..4]);
|
len.copy_from_slice(&raw[..4]);
|
||||||
let hlen = u32::from_le_bytes(len) as usize;
|
let hlen = u32::from_le_bytes(len) as usize;
|
||||||
if raw.len() < 4 + hlen {
|
if raw.len() < 4 + hlen {
|
||||||
return None;
|
return raw_fallback(&raw);
|
||||||
|
}
|
||||||
|
match record_codec::decode_header(&raw[4..4 + hlen]) {
|
||||||
|
Some(hdr) => {
|
||||||
|
let body = raw[4 + hlen..].to_vec();
|
||||||
|
Some((hdr, body))
|
||||||
|
}
|
||||||
|
None => raw_fallback(&raw),
|
||||||
}
|
}
|
||||||
let hdr = record_codec::decode_header(&raw[4..4 + hlen])?;
|
|
||||||
let body = raw[4 + hlen..].to_vec();
|
|
||||||
Some((hdr, body))
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Raw backend write (non-record payloads, e.g. ACL/xattr/volume buckets).
|
/// Raw backend write (non-record payloads, e.g. ACL/xattr/volume buckets).
|
||||||
@@ -472,6 +526,167 @@ impl<B: CubeBackend> CubeStore<B> {
|
|||||||
out.sort();
|
out.sort();
|
||||||
out
|
out
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// PDF Package 2 API: `scan_by_flag` — associative storage lookup.
|
||||||
|
///
|
||||||
|
/// Returns every coordinate whose decoded [`CubeHeader`] carries `flag`
|
||||||
|
/// set. This is the "query by metatag/flag, not by path" primitive the
|
||||||
|
/// source PDF describes (§"associative storage"): records are addressed
|
||||||
|
/// by *what they are* (a flag bit) rather than *where they live* (a
|
||||||
|
/// coordinate path). `cube-os-*` services that emit typed records
|
||||||
|
/// (e.g. a log line tagged `doc_type = "klog"`) become discoverable by
|
||||||
|
/// event type without knowing their CZYX address in advance.
|
||||||
|
///
|
||||||
|
/// Records written through the RAW path (`rawput`, used by the OS layers)
|
||||||
|
/// carry only a synthesized `size_bytes` header, so they will NOT match
|
||||||
|
/// a content flag unless the writer also set one. The associative query
|
||||||
|
/// is therefore most powerful when records are written through
|
||||||
|
/// [`CubeStore::put_record`] with a populated [`CubeHeader`].
|
||||||
|
pub fn scan_by_flag(&self, flag: u16) -> Vec<Czyx> {
|
||||||
|
let mut out: Vec<Czyx> = self
|
||||||
|
.backend
|
||||||
|
.keys()
|
||||||
|
.into_iter()
|
||||||
|
.filter(|k| {
|
||||||
|
self.get_record(k)
|
||||||
|
.map(|(h, _)| h.flags.has(flag))
|
||||||
|
.unwrap_or(false)
|
||||||
|
})
|
||||||
|
.collect();
|
||||||
|
out.sort();
|
||||||
|
out
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Query records by per-word flag (the 16-bit WordFlags metadata stripe
|
||||||
|
/// carried in the header, tag 14). Parallel to [`scan_by_flag`] but on
|
||||||
|
/// `word_flags`, so per-word metadata (encrypted/compressed/stego/type/
|
||||||
|
/// arrangement/continuation) is queryable without decoding the body.
|
||||||
|
pub fn scan_by_word_flag(&self, flag: u16) -> Vec<Czyx> {
|
||||||
|
let mut out: Vec<Czyx> = self
|
||||||
|
.backend
|
||||||
|
.keys()
|
||||||
|
.into_iter()
|
||||||
|
.filter(|k| {
|
||||||
|
self.get_record(k)
|
||||||
|
.map(|(h, _)| h.word_flags.has(flag))
|
||||||
|
.unwrap_or(false)
|
||||||
|
})
|
||||||
|
.collect();
|
||||||
|
out.sort();
|
||||||
|
out
|
||||||
|
}
|
||||||
|
|
||||||
|
/// PDF Package 2 API: `scan_by_type` — query records by `doc_type`
|
||||||
|
/// (the "event type" / file-extension analogue the PDF calls Flag 2).
|
||||||
|
///
|
||||||
|
/// This is the concrete "log lookup by query" the OS layers want: instead
|
||||||
|
/// of addressing `/cubefs/c200/z004/...` directly, you ask "every record
|
||||||
|
/// whose doc_type is `klog`" and get back all matching coordinates. The
|
||||||
|
/// comparison is exact-match (case-sensitive) against the header field.
|
||||||
|
pub fn scan_by_type(&self, doc_type: &str) -> Vec<Czyx> {
|
||||||
|
let mut out: Vec<Czyx> = self
|
||||||
|
.backend
|
||||||
|
.keys()
|
||||||
|
.into_iter()
|
||||||
|
.filter(|k| {
|
||||||
|
self.get_record(k)
|
||||||
|
.map(|(h, _)| h.doc_type.as_deref() == Some(doc_type))
|
||||||
|
.unwrap_or(false)
|
||||||
|
})
|
||||||
|
.collect();
|
||||||
|
out.sort();
|
||||||
|
out
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Return `(label, header, body)` for every record matching `flag`.
|
||||||
|
/// Convenience wrapper over [`CubeStore::scan_by_flag`] that also pulls
|
||||||
|
/// the decoded payload so a caller (e.g. a `cubelog` query tool) can
|
||||||
|
/// present the matching records directly.
|
||||||
|
pub fn query_by_flag(&self, flag: u16) -> Vec<(Czyx, CubeHeader, Vec<u8>)> {
|
||||||
|
self.scan_by_flag(flag)
|
||||||
|
.into_iter()
|
||||||
|
.filter_map(|k| self.get_record(&k).map(|(h, b)| (k, h, b)))
|
||||||
|
.collect()
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Return `(label, header, body)` for every record whose `doc_type`
|
||||||
|
/// matches. See [`CubeStore::scan_by_type`] for the matching semantics.
|
||||||
|
pub fn query_by_type(&self, doc_type: &str) -> Vec<(Czyx, CubeHeader, Vec<u8>)> {
|
||||||
|
self.scan_by_type(doc_type)
|
||||||
|
.into_iter()
|
||||||
|
.filter_map(|k| self.get_record(&k).map(|(h, b)| (k, h, b)))
|
||||||
|
.collect()
|
||||||
|
}
|
||||||
|
|
||||||
|
/// PDF Package 2 / OS-in-CUBE: reconstruct a *directory hierarchy from
|
||||||
|
/// metatags*. Returns every coordinate whose header carries a `path`
|
||||||
|
/// metatag equal to `path` (exact) — i.e. "the file at this path".
|
||||||
|
///
|
||||||
|
/// Combined with [`CubeStore::scan_by_path_prefix`], this lets the cube
|
||||||
|
/// answer "everything under /etc" WITHOUT the filesystem supporting
|
||||||
|
/// recursive nesting: the path is stored as a flag-addressed field, not
|
||||||
|
/// as an inode tree. This is the source PDF's prescribed model ("operate
|
||||||
|
/// on CZYX records and Null-space flags rather than paths and inodes").
|
||||||
|
pub fn scan_by_path(&self, path: &str) -> Vec<Czyx> {
|
||||||
|
let mut out: Vec<Czyx> = self
|
||||||
|
.backend
|
||||||
|
.keys()
|
||||||
|
.into_iter()
|
||||||
|
.filter(|k| {
|
||||||
|
self.get_record(k)
|
||||||
|
.map(|(h, _)| h.path.as_deref() == Some(path))
|
||||||
|
.unwrap_or(false)
|
||||||
|
})
|
||||||
|
.collect();
|
||||||
|
out.sort();
|
||||||
|
out
|
||||||
|
}
|
||||||
|
|
||||||
|
/// "List a directory": every record whose `path` metatag is *under* the
|
||||||
|
/// given directory prefix (e.g. `scan_by_path_prefix("/etc")` returns
|
||||||
|
/// `/etc/passwd`, `/etc/network/interfaces`, ...). Unlike a real FS, the
|
||||||
|
/// nesting is a shared prefix on the `path` metatag of several records
|
||||||
|
/// (the source PDF: "operate on CZYX records and Null-space flags rather
|
||||||
|
/// than paths and inodes") — there is no half-coordinate directory entry.
|
||||||
|
///
|
||||||
|
/// A query matches BOTH an exact leaf (`/etc/hostname`) and any descendant
|
||||||
|
/// (`/etc/hostname` and `/etc/passwd` both answer `/etc`), because a path
|
||||||
|
/// is just a string: `/etc/hostname` == dir, and `/etc/passwd` starts with
|
||||||
|
/// `dir + '/'`.
|
||||||
|
pub fn scan_by_path_prefix(&self, dir: &str) -> Vec<Czyx> {
|
||||||
|
let sep = format!("{dir}/");
|
||||||
|
let mut out: Vec<Czyx> = self
|
||||||
|
.backend
|
||||||
|
.keys()
|
||||||
|
.into_iter()
|
||||||
|
.filter(|k| {
|
||||||
|
self.get_record(k)
|
||||||
|
.map(|(h, _)| match &h.path {
|
||||||
|
Some(p) => p == dir || p.starts_with(&sep),
|
||||||
|
None => false,
|
||||||
|
})
|
||||||
|
.unwrap_or(false)
|
||||||
|
})
|
||||||
|
.collect();
|
||||||
|
out.sort();
|
||||||
|
out
|
||||||
|
}
|
||||||
|
|
||||||
|
/// `(label, header, body)` for [`CubeStore::scan_by_path`].
|
||||||
|
pub fn query_by_path(&self, path: &str) -> Vec<(Czyx, CubeHeader, Vec<u8>)> {
|
||||||
|
self.scan_by_path(path)
|
||||||
|
.into_iter()
|
||||||
|
.filter_map(|k| self.get_record(&k).map(|(h, b)| (k, h, b)))
|
||||||
|
.collect()
|
||||||
|
}
|
||||||
|
|
||||||
|
/// `(label, header, body)` for [`CubeStore::scan_by_path_prefix`].
|
||||||
|
pub fn query_by_path_prefix(&self, dir: &str) -> Vec<(Czyx, CubeHeader, Vec<u8>)> {
|
||||||
|
self.scan_by_path_prefix(dir)
|
||||||
|
.into_iter()
|
||||||
|
.filter_map(|k| self.get_record(&k).map(|(h, b)| (k, h, b)))
|
||||||
|
.collect()
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
#[cfg(test)]
|
#[cfg(test)]
|
||||||
@@ -501,6 +716,25 @@ mod tests {
|
|||||||
assert!(rh.flags.has(cubecoords::HeaderFlags::HAS_ASSOCIATIONS));
|
assert!(rh.flags.has(cubecoords::HeaderFlags::HAS_ASSOCIATIONS));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn word_flags_roundtrip() {
|
||||||
|
let mut h = CubeHeader::new();
|
||||||
|
h.word_flags = cubecoords::WordFlags::from_bits(
|
||||||
|
cubecoords::WordFlags::ENCRYPTED
|
||||||
|
| cubecoords::WordFlags::CONTINUATION
|
||||||
|
| (cubecoords::WordFlags::TYPE_MASK & 0b11),
|
||||||
|
);
|
||||||
|
let enc = super::record_codec::encode_header(&h);
|
||||||
|
let dec = super::record_codec::decode_header(&enc).unwrap();
|
||||||
|
assert_eq!(dec.word_flags.bits(), h.word_flags.bits(), "word_flags must round-trip");
|
||||||
|
// refresh_flags recomputes only the per-RECORD HeaderFlags; the separate
|
||||||
|
// per-word word_flags field must be preserved untouched.
|
||||||
|
let mut dec2 = dec;
|
||||||
|
dec2.refresh_flags();
|
||||||
|
assert_eq!(dec2.word_flags.bits(), h.word_flags.bits());
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn missing_record_is_none() {
|
fn missing_record_is_none() {
|
||||||
let store = CubeStore::new(HashBackend::new());
|
let store = CubeStore::new(HashBackend::new());
|
||||||
@@ -515,4 +749,286 @@ mod tests {
|
|||||||
assert_eq!(store.get_raw(&Czyx::new(0, 0, 0, 0)), Some(vec![1]));
|
assert_eq!(store.get_raw(&Czyx::new(0, 0, 0, 0)), Some(vec![1]));
|
||||||
assert_eq!(store.get_raw(&Czyx::new(0, 0, 0, 1)), Some(vec![2]));
|
assert_eq!(store.get_raw(&Czyx::new(0, 0, 0, 1)), Some(vec![2]));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Regression (2026-08-13, OS-in-CUBE migration): a payload written via the
|
||||||
|
/// RAW path (`put_raw`, i.e. the daemon's `rawput` verb used by OS-layer
|
||||||
|
/// services) must still be READABLE through `get_record`, because that is
|
||||||
|
/// what `cubefs` getattr/read and the `stat` verb go through. Before the
|
||||||
|
/// fix these records reported size 0 and read back empty — real bytes were
|
||||||
|
/// silently invisible through the filesystem.
|
||||||
|
#[test]
|
||||||
|
fn get_record_surfaces_raw_unenveloped_payloads() {
|
||||||
|
let mut store = CubeStore::new(HashBackend::new());
|
||||||
|
let coord = Czyx::new(200, 70, 1, 1);
|
||||||
|
let payload = b"CUBELINUX OS PROCESS SNAPSHOT\nprocs: 118\n".to_vec();
|
||||||
|
store.put_raw(coord, payload.clone());
|
||||||
|
|
||||||
|
let (hdr, body) = store
|
||||||
|
.get_record(&coord)
|
||||||
|
.expect("raw payload must be visible as a record, not vanish");
|
||||||
|
assert_eq!(body, payload, "body must round-trip byte-for-byte");
|
||||||
|
assert_eq!(
|
||||||
|
hdr.size_bytes,
|
||||||
|
Some(payload.len() as u64),
|
||||||
|
"synthesized header must report the true size so getattr is correct"
|
||||||
|
);
|
||||||
|
|
||||||
|
// A short payload (< 4 bytes, cannot even hold a length prefix) is the
|
||||||
|
// other edge the old code dropped: counters like "118" land here.
|
||||||
|
let short = Czyx::new(200, 70, 2, 1);
|
||||||
|
store.put_raw(short, b"118".to_vec());
|
||||||
|
let (h2, b2) = store.get_record(&short).expect("short raw payload visible");
|
||||||
|
assert_eq!(b2, b"118");
|
||||||
|
assert_eq!(h2.size_bytes, Some(3));
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The envelope path must be unaffected by the raw fallback: a properly
|
||||||
|
/// stored record still decodes its real header (not a synthesized one).
|
||||||
|
#[test]
|
||||||
|
fn get_record_still_prefers_the_real_envelope() {
|
||||||
|
let mut store = CubeStore::new(HashBackend::new());
|
||||||
|
let coord = Czyx::new(201, 5, 1, 1);
|
||||||
|
let mut hdr = CubeHeader::new();
|
||||||
|
hdr.title = Some("real-record".to_string());
|
||||||
|
store.put_record(coord, &hdr, b"payload");
|
||||||
|
|
||||||
|
let (got, body) = store.get_record(&coord).expect("enveloped record");
|
||||||
|
assert_eq!(body, b"payload");
|
||||||
|
assert_eq!(
|
||||||
|
got.title.as_deref(),
|
||||||
|
Some("real-record"),
|
||||||
|
"must decode the true header, not fall back to raw"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// PDF Package 2: `scan_by_flag` finds records by flag bit, independent of
|
||||||
|
/// their coordinate address. A record tagged by `DOC_TYPE` must surface
|
||||||
|
/// when queried for that flag and be absent otherwise.
|
||||||
|
#[test]
|
||||||
|
fn scan_by_flag_finds_typed_records() {
|
||||||
|
let mut store = CubeStore::new(HashBackend::new());
|
||||||
|
|
||||||
|
// A "klog" typed event record.
|
||||||
|
let mut klog = CubeHeader::new();
|
||||||
|
klog.doc_type = Some("klog".into());
|
||||||
|
klog.refresh_flags();
|
||||||
|
store.put_record(Czyx::new(200, 4, 2, 1), &klog, b"kernel: eth0 up");
|
||||||
|
|
||||||
|
// An untyped raw record (the common OS-layer case).
|
||||||
|
store.put_raw(Czyx::new(200, 70, 1, 1), b"118".to_vec());
|
||||||
|
|
||||||
|
let by_doc_type = store.scan_by_flag(cubecoords::HeaderFlags::DOC_TYPE);
|
||||||
|
assert_eq!(by_doc_type, vec![Czyx::new(200, 4, 2, 1)]);
|
||||||
|
|
||||||
|
// The raw record carries only a synthesized size flag, so it must NOT
|
||||||
|
// match DOC_TYPE.
|
||||||
|
assert!(!by_doc_type.contains(&Czyx::new(200, 70, 1, 1)));
|
||||||
|
|
||||||
|
let by_size = store.scan_by_flag(cubecoords::HeaderFlags::SIZE_BYTES);
|
||||||
|
assert!(
|
||||||
|
!by_size.is_empty(),
|
||||||
|
"synthesized raw headers carry SIZE_BYTES"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn scan_by_word_flag_finds_records() {
|
||||||
|
let mut store = CubeStore::new(HashBackend::new());
|
||||||
|
let mut h = CubeHeader::new();
|
||||||
|
h.word_flags = cubecoords::WordFlags::from_bits(cubecoords::WordFlags::ENCRYPTED);
|
||||||
|
store.put_record(Czyx::new(200, 5, 1, 1), &h, b"secret body");
|
||||||
|
|
||||||
|
let mut h2 = CubeHeader::new();
|
||||||
|
h2.doc_type = Some("plain".into());
|
||||||
|
h2.refresh_flags();
|
||||||
|
store.put_record(Czyx::new(200, 5, 2, 2), &h2, b"plain body");
|
||||||
|
|
||||||
|
let enc = store.scan_by_word_flag(cubecoords::WordFlags::ENCRYPTED);
|
||||||
|
assert_eq!(enc, vec![Czyx::new(200, 5, 1, 1)]);
|
||||||
|
assert!(!enc.contains(&Czyx::new(200, 5, 2, 2)));
|
||||||
|
}
|
||||||
|
|
||||||
|
/// `scan_by_type` is the concrete "log lookup by query" — pull every record
|
||||||
|
/// of a given event type without knowing its coordinate.
|
||||||
|
#[test]
|
||||||
|
fn scan_by_type_event_lookup() {
|
||||||
|
let mut store = CubeStore::new(HashBackend::new());
|
||||||
|
let mut a = CubeHeader::new();
|
||||||
|
a.doc_type = Some("klog".into());
|
||||||
|
let mut b = CubeHeader::new();
|
||||||
|
b.doc_type = Some("klog".into());
|
||||||
|
let mut c = CubeHeader::new();
|
||||||
|
c.doc_type = Some("state".into());
|
||||||
|
store.put_record(Czyx::new(200, 4, 2, 1), &a, b"k1");
|
||||||
|
store.put_record(Czyx::new(200, 4, 2, 2), &b, b"k2");
|
||||||
|
store.put_record(Czyx::new(200, 1, 1, 1), &c, b"s1");
|
||||||
|
|
||||||
|
let klogs = store.scan_by_type("klog");
|
||||||
|
assert_eq!(klogs.len(), 2);
|
||||||
|
assert!(klogs.contains(&Czyx::new(200, 4, 2, 1)));
|
||||||
|
assert!(klogs.contains(&Czyx::new(200, 4, 2, 2)));
|
||||||
|
|
||||||
|
// query_by_type returns the decoded payloads too.
|
||||||
|
let klogs_full = store.query_by_type("klog");
|
||||||
|
assert_eq!(klogs_full.len(), 2);
|
||||||
|
assert!(klogs_full.iter().any(|(_, _, b)| b == b"k1"));
|
||||||
|
assert!(klogs_full.iter().any(|(_, _, b)| b == b"k2"));
|
||||||
|
|
||||||
|
assert_eq!(store.scan_by_type("state"), vec![Czyx::new(200, 1, 1, 1)]);
|
||||||
|
assert!(store.scan_by_type("nonexistent").is_empty());
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Path metatags reconstruct a nested directory hierarchy WITHOUT the FS
|
||||||
|
/// needing recursive inode trees (source PDF: "operate on CZYX records and
|
||||||
|
/// Null-space flags rather than paths and inodes"). `/etc/passwd` and
|
||||||
|
/// `/etc/network/interfaces` live at unrelated coordinates but share the
|
||||||
|
/// `/etc` prefix, so `scan_by_path_prefix("/etc")` finds both.
|
||||||
|
#[test]
|
||||||
|
fn scan_by_path_prefix_lists_directory() {
|
||||||
|
let mut store = CubeStore::new(HashBackend::new());
|
||||||
|
let mut mk = |c: u8, path: &str, dt: &str| {
|
||||||
|
let mut h = CubeHeader::new();
|
||||||
|
h.path = Some(path.to_string());
|
||||||
|
h.doc_type = Some(dt.to_string());
|
||||||
|
h.refresh_flags();
|
||||||
|
store.put_record(Czyx::new(200, 1, c, 1), &h, b"body");
|
||||||
|
};
|
||||||
|
mk(1, "/etc/passwd", "file");
|
||||||
|
mk(2, "/etc/network/interfaces", "file");
|
||||||
|
mk(3, "/etc/hosts", "file");
|
||||||
|
mk(4, "/usr/bin/ls", "file");
|
||||||
|
|
||||||
|
// Exact match.
|
||||||
|
assert_eq!(
|
||||||
|
store.scan_by_path("/etc/passwd"),
|
||||||
|
vec![Czyx::new(200, 1, 1, 1)]
|
||||||
|
);
|
||||||
|
|
||||||
|
// Directory listing via metatag prefix. Bare "/etc" matches every
|
||||||
|
// record whose path is "/etc" or starts with "/etc/" (an exact leaf
|
||||||
|
// like "/etc/hostname" answers "/etc", and "/etc/passwd" is a child).
|
||||||
|
let etc = store.scan_by_path_prefix("/etc");
|
||||||
|
assert_eq!(etc.len(), 3);
|
||||||
|
assert!(etc.contains(&Czyx::new(200, 1, 1, 1)));
|
||||||
|
assert!(etc.contains(&Czyx::new(200, 1, 2, 1)));
|
||||||
|
assert!(etc.contains(&Czyx::new(200, 1, 3, 1)));
|
||||||
|
assert!(
|
||||||
|
!etc.contains(&Czyx::new(200, 1, 4, 1)),
|
||||||
|
"/usr must not appear under /etc"
|
||||||
|
);
|
||||||
|
|
||||||
|
// An exact leaf answers its own parent directory query.
|
||||||
|
assert_eq!(
|
||||||
|
store.scan_by_path_prefix("/etc/hosts"),
|
||||||
|
vec![Czyx::new(200, 1, 3, 1)]
|
||||||
|
);
|
||||||
|
|
||||||
|
// Empty prefix = all paths.
|
||||||
|
assert_eq!(store.scan_by_path_prefix("").len(), 4);
|
||||||
|
|
||||||
|
// The path flag bit is set so the record is also flag-discoverable.
|
||||||
|
let by_path_flag = store.scan_by_flag(cubecoords::HeaderFlags::HAS_PATH);
|
||||||
|
assert_eq!(by_path_flag.len(), 4);
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---- HashBackend trait-level tests (PDF spec: HashMap<u32, Vec<u8>>) ----
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn hash_backend_put_get_delete_roundtrip() {
|
||||||
|
let mut b = HashBackend::new();
|
||||||
|
let k = Czyx::new(10, 20, 30, 40);
|
||||||
|
b.put(k, b"payload".to_vec());
|
||||||
|
assert_eq!(b.get(&k), Some(b"payload".to_vec()));
|
||||||
|
b.delete(&k);
|
||||||
|
assert_eq!(b.get(&k), None);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn hash_backend_overwrite_replaces_value() {
|
||||||
|
let mut b = HashBackend::new();
|
||||||
|
let k = Czyx::new(0, 1, 2, 3);
|
||||||
|
b.put(k, b"old".to_vec());
|
||||||
|
b.put(k, b"new".to_vec());
|
||||||
|
assert_eq!(b.get(&k), Some(b"new".to_vec()));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn hash_backend_keys_sorted_order() {
|
||||||
|
let mut b = HashBackend::new();
|
||||||
|
for c in 1u8..=3 {
|
||||||
|
b.put(Czyx::new(50, c, 10, 1), vec![c]);
|
||||||
|
}
|
||||||
|
let keys = b.keys();
|
||||||
|
assert_eq!(keys.len(), 3);
|
||||||
|
// HashMap iteration is unordered; keys() must sort for stable output.
|
||||||
|
let mut sorted = keys.clone();
|
||||||
|
sorted.sort();
|
||||||
|
assert_eq!(keys, sorted);
|
||||||
|
for k in &keys {
|
||||||
|
assert_eq!(k.c, 50);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn hash_backend_scan_prefix_filters_correctly() {
|
||||||
|
let mut b = HashBackend::new();
|
||||||
|
b.put(Czyx::new(1, 2, 3, 4), b"a".to_vec());
|
||||||
|
b.put(Czyx::new(1, 9, 9, 9), b"b".to_vec());
|
||||||
|
b.put(Czyx::new(2, 0, 0, 0), b"c".to_vec());
|
||||||
|
b.put(Czyx::new(1, 2, 9, 9), b"d".to_vec());
|
||||||
|
|
||||||
|
// C=1 only.
|
||||||
|
let c1 = b.scan_prefix(1, None, None);
|
||||||
|
assert_eq!(c1.len(), 3);
|
||||||
|
for k in &c1 {
|
||||||
|
assert_eq!(k.c, 1);
|
||||||
|
}
|
||||||
|
|
||||||
|
// C=1, Z=2 only.
|
||||||
|
let c1z2 = b.scan_prefix(1, Some(2), None);
|
||||||
|
assert_eq!(c1z2.len(), 2);
|
||||||
|
for k in &c1z2 {
|
||||||
|
assert_eq!(k.c, 1);
|
||||||
|
assert_eq!(k.z, 2);
|
||||||
|
}
|
||||||
|
|
||||||
|
// C=2 only.
|
||||||
|
let c2 = b.scan_prefix(2, None, None);
|
||||||
|
assert_eq!(c2, vec![Czyx::new(2, 0, 0, 0)]);
|
||||||
|
|
||||||
|
// Non-existent C.
|
||||||
|
assert!(b.scan_prefix(99, None, None).is_empty());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn hash_backend_empty_keys_and_scan() {
|
||||||
|
let b = HashBackend::new();
|
||||||
|
assert!(b.keys().is_empty());
|
||||||
|
assert!(b.scan_prefix(0, None, None).is_empty());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn hash_backend_null_coord_is_valid_key() {
|
||||||
|
let mut b = HashBackend::new();
|
||||||
|
let null = Czyx::new(0, 0, 0, 0);
|
||||||
|
b.put(null, b"null-cube".to_vec());
|
||||||
|
assert_eq!(b.get(&null), Some(b"null-cube".to_vec()));
|
||||||
|
b.delete(&null);
|
||||||
|
assert_eq!(b.get(&null), None);
|
||||||
|
|
||||||
|
// Null cube (0,0,0,0) is distinct from a nearby coord.
|
||||||
|
let near = Czyx::new(0, 0, 0, 1);
|
||||||
|
b.put(near, b"near".to_vec());
|
||||||
|
assert_eq!(b.get(&near), Some(b"near".to_vec()));
|
||||||
|
assert_eq!(b.get(&null), None);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn hash_backend_multiple_backends_are_independent() {
|
||||||
|
let b1 = HashBackend::new();
|
||||||
|
let mut b2 = HashBackend::new();
|
||||||
|
b2.put(Czyx::new(1, 1, 1, 1), b"x".to_vec());
|
||||||
|
assert!(b1.get(&Czyx::new(1, 1, 1, 1)).is_none());
|
||||||
|
assert_eq!(b2.get(&Czyx::new(1, 1, 1, 1)), Some(b"x".to_vec()));
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -16,19 +16,3 @@ cubestore = { path = "../cubestore" }
|
|||||||
cubefs = { path = "../cubefs" }
|
cubefs = { path = "../cubefs" }
|
||||||
cubecode = { path = "../cubecode" }
|
cubecode = { path = "../cubecode" }
|
||||||
cubecrypt = { path = "../cubecrypt" }
|
cubecrypt = { path = "../cubecrypt" }
|
||||||
|
|
||||||
[[bin]]
|
|
||||||
name = "cube"
|
|
||||||
path = "src/bin/cube.rs"
|
|
||||||
|
|
||||||
[[bin]]
|
|
||||||
name = "cube-demo"
|
|
||||||
path = "src/bin/cube_demo.rs"
|
|
||||||
|
|
||||||
[[bin]]
|
|
||||||
name = "cube-server"
|
|
||||||
path = "src/bin/cube-server.rs"
|
|
||||||
|
|
||||||
[[bin]]
|
|
||||||
name = "cubec"
|
|
||||||
path = "src/bin/cubec.rs"
|
|
||||||
|
|||||||
@@ -76,9 +76,11 @@ the spec never defined.
|
|||||||
## Binaries
|
## Binaries
|
||||||
|
|
||||||
* `cube-demo` — self-contained tour (write two linked cells, run; seal a
|
* `cube-demo` — self-contained tour (write two linked cells, run; seal a
|
||||||
record, reopen + run). Prints evidence at each step.
|
record, reopen + run). Prints evidence at each step. (in `cubesys`)
|
||||||
* `cube` — CLI: `write`, `run`, `ls`, `stat`, `seal`, `open` over one store,
|
* `cube` — CLI: `write`, `run`, `ls`, `stat`, `seal`, `open` over one store,
|
||||||
interactively (`repl`), from a file (`script`), or as the demo (`demo`).
|
interactively (`repl`), from a file (`script`), or as the demo (`demo`).
|
||||||
|
Now a standalone crate (`cubecli/`); identical behavior to the former
|
||||||
|
`cubesys/src/bin/cube.rs`.
|
||||||
* `cube-server` — long-lived daemon: holds ONE `CubeStore` for its whole
|
* `cube-server` — long-lived daemon: holds ONE `CubeStore` for its whole
|
||||||
lifetime and serves the same command language over a Unix-domain socket.
|
lifetime and serves the same command language over a Unix-domain socket.
|
||||||
Snapshots the store to a JSON file on every request so the cube survives
|
Snapshots the store to a JSON file on every request so the cube survives
|
||||||
|
|||||||
+805
-18
@@ -15,7 +15,7 @@ use crate::audit::{Audit, OP_DELETE, OP_GRANT, OP_OPEN, OP_READ, OP_REVOKE, OP_S
|
|||||||
use crate::grants::{grant, grant_allows, perms_from_str, revoke, Owner, Perm};
|
use crate::grants::{grant, grant_allows, perms_from_str, revoke, Owner, Perm};
|
||||||
use crate::store::ConcurrentStore;
|
use crate::store::ConcurrentStore;
|
||||||
use crate::tenant::{TenantIdentity, TenantSession};
|
use crate::tenant::{TenantIdentity, TenantSession};
|
||||||
use cubecode::{CodeCell, Kind, Op, Vm};
|
use cubecode::{CodeCell, Kind, Op, RunResult, Vm};
|
||||||
use cubecoords::{CubeHeader, Czyx};
|
use cubecoords::{CubeHeader, Czyx};
|
||||||
use cubecrypt::{CubeEnv, KeySlot, Selector, TransformId};
|
use cubecrypt::{CubeEnv, KeySlot, Selector, TransformId};
|
||||||
use cubestore::{CubeStore, HashBackend};
|
use cubestore::{CubeStore, HashBackend};
|
||||||
@@ -448,10 +448,218 @@ impl Session {
|
|||||||
))
|
))
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
"query-path" => {
|
||||||
|
// Reconstruct a directory listing from path metatags (no FS
|
||||||
|
// inode tree needed). `query-path /etc` returns every record
|
||||||
|
// whose `path` metatag is under /etc — the PDF's "operate on
|
||||||
|
// CZYX records and Null-space flags rather than paths/inodes".
|
||||||
|
let dir = it
|
||||||
|
.next()
|
||||||
|
.ok_or_else(|| "query-path needs <dir-prefix>".to_string())?;
|
||||||
|
let matches = store.query_by_path_prefix(dir);
|
||||||
|
if matches.is_empty() {
|
||||||
|
Ok(format!("query-path {dir} -> (no matches)"))
|
||||||
|
} else {
|
||||||
|
let mut names: Vec<String> = matches
|
||||||
|
.iter()
|
||||||
|
.filter_map(|c| {
|
||||||
|
store.get_record(c).map(|(h, _)| {
|
||||||
|
let p = h.path.as_deref().unwrap_or("");
|
||||||
|
format!("{}:{}", c.pack_u32(), p)
|
||||||
|
})
|
||||||
|
})
|
||||||
|
.collect();
|
||||||
|
names.sort();
|
||||||
|
Ok(format!(
|
||||||
|
"query-path {dir} -> {} matches:\n {}",
|
||||||
|
names.len(),
|
||||||
|
names.join("\n ")
|
||||||
|
))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
"scan-word-flags" => {
|
||||||
|
let a = it
|
||||||
|
.next()
|
||||||
|
.ok_or_else(|| "scan-word-flags needs <flag> (name or 0xbits)".to_string())?;
|
||||||
|
let bits = parse_word_flag(a)?;
|
||||||
|
let coords = store.scan_by_word_flag(bits);
|
||||||
|
if coords.is_empty() {
|
||||||
|
Ok(format!("scan-word-flags {a} (0x{bits:04x}) -> (no matches)"))
|
||||||
|
} else {
|
||||||
|
let names: Vec<String> =
|
||||||
|
coords.iter().map(|c| c.pack_u32().to_string()).collect();
|
||||||
|
Ok(format!(
|
||||||
|
"scan-word-flags {a} (0x{bits:04x}) -> {} matches:\n {}",
|
||||||
|
coords.len(),
|
||||||
|
names.join("\n ")
|
||||||
|
))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
"show-flags" => {
|
||||||
|
let cs = it.next().ok_or_else(|| "show-flags needs <C.Z.Y.X>".to_string())?;
|
||||||
|
let coord = parse_coord(cs).ok_or_else(|| format!("bad coordinate '{cs}' (want C.Z.Y.X)"))?;
|
||||||
|
match store.get_record(&coord) {
|
||||||
|
None => Ok(format!("show-flags {cs} -> (no record)")),
|
||||||
|
Some((h, body)) => Ok(format!(
|
||||||
|
"show-flags {cs}\n header_flags: 0x{:04x} {}\n word_flags: 0x{:04x} {}\n size: {} bytes",
|
||||||
|
h.flags.bits(), header_flag_names(h.flags.bits()),
|
||||||
|
h.word_flags.bits(), word_flag_names(h.word_flags.bits()),
|
||||||
|
body.len()))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// Session-message / note log: a searchable, per-session, per-day
|
||||||
|
// note area stored as CZYX records carrying WordFlags metadata
|
||||||
|
// (the structured replacement for the hand-maintained RESUME-*.md
|
||||||
|
// files and the Hermes session store).
|
||||||
|
"note" | "notes" => crate::notes::note_command(store, line),
|
||||||
|
// Project threads: list projects / walk a project's associated notes
|
||||||
|
// (the first-class-association model for tracking work between sessions).
|
||||||
|
"project" | "projects" => crate::notes::project_command(store, line),
|
||||||
|
"trace-capture" => {
|
||||||
|
let src_s = it.next().ok_or_else(|| "trace-capture needs <source C.Z.Y.X> <target C.Z.Y.X>".to_string())?;
|
||||||
|
let tgt_s = it.next().ok_or_else(|| "trace-capture needs <target C.Z.Y.X>".to_string())?;
|
||||||
|
let src = parse_coord(src_s).ok_or_else(|| format!("bad source coord '{src_s}'"))?;
|
||||||
|
let tgt = parse_coord(tgt_s).ok_or_else(|| format!("bad target coord '{tgt_s}'"))?;
|
||||||
|
let snap = store.read_snapshot();
|
||||||
|
let mut vm = cubecode::Vm::new(snap);
|
||||||
|
let (result, trace) = vm.run_captured(src);
|
||||||
|
let out = vm.output().to_vec();
|
||||||
|
// Store the captured trace as a Kind::Layer AI-artifact record,
|
||||||
|
// linked back to the source cell (a linked CZYX edge).
|
||||||
|
let mut h = CubeHeader::new();
|
||||||
|
h.title = Some(format!("trace:{}", src.pack_u32()));
|
||||||
|
h.doc_type = Some(cubecode::Kind::Layer.as_str().to_string());
|
||||||
|
h.linked_records = vec![src];
|
||||||
|
h.refresh_flags();
|
||||||
|
store.put_record(tgt, &h, &cubecode::encode(&trace));
|
||||||
|
Ok(format!(
|
||||||
|
"trace-capture {src_s} -> {tgt_s} (layer, {} ops, {:?}) output {} bytes",
|
||||||
|
trace.len(),
|
||||||
|
result,
|
||||||
|
out.len()
|
||||||
|
))
|
||||||
|
}
|
||||||
|
"trace-replay" => {
|
||||||
|
let tgt_s = it.next().ok_or_else(|| "trace-replay needs <C.Z.Y.X>".to_string())?;
|
||||||
|
let tgt = parse_coord(tgt_s).ok_or_else(|| format!("bad coord '{tgt_s}'"))?;
|
||||||
|
let snap = store.read_snapshot();
|
||||||
|
match cubecode::replay_trace(&snap, tgt) {
|
||||||
|
Some((result, out)) => Ok(format!(
|
||||||
|
"trace-replay {tgt_s}: {:?}\n output ({} bytes): {}",
|
||||||
|
result,
|
||||||
|
out.len(),
|
||||||
|
String::from_utf8_lossy(&out)
|
||||||
|
)),
|
||||||
|
None => Err(format!("trace-replay {tgt_s}: no valid trace record at that coord")),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
"trace-verify" => {
|
||||||
|
let src_s = it.next().ok_or_else(|| "trace-verify needs <source> <trace> <golden>".to_string())?;
|
||||||
|
let tr_s = it.next().ok_or_else(|| "trace-verify needs <trace>".to_string())?;
|
||||||
|
let go_s = it.next().ok_or_else(|| "trace-verify needs <golden>".to_string())?;
|
||||||
|
let src = parse_coord(src_s).ok_or_else(|| format!("bad coord '{src_s}'"))?;
|
||||||
|
let tr = parse_coord(tr_s).ok_or_else(|| format!("bad coord '{tr_s}'"))?;
|
||||||
|
let go = parse_coord(go_s).ok_or_else(|| format!("bad coord '{go_s}'"))?;
|
||||||
|
let snap = store.read_snapshot();
|
||||||
|
match cubecode::verify_golden(&snap, src, tr, go) {
|
||||||
|
Ok(()) => Ok(format!(
|
||||||
|
"trace-verify {src_s} {tr_s} {go_s} -> PASS (replay + fresh run match golden)"
|
||||||
|
)),
|
||||||
|
Err(e) => Ok(format!("trace-verify {src_s} {tr_s} {go_s} -> FAIL: {e}")),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
"trace-list" => {
|
||||||
|
let snap = store.read_snapshot();
|
||||||
|
let mut rows = Vec::new();
|
||||||
|
for k in snap.keys() {
|
||||||
|
if let Some((h, _)) = snap.get_record(&k) {
|
||||||
|
let dt = h.doc_type.clone().unwrap_or_default();
|
||||||
|
let is_artifact = matches!(dt.as_str(), "layer" | "golden" | "checkpoint" | "variant")
|
||||||
|
|| h.title.as_deref().map_or(false, |t| t.starts_with("trace:") || t.starts_with("golden:"));
|
||||||
|
if is_artifact {
|
||||||
|
let links = h.linked_records
|
||||||
|
.iter().map(|c| c.pack_u32().to_string()).collect::<Vec<_>>().join(",");
|
||||||
|
rows.push(format!("{} [{}] {} -> links: {links}",
|
||||||
|
k.pack_u32(), dt, h.title.as_deref().unwrap_or("")));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if rows.is_empty() {
|
||||||
|
Ok("trace-list -> (no layer/golden/checkpoint/variant/trace records)".to_string())
|
||||||
|
} else {
|
||||||
|
Ok(format!("trace-list -> {} records:\n {}", rows.len(), rows.join("\n ")))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
"trace-links" => {
|
||||||
|
let cs = it.next().ok_or_else(|| "trace-links needs <C.Z.Y.X> [depth]".to_string())?;
|
||||||
|
let coord = parse_coord(cs).ok_or_else(|| format!("bad coord '{cs}'"))?;
|
||||||
|
let depth = it.next().and_then(|d| d.parse::<usize>().ok()).unwrap_or(3);
|
||||||
|
let snap = store.read_snapshot();
|
||||||
|
let hops = cubecode::lineage(&snap, coord, depth);
|
||||||
|
if hops.is_empty() {
|
||||||
|
return Ok(format!("trace-links {cs} -> (no record)"));
|
||||||
|
}
|
||||||
|
let mut lines = Vec::new();
|
||||||
|
for h in &hops {
|
||||||
|
let links = h.links.iter().map(|c| c.pack_u32().to_string()).collect::<Vec<_>>().join(" ");
|
||||||
|
lines.push(format!(" {} [{}] {} -> {links}", h.coord.pack_u32(), h.doc_type, h.title));
|
||||||
|
}
|
||||||
|
Ok(format!("trace-links {cs} (depth {depth}) -> {} hops:\n{}", hops.len(), lines.join("\n")))
|
||||||
|
}
|
||||||
|
"golden-capture" => {
|
||||||
|
let src_s = it.next().ok_or_else(|| "golden-capture needs <source> <trace> <golden>".to_string())?;
|
||||||
|
let tr_s = it.next().ok_or_else(|| "golden-capture needs <trace>".to_string())?;
|
||||||
|
let go_s = it.next().ok_or_else(|| "golden-capture needs <golden>".to_string())?;
|
||||||
|
let name = it.next().unwrap_or("golden").to_string();
|
||||||
|
let src = parse_coord(src_s).ok_or_else(|| format!("bad coord '{src_s}'"))?;
|
||||||
|
let tr = parse_coord(tr_s).ok_or_else(|| format!("bad coord '{tr_s}'"))?;
|
||||||
|
let go = parse_coord(go_s).ok_or_else(|| format!("bad coord '{go_s}'"))?;
|
||||||
|
let snap = store.read_snapshot();
|
||||||
|
let mut vm = cubecode::Vm::new(snap);
|
||||||
|
let (_r, trace) = vm.run_captured(src);
|
||||||
|
let out = vm.output().to_vec();
|
||||||
|
// Trace record: Kind::Layer, linked to the source.
|
||||||
|
let mut h = CubeHeader::new();
|
||||||
|
h.title = Some(format!("trace:{name}"));
|
||||||
|
h.doc_type = Some(cubecode::Kind::Layer.as_str().to_string());
|
||||||
|
h.linked_records = vec![src];
|
||||||
|
h.refresh_flags();
|
||||||
|
store.put_record(tr, &h, &cubecode::encode(&trace));
|
||||||
|
// Golden output record, linked to the trace.
|
||||||
|
let mut hg = CubeHeader::new();
|
||||||
|
hg.title = Some(format!("golden:{name}"));
|
||||||
|
hg.doc_type = Some("golden".into());
|
||||||
|
hg.linked_records = vec![tr];
|
||||||
|
hg.refresh_flags();
|
||||||
|
store.put_record(go, &hg, &out);
|
||||||
|
Ok(format!(
|
||||||
|
"golden-capture {src_s} -> {tr_s} (layer) + {go_s} (golden), output {} bytes",
|
||||||
|
out.len()
|
||||||
|
))
|
||||||
|
}
|
||||||
"prog" => {
|
"prog" => {
|
||||||
let path = it.next().ok_or_else(|| "prog needs <path>".to_string())?;
|
let path = it.next().ok_or_else(|| "prog needs <path>".to_string())?;
|
||||||
let mut ops: Vec<Op> = Vec::new();
|
let mut ops: Vec<Op> = Vec::new();
|
||||||
|
// Optional behavior-descriptor flags: `K=pure` `K=io` `K=hot`.
|
||||||
|
// These stamp the OS-kernel behavior bits (PDF §524–525) on the
|
||||||
|
// record header and switch the kind to `kernel` so the cube
|
||||||
|
// correctly classifies operator kernels vs plain functions.
|
||||||
|
let mut descriptor: Option<cubecode::Behavior> = None;
|
||||||
while let Some(tok) = it.next() {
|
while let Some(tok) = it.next() {
|
||||||
|
if let Some(flag) = tok.strip_prefix("K=") {
|
||||||
|
let bit = match flag {
|
||||||
|
"pure" => cubecode::Behavior::PURE,
|
||||||
|
"io" => cubecode::Behavior::IO_HEAVY,
|
||||||
|
"alloc" => cubecode::Behavior::ALLOCATES,
|
||||||
|
"net" => cubecode::Behavior::NETWORK,
|
||||||
|
"sec" => cubecode::Behavior::SECURITY_SENSITIVE,
|
||||||
|
"hot" => cubecode::Behavior::HOT_PATH,
|
||||||
|
other => return Err(format!("prog: unknown descriptor K={other}")),
|
||||||
|
};
|
||||||
|
let cur = descriptor.unwrap_or_default();
|
||||||
|
descriptor = Some(cubecode::Behavior(cur.0 | bit));
|
||||||
|
continue;
|
||||||
|
}
|
||||||
let arg = if takes_arg(tok) {
|
let arg = if takes_arg(tok) {
|
||||||
it.next()
|
it.next()
|
||||||
.and_then(|a| a.parse::<u8>().ok())
|
.and_then(|a| a.parse::<u8>().ok())
|
||||||
@@ -464,11 +672,13 @@ impl Session {
|
|||||||
if ops.is_empty() {
|
if ops.is_empty() {
|
||||||
return Err("prog: no ops given".to_string());
|
return Err("prog: no ops given".to_string());
|
||||||
}
|
}
|
||||||
|
let is_kernel = descriptor.is_some();
|
||||||
|
let kind = if is_kernel { Kind::Kernel } else { Kind::Fn };
|
||||||
let name = path.rsplit('/').next().unwrap_or(path);
|
let name = path.rsplit('/').next().unwrap_or(path);
|
||||||
// Compute the exact record bytes `put_record` would write, using
|
// Compute the exact record bytes `put_record` would write, using
|
||||||
// a throwaway store so we can buffer (or apply) them without
|
// a throwaway store so we can buffer (or apply) them without
|
||||||
// duplicating the record codec.
|
// duplicating the record codec.
|
||||||
let coord = scratch_code_coord(path, Kind::Fn, name, &ops)?;
|
let coord = scratch_code_coord(path, kind, name, &ops)?;
|
||||||
// Task 6: reject overwriting a record owned by a different owner.
|
// Task 6: reject overwriting a record owned by a different owner.
|
||||||
if let Some(msg) = self.admit_mutate(coord, Perm::Write) {
|
if let Some(msg) = self.admit_mutate(coord, Perm::Write) {
|
||||||
self.audit_now(OP_WRITE, coord, false);
|
self.audit_now(OP_WRITE, coord, false);
|
||||||
@@ -478,28 +688,341 @@ impl Session {
|
|||||||
let owner = self.identity.as_ref().map(|i| i.owner_local.as_str());
|
let owner = self.identity.as_ref().map(|i| i.owner_local.as_str());
|
||||||
let value = {
|
let value = {
|
||||||
let mut scratch = CubeStore::new(HashBackend::new());
|
let mut scratch = CubeStore::new(HashBackend::new());
|
||||||
crate::store_code_cell(&mut scratch, path, Kind::Fn, name, &[], &ops, owner)
|
crate::store_code_cell(
|
||||||
.map_err(|e| e.to_string())?;
|
&mut scratch,
|
||||||
|
path,
|
||||||
|
kind,
|
||||||
|
name,
|
||||||
|
&[],
|
||||||
|
&ops,
|
||||||
|
owner,
|
||||||
|
descriptor,
|
||||||
|
)
|
||||||
|
.map_err(|e| e.to_string())?;
|
||||||
scratch.get_raw(&coord).unwrap_or_default()
|
scratch.get_raw(&coord).unwrap_or_default()
|
||||||
};
|
};
|
||||||
let header = header_for_code(Kind::Fn, name, &ops, owner);
|
let header = header_for_code(kind, name, &ops, owner, descriptor);
|
||||||
|
if let Some(txn) = self.txn.as_mut() {
|
||||||
|
txn.ops.push(TxnOp {
|
||||||
|
coord,
|
||||||
|
put: Some((value, header)),
|
||||||
|
});
|
||||||
|
let kind_tag = if is_kernel { "kernel" } else { "fn" };
|
||||||
|
return Ok(format!(
|
||||||
|
"buffered prog {path} ({kind_tag}, {} ops) — commit to apply",
|
||||||
|
ops.len()
|
||||||
|
));
|
||||||
|
}
|
||||||
|
let coord = store
|
||||||
|
.put_code_cell(path, kind, name, &[], &ops, owner, descriptor)
|
||||||
|
.map_err(|e| e.to_string())?;
|
||||||
|
let kind_tag = if is_kernel { "kernel" } else { "fn" };
|
||||||
|
Ok(format!(
|
||||||
|
"wrote program {path} -> coord {} ({kind_tag}, {} ops)",
|
||||||
|
coord.pack_u32(),
|
||||||
|
ops.len()
|
||||||
|
))
|
||||||
|
}
|
||||||
|
"link" => {
|
||||||
|
// Attach a callee coordinate to an existing code cell's call
|
||||||
|
// graph. `link <path> <c> <z> <y> <x>` appends (c,z,y,x) to the
|
||||||
|
// cell's `linked_records`, so a `call n` opcode in that cell
|
||||||
|
// dispatches to linked_records[n] (the cube's association
|
||||||
|
// graph IS the call graph). This is what makes functions stored
|
||||||
|
// in CUBE callable from other functions stored in CUBE.
|
||||||
|
let path = it.next().ok_or_else(|| "link needs <path>".to_string())?;
|
||||||
|
let c = parse_u8(it.next(), "link needs <c>")?;
|
||||||
|
let z = parse_u8(it.next(), "link needs <z>")?;
|
||||||
|
let y = parse_u8(it.next(), "link needs <y>")?;
|
||||||
|
let x = parse_u8(it.next(), "link needs <x>")?;
|
||||||
|
let target = Czyx::new(c, z, y, x);
|
||||||
|
let coord = crate::path_to_czyx(path).map_err(|e| e.to_string())?;
|
||||||
|
if let Some(msg) = self.admit_mutate(coord, Perm::Write) {
|
||||||
|
self.audit_now(OP_WRITE, coord, false);
|
||||||
|
return Err(msg);
|
||||||
|
}
|
||||||
|
self.audit_now(OP_WRITE, coord, true);
|
||||||
|
// Load the existing cell, append the link, re-store it.
|
||||||
|
let cell = crate::load_code_cell(&store.read_snapshot(), path)
|
||||||
|
.map_err(|e| format!("link: cannot load {path}: {e:?}"))?;
|
||||||
|
let mut links = cell.links().to_vec();
|
||||||
|
if links.contains(&target) {
|
||||||
|
return Ok(format!(
|
||||||
|
"link {path}: {target:?} already linked (degree {})",
|
||||||
|
links.len()
|
||||||
|
));
|
||||||
|
}
|
||||||
|
links.push(target);
|
||||||
|
let code = cubecode::decode(&cell.body())
|
||||||
|
.map_err(|e| format!("link: bad bytecode in {path}: {e:?}"))?;
|
||||||
|
let owner = self.identity.as_ref().map(|i| i.owner_local.as_str());
|
||||||
|
store
|
||||||
|
.put_code_cell(
|
||||||
|
path,
|
||||||
|
cell.kind(),
|
||||||
|
cell.name().unwrap_or(path),
|
||||||
|
&links,
|
||||||
|
&code,
|
||||||
|
owner,
|
||||||
|
None,
|
||||||
|
)
|
||||||
|
.map_err(|e| e.to_string())?;
|
||||||
|
Ok(format!(
|
||||||
|
"linked {path} -> {target:?} (call-graph degree now {})",
|
||||||
|
links.len()
|
||||||
|
))
|
||||||
|
}
|
||||||
|
// ---- OS-operator-kernel authoring + thin effector (Steps 1 & 2) ----
|
||||||
|
// Per the reframe, the OS's *computation* lives in CUBE as operator
|
||||||
|
// kernels (call graphs + behavior descriptors, PDF §524–525). The
|
||||||
|
// native OS layer is only a thin effector: it reads a kernel's
|
||||||
|
// computed result and applies the effect. The cubevm never does
|
||||||
|
// store-IO itself. `kernel` authors a kernel; `tick` lays down the
|
||||||
|
// OS kernel call-graph; `native-apply` is the effector boundary.
|
||||||
|
"kernel" => {
|
||||||
|
// `kernel <path> [K=pure|io|hot ...] <op> <arg> ...`
|
||||||
|
// Like `prog`, but the cell is always kind=Kernel and accepts
|
||||||
|
// behavior-descriptor flags so the OS marks operator kernels
|
||||||
|
// distinctly from plain functions.
|
||||||
|
let path = it.next().ok_or_else(|| "kernel needs <path>".to_string())?;
|
||||||
|
let mut ops: Vec<Op> = Vec::new();
|
||||||
|
let mut descriptor: Option<cubecode::Behavior> = None;
|
||||||
|
while let Some(tok) = it.next() {
|
||||||
|
if let Some(flag) = tok.strip_prefix("K=") {
|
||||||
|
let bit = match flag {
|
||||||
|
"pure" => cubecode::Behavior::PURE,
|
||||||
|
"io" => cubecode::Behavior::IO_HEAVY,
|
||||||
|
"alloc" => cubecode::Behavior::ALLOCATES,
|
||||||
|
"net" => cubecode::Behavior::NETWORK,
|
||||||
|
"sec" => cubecode::Behavior::SECURITY_SENSITIVE,
|
||||||
|
"hot" => cubecode::Behavior::HOT_PATH,
|
||||||
|
other => return Err(format!("kernel: unknown descriptor K={other}")),
|
||||||
|
};
|
||||||
|
let cur = descriptor.unwrap_or_default();
|
||||||
|
descriptor = Some(cubecode::Behavior(cur.0 | bit));
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
let arg = if takes_arg(tok) {
|
||||||
|
it.next()
|
||||||
|
.and_then(|a| a.parse::<u8>().ok())
|
||||||
|
.ok_or_else(|| format!("kernel: {tok} needs a u8 argument"))?
|
||||||
|
} else {
|
||||||
|
0
|
||||||
|
};
|
||||||
|
ops.push(make_op(tok, arg)?);
|
||||||
|
}
|
||||||
|
if ops.is_empty() {
|
||||||
|
return Err("kernel: no ops given".to_string());
|
||||||
|
}
|
||||||
|
let name = path.rsplit('/').next().unwrap_or(path);
|
||||||
|
let coord = scratch_code_coord(path, Kind::Kernel, name, &ops)?;
|
||||||
|
if let Some(msg) = self.admit_mutate(coord, Perm::Write) {
|
||||||
|
self.audit_now(OP_WRITE, coord, false);
|
||||||
|
return Err(msg);
|
||||||
|
}
|
||||||
|
self.audit_now(OP_WRITE, coord, true);
|
||||||
|
let owner = self.identity.as_ref().map(|i| i.owner_local.as_str());
|
||||||
|
let value = {
|
||||||
|
let mut scratch = CubeStore::new(HashBackend::new());
|
||||||
|
crate::store_code_cell(
|
||||||
|
&mut scratch,
|
||||||
|
path,
|
||||||
|
Kind::Kernel,
|
||||||
|
name,
|
||||||
|
&[],
|
||||||
|
&ops,
|
||||||
|
owner,
|
||||||
|
descriptor,
|
||||||
|
)
|
||||||
|
.map_err(|e| e.to_string())?;
|
||||||
|
scratch.get_raw(&coord).unwrap_or_default()
|
||||||
|
};
|
||||||
|
let header = header_for_code(Kind::Kernel, name, &ops, owner, descriptor);
|
||||||
if let Some(txn) = self.txn.as_mut() {
|
if let Some(txn) = self.txn.as_mut() {
|
||||||
txn.ops.push(TxnOp {
|
txn.ops.push(TxnOp {
|
||||||
coord,
|
coord,
|
||||||
put: Some((value, header)),
|
put: Some((value, header)),
|
||||||
});
|
});
|
||||||
return Ok(format!(
|
return Ok(format!(
|
||||||
"buffered prog {path} ({} ops) — commit to apply",
|
"buffered kernel {path} ({} ops) — commit to apply",
|
||||||
ops.len()
|
ops.len()
|
||||||
));
|
));
|
||||||
}
|
}
|
||||||
let coord = store
|
let coord = store
|
||||||
.put_code_cell(path, Kind::Fn, name, &[], &ops, owner)
|
.put_code_cell(path, Kind::Kernel, name, &[], &ops, owner, descriptor)
|
||||||
.map_err(|e| e.to_string())?;
|
.map_err(|e| e.to_string())?;
|
||||||
Ok(format!(
|
Ok(format!(
|
||||||
"wrote program {path} -> coord {} ({} ops)",
|
"wrote kernel {path} -> coord {} ({} ops, descriptors={:?})",
|
||||||
coord.pack_u32(),
|
coord.pack_u32(),
|
||||||
ops.len()
|
ops.len(),
|
||||||
|
descriptor.map(|b| b.tags()).unwrap_or_default()
|
||||||
|
))
|
||||||
|
}
|
||||||
|
"tick" => {
|
||||||
|
// Lay down the OS operator-kernel call graph (Step 1). Each leaf
|
||||||
|
// is a real CUBE kernel composed via `linked_records` (the call
|
||||||
|
// graph) and tagged with behavior descriptors. `cube-os-tick`
|
||||||
|
// calls the three leaves in order. Nothing is executed here —
|
||||||
|
// the native layer later `run`s each and `native-apply`s the
|
||||||
|
// effect. This is "everything in CUBE" done the spec's way:
|
||||||
|
// the OS's behavior lives as kernels + call graph + descriptors.
|
||||||
|
let c = cubecode::C_OS_KERNEL;
|
||||||
|
let cfg = format!("/c{c}/z001/y001/x001"); // normalize-config: pure
|
||||||
|
let decide = format!("/c{c}/z001/y001/x002"); // decide-snapshot: io
|
||||||
|
let summarize = format!("/c{c}/z001/y001/x003"); // summarize-procs: pure+hot
|
||||||
|
let tick = format!("/c{c}/z001/y001/x004"); // cube-os-tick: hot (calls 0..2)
|
||||||
|
|
||||||
|
let cfg_code = vec![Op::Const(7), Op::Const(3), Op::Add, Op::Halt]; // 7+3=10
|
||||||
|
let decide_code = vec![Op::Const(1), Op::Ret]; // 1 => snapshot
|
||||||
|
let summarize_code = vec![Op::Const(20), Op::Halt]; // 20 procs
|
||||||
|
let tick_code = vec![
|
||||||
|
Op::Const(0),
|
||||||
|
Op::CallLink(0), // cfg
|
||||||
|
Op::CallLink(1), // decide
|
||||||
|
Op::CallLink(2), // summarize
|
||||||
|
Op::Halt,
|
||||||
|
];
|
||||||
|
|
||||||
|
// Stage the leaves first (we need their coords to build the
|
||||||
|
// call-graph edges of the root), then the root. `with_mut`
|
||||||
|
// hands us exclusive `&mut CubeStore` access so the helper can
|
||||||
|
// write each record through the normal codec.
|
||||||
|
let cfg_c = store
|
||||||
|
.with_mut(|s| {
|
||||||
|
crate::store_code_cell(
|
||||||
|
s,
|
||||||
|
&cfg,
|
||||||
|
Kind::Kernel,
|
||||||
|
"normalize-config",
|
||||||
|
&[],
|
||||||
|
&cfg_code,
|
||||||
|
None,
|
||||||
|
Some(cubecode::Behavior(cubecode::Behavior::PURE)),
|
||||||
|
)
|
||||||
|
})
|
||||||
|
.map_err(|e: crate::SysError| e.to_string())?;
|
||||||
|
let dec_c = store
|
||||||
|
.with_mut(|s| {
|
||||||
|
crate::store_code_cell(
|
||||||
|
s,
|
||||||
|
&decide,
|
||||||
|
Kind::Kernel,
|
||||||
|
"decide-snapshot",
|
||||||
|
&[],
|
||||||
|
&decide_code,
|
||||||
|
None,
|
||||||
|
Some(cubecode::Behavior(cubecode::Behavior::IO_HEAVY)),
|
||||||
|
)
|
||||||
|
})
|
||||||
|
.map_err(|e: crate::SysError| e.to_string())?;
|
||||||
|
let sum_c = store
|
||||||
|
.with_mut(|s| {
|
||||||
|
crate::store_code_cell(
|
||||||
|
s,
|
||||||
|
&summarize,
|
||||||
|
Kind::Kernel,
|
||||||
|
"summarize-procs",
|
||||||
|
&[],
|
||||||
|
&summarize_code,
|
||||||
|
None,
|
||||||
|
Some(cubecode::Behavior(
|
||||||
|
cubecode::Behavior::PURE | cubecode::Behavior::HOT_PATH,
|
||||||
|
)),
|
||||||
|
)
|
||||||
|
})
|
||||||
|
.map_err(|e: crate::SysError| e.to_string())?;
|
||||||
|
let tick_c = store
|
||||||
|
.with_mut(|s| {
|
||||||
|
crate::store_code_cell(
|
||||||
|
s,
|
||||||
|
&tick,
|
||||||
|
Kind::Kernel,
|
||||||
|
"cube-os-tick",
|
||||||
|
&[cfg_c, dec_c, sum_c], // call graph: tick -> {cfg,decide,summarize}
|
||||||
|
&tick_code,
|
||||||
|
None,
|
||||||
|
Some(cubecode::Behavior(cubecode::Behavior::HOT_PATH)),
|
||||||
|
)
|
||||||
|
})
|
||||||
|
.map_err(|e: crate::SysError| e.to_string())?;
|
||||||
|
|
||||||
|
Ok(format!(
|
||||||
|
"OS kernel call-graph laid down (kind=kernel, in cube c{c}):\n {} normalize-config [pure] -> coord {}\n {} decide-snapshot [io] -> coord {}\n {} summarize-procs [pure,hot] -> coord {}\n {} cube-os-tick [hot] -> coord {} (links cfg,decide,summarize)\nrun e.g.: run {}\nthen effector: native-apply {}",
|
||||||
|
cfg, cfg_c.pack_u32(), decide, dec_c.pack_u32(), summarize,
|
||||||
|
sum_c.pack_u32(), tick, tick_c.pack_u32(), tick_c.pack_u32(),
|
||||||
|
tick
|
||||||
|
))
|
||||||
|
}
|
||||||
|
"native-apply" => {
|
||||||
|
// Step 2: the thin effector. The decision/computation already
|
||||||
|
let path = it
|
||||||
|
.next()
|
||||||
|
.ok_or_else(|| "native-apply needs <kernel-path>".to_string())?;
|
||||||
|
let cell = crate::load_code_cell(&store.read_snapshot(), path)
|
||||||
|
.map_err(|e| format!("native-apply: cannot load {path}: {e}"))?;
|
||||||
|
if cell.kind() != Kind::Kernel {
|
||||||
|
return Err(format!(
|
||||||
|
"native-apply: {path} is kind={:?}, expected a kernel",
|
||||||
|
cell.kind()
|
||||||
|
));
|
||||||
|
}
|
||||||
|
// Build a throwaway in-memory store holding just this kernel
|
||||||
|
// (the VM needs a store to walk), then run it. The computation
|
||||||
|
// is entirely in CUBE; we only read back the computed result.
|
||||||
|
let coord = crate::path_to_czyx(path).map_err(|e| e.to_string())?;
|
||||||
|
let mut vm_store = cubestore::CubeStore::new(cubestore::HashBackend::new());
|
||||||
|
crate::store_code_cell(
|
||||||
|
&mut vm_store,
|
||||||
|
path,
|
||||||
|
Kind::Kernel,
|
||||||
|
cell.name().unwrap_or(path),
|
||||||
|
cell.links(),
|
||||||
|
&cell.code,
|
||||||
|
None,
|
||||||
|
None,
|
||||||
|
)
|
||||||
|
.map_err(|e| format!("native-apply: stage failed: {e}"))?;
|
||||||
|
let mut vm = Vm::new(vm_store);
|
||||||
|
let result = match vm.run(coord) {
|
||||||
|
RunResult::Halted { top } => top,
|
||||||
|
other => {
|
||||||
|
return Err(format!(
|
||||||
|
"native-apply: kernel did not halt cleanly: {other:?}"
|
||||||
|
))
|
||||||
|
}
|
||||||
|
};
|
||||||
|
let r = result.unwrap_or(0);
|
||||||
|
let effect = match cell.name() {
|
||||||
|
Some("decide-snapshot") => {
|
||||||
|
if r != 0 {
|
||||||
|
format!(
|
||||||
|
"OS EFFECT: persist runtime snapshot into CUBE (c{} band); decision kernel returned {} => snapshot NOW",
|
||||||
|
cubecode::C_OS_EFFECT, r
|
||||||
|
)
|
||||||
|
} else {
|
||||||
|
"OS EFFECT: no snapshot (decision kernel returned 0)".to_string()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
_ => format!(
|
||||||
|
"OS EFFECT: apply computed result {} from kernel {}@{}",
|
||||||
|
r,
|
||||||
|
cell.name().unwrap_or("?"),
|
||||||
|
path
|
||||||
|
),
|
||||||
|
};
|
||||||
|
Ok(format!(
|
||||||
|
"native-apply {} (kind=kernel, descriptors={:?}):\n computed result = {}\n {}",
|
||||||
|
path,
|
||||||
|
cubecode::Behavior::from_flags(
|
||||||
|
store
|
||||||
|
.read_snapshot()
|
||||||
|
.get_record(&coord)
|
||||||
|
.map(|(h, _)| h.flags.bits())
|
||||||
|
.unwrap_or(0)
|
||||||
|
)
|
||||||
|
.tags(),
|
||||||
|
r,
|
||||||
|
effect
|
||||||
))
|
))
|
||||||
}
|
}
|
||||||
"write" => {
|
"write" => {
|
||||||
@@ -520,11 +1043,20 @@ impl Session {
|
|||||||
let owner = self.identity.as_ref().map(|i| i.owner_local.as_str());
|
let owner = self.identity.as_ref().map(|i| i.owner_local.as_str());
|
||||||
let value = {
|
let value = {
|
||||||
let mut scratch = CubeStore::new(HashBackend::new());
|
let mut scratch = CubeStore::new(HashBackend::new());
|
||||||
crate::store_code_cell(&mut scratch, path, Kind::Fn, name, &[], &code, owner)
|
crate::store_code_cell(
|
||||||
.map_err(|e| e.to_string())?;
|
&mut scratch,
|
||||||
|
path,
|
||||||
|
Kind::Fn,
|
||||||
|
name,
|
||||||
|
&[],
|
||||||
|
&code,
|
||||||
|
owner,
|
||||||
|
None,
|
||||||
|
)
|
||||||
|
.map_err(|e| e.to_string())?;
|
||||||
scratch.get_raw(&coord).unwrap_or_default()
|
scratch.get_raw(&coord).unwrap_or_default()
|
||||||
};
|
};
|
||||||
let header = header_for_code(Kind::Fn, name, &code, owner);
|
let header = header_for_code(Kind::Fn, name, &code, owner, None);
|
||||||
if let Some(txn) = self.txn.as_mut() {
|
if let Some(txn) = self.txn.as_mut() {
|
||||||
txn.ops.push(TxnOp {
|
txn.ops.push(TxnOp {
|
||||||
coord,
|
coord,
|
||||||
@@ -536,7 +1068,7 @@ impl Session {
|
|||||||
));
|
));
|
||||||
}
|
}
|
||||||
let coord = store
|
let coord = store
|
||||||
.put_code_cell(path, Kind::Fn, name, &[], &code, owner)
|
.put_code_cell(path, Kind::Fn, name, &[], &code, owner, None)
|
||||||
.map_err(|e| e.to_string())?;
|
.map_err(|e| e.to_string())?;
|
||||||
Ok(format!("wrote {path} -> coord {}", coord.pack_u32()))
|
Ok(format!("wrote {path} -> coord {}", coord.pack_u32()))
|
||||||
}
|
}
|
||||||
@@ -596,6 +1128,66 @@ impl Session {
|
|||||||
store.delete_raw(&coord);
|
store.delete_raw(&coord);
|
||||||
Ok(format!("ok: deleted {}", coord.pack_u32()))
|
Ok(format!("ok: deleted {}", coord.pack_u32()))
|
||||||
}
|
}
|
||||||
|
// --- Enveloped coordinate API (PDF Package 2: `put(C,Z,Y,X,
|
||||||
|
// bytes, flags)`). Unlike `rawput`, this writes a full record
|
||||||
|
// with a `CubeHeader`, so the record carries metatags
|
||||||
|
// (doc_type / title) that the associative `query` verb can
|
||||||
|
// target by event type. `rawput` stays for bulk/append payloads
|
||||||
|
// (e.g. the rolling kernel-log window) that do not need header
|
||||||
|
// metadata; `put` is for addressed, discoverable records. ---
|
||||||
|
"put" => {
|
||||||
|
let c = parse_u8(it.next(), "put needs <c>")?;
|
||||||
|
let z = parse_u8(it.next(), "put needs <z>")?;
|
||||||
|
let y = parse_u8(it.next(), "put needs <y>")?;
|
||||||
|
let x = parse_u8(it.next(), "put needs <x>")?;
|
||||||
|
let hex = it
|
||||||
|
.next()
|
||||||
|
.ok_or_else(|| "put needs <hex-bytes>".to_string())?;
|
||||||
|
let val = hex_decode(hex).ok_or_else(|| "put: value must be hex".to_string())?;
|
||||||
|
let coord = Czyx::new(c, z, y, x);
|
||||||
|
// Optional metatags: `doc_type=<t>` and/or `title=<t>` may
|
||||||
|
// follow. These are what `query <doc_type>` matches on.
|
||||||
|
let mut header = CubeHeader::new();
|
||||||
|
for tok in it {
|
||||||
|
if let Some(v) = tok.strip_prefix("doc_type=") {
|
||||||
|
header.doc_type = Some(v.to_string());
|
||||||
|
} else if let Some(v) = tok.strip_prefix("title=") {
|
||||||
|
header.title = Some(v.to_string());
|
||||||
|
} else if let Some(v) = tok.strip_prefix("path=") {
|
||||||
|
// POSIX path metatag: the original filesystem path.
|
||||||
|
// Lets scan_by_path_prefix reconstruct nesting from flags.
|
||||||
|
header.path = Some(v.to_string());
|
||||||
|
} else {
|
||||||
|
return Err(format!("put: unknown option {tok}"));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
header.created_at = Some(
|
||||||
|
std::time::SystemTime::now()
|
||||||
|
.duration_since(std::time::UNIX_EPOCH)
|
||||||
|
.map(|d| d.as_secs())
|
||||||
|
.unwrap_or(0),
|
||||||
|
);
|
||||||
|
header.refresh_flags();
|
||||||
|
let doc_type_dbg = header.doc_type.clone();
|
||||||
|
if let Some(txn) = self.txn.as_mut() {
|
||||||
|
txn.ops.push(TxnOp {
|
||||||
|
coord,
|
||||||
|
put: Some((val, header)),
|
||||||
|
});
|
||||||
|
Ok(format!(
|
||||||
|
"buffered put {} (doc_type={:?}) — commit to apply",
|
||||||
|
coord.pack_u32(),
|
||||||
|
doc_type_dbg
|
||||||
|
))
|
||||||
|
} else {
|
||||||
|
store.put_record(coord, &header, &val);
|
||||||
|
Ok(format!(
|
||||||
|
"ok: wrote {} (doc_type={:?})",
|
||||||
|
coord.pack_u32(),
|
||||||
|
doc_type_dbg
|
||||||
|
))
|
||||||
|
}
|
||||||
|
}
|
||||||
"rawkeys" => {
|
"rawkeys" => {
|
||||||
let ks: Vec<String> = store
|
let ks: Vec<String> = store
|
||||||
.keys()
|
.keys()
|
||||||
@@ -752,7 +1344,20 @@ impl Session {
|
|||||||
let dir = it.next().ok_or_else(|| "ls needs <dir>".to_string())?;
|
let dir = it.next().ok_or_else(|| "ls needs <dir>".to_string())?;
|
||||||
// R5: directory reads are metadata reads — honor the read gate
|
// R5: directory reads are metadata reads — honor the read gate
|
||||||
// so an attacker can't enumerate a victim's records by name.
|
// so an attacker can't enumerate a victim's records by name.
|
||||||
let dir_coord = crate::path_to_czyx(dir).map_err(|e| e.to_string())?;
|
// Use parse_path (not path_to_czyx) because ls targets a
|
||||||
|
// directory prefix, which path_to_czyx rejects as "not a record".
|
||||||
|
let parsed = cubefs::path::parse_path(dir)
|
||||||
|
.map_err(|e| format!("ls: bad path {dir}: {e}"))?;
|
||||||
|
// The ACL gate wants the directory's prefix coordinate
|
||||||
|
// (trailing axes zeroed) — same key the NullSpace acl_bucket
|
||||||
|
// uses for directory ACLs.
|
||||||
|
let dir_coord = match parsed.axes.len() {
|
||||||
|
0 => Czyx::new(0, 0, 0, 0), // root
|
||||||
|
1 => Czyx::new(parsed.axes[0], 0, 0, 0),
|
||||||
|
2 => Czyx::new(parsed.axes[0], parsed.axes[1], 0, 0),
|
||||||
|
3 => Czyx::new(parsed.axes[0], parsed.axes[1], parsed.axes[2], 0),
|
||||||
|
_ => return Err("ls: path too deep".to_string()),
|
||||||
|
};
|
||||||
if let Some(msg) = self.admit_read(dir_coord) {
|
if let Some(msg) = self.admit_read(dir_coord) {
|
||||||
self.audit_now(OP_READ, dir_coord, false);
|
self.audit_now(OP_READ, dir_coord, false);
|
||||||
return Err(msg);
|
return Err(msg);
|
||||||
@@ -783,8 +1388,15 @@ impl Session {
|
|||||||
.getattr(path)
|
.getattr(path)
|
||||||
.map_err(|e| format!("stat {path}: {e:?}"))?;
|
.map_err(|e| format!("stat {path}: {e:?}"))?;
|
||||||
Ok(format!(
|
Ok(format!(
|
||||||
"stat {path} -> ino={} kind={:?} size={} mode={:o}",
|
"stat {path} -> ino={} kind={:?} size={} mode={:o} word_flags=0x{:04x} {} header_flags=0x{:04x} {}",
|
||||||
a.ino, a.kind, a.size, a.mode
|
a.ino,
|
||||||
|
a.kind,
|
||||||
|
a.size,
|
||||||
|
a.mode,
|
||||||
|
a.word_flags,
|
||||||
|
word_flag_names(a.word_flags),
|
||||||
|
a.header_flags,
|
||||||
|
header_flag_names(a.header_flags)
|
||||||
))
|
))
|
||||||
}
|
}
|
||||||
"keyinit" => {
|
"keyinit" => {
|
||||||
@@ -952,19 +1564,31 @@ pub fn txn_snapshot(s: &Session) -> CubeStore<HashBackend> {
|
|||||||
/// writing — used to buffer `prog`/`write` mutations during a transaction.
|
/// writing — used to buffer `prog`/`write` mutations during a transaction.
|
||||||
fn scratch_code_coord(path: &str, kind: Kind, name: &str, code: &[Op]) -> Result<Czyx, String> {
|
fn scratch_code_coord(path: &str, kind: Kind, name: &str, code: &[Op]) -> Result<Czyx, String> {
|
||||||
let mut scratch = CubeStore::new(HashBackend::new());
|
let mut scratch = CubeStore::new(HashBackend::new());
|
||||||
crate::store_code_cell(&mut scratch, path, kind, name, &[], code, None)
|
crate::store_code_cell(&mut scratch, path, kind, name, &[], code, None, None)
|
||||||
.map_err(|e| e.to_string())
|
.map_err(|e| e.to_string())
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Build the `CubeHeader` a `store_code_cell` call would attach (mirrors
|
/// Build the `CubeHeader` a `store_code_cell` call would attach (mirrors
|
||||||
/// `crate::store_code_cell`), so a buffered txn put carries the same header.
|
/// `crate::store_code_cell`), so a buffered txn put carries the same header.
|
||||||
/// `owner` (when set) is stamped on `owner_local_user` for Task 6 enforcement.
|
/// `owner` (when set) is stamped on `owner_local_user` for Task 6 enforcement.
|
||||||
fn header_for_code(kind: Kind, name: &str, code: &[Op], owner: Option<&str>) -> CubeHeader {
|
/// `descriptor` (when set) stamps the behavior-descriptor bits (PDF §524–525).
|
||||||
|
fn header_for_code(
|
||||||
|
kind: Kind,
|
||||||
|
name: &str,
|
||||||
|
code: &[Op],
|
||||||
|
owner: Option<&str>,
|
||||||
|
descriptor: Option<cubecode::Behavior>,
|
||||||
|
) -> CubeHeader {
|
||||||
let mut h = CubeHeader::new();
|
let mut h = CubeHeader::new();
|
||||||
h.title = Some(name.to_string());
|
h.title = Some(name.to_string());
|
||||||
h.doc_type = Some(kind.as_str().to_string());
|
h.doc_type = Some(kind.as_str().to_string());
|
||||||
h.linked_records = Vec::new();
|
h.linked_records = Vec::new();
|
||||||
h.owner_local_user = owner.map(|o| o.to_string());
|
h.owner_local_user = owner.map(|o| o.to_string());
|
||||||
|
// OS-created records carry per-word flags (tag 14).
|
||||||
|
h.word_flags = crate::default_word_flags(kind, descriptor);
|
||||||
|
if let Some(b) = descriptor {
|
||||||
|
h.flags.0 |= b.to_flags();
|
||||||
|
}
|
||||||
if h.doc_type.as_deref() == Some("fn") {
|
if h.doc_type.as_deref() == Some("fn") {
|
||||||
h.size_bytes = Some(cubecode::encode(code).len() as u64);
|
h.size_bytes = Some(cubecode::encode(code).len() as u64);
|
||||||
}
|
}
|
||||||
@@ -992,6 +1616,72 @@ pub fn parse_coord(s: &str) -> Option<cubecoords::Czyx> {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/// Parse a single `u8` axis token: decimal (`7`) or hex (`0x07`).
|
/// Parse a single `u8` axis token: decimal (`7`) or hex (`0x07`).
|
||||||
|
|
||||||
|
fn header_flag_names(bits: u16) -> String {
|
||||||
|
let mut v: Vec<&str> = Vec::new();
|
||||||
|
if bits & cubecoords::HeaderFlags::TITLE != 0 { v.push("title"); }
|
||||||
|
if bits & cubecoords::HeaderFlags::DOC_TYPE != 0 { v.push("doc_type"); }
|
||||||
|
if bits & cubecoords::HeaderFlags::CREATED_AT != 0 { v.push("created_at"); }
|
||||||
|
if bits & cubecoords::HeaderFlags::SIZE_BYTES != 0 { v.push("size_bytes"); }
|
||||||
|
if bits & cubecoords::HeaderFlags::PERM_ROOT_ONLY != 0 { v.push("root_only"); }
|
||||||
|
if bits & cubecoords::HeaderFlags::PERM_LOCAL_USER != 0 { v.push("local_user"); }
|
||||||
|
if bits & cubecoords::HeaderFlags::PERM_REMOTE_USER != 0 { v.push("remote_user"); }
|
||||||
|
if bits & cubecoords::HeaderFlags::HAS_ASSOCIATIONS != 0 { v.push("associations"); }
|
||||||
|
if bits & cubecoords::HeaderFlags::HAS_PATH != 0 { v.push("path"); }
|
||||||
|
if v.is_empty() { String::new() } else { format!("({})", v.join("|")) }
|
||||||
|
}
|
||||||
|
|
||||||
|
fn word_flag_names(bits: u16) -> String {
|
||||||
|
let mut v: Vec<&str> = Vec::new();
|
||||||
|
let arr = bits & cubecoords::WordFlags::ARR_MASK;
|
||||||
|
if arr == 0 { v.push("tri6"); } else if arr == 1 { v.push("pair"); }
|
||||||
|
else if arr == 2 { v.push("triad"); } else { v.push("quad"); }
|
||||||
|
if bits & cubecoords::WordFlags::START_RECORD != 0 { v.push("start"); }
|
||||||
|
if bits & cubecoords::WordFlags::END_RECORD != 0 { v.push("end"); }
|
||||||
|
if bits & cubecoords::WordFlags::IS_HEADER != 0 { v.push("header"); }
|
||||||
|
if bits & cubecoords::WordFlags::CONTINUATION != 0 { v.push("cont"); }
|
||||||
|
let ty = (bits & cubecoords::WordFlags::TYPE_MASK) >> 6;
|
||||||
|
match ty { 0 => v.push("type=text"), 1 => v.push("type=binary"), 2 => v.push("type=code"), _ => v.push("type=ai") }
|
||||||
|
if bits & cubecoords::WordFlags::PERM_ROOT != 0 { v.push("root"); }
|
||||||
|
if bits & cubecoords::WordFlags::PERM_USER != 0 { v.push("user"); }
|
||||||
|
if bits & cubecoords::WordFlags::ASSOC_EDGE != 0 { v.push("assoc"); }
|
||||||
|
if bits & cubecoords::WordFlags::ENCRYPTED != 0 { v.push("encrypted"); }
|
||||||
|
if bits & cubecoords::WordFlags::COMPRESSED != 0 { v.push("compressed"); }
|
||||||
|
if bits & cubecoords::WordFlags::STEGO != 0 { v.push("stego"); }
|
||||||
|
if bits & cubecoords::WordFlags::NULL_LOOKUP != 0 { v.push("null"); }
|
||||||
|
if bits & cubecoords::WordFlags::CHECKSUM != 0 { v.push("checksum"); }
|
||||||
|
format!("({})", v.join("|"))
|
||||||
|
}
|
||||||
|
|
||||||
|
fn parse_word_flag(a: &str) -> Result<u16, String> {
|
||||||
|
let lower = a.to_lowercase();
|
||||||
|
let bits = match lower.as_str() {
|
||||||
|
"start" => cubecoords::WordFlags::START_RECORD,
|
||||||
|
"end" => cubecoords::WordFlags::END_RECORD,
|
||||||
|
"header" => cubecoords::WordFlags::IS_HEADER,
|
||||||
|
"cont" | "continuation" => cubecoords::WordFlags::CONTINUATION,
|
||||||
|
"text" | "type-text" => 0 << 6,
|
||||||
|
"binary" | "type-binary" => 1 << 6,
|
||||||
|
"code" | "type-code" => 2 << 6,
|
||||||
|
"ai" | "type-ai" => 3 << 6,
|
||||||
|
"root" => cubecoords::WordFlags::PERM_ROOT,
|
||||||
|
"user" => cubecoords::WordFlags::PERM_USER,
|
||||||
|
"assoc" | "assoc-edge" => cubecoords::WordFlags::ASSOC_EDGE,
|
||||||
|
"encrypted" | "enc" => cubecoords::WordFlags::ENCRYPTED,
|
||||||
|
"compressed" | "comp" => cubecoords::WordFlags::COMPRESSED,
|
||||||
|
"stego" => cubecoords::WordFlags::STEGO,
|
||||||
|
"null" | "null-lookup" => cubecoords::WordFlags::NULL_LOOKUP,
|
||||||
|
"checksum" => cubecoords::WordFlags::CHECKSUM,
|
||||||
|
other => match other.strip_prefix("0x") {
|
||||||
|
Some(hex) => return u16::from_str_radix(hex, 16)
|
||||||
|
.map_err(|_| format!("invalid hex word flag '{a}'")),
|
||||||
|
None => return other.parse::<u16>()
|
||||||
|
.map_err(|_| format!("unknown word flag '{a}'")),
|
||||||
|
},
|
||||||
|
};
|
||||||
|
Ok(bits)
|
||||||
|
}
|
||||||
|
|
||||||
fn parse_u8(t: Option<&str>, what: &str) -> Result<u8, String> {
|
fn parse_u8(t: Option<&str>, what: &str) -> Result<u8, String> {
|
||||||
let t = t.ok_or_else(|| what.to_string())?;
|
let t = t.ok_or_else(|| what.to_string())?;
|
||||||
t.parse::<u8>()
|
t.parse::<u8>()
|
||||||
@@ -1101,6 +1791,82 @@ mod tests {
|
|||||||
Session::with_store(Arc::new(ConcurrentStore::memory()))
|
Session::with_store(Arc::new(ConcurrentStore::memory()))
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn os_kernels_live_in_cube_with_call_graph_and_descriptors() {
|
||||||
|
// Step 1: OS operator behavior lives in CUBE as kernels, composed via
|
||||||
|
// a call graph (linked_records) and tagged with behavior descriptors.
|
||||||
|
// Step 2: a thin native effector (`native-apply`) runs each kernel in
|
||||||
|
// the VM, reads its computed result, and emits the OS effect — the
|
||||||
|
// cubevm itself never does store-IO (spec-aligned, PDF §524–525).
|
||||||
|
let mut s = session();
|
||||||
|
|
||||||
|
// Lay down the OS kernel call graph.
|
||||||
|
let out = s.exec("tick").expect("tick should lay down kernels");
|
||||||
|
assert!(
|
||||||
|
out.contains("normalize-config"),
|
||||||
|
"cfg kernel missing: {out}"
|
||||||
|
);
|
||||||
|
assert!(
|
||||||
|
out.contains("decide-snapshot"),
|
||||||
|
"decide kernel missing: {out}"
|
||||||
|
);
|
||||||
|
assert!(
|
||||||
|
out.contains("summarize-procs"),
|
||||||
|
"summarize kernel missing: {out}"
|
||||||
|
);
|
||||||
|
assert!(out.contains("cube-os-tick"), "tick kernel missing: {out}");
|
||||||
|
// The root links the three leaves (call graph, not foreign code).
|
||||||
|
assert!(
|
||||||
|
out.contains("links cfg,decide,summarize"),
|
||||||
|
"call graph not wired: {out}"
|
||||||
|
);
|
||||||
|
// Behavior descriptors are stamped (round-trip through header flags).
|
||||||
|
assert!(
|
||||||
|
out.contains("[pure]") && out.contains("[io]") && out.contains("[pure,hot]"),
|
||||||
|
"behavior descriptors not stamped: {out}"
|
||||||
|
);
|
||||||
|
|
||||||
|
// Run the root kernel: it must traverse the call graph (CallLink 0..2)
|
||||||
|
// and return, proving the OS's behavior lives as addressable kernels.
|
||||||
|
let run_out = s
|
||||||
|
.exec("run /c210/z001/y001/x004")
|
||||||
|
.expect("tick kernel must run");
|
||||||
|
assert!(
|
||||||
|
run_out.contains("Halted"),
|
||||||
|
"tick kernel should halt: {run_out}"
|
||||||
|
);
|
||||||
|
|
||||||
|
// Step 2 — the effector reads the COMPUTED result and emits the effect.
|
||||||
|
let eff = s
|
||||||
|
.exec("native-apply /c210/z001/y001/x002")
|
||||||
|
.expect("effector must run decide-snapshot");
|
||||||
|
assert!(
|
||||||
|
eff.contains("computed result = 1"),
|
||||||
|
"decide kernel result wrong: {eff}"
|
||||||
|
);
|
||||||
|
assert!(
|
||||||
|
eff.contains("OS EFFECT"),
|
||||||
|
"effector must emit OS EFFECT: {eff}"
|
||||||
|
);
|
||||||
|
assert!(
|
||||||
|
eff.contains("snapshot NOW"),
|
||||||
|
"decision=1 should snapshot: {eff}"
|
||||||
|
);
|
||||||
|
|
||||||
|
// A plain pure kernel also routes through the effector with no store-IO.
|
||||||
|
let eff2 = s
|
||||||
|
.exec("native-apply /c210/z001/y001/x003")
|
||||||
|
.expect("effector must run summarize-procs");
|
||||||
|
assert!(
|
||||||
|
eff2.contains("computed result = 20"),
|
||||||
|
"summarize result wrong: {eff2}"
|
||||||
|
);
|
||||||
|
assert!(
|
||||||
|
eff2.contains("descriptors=[\"pure\", \"hot\"]"),
|
||||||
|
"descriptor readback wrong: {eff2}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn begin_commit_applies_buffered_writes() {
|
fn begin_commit_applies_buffered_writes() {
|
||||||
let mut s = session();
|
let mut s = session();
|
||||||
@@ -1803,4 +2569,25 @@ mod tests {
|
|||||||
);
|
);
|
||||||
let _ = std::fs::remove_dir_all(&dir);
|
let _ = std::fs::remove_dir_all(&dir);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn header_for_code_sets_word_flags() {
|
||||||
|
// Fn cell with a security-sensitive descriptor -> per-word flags set.
|
||||||
|
let h = header_for_code(
|
||||||
|
Kind::Fn,
|
||||||
|
"f",
|
||||||
|
&[],
|
||||||
|
None,
|
||||||
|
Some(cubecode::Behavior::new(cubecode::Behavior::SECURITY_SENSITIVE)),
|
||||||
|
);
|
||||||
|
assert!(h.word_flags.has(cubecoords::WordFlags::START_RECORD));
|
||||||
|
assert!(h.word_flags.has(cubecoords::WordFlags::END_RECORD));
|
||||||
|
assert!(h.word_flags.has(cubecoords::WordFlags::IS_HEADER));
|
||||||
|
assert!(h.word_flags.has(cubecoords::WordFlags::ENCRYPTED), "sec descriptor -> ENCRYPTED");
|
||||||
|
assert_eq!((h.word_flags.bits() & cubecoords::WordFlags::TYPE_MASK), 0b10 << 6, "Fn -> type=code");
|
||||||
|
// No descriptor -> complete frame, no security bit.
|
||||||
|
let h2 = header_for_code(Kind::Fn, "g", &[], None, None);
|
||||||
|
assert!(h2.word_flags.has(cubecoords::WordFlags::START_RECORD));
|
||||||
|
assert!(!h2.word_flags.has(cubecoords::WordFlags::ENCRYPTED));
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -276,6 +276,7 @@ fn write_bucket(store: &ConcurrentStore, grants: &[Grant]) {
|
|||||||
pub fn grant_header() -> CubeHeader {
|
pub fn grant_header() -> CubeHeader {
|
||||||
CubeHeader {
|
CubeHeader {
|
||||||
flags: cubecoords::HeaderFlags(cubecoords::HeaderFlags::DOC_TYPE),
|
flags: cubecoords::HeaderFlags(cubecoords::HeaderFlags::DOC_TYPE),
|
||||||
|
word_flags: cubecoords::WordFlags::default(),
|
||||||
title: None,
|
title: None,
|
||||||
doc_type: Some("grant-table".to_string()),
|
doc_type: Some("grant-table".to_string()),
|
||||||
created_at: None,
|
created_at: None,
|
||||||
@@ -283,6 +284,7 @@ pub fn grant_header() -> CubeHeader {
|
|||||||
owner_local_user: None,
|
owner_local_user: None,
|
||||||
owner_remote_user: None,
|
owner_remote_user: None,
|
||||||
linked_records: Vec::new(),
|
linked_records: Vec::new(),
|
||||||
|
path: None,
|
||||||
total_accesses: 0,
|
total_accesses: 0,
|
||||||
total_remote_accesses: 0,
|
total_remote_accesses: 0,
|
||||||
last_access: None,
|
last_access: None,
|
||||||
|
|||||||
@@ -54,6 +54,11 @@ pub mod commands;
|
|||||||
pub mod grants;
|
pub mod grants;
|
||||||
/// Length-framed Unix-domain-socket transport shared by client and server.
|
/// Length-framed Unix-domain-socket transport shared by client and server.
|
||||||
pub mod net;
|
pub mod net;
|
||||||
|
/// Session-message / note log: a searchable, per-session, per-day note area
|
||||||
|
/// stored as CZYX records carrying WordFlags metadata. This is the structured
|
||||||
|
/// replacement for the hand-maintained `RESUME-*.md` files so a new session
|
||||||
|
/// can review prior sessions' messages over the cube store (see [`notes`]).
|
||||||
|
pub mod notes;
|
||||||
/// Dependency-free JSON snapshot load/dump for daemon store persistence.
|
/// Dependency-free JSON snapshot load/dump for daemon store persistence.
|
||||||
pub mod persist;
|
pub mod persist;
|
||||||
/// Concurrent, durable store: mutex-wrapped [`CubeStore`] + NDJSON write-ahead
|
/// Concurrent, durable store: mutex-wrapped [`CubeStore`] + NDJSON write-ahead
|
||||||
@@ -133,6 +138,35 @@ pub fn load_code_cell<B: CubeBackend>(
|
|||||||
|
|
||||||
/// Build a [`CodeCell`] from parts and write it back as a record at `path`,
|
/// Build a [`CodeCell`] from parts and write it back as a record at `path`,
|
||||||
/// so code written by the VM (or tooling) is visible to cubefs at that path.
|
/// so code written by the VM (or tooling) is visible to cubefs at that path.
|
||||||
|
#[allow(clippy::too_many_arguments)]
|
||||||
|
/// Per-word flags (tag 14) stamped on OS-created records: a complete single
|
||||||
|
/// record frame + a type note (code cells = type=code) + a security note, so the
|
||||||
|
/// store actually carries WordFlags metadata (not just stores it).
|
||||||
|
pub(crate) fn default_word_flags(
|
||||||
|
kind: Kind,
|
||||||
|
descriptor: Option<cubecode::Behavior>,
|
||||||
|
) -> cubecoords::WordFlags {
|
||||||
|
let mut wf = cubecoords::WordFlags::from_bits(
|
||||||
|
cubecoords::WordFlags::START_RECORD
|
||||||
|
| cubecoords::WordFlags::END_RECORD
|
||||||
|
| cubecoords::WordFlags::IS_HEADER,
|
||||||
|
);
|
||||||
|
if matches!(kind, Kind::Fn | Kind::Kernel) {
|
||||||
|
wf.0 |= 0b10 << 6; // type=code
|
||||||
|
}
|
||||||
|
if let Some(b) = descriptor {
|
||||||
|
if b.0 & cubecode::Behavior::SECURITY_SENSITIVE != 0 {
|
||||||
|
wf.0 |= cubecoords::WordFlags::ENCRYPTED;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
wf
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Store a CUBEVM code cell at the coordinate derived from `path`. Builds the
|
||||||
|
/// record header (title/doc_type/owner) and stamps per-word flags (tag 14) via
|
||||||
|
/// [`default_word_flags`], so OS-created code cells carry WordFlags metadata.
|
||||||
|
/// Returns the resolved coordinate; used by the `prog`/`write`/`kernel`/`fn`
|
||||||
|
/// OS commands and the `cubefs` socket backend.
|
||||||
pub fn store_code_cell<B: CubeBackend>(
|
pub fn store_code_cell<B: CubeBackend>(
|
||||||
store: &mut CubeStore<B>,
|
store: &mut CubeStore<B>,
|
||||||
path: &str,
|
path: &str,
|
||||||
@@ -141,6 +175,10 @@ pub fn store_code_cell<B: CubeBackend>(
|
|||||||
links: &[Czyx],
|
links: &[Czyx],
|
||||||
code: &[cubecode::Op],
|
code: &[cubecode::Op],
|
||||||
owner: Option<&str>,
|
owner: Option<&str>,
|
||||||
|
// Behavior-descriptor flags (PDF §524–525: pure / I/O heavy / hot path) to
|
||||||
|
// stamp on the record header's out-of-band bits. `None` leaves the field
|
||||||
|
// at zero. See `cubecode::Behavior`.
|
||||||
|
descriptor: Option<cubecode::Behavior>,
|
||||||
) -> Result<Czyx, SysError> {
|
) -> Result<Czyx, SysError> {
|
||||||
let coord = path_to_czyx(path)?;
|
let coord = path_to_czyx(path)?;
|
||||||
let mut h = CubeHeader::new();
|
let mut h = CubeHeader::new();
|
||||||
@@ -148,6 +186,10 @@ pub fn store_code_cell<B: CubeBackend>(
|
|||||||
h.doc_type = Some(kind.as_str().to_string());
|
h.doc_type = Some(kind.as_str().to_string());
|
||||||
h.linked_records = links.to_vec();
|
h.linked_records = links.to_vec();
|
||||||
h.owner_local_user = owner.map(|o| o.to_string());
|
h.owner_local_user = owner.map(|o| o.to_string());
|
||||||
|
h.word_flags = default_word_flags(kind, descriptor);
|
||||||
|
if let Some(b) = descriptor {
|
||||||
|
h.flags.0 |= b.to_flags();
|
||||||
|
}
|
||||||
if h.doc_type.as_deref() == Some("fn") {
|
if h.doc_type.as_deref() == Some("fn") {
|
||||||
h.size_bytes = Some(cubecode::encode(code).len() as u64);
|
h.size_bytes = Some(cubecode::encode(code).len() as u64);
|
||||||
}
|
}
|
||||||
@@ -187,6 +229,7 @@ mod tests {
|
|||||||
&[],
|
&[],
|
||||||
&[Op::Const(2), Op::Const(3), Op::Add, Op::Halt],
|
&[Op::Const(2), Op::Const(3), Op::Add, Op::Halt],
|
||||||
None,
|
None,
|
||||||
|
None,
|
||||||
)
|
)
|
||||||
.unwrap();
|
.unwrap();
|
||||||
|
|
||||||
@@ -300,6 +343,7 @@ pub mod demo {
|
|||||||
&[],
|
&[],
|
||||||
&double_code,
|
&double_code,
|
||||||
None,
|
None,
|
||||||
|
None,
|
||||||
)
|
)
|
||||||
.expect("store double");
|
.expect("store double");
|
||||||
let entry_coord = super::store_code_cell(
|
let entry_coord = super::store_code_cell(
|
||||||
@@ -310,6 +354,7 @@ pub mod demo {
|
|||||||
&[double_coord],
|
&[double_coord],
|
||||||
&entry_code,
|
&entry_code,
|
||||||
None,
|
None,
|
||||||
|
None,
|
||||||
)
|
)
|
||||||
.expect("store entry");
|
.expect("store entry");
|
||||||
println!(" wrote {double} -> coord {}", double_coord.pack_u32());
|
println!(" wrote {double} -> coord {}", double_coord.pack_u32());
|
||||||
|
|||||||
@@ -0,0 +1,772 @@
|
|||||||
|
//! CUBELinux session-message / note log with **categories** and **projects**.
|
||||||
|
//!
|
||||||
|
//! The "message save path" helper. Each note is a CZYX record with a text-type
|
||||||
|
//! [`WordFlags`] stamp, a `created_at` day, and the session label as the owner.
|
||||||
|
//! Two extra, searchable dimensions layer on top:
|
||||||
|
//! * **category** — encoded in `doc_type` as `note:<category>` (so
|
||||||
|
//! [`scan_by_type`] / filtering slices notes by kind: design, impl, cube, ...).
|
||||||
|
//! * **project** — notes are *associated* (via `linked_records`) to a
|
||||||
|
//! `project` record (`doc_type = "project"`), so `cube project walk`
|
||||||
|
//! follows the links into a project's timeline/thread. This is the
|
||||||
|
//! CUBELinux first-class-association model (EDG) applied to project work.
|
||||||
|
//!
|
||||||
|
//! Cross-session: `created_at` (day) + session label + category + project make
|
||||||
|
//! any note recoverable as `note search <text> [--project p] [--cat c]`, and a
|
||||||
|
//! new harness session can `note list --project p` / `project walk` to see where
|
||||||
|
//! a project left off.
|
||||||
|
|
||||||
|
use std::time::{SystemTime, UNIX_EPOCH};
|
||||||
|
|
||||||
|
use cubecoords::{CubeHeader, Czyx, WordFlags};
|
||||||
|
|
||||||
|
use crate::store::ConcurrentStore;
|
||||||
|
|
||||||
|
/// Base `doc_type` for a note without a category.
|
||||||
|
pub const NOTE_DOC_TYPE: &str = "note";
|
||||||
|
/// `doc_type` for a project record.
|
||||||
|
pub const PROJECT_DOC_TYPE: &str = "project";
|
||||||
|
|
||||||
|
/// The default session label for CLI-created notes (today's calendar date).
|
||||||
|
pub fn default_session() -> String {
|
||||||
|
date_string(now())
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The WordFlags stamped on a note: a complete single-record frame with the
|
||||||
|
/// text data type (type bits 00). Other metadata bits (encrypted, assoc, ...)
|
||||||
|
/// are left unset for the caller to raise as needed.
|
||||||
|
pub fn note_word_flags() -> WordFlags {
|
||||||
|
WordFlags::from_bits(
|
||||||
|
WordFlags::START_RECORD | WordFlags::END_RECORD | WordFlags::IS_HEADER,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Current unix time (seconds).
|
||||||
|
pub fn now() -> u64 {
|
||||||
|
SystemTime::now()
|
||||||
|
.duration_since(UNIX_EPOCH)
|
||||||
|
.map(|d| d.as_secs())
|
||||||
|
.unwrap_or(0)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A returned note (coordinate + header + body bytes).
|
||||||
|
#[derive(Clone, Debug)]
|
||||||
|
pub struct Note {
|
||||||
|
/// The note's coordinate (`C.Z.Y.X`).
|
||||||
|
pub coord: Czyx,
|
||||||
|
/// The note's record header (title, session label, WordFlags, category).
|
||||||
|
pub header: CubeHeader,
|
||||||
|
/// The note's body text as raw bytes.
|
||||||
|
pub body: Vec<u8>,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Query dimensions for listing/searching notes. `since`/`until` are
|
||||||
|
/// day-of-epoch (see [`date_to_day`]); the rest are exact-match labels.
|
||||||
|
#[derive(Clone, Default, Debug)]
|
||||||
|
pub struct NoteFilter {
|
||||||
|
/// Session label (owner_local_user).
|
||||||
|
pub session: Option<String>,
|
||||||
|
/// Project name (note must be associated to that project record).
|
||||||
|
pub project: Option<String>,
|
||||||
|
/// Category (doc_type `note:<category>`).
|
||||||
|
pub category: Option<String>,
|
||||||
|
/// Earliest day-of-epoch, inclusive.
|
||||||
|
pub since: Option<u64>,
|
||||||
|
/// Latest day-of-epoch, inclusive.
|
||||||
|
pub until: Option<u64>,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The category portion of a note's `doc_type`, if any (`note:<category>`).
|
||||||
|
pub fn category_of(doc_type: &str) -> Option<&str> {
|
||||||
|
doc_type.strip_prefix(&format!("{NOTE_DOC_TYPE}:"))
|
||||||
|
}
|
||||||
|
|
||||||
|
fn fnv1a(bytes: &[u8]) -> u64 {
|
||||||
|
let mut h: u64 = 0xcbf2_9ce4_8422_2325;
|
||||||
|
for &b in bytes {
|
||||||
|
h ^= b as u64;
|
||||||
|
h = h.wrapping_mul(0x0000_0100_0000_01b3);
|
||||||
|
}
|
||||||
|
h
|
||||||
|
}
|
||||||
|
|
||||||
|
fn non_zero(b: u8) -> u8 {
|
||||||
|
if b == 0 {
|
||||||
|
1
|
||||||
|
} else {
|
||||||
|
b
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The single-byte notes class for a session label, so `scan_prefix(class,
|
||||||
|
/// None, None)` returns exactly that session's notes.
|
||||||
|
fn session_class(session: &str) -> u8 {
|
||||||
|
let h = fnv1a(session.as_bytes());
|
||||||
|
1 + ((h >> 24) as u8 & 0x7f) // 1..=128
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Project records live in the upper class range (128..=255) so they never
|
||||||
|
/// collide with note session classes (1..=128).
|
||||||
|
fn project_class(name: &str) -> u8 {
|
||||||
|
let h = fnv1a(name.as_bytes());
|
||||||
|
128 + ((h >> 24) as u8 & 0x7f)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Derive a deterministic, collision-light coordinate for a note: the C axis
|
||||||
|
/// is the session class, Z/Y/X come from the note content + a salt.
|
||||||
|
fn note_coord(session: &str, created: u64, subject: &str, body: &str, salt: u64) -> Czyx {
|
||||||
|
let mut mix = fnv1a(session.as_bytes());
|
||||||
|
mix ^= created.wrapping_mul(0x9e37_79b9);
|
||||||
|
mix ^= fnv1a(subject.as_bytes());
|
||||||
|
mix ^= fnv1a(body.as_bytes());
|
||||||
|
mix ^= salt.wrapping_mul(0x85eb_ca6b);
|
||||||
|
Czyx::new(
|
||||||
|
session_class(session),
|
||||||
|
non_zero((mix >> 16) as u8),
|
||||||
|
non_zero((mix >> 8) as u8),
|
||||||
|
non_zero(mix as u8 ^ (salt as u8)),
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Derive the deterministic coordinate of a project record.
|
||||||
|
fn project_coord(name: &str, salt: u64) -> Czyx {
|
||||||
|
let mut mix = fnv1a(name.as_bytes());
|
||||||
|
mix ^= salt.wrapping_mul(0x8549_b2a1);
|
||||||
|
Czyx::new(
|
||||||
|
project_class(name),
|
||||||
|
non_zero((mix >> 16) as u8),
|
||||||
|
non_zero((mix >> 8) as u8),
|
||||||
|
non_zero(mix as u8 ^ (salt as u8)),
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Find an existing project record by name, without creating it.
|
||||||
|
pub fn find_project(store: &ConcurrentStore, name: &str) -> Option<Czyx> {
|
||||||
|
store
|
||||||
|
.read_snapshot()
|
||||||
|
.query_by_type(PROJECT_DOC_TYPE)
|
||||||
|
.into_iter()
|
||||||
|
.find(|(_, h, _)| {
|
||||||
|
h.title.as_deref() == Some(name)
|
||||||
|
|| h.doc_type.as_deref() == Some(PROJECT_DOC_TYPE) && h.path.as_deref() == Some(name)
|
||||||
|
})
|
||||||
|
.map(|(c, _, _)| c)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Find-or-create a project record named `name`, returning its coordinate.
|
||||||
|
/// Notes tagged with a project are associated to it via `linked_records`.
|
||||||
|
pub fn ensure_project(store: &ConcurrentStore, name: &str, session: &str) -> Result<Czyx, String> {
|
||||||
|
if let Some(c) = find_project(store, name) {
|
||||||
|
return Ok(c);
|
||||||
|
}
|
||||||
|
let mut salt = 0u64;
|
||||||
|
let mut coord = project_coord(name, salt);
|
||||||
|
while store.get_record(&coord).is_some() && salt < 4096 {
|
||||||
|
salt += 1;
|
||||||
|
coord = project_coord(name, salt);
|
||||||
|
}
|
||||||
|
if salt >= 4096 {
|
||||||
|
return Err("project store: could not find a free coordinate".into());
|
||||||
|
}
|
||||||
|
let mut h = CubeHeader::new();
|
||||||
|
h.title = Some(name.to_string());
|
||||||
|
h.path = Some(name.to_string());
|
||||||
|
h.doc_type = Some(PROJECT_DOC_TYPE.to_string());
|
||||||
|
h.created_at = Some(now());
|
||||||
|
h.owner_local_user = Some(session.to_string());
|
||||||
|
h.refresh_flags();
|
||||||
|
store.put_record(coord, &h, &[]);
|
||||||
|
Ok(coord)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Store a note record under `session`, with a `subject` title and `body`
|
||||||
|
/// text. `project` (if given) associates the note to a project record; `category`
|
||||||
|
/// (if given) sets the note's `doc_type` to `note:<category>`. Returns the note's
|
||||||
|
/// coordinate.
|
||||||
|
pub fn store_note(
|
||||||
|
store: &ConcurrentStore,
|
||||||
|
session: &str,
|
||||||
|
subject: &str,
|
||||||
|
body: &str,
|
||||||
|
project: Option<&str>,
|
||||||
|
category: Option<&str>,
|
||||||
|
) -> Result<Czyx, String> {
|
||||||
|
let created = now();
|
||||||
|
let proj_coord = match project {
|
||||||
|
Some(p) => Some(ensure_project(store, p, session)?),
|
||||||
|
None => None,
|
||||||
|
};
|
||||||
|
let mut salt = 0u64;
|
||||||
|
let mut coord = note_coord(session, created, subject, body, salt);
|
||||||
|
while store.get_record(&coord).is_some() && salt < 4096 {
|
||||||
|
salt += 1;
|
||||||
|
coord = note_coord(session, created, subject, body, salt);
|
||||||
|
}
|
||||||
|
if salt >= 4096 {
|
||||||
|
return Err("note store: could not find a free coordinate (4096 tries)".into());
|
||||||
|
}
|
||||||
|
let mut h = CubeHeader::new();
|
||||||
|
h.title = Some(subject.to_string());
|
||||||
|
h.doc_type = Some(match category {
|
||||||
|
Some(c) => format!("{NOTE_DOC_TYPE}:{c}"),
|
||||||
|
None => NOTE_DOC_TYPE.to_string(),
|
||||||
|
});
|
||||||
|
h.created_at = Some(created);
|
||||||
|
h.owner_local_user = Some(session.to_string());
|
||||||
|
if let Some(pc) = proj_coord {
|
||||||
|
h.linked_records = vec![pc];
|
||||||
|
}
|
||||||
|
h.word_flags = note_word_flags();
|
||||||
|
h.refresh_flags();
|
||||||
|
store.put_record(coord, &h, body.as_bytes());
|
||||||
|
Ok(coord)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// All note records (`doc_type == "note"` or `"note:<cat>"`), regardless of
|
||||||
|
/// category, so a category filter is applied in Rust (the store's exact
|
||||||
|
/// `scan_by_type("note")` would miss categorized notes).
|
||||||
|
fn all_notes(snap: &cubestore::CubeStore<cubestore::HashBackend>) -> Vec<Note> {
|
||||||
|
snap.keys()
|
||||||
|
.into_iter()
|
||||||
|
.filter_map(|coord| {
|
||||||
|
let (header, body) = snap.get_record(&coord)?;
|
||||||
|
let dt = header.doc_type.as_deref().unwrap_or("");
|
||||||
|
if dt == NOTE_DOC_TYPE || dt.starts_with(&format!("{NOTE_DOC_TYPE}:")) {
|
||||||
|
Some(Note {
|
||||||
|
coord,
|
||||||
|
header,
|
||||||
|
body,
|
||||||
|
})
|
||||||
|
} else {
|
||||||
|
None
|
||||||
|
}
|
||||||
|
})
|
||||||
|
.collect()
|
||||||
|
}
|
||||||
|
|
||||||
|
/// List notes, newest first, filtered by [`NoteFilter`].
|
||||||
|
pub fn list_notes(store: &ConcurrentStore, filter: Option<&NoteFilter>) -> Vec<Note> {
|
||||||
|
let snap = store.read_snapshot();
|
||||||
|
let mut out: Vec<Note> = all_notes(&snap)
|
||||||
|
.into_iter()
|
||||||
|
.filter(|n| note_matches_filtered(store, n, filter))
|
||||||
|
.collect();
|
||||||
|
out.sort_by(|a, b| b.header.created_at.cmp(&a.header.created_at));
|
||||||
|
out
|
||||||
|
}
|
||||||
|
|
||||||
|
fn note_matches_filtered(store: &ConcurrentStore, n: &Note, filter: Option<&NoteFilter>) -> bool {
|
||||||
|
let Some(f) = filter else {
|
||||||
|
return true;
|
||||||
|
};
|
||||||
|
if let Some(s) = &f.session {
|
||||||
|
if n.header.owner_local_user.as_deref() != Some(s.as_str()) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if let Some(cat) = &f.category {
|
||||||
|
let expected = format!("{NOTE_DOC_TYPE}:{cat}");
|
||||||
|
if n.header.doc_type.as_deref() != Some(expected.as_str()) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if let Some(sd) = f.since {
|
||||||
|
if n.header.created_at.map(|c| c / 86_400).unwrap_or(0) < sd {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if let Some(ud) = f.until {
|
||||||
|
if n.header.created_at.map(|c| c / 86_400).unwrap_or(u64::MAX) > ud {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if let Some(p) = &f.project {
|
||||||
|
if let Some(pc) = find_project(store, p) {
|
||||||
|
if !n.header.linked_records.contains(&pc) && n.header.path.as_deref() != Some(p.as_str()) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
true
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Search all notes (session + body) for `needle`, filtered, newest first.
|
||||||
|
pub fn search_notes(store: &ConcurrentStore, needle: &str, filter: Option<&NoteFilter>) -> Vec<Note> {
|
||||||
|
let needle = needle.to_lowercase();
|
||||||
|
let snap = store.read_snapshot();
|
||||||
|
let mut out: Vec<Note> = all_notes(&snap)
|
||||||
|
.into_iter()
|
||||||
|
.filter(|n| note_matches_filtered(store, n, filter))
|
||||||
|
.filter(|n| {
|
||||||
|
let subject = n.header.title.as_deref().unwrap_or("").to_lowercase();
|
||||||
|
let b = String::from_utf8_lossy(&n.body).to_lowercase();
|
||||||
|
subject.contains(&needle) || b.contains(&needle)
|
||||||
|
})
|
||||||
|
.collect();
|
||||||
|
out.sort_by(|a, b| b.header.created_at.cmp(&a.header.created_at));
|
||||||
|
out
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Fetch a single note by coordinate.
|
||||||
|
pub fn show_note(store: &ConcurrentStore, coord: Czyx) -> Option<Note> {
|
||||||
|
store.get_record(&coord).map(|(header, body)| Note {
|
||||||
|
coord,
|
||||||
|
header,
|
||||||
|
body,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
/// List all project records, newest first.
|
||||||
|
pub fn project_list(store: &ConcurrentStore) -> Vec<(Czyx, CubeHeader, Vec<u8>)> {
|
||||||
|
let mut v = store.read_snapshot().query_by_type(PROJECT_DOC_TYPE);
|
||||||
|
v.sort_by(|a, b| b.1.created_at.cmp(&a.1.created_at));
|
||||||
|
v
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Return the notes associated to a project record (its thread/timeline),
|
||||||
|
/// newest first.
|
||||||
|
pub fn project_walk(store: &ConcurrentStore, coord: Czyx) -> Vec<Note> {
|
||||||
|
let snap = store.read_snapshot();
|
||||||
|
let mut out: Vec<Note> = all_notes(&snap)
|
||||||
|
.into_iter()
|
||||||
|
.filter(|n| n.header.linked_records.contains(&coord))
|
||||||
|
.collect();
|
||||||
|
out.sort_by(|a, b| b.header.created_at.cmp(&a.header.created_at));
|
||||||
|
out
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Format a unix timestamp as `YYYY-MM-DD` (UTC).
|
||||||
|
pub fn date_string(ts: u64) -> String {
|
||||||
|
let days = (ts / 86_400) as i64;
|
||||||
|
let (y, m, d) = civil_from_days(days);
|
||||||
|
format!("{y:04}-{m:02}-{d:02}")
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Parse `YYYY-MM-DD` into a day-of-epoch (days since 1970-01-01).
|
||||||
|
pub fn date_to_day(s: &str) -> Option<u64> {
|
||||||
|
let mut it = s.split('-');
|
||||||
|
let y: i64 = it.next()?.parse().ok()?;
|
||||||
|
let m: u32 = it.next()?.parse().ok()?;
|
||||||
|
let d: u32 = it.next()?.parse().ok()?;
|
||||||
|
if it.next().is_some() || !(1..=12).contains(&m) || !(1..=31).contains(&d) {
|
||||||
|
return None;
|
||||||
|
}
|
||||||
|
Some(days_from_civil(y, m, d))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Days since 1970-01-01 for a civil date (Howard Hinnant's algorithm).
|
||||||
|
fn days_from_civil(y: i64, m: u32, d: u32) -> u64 {
|
||||||
|
let yy = if m <= 2 { y - 1 } else { y };
|
||||||
|
let era = if yy >= 0 { yy } else { yy - 399 } / 400;
|
||||||
|
let yoe = yy - era * 400;
|
||||||
|
let mp = (m as i64 + 9) % 12;
|
||||||
|
let doy = (153 * mp + 2) / 5 + d as i64 - 1;
|
||||||
|
let doe = yoe * 365 + yoe / 4 - yoe / 100 + doy;
|
||||||
|
(era * 146_097 + doe - 719_468).max(0) as u64
|
||||||
|
}
|
||||||
|
|
||||||
|
fn civil_from_days(days: i64) -> (i64, u32, u32) {
|
||||||
|
let z = days + 719_468;
|
||||||
|
let era = if z >= 0 { z } else { z - 146_096 } / 146_097;
|
||||||
|
let doe = z - era * 146_097;
|
||||||
|
let yoe = (doe - doe / 1460 + doe / 36_524 - doe / 146_096) / 365;
|
||||||
|
let y = yoe + era * 400;
|
||||||
|
let doy = doe - (365 * yoe + yoe / 4 - yoe / 100);
|
||||||
|
let mp = (5 * doy + 2) / 153;
|
||||||
|
let d = (doy - (153 * mp + 2) / 5 + 1) as u32;
|
||||||
|
let m = if mp < 10 { mp + 3 } else { mp - 9 } as u32;
|
||||||
|
(if m <= 2 { y + 1 } else { y }, m, d)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Render a coordinate as `C.Z.Y.X` (the [`crate::commands::parse_coord`]
|
||||||
|
/// form), so `note show <coord>` round-trips.
|
||||||
|
fn coord_str(c: Czyx) -> String {
|
||||||
|
format!("{}.{}.{}.{}", c.c, c.z, c.y, c.x)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Subject = the first line-ish portion of the note text (first 60 chars).
|
||||||
|
fn subject_of(text: &str) -> String {
|
||||||
|
let s = text.trim();
|
||||||
|
let s: String = s.chars().take(60).collect();
|
||||||
|
if s.len() < text.trim().len() {
|
||||||
|
format!("{s}…")
|
||||||
|
} else {
|
||||||
|
s
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn format_notes(notes: &[Note]) -> String {
|
||||||
|
if notes.is_empty() {
|
||||||
|
return "(no notes)".to_string();
|
||||||
|
}
|
||||||
|
let lines: Vec<String> = notes
|
||||||
|
.iter()
|
||||||
|
.map(|n| {
|
||||||
|
let subj = n.header.title.as_deref().unwrap_or("");
|
||||||
|
let when = n
|
||||||
|
.header
|
||||||
|
.created_at
|
||||||
|
.map(|t| date_string(t))
|
||||||
|
.unwrap_or_else(|| "?".into());
|
||||||
|
let cat = n
|
||||||
|
.header
|
||||||
|
.doc_type
|
||||||
|
.as_deref()
|
||||||
|
.and_then(category_of)
|
||||||
|
.map(|c| format!(" [{c}]"))
|
||||||
|
.unwrap_or_default();
|
||||||
|
format!(" {} [{}]{} {}", coord_str(n.coord), when, cat, subj)
|
||||||
|
})
|
||||||
|
.collect();
|
||||||
|
format!("{} note(s):\n{}", notes.len(), lines.join("\n"))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Parse `--key value` flags out of a command string; returns `(flags,
|
||||||
|
/// remainingPositionalJoined)`. A flag with no following token gets `"true"`.
|
||||||
|
fn split_flags(s: &str) -> (Vec<(String, String)>, String) {
|
||||||
|
let tokens: Vec<&str> = s.split_whitespace().collect();
|
||||||
|
let mut flags = Vec::new();
|
||||||
|
let mut pos: Vec<&str> = Vec::new();
|
||||||
|
let mut i = 0;
|
||||||
|
while i < tokens.len() {
|
||||||
|
let t = tokens[i];
|
||||||
|
if let Some(k) = t.strip_prefix("--") {
|
||||||
|
let (key, val) = if i + 1 < tokens.len() && !tokens[i + 1].starts_with("--") {
|
||||||
|
i += 1;
|
||||||
|
(k.to_string(), tokens[i].to_string())
|
||||||
|
} else {
|
||||||
|
(k.to_string(), "true".to_string())
|
||||||
|
};
|
||||||
|
flags.push((key, val));
|
||||||
|
} else {
|
||||||
|
pos.push(t);
|
||||||
|
}
|
||||||
|
i += 1;
|
||||||
|
}
|
||||||
|
(flags, pos.join(" "))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Process one `note`/`notes` command line against `store`.
|
||||||
|
///
|
||||||
|
/// Syntax (REPL/CLI):
|
||||||
|
/// `note <text>` quick note
|
||||||
|
/// `note add [--project P] [--cat C] <text>` add a note to a project/category
|
||||||
|
/// `note list [session] [--project P] [--cat C] [--since D] [--until D]`
|
||||||
|
/// `note search <text> [--project P] [--cat C]`
|
||||||
|
/// `note show <C.Z.Y.X>`
|
||||||
|
/// `project list` / `project show <C.Z.Y.X>` (see [`project_command`])
|
||||||
|
pub fn note_command(store: &ConcurrentStore, line: &str) -> Result<String, String> {
|
||||||
|
let body = line
|
||||||
|
.splitn(2, char::is_whitespace)
|
||||||
|
.nth(1)
|
||||||
|
.unwrap_or("")
|
||||||
|
.trim()
|
||||||
|
.to_string();
|
||||||
|
let Some((head, rest)) = split_head(&body) else {
|
||||||
|
let session = default_session();
|
||||||
|
let notes = list_notes(store, Some(&NoteFilter { session: Some(session), ..Default::default() }));
|
||||||
|
return Ok(format_notes(¬es));
|
||||||
|
};
|
||||||
|
match head {
|
||||||
|
"list" => {
|
||||||
|
let (flags, pos) = split_flags(rest);
|
||||||
|
let f = filter_from_flags(&flags);
|
||||||
|
let session = if pos.is_empty() { None } else { Some(pos.clone()) };
|
||||||
|
let f = NoteFilter { session, ..f };
|
||||||
|
let notes = list_notes(store, Some(&f));
|
||||||
|
Ok(format_notes(¬es))
|
||||||
|
}
|
||||||
|
"search" => {
|
||||||
|
let (flags, pos) = split_flags(rest);
|
||||||
|
let f = filter_from_flags(&flags);
|
||||||
|
if pos.is_empty() {
|
||||||
|
return Err("note search needs <text>".to_string());
|
||||||
|
}
|
||||||
|
let notes = search_notes(store, &pos, Some(&f));
|
||||||
|
Ok(format!("search '{pos}':\n{}", format_notes(¬es)))
|
||||||
|
}
|
||||||
|
"tag" => {
|
||||||
|
let mut parts = rest.split_whitespace();
|
||||||
|
let cs = parts.next().unwrap_or("");
|
||||||
|
let cat = parts.next().unwrap_or("");
|
||||||
|
let coord = crate::commands::parse_coord(cs)
|
||||||
|
.ok_or_else(|| format!("bad coordinate '{cs}' (want C.Z.Y.X)"))?;
|
||||||
|
if cat.is_empty() {
|
||||||
|
return Err("note tag needs <coord> <cat>".to_string());
|
||||||
|
}
|
||||||
|
match store.get_record(&coord) {
|
||||||
|
Some((mut header, body)) => {
|
||||||
|
header.doc_type = Some(format!("{NOTE_DOC_TYPE}:{cat}"));
|
||||||
|
store.put_record(coord, &header, &body);
|
||||||
|
Ok(format!("note {} tagged {cat}", coord_str(coord)))
|
||||||
|
}
|
||||||
|
None => Ok(format!("note {} -> (no note)", coord_str(coord))),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
"show" => {
|
||||||
|
let cs = rest.trim();
|
||||||
|
let coord = crate::commands::parse_coord(cs)
|
||||||
|
.ok_or_else(|| format!("bad coordinate '{cs}' (want C.Z.Y.X)"))?;
|
||||||
|
match show_note(store, coord) {
|
||||||
|
Some(n) => {
|
||||||
|
let subj = n.header.title.as_deref().unwrap_or("");
|
||||||
|
let when = n
|
||||||
|
.header
|
||||||
|
.created_at
|
||||||
|
.map(|t| date_string(t))
|
||||||
|
.unwrap_or_else(|| "?".into());
|
||||||
|
Ok(format!(
|
||||||
|
"note {cs} [{}] {}\n{}",
|
||||||
|
when,
|
||||||
|
subj,
|
||||||
|
String::from_utf8_lossy(&n.body)
|
||||||
|
))
|
||||||
|
}
|
||||||
|
None => Ok(format!("note {cs} -> (no note)")),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
"add" | _ => {
|
||||||
|
let (flags, pos) = split_flags(rest);
|
||||||
|
let mut project = None;
|
||||||
|
let mut category = None;
|
||||||
|
for (k, v) in &flags {
|
||||||
|
match k.as_str() {
|
||||||
|
"project" | "p" => project = Some(v.clone()),
|
||||||
|
"cat" | "category" | "c" => category = Some(v.clone()),
|
||||||
|
_ => {}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
let text = pos.trim();
|
||||||
|
if text.is_empty() {
|
||||||
|
return Err("note needs <text>".to_string());
|
||||||
|
}
|
||||||
|
let session = default_session();
|
||||||
|
let subject = subject_of(text);
|
||||||
|
let coord = store_note(store, &session, &subject, text, project.as_deref(), category.as_deref())?;
|
||||||
|
let proj = project.map(|p| format!(" (project {p})")).unwrap_or_default();
|
||||||
|
Ok(format!("note saved at {} (session {}){proj}", coord_str(coord), session))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn filter_from_flags(flags: &[(String, String)]) -> NoteFilter {
|
||||||
|
let mut f = NoteFilter::default();
|
||||||
|
for (k, v) in flags {
|
||||||
|
match k.as_str() {
|
||||||
|
"project" | "p" => f.project = Some(v.clone()),
|
||||||
|
"cat" | "category" | "c" => f.category = Some(v.clone()),
|
||||||
|
"since" => f.since = date_to_day(v),
|
||||||
|
"until" => f.until = date_to_day(v),
|
||||||
|
_ => {}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
f
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Set a "resume" marker for a project. Stored as a note tagged
|
||||||
|
/// `category = "resume"` and associated to the project, so a new session can
|
||||||
|
/// ask "where did this project leave off?" and get it back.
|
||||||
|
pub fn project_resume(
|
||||||
|
store: &ConcurrentStore,
|
||||||
|
name: &str,
|
||||||
|
text: &str,
|
||||||
|
session: &str,
|
||||||
|
) -> Result<Czyx, String> {
|
||||||
|
let subject = format!("resume: {name}");
|
||||||
|
store_note(store, session, &subject, text, Some(name), Some("resume"))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Build a compact "project state" summary for context injection: the project
|
||||||
|
/// name, its note count, the resume marker (if any), and the latest notes with
|
||||||
|
/// date/category. A new session reads this to know where a project left off.
|
||||||
|
pub fn project_context(store: &ConcurrentStore, name: &str) -> String {
|
||||||
|
let Some(coord) = find_project(store, name) else {
|
||||||
|
return format!("(no project '{name}')");
|
||||||
|
};
|
||||||
|
let notes = project_walk(store, coord);
|
||||||
|
let mut out = String::new();
|
||||||
|
out.push_str(&format!("Project: {name}\n"));
|
||||||
|
out.push_str(&format!(" {} note(s); thread {}.\n", notes.len(), coord_str(coord)));
|
||||||
|
if let Some(r) = notes
|
||||||
|
.iter()
|
||||||
|
.find(|n| category_of(n.header.doc_type.as_deref().unwrap_or("")) == Some("resume"))
|
||||||
|
{
|
||||||
|
out.push_str(&format!(
|
||||||
|
" RESUME: {}\n",
|
||||||
|
String::from_utf8_lossy(&r.body).trim()
|
||||||
|
));
|
||||||
|
}
|
||||||
|
out.push_str(" Latest:\n");
|
||||||
|
for n in notes.iter().take(10) {
|
||||||
|
let subj = n.header.title.as_deref().unwrap_or("");
|
||||||
|
let when = n.header.created_at.map(|t| date_string(t)).unwrap_or_else(|| "?".into());
|
||||||
|
let cat = category_of(n.header.doc_type.as_deref().unwrap_or(""))
|
||||||
|
.map(|c| format!("[{c}] "))
|
||||||
|
.unwrap_or_default();
|
||||||
|
out.push_str(&format!(" {when} {cat}{subj}\n"));
|
||||||
|
}
|
||||||
|
out
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Process a `project` command line: `project list`, `project show <C.Z.Y.X>`,
|
||||||
|
/// `project resume <name> <text>`, `project context <name>`.
|
||||||
|
pub fn project_command(store: &ConcurrentStore, line: &str) -> Result<String, String> {
|
||||||
|
let body = line
|
||||||
|
.splitn(2, char::is_whitespace)
|
||||||
|
.nth(1)
|
||||||
|
.unwrap_or("")
|
||||||
|
.trim()
|
||||||
|
.to_string();
|
||||||
|
let Some((head, rest)) = split_head(&body) else {
|
||||||
|
return Ok(format_projects(project_list(store)));
|
||||||
|
};
|
||||||
|
match head {
|
||||||
|
"list" => Ok(format_projects(project_list(store))),
|
||||||
|
"show" | "walk" => {
|
||||||
|
let cs = rest.trim();
|
||||||
|
let coord = crate::commands::parse_coord(cs)
|
||||||
|
.ok_or_else(|| format!("bad coordinate '{cs}' (want C.Z.Y.X)"))?;
|
||||||
|
match show_note(store, coord) {
|
||||||
|
Some(p) => {
|
||||||
|
let name = p.header.title.as_deref().unwrap_or("?");
|
||||||
|
let notes = project_walk(store, coord);
|
||||||
|
Ok(format!(
|
||||||
|
"project {cs} \"{name}\" — {} note(s):\n{}",
|
||||||
|
notes.len(),
|
||||||
|
format_notes(¬es)
|
||||||
|
))
|
||||||
|
}
|
||||||
|
None => Ok(format!("project {cs} -> (no project record)")),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
"resume" => {
|
||||||
|
let (_, pos) = split_flags(rest);
|
||||||
|
let mut words = pos.splitn(2, char::is_whitespace);
|
||||||
|
let name = words.next().unwrap_or("").to_string();
|
||||||
|
let text = words.next().unwrap_or("").trim().to_string();
|
||||||
|
if name.is_empty() || text.is_empty() {
|
||||||
|
return Err("project resume needs <name> <text>".to_string());
|
||||||
|
}
|
||||||
|
let session = default_session();
|
||||||
|
let coord = project_resume(store, &name, &text, &session)?;
|
||||||
|
Ok(format!("resume set for '{name}' at {} (session {})", coord_str(coord), session))
|
||||||
|
}
|
||||||
|
"context" => {
|
||||||
|
let name = rest.trim();
|
||||||
|
if name.is_empty() {
|
||||||
|
return Err("project context needs <name>".to_string());
|
||||||
|
}
|
||||||
|
Ok(project_context(store, name))
|
||||||
|
}
|
||||||
|
_ => Err(format!("project: unknown subcommand '{head}' (want list|show|resume|context)")),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn format_projects(projects: Vec<(Czyx, CubeHeader, Vec<u8>)>) -> String {
|
||||||
|
if projects.is_empty() {
|
||||||
|
return "(no projects)".to_string();
|
||||||
|
}
|
||||||
|
let lines: Vec<String> = projects
|
||||||
|
.iter()
|
||||||
|
.map(|(c, h, _)| {
|
||||||
|
let name = h.title.as_deref().unwrap_or("");
|
||||||
|
let when = h.created_at.map(|t| date_string(t)).unwrap_or_else(|| "?".into());
|
||||||
|
format!(" {} [{}] {}", coord_str(*c), when, name)
|
||||||
|
})
|
||||||
|
.collect();
|
||||||
|
format!("{} project(s):\n{}", projects.len(), lines.join("\n"))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Split the first whitespace-delimited token from the rest of a string.
|
||||||
|
fn split_head(s: &str) -> Option<(&str, &str)> {
|
||||||
|
let idx = s.find(char::is_whitespace)?;
|
||||||
|
Some((&s[..idx], &s[idx..].trim()))
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn store_and_list_a_note() {
|
||||||
|
let store = ConcurrentStore::memory();
|
||||||
|
let coord = store_note(&store, "test-session", "hello", "world note", None, None).unwrap();
|
||||||
|
assert!(store.get_record(&coord).is_some());
|
||||||
|
let all = list_notes(&store, None);
|
||||||
|
assert_eq!(all.len(), 1);
|
||||||
|
assert_eq!(all[0].header.doc_type.as_deref(), Some(NOTE_DOC_TYPE));
|
||||||
|
assert_eq!(all[0].header.owner_local_user.as_deref(), Some("test-session"));
|
||||||
|
assert_eq!(all[0].body, b"world note");
|
||||||
|
assert!(all[0].header.word_flags.has(WordFlags::START_RECORD));
|
||||||
|
assert!(!all[0].header.word_flags.has(WordFlags::ENCRYPTED));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn session_and_day_filters() {
|
||||||
|
let store = ConcurrentStore::memory();
|
||||||
|
store_note(&store, "a", "one", "first", None, None).unwrap();
|
||||||
|
store_note(&store, "b", "two", "second", None, None).unwrap();
|
||||||
|
assert_eq!(
|
||||||
|
list_notes(&store, Some(&NoteFilter { session: Some("a".into()), ..Default::default() })).len(),
|
||||||
|
1
|
||||||
|
);
|
||||||
|
assert_eq!(list_notes(&store, None).len(), 2);
|
||||||
|
let today = now() / 86_400;
|
||||||
|
assert_eq!(
|
||||||
|
list_notes(&store, Some(&NoteFilter { since: Some(today), ..Default::default() })).len(),
|
||||||
|
2
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn category_and_project_filters() {
|
||||||
|
let store = ConcurrentStore::memory();
|
||||||
|
let a = store_note(&store, "s", "design", "widget design", Some("cubelinux"), Some("design")).unwrap();
|
||||||
|
let b = store_note(&store, "s", "impl", "widget impl", None, Some("impl")).unwrap();
|
||||||
|
let c = store_note(&store, "s", "cube", "cube notes", None, None).unwrap();
|
||||||
|
// category filter
|
||||||
|
assert_eq!(
|
||||||
|
list_notes(&store, Some(&NoteFilter { category: Some("design".into()), ..Default::default() })).len(),
|
||||||
|
1
|
||||||
|
);
|
||||||
|
// all categorized + base notes are still found by an unfiltered list
|
||||||
|
assert_eq!(
|
||||||
|
list_notes(&store, Some(&NoteFilter { session: Some("s".into()), ..Default::default() })).len(),
|
||||||
|
3
|
||||||
|
);
|
||||||
|
// project filter (a is linked to project cubelinux)
|
||||||
|
let pc = ensure_project(&store, "cubelinux", "s").unwrap();
|
||||||
|
assert_eq!(store.get_record(&a).unwrap().0.linked_records, vec![pc]);
|
||||||
|
assert_eq!(store.get_record(&b).unwrap().0.linked_records, vec![]);
|
||||||
|
assert_eq!(store.get_record(&c).unwrap().0.linked_records, vec![]);
|
||||||
|
let proj_notes = list_notes(&store, Some(&NoteFilter { project: Some("cubelinux".into()), ..Default::default() }));
|
||||||
|
assert_eq!(proj_notes.len(), 1);
|
||||||
|
assert_eq!(proj_notes[0].header.title.as_deref(), Some("design"));
|
||||||
|
// project walk returns the thread
|
||||||
|
assert_eq!(project_walk(&store, pc).len(), 1);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn full_text_search_and_dates() {
|
||||||
|
let store = ConcurrentStore::memory();
|
||||||
|
store_note(&store, "s", "kernel", "the ipu4 camera driver linked", None, None).unwrap();
|
||||||
|
store_note(&store, "s", "mail", "opendkim signing works", None, None).unwrap();
|
||||||
|
assert_eq!(search_notes(&store, "ipu4", None).len(), 1);
|
||||||
|
assert_eq!(search_notes(&store, "opendkim", None).len(), 1);
|
||||||
|
assert_eq!(search_notes(&store, "nomatchxyz", None).len(), 0);
|
||||||
|
assert_eq!(date_to_day("2026-09-08"), Some(20_704));
|
||||||
|
assert_eq!(date_string(20_704 * 86_400), "2026-09-08");
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn project_resume_and_context() {
|
||||||
|
let store = ConcurrentStore::memory();
|
||||||
|
store_note(&store, "s", "design", "widget design", Some("p"), Some("design")).unwrap();
|
||||||
|
project_resume(&store, "p", "at the design review", "s").unwrap();
|
||||||
|
let ctx = project_context(&store, "p");
|
||||||
|
assert!(ctx.contains("Project: p"));
|
||||||
|
assert!(ctx.contains("RESUME: at the design review"));
|
||||||
|
assert!(ctx.contains("[design]"));
|
||||||
|
}
|
||||||
|
}
|
||||||
+22
-1
@@ -574,6 +574,25 @@ impl ConcurrentStore {
|
|||||||
out
|
out
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Reconstruct a directory listing from `path` metatags: every record
|
||||||
|
/// whose `path` header field is under `dir` (the PDF's "operate on CZYX
|
||||||
|
/// records and Null-space flags rather than paths and inodes"). Delegates
|
||||||
|
/// to [`CubeStore::scan_by_path_prefix`].
|
||||||
|
pub fn query_by_path_prefix(&self, dir: &str) -> Vec<Czyx> {
|
||||||
|
self.inner.read().unwrap().scan_by_path_prefix(dir)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Exact-path metatag lookup. Delegates to [`CubeStore::scan_by_path`].
|
||||||
|
pub fn query_by_path(&self, path: &str) -> Vec<Czyx> {
|
||||||
|
self.inner.read().unwrap().scan_by_path(path)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Query records by per-word flag (the 16-bit WordFlags metadata stripe in
|
||||||
|
/// the header, tag 14). Delegates to [`CubeStore::scan_by_word_flag`].
|
||||||
|
pub fn scan_by_word_flag(&self, flag: u16) -> Vec<Czyx> {
|
||||||
|
self.inner.read().unwrap().scan_by_word_flag(flag)
|
||||||
|
}
|
||||||
|
|
||||||
/// The `owner_local_user` stamped on the record at `key`, if it has one.
|
/// The `owner_local_user` stamped on the record at `key`, if it has one.
|
||||||
/// Used by owner enforcement (Task 6): a mutating command may only
|
/// Used by owner enforcement (Task 6): a mutating command may only
|
||||||
/// overwrite a record whose owner matches the session's identity owner.
|
/// overwrite a record whose owner matches the session's identity owner.
|
||||||
@@ -683,6 +702,7 @@ impl ConcurrentStore {
|
|||||||
/// Store a code cell at `path` (the path->code bridge), durability-logged.
|
/// Store a code cell at `path` (the path->code bridge), durability-logged.
|
||||||
/// `owner` (when set) is stamped on the record's `owner_local_user` field
|
/// `owner` (when set) is stamped on the record's `owner_local_user` field
|
||||||
/// so owner enforcement (Task 6) can later reject cross-owner overwrites.
|
/// so owner enforcement (Task 6) can later reject cross-owner overwrites.
|
||||||
|
#[allow(clippy::too_many_arguments)]
|
||||||
pub fn put_code_cell(
|
pub fn put_code_cell(
|
||||||
&self,
|
&self,
|
||||||
path: &str,
|
path: &str,
|
||||||
@@ -691,9 +711,10 @@ impl ConcurrentStore {
|
|||||||
links: &[Czyx],
|
links: &[Czyx],
|
||||||
code: &[Op],
|
code: &[Op],
|
||||||
owner: Option<&str>,
|
owner: Option<&str>,
|
||||||
|
descriptor: Option<cubecode::Behavior>,
|
||||||
) -> Result<Czyx, crate::SysError> {
|
) -> Result<Czyx, crate::SysError> {
|
||||||
let coord = self.with_mut(|store| {
|
let coord = self.with_mut(|store| {
|
||||||
crate::store_code_cell(store, path, kind, name, links, code, owner)
|
crate::store_code_cell(store, path, kind, name, links, code, owner, descriptor)
|
||||||
})?;
|
})?;
|
||||||
if let Some(v) = self.get_raw(&coord) {
|
if let Some(v) = self.get_raw(&coord) {
|
||||||
self.log_put(coord, v);
|
self.log_put(coord, v);
|
||||||
|
|||||||
@@ -0,0 +1,11 @@
|
|||||||
|
[package]
|
||||||
|
name = "cubetrace"
|
||||||
|
version = "0.1.0"
|
||||||
|
edition.workspace = true
|
||||||
|
license.workspace = true
|
||||||
|
description = "CUBELinux-2 tracing package: wraps a DBI engine (via FFI seam) and streams trace events (basic blocks, syscalls) into cubestore, tagging each with CZYX coordinates and header flags (PDF Package 4, §547)."
|
||||||
|
|
||||||
|
[dependencies]
|
||||||
|
cubecoords = { path = "../cubecoords" }
|
||||||
|
cubestore = { path = "../cubestore" }
|
||||||
|
cubecode = { path = "../cubecode" }
|
||||||
@@ -0,0 +1,306 @@
|
|||||||
|
//! CUBELinux-2 tracing package.
|
||||||
|
//!
|
||||||
|
//! Per the PDF (Package 4, §547 / §1113): *"cubetrace: wraps a DBI engine
|
||||||
|
//! (via FFI) and streams trace events (basic blocks, syscalls) into
|
||||||
|
//! cubestore, tagging each with CZYX coordinates and header flags."*
|
||||||
|
//!
|
||||||
|
//! This crate is the *head* of the Package‑4 pipeline (cubetrace → cubeai →
|
||||||
|
//! cubedbt). It is dependency-free and exercises today on a `Null` (offline)
|
||||||
|
//! source so it builds and is testable without an external DBI engine; the
|
||||||
|
//! real engine is reached through the documented `DbiEngine` FFI seam.
|
||||||
|
//!
|
||||||
|
//! Model
|
||||||
|
//! -----
|
||||||
|
//! * A [`TraceEvent`] is a basic block / syscall / metadata observation with a
|
||||||
|
//! timestamp, a target coordinate, and policy/behavior header flags.
|
||||||
|
//! * A [`Tracer`] streams events into a [`CubeStore`] keyed by CZYX (the
|
||||||
|
//! `c240` trace band), tagging each record's header with time, environment,
|
||||||
|
//! and policy flags (§976).
|
||||||
|
//! * `replay_all` deterministically replays the stored stream in timestamp
|
||||||
|
//! order (§977/§1118: "deterministic replay first").
|
||||||
|
|
||||||
|
use cubecode::{Behavior, Kind};
|
||||||
|
use cubecoords::{CubeHeader, Czyx};
|
||||||
|
use cubestore::{CubeBackend, CubeStore, HashBackend};
|
||||||
|
|
||||||
|
/// `C` axis band where trace events are stored (keyed CZYX, §975).
|
||||||
|
pub const C_TRACE: u8 = 240;
|
||||||
|
|
||||||
|
/// Kind of traced observation.
|
||||||
|
#[derive(Copy, Clone, Debug, PartialEq, Eq)]
|
||||||
|
pub enum EventKind {
|
||||||
|
/// A basic block executed (carries the block's code/length).
|
||||||
|
BasicBlock,
|
||||||
|
/// A syscall entered/exited (carries the syscall number).
|
||||||
|
Syscall,
|
||||||
|
/// Free-form metadata about the run (policy, environment).
|
||||||
|
Meta,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A single traced observation: what happened, when, where, and with what
|
||||||
|
/// policy/behavior tags. Serialized into a trace record's body.
|
||||||
|
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||||
|
pub struct TraceEvent {
|
||||||
|
pub kind: EventKind,
|
||||||
|
/// Monotonic timestamp (ns from trace start).
|
||||||
|
pub ts: u64,
|
||||||
|
/// Target CZYX the event is about (the block/syscall coordinate).
|
||||||
|
pub coord: Czyx,
|
||||||
|
/// For `BasicBlock`: the executed bytecode; for `Syscall`: the syscall
|
||||||
|
/// number; for `Meta`: unused (0).
|
||||||
|
pub payload: Vec<u8>,
|
||||||
|
/// Behavior/policy descriptors to stamp onto the record header.
|
||||||
|
pub behavior: Behavior,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl TraceEvent {
|
||||||
|
/// Serialize to a stable byte form (no external deps).
|
||||||
|
pub fn to_bytes(&self) -> Vec<u8> {
|
||||||
|
let mut out = Vec::new();
|
||||||
|
out.push(self.kind as u8);
|
||||||
|
out.extend_from_slice(&self.ts.to_le_bytes());
|
||||||
|
out.extend_from_slice(&CZYX_BYTES);
|
||||||
|
out.extend_from_slice(&self.coord.pack_u32().to_le_bytes());
|
||||||
|
out.extend_from_slice(&self.behavior.to_flags().to_le_bytes());
|
||||||
|
out.push(self.payload.len() as u8);
|
||||||
|
out.extend_from_slice(&self.payload);
|
||||||
|
out
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Inverse of [`to_bytes`]. Returns `None` on malformed input.
|
||||||
|
pub fn from_bytes(b: &[u8]) -> Option<TraceEvent> {
|
||||||
|
let mut i = 0;
|
||||||
|
let kind = match *b.get(i)? {
|
||||||
|
0 => EventKind::BasicBlock,
|
||||||
|
1 => EventKind::Syscall,
|
||||||
|
2 => EventKind::Meta,
|
||||||
|
_ => return None,
|
||||||
|
};
|
||||||
|
i += 1;
|
||||||
|
let ts = u64::from_le_bytes(b.get(i..i + 8)?.try_into().ok()?);
|
||||||
|
i += 8;
|
||||||
|
i += 4; // skip CZYX_BYTES sentinel
|
||||||
|
let packed = u32::from_le_bytes(b.get(i..i + 4)?.try_into().ok()?);
|
||||||
|
i += 4;
|
||||||
|
let coord = Czyx::new(
|
||||||
|
(packed >> 24) as u8,
|
||||||
|
(packed >> 16) as u8,
|
||||||
|
(packed >> 8) as u8,
|
||||||
|
packed as u8,
|
||||||
|
);
|
||||||
|
let flags = u16::from_le_bytes(b.get(i..i + 2)?.try_into().ok()?);
|
||||||
|
i += 2;
|
||||||
|
let plen = *b.get(i)? as usize;
|
||||||
|
i += 1;
|
||||||
|
let payload = b.get(i..i + plen)?.to_vec();
|
||||||
|
Some(TraceEvent {
|
||||||
|
kind,
|
||||||
|
ts,
|
||||||
|
coord,
|
||||||
|
payload,
|
||||||
|
behavior: Behavior::from_flags(flags),
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Sentinel to make the on-wire format self-identifying as a CZYX record.
|
||||||
|
const CZYX_BYTES: [u8; 4] = *b"CZYX";
|
||||||
|
|
||||||
|
/// Source of trace events. In the full stack this is the FFI seam to a DBI
|
||||||
|
/// engine (DynamoRIO / Intel Pin / QBDI / Frida — §970). Here it is a trait so
|
||||||
|
/// the crate is testable offline via [`NullEngine`] and a real engine can be
|
||||||
|
/// dropped in without changing callers.
|
||||||
|
pub trait DbiEngine {
|
||||||
|
/// Attach to a target (pid or path). Returns `Err` if the engine can't.
|
||||||
|
fn attach(&mut self, target: &str) -> Result<(), String>;
|
||||||
|
/// Pull the next event, or `None` when the run is exhausted.
|
||||||
|
fn next_event(&mut self) -> Option<TraceEvent>;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Offline engine: replays a pre-recorded event vector. Stands in for a live
|
||||||
|
/// DBI backend in tests and headless environments.
|
||||||
|
pub struct NullEngine {
|
||||||
|
events: Vec<TraceEvent>,
|
||||||
|
idx: usize,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl NullEngine {
|
||||||
|
pub fn new(events: Vec<TraceEvent>) -> Self {
|
||||||
|
NullEngine { events, idx: 0 }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl DbiEngine for NullEngine {
|
||||||
|
fn attach(&mut self, _target: &str) -> Result<(), String> {
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
fn next_event(&mut self) -> Option<TraceEvent> {
|
||||||
|
if self.idx < self.events.len() {
|
||||||
|
let e = self.events[self.idx].clone();
|
||||||
|
self.idx += 1;
|
||||||
|
Some(e)
|
||||||
|
} else {
|
||||||
|
None
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Streams trace events into a [`CubeStore`] (the `c240` band), tagging each
|
||||||
|
/// record with time, environment, and policy header flags (PDF §976).
|
||||||
|
pub struct Tracer<B: CubeBackend> {
|
||||||
|
store: CubeStore<B>,
|
||||||
|
next_x: u8,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl<B: CubeBackend> Tracer<B> {
|
||||||
|
pub fn new(store: CubeStore<B>) -> Self {
|
||||||
|
Tracer { store, next_x: 1 }
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Drain an engine into the store, returning the number of events stored.
|
||||||
|
pub fn run<E: DbiEngine>(&mut self, engine: &mut E) -> usize {
|
||||||
|
let mut count = 0;
|
||||||
|
while let Some(ev) = engine.next_event() {
|
||||||
|
self.store_event(&ev);
|
||||||
|
count += 1;
|
||||||
|
}
|
||||||
|
count
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Store one event under a fresh CZYX coordinate in the `c240` band.
|
||||||
|
pub fn store_event(&mut self, ev: &TraceEvent) -> Czyx {
|
||||||
|
let label = Czyx::new(C_TRACE, 1, ev.kind as u8, self.next_x);
|
||||||
|
self.next_x = self.next_x.wrapping_add(1).max(1);
|
||||||
|
let mut h = CubeHeader::new();
|
||||||
|
h.title = Some(format!("{:?}:{:?}", ev.kind, ev.coord));
|
||||||
|
h.doc_type = Some(Kind::Other.as_str().into());
|
||||||
|
h.size_bytes = Some(ev.to_bytes().len() as u64);
|
||||||
|
h.flags.0 |= ev.behavior.to_flags();
|
||||||
|
h.refresh_flags();
|
||||||
|
self.store.put_record(label, &h, &ev.to_bytes());
|
||||||
|
label
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Borrow the backing store (e.g. to hand to `cubeai`/others).
|
||||||
|
pub fn store(&self) -> &CubeStore<B> {
|
||||||
|
&self.store
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Deterministic replay: collect every trace event from the `c240` band and
|
||||||
|
/// return them in timestamp order (PDF §977/§1118: "deterministic replay
|
||||||
|
/// first"). The trace is the immutable cube record; replay never mutates it.
|
||||||
|
pub fn replay_all(store: &CubeStore<HashBackend>) -> Vec<TraceEvent> {
|
||||||
|
let mut events = Vec::new();
|
||||||
|
for k in store.keys() {
|
||||||
|
if k.c != C_TRACE {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
if let Some((_, body)) = store.get_record(&k) {
|
||||||
|
if let Some(ev) = TraceEvent::from_bytes(&body) {
|
||||||
|
events.push(ev);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
events.sort_by_key(|e| e.ts);
|
||||||
|
events
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
|
||||||
|
fn ev(kind: EventKind, ts: u64, coord: Czyx, beh: Behavior) -> TraceEvent {
|
||||||
|
TraceEvent {
|
||||||
|
kind,
|
||||||
|
ts,
|
||||||
|
coord,
|
||||||
|
payload: vec![kind as u8],
|
||||||
|
behavior: beh,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn behavior(flag: u16) -> Behavior {
|
||||||
|
Behavior(flag)
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn event_round_trips_through_bytes() {
|
||||||
|
let e = ev(
|
||||||
|
EventKind::Syscall,
|
||||||
|
42,
|
||||||
|
Czyx::new(1, 2, 3, 4),
|
||||||
|
behavior(Behavior::HOT_PATH),
|
||||||
|
);
|
||||||
|
let back = TraceEvent::from_bytes(&e.to_bytes()).expect("decodes");
|
||||||
|
assert_eq!(e, back);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn tracer_streams_to_c240() {
|
||||||
|
let mut engine = NullEngine::new(vec![
|
||||||
|
ev(
|
||||||
|
EventKind::BasicBlock,
|
||||||
|
10,
|
||||||
|
Czyx::new(1, 0, 0, 1),
|
||||||
|
behavior(Behavior::PURE),
|
||||||
|
),
|
||||||
|
ev(
|
||||||
|
EventKind::Syscall,
|
||||||
|
20,
|
||||||
|
Czyx::new(1, 0, 0, 2),
|
||||||
|
behavior(Behavior::IO_HEAVY),
|
||||||
|
),
|
||||||
|
ev(
|
||||||
|
EventKind::Meta,
|
||||||
|
5,
|
||||||
|
Czyx::new(1, 0, 0, 3),
|
||||||
|
Behavior::default(),
|
||||||
|
),
|
||||||
|
]);
|
||||||
|
let mut tracer = Tracer::new(CubeStore::new(HashBackend::new()));
|
||||||
|
assert_eq!(tracer.run(&mut engine), 3);
|
||||||
|
// All records landed in the c240 band.
|
||||||
|
let stored: usize = tracer
|
||||||
|
.store()
|
||||||
|
.keys()
|
||||||
|
.into_iter()
|
||||||
|
.filter(|k| k.c == C_TRACE)
|
||||||
|
.count();
|
||||||
|
assert_eq!(stored, 3);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn replay_is_deterministic_and_ordered() {
|
||||||
|
let mut engine = NullEngine::new(vec![
|
||||||
|
ev(
|
||||||
|
EventKind::BasicBlock,
|
||||||
|
30,
|
||||||
|
Czyx::new(1, 0, 0, 1),
|
||||||
|
behavior(Behavior::PURE),
|
||||||
|
),
|
||||||
|
ev(
|
||||||
|
EventKind::Syscall,
|
||||||
|
10,
|
||||||
|
Czyx::new(1, 0, 0, 2),
|
||||||
|
behavior(Behavior::IO_HEAVY),
|
||||||
|
),
|
||||||
|
ev(
|
||||||
|
EventKind::Meta,
|
||||||
|
20,
|
||||||
|
Czyx::new(1, 0, 0, 3),
|
||||||
|
Behavior::default(),
|
||||||
|
),
|
||||||
|
]);
|
||||||
|
let mut tracer = Tracer::new(CubeStore::new(HashBackend::new()));
|
||||||
|
tracer.run(&mut engine);
|
||||||
|
let replayed = replay_all(tracer.store());
|
||||||
|
assert_eq!(replayed.len(), 3);
|
||||||
|
// Sorted by ts: 10 (Syscall), 20 (Meta), 30 (BasicBlock).
|
||||||
|
assert_eq!(replayed[0].kind, EventKind::Syscall);
|
||||||
|
assert_eq!(replayed[1].kind, EventKind::Meta);
|
||||||
|
assert_eq!(replayed[2].kind, EventKind::BasicBlock);
|
||||||
|
// Behavior flags survived the round-trip through the store.
|
||||||
|
assert_eq!(replayed[0].behavior, Behavior(Behavior::IO_HEAVY));
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,49 @@
|
|||||||
|
# CUBE notes ↔ model integration: `cube-notes-agent`
|
||||||
|
|
||||||
|
The CUBELinux-2 note/message-save-path is a durable, searchable log that a small
|
||||||
|
local LLM reads, reviews, and writes back to — a closed loop that is **grounded
|
||||||
|
in CUBE** (the model never invents findings) and **incremental** (each run reviews
|
||||||
|
only what has not been seen).
|
||||||
|
|
||||||
|
## The notes system (`cubesys::notes`)
|
||||||
|
- WordFlags-tagged **CZYX** note log. Each note has `doc_type = note:<category>`
|
||||||
|
(categories like `finding`, `action`, `checkpoint`, `resume`, `task`, …) and an
|
||||||
|
optional **project** association via `linked_records`.
|
||||||
|
- `cube note add|list|search|show`; `cube project list|show|resume|context`.
|
||||||
|
- `cube note tag <coord> <cat>` reclassifies a note in place (Puts are durable;
|
||||||
|
`delete_raw` is not — the checkpoint/delta only records Puts of surviving keys,
|
||||||
|
so a deleted key resurrects from the base snapshot on reload).
|
||||||
|
- Durable `ConcurrentStore` (WAL + delta + checkpoint). Short-lived processes must
|
||||||
|
`store.checkpoint()` before exit or buffered writes are lost.
|
||||||
|
|
||||||
|
## Model ↔ CUBE
|
||||||
|
- `cube-notes-mcp.py` (stdio MCP server) exposes `mcp__cubenotes__note_*` /
|
||||||
|
`project_*` tools to the DeepSeek Harness. The harness registers the mcp-client at
|
||||||
|
the **bundle** layer (`inject: [tools]`); profile-layer rows can't inject the tools
|
||||||
|
service, so the plugin silently never mounts — the bundle registration was the fix
|
||||||
|
that made `mcp__cubenotes__note_*` live.
|
||||||
|
- `tools/cube-notes-agent.py` (headless, cron-driven, no control panel) treats CUBE as
|
||||||
|
the agent's memory:
|
||||||
|
- every action is written as a note (`category=action`) so a run can be replayed;
|
||||||
|
- a review run reads **only uncovered** findings (`category=finding`), asks the model
|
||||||
|
to rank them, then writes a grounded report and marks them covered with a
|
||||||
|
`checkpoint` note — true incremental, non-destructive review;
|
||||||
|
- the report is composed in Python from the finding notes **verbatim**, with a
|
||||||
|
deterministic severity (HIGH/MEDIUM/LOW) and priority order, so no content is
|
||||||
|
invented. Report ⇒ a review `.txt` (`/root/workspace/cube-agent/report.txt`).
|
||||||
|
|
||||||
|
## Success: tracing errors from prior run data saved in CUBE
|
||||||
|
- The findings/action trace already stored in CUBE (from prior runs) is the source of
|
||||||
|
truth. The broad scan surfaced **7 un-categorized logs** that had been logged but
|
||||||
|
never tagged `finding`: `GIT-SHALLOW-CLONE`, `POSTFIX-TLS-CERTS`,
|
||||||
|
`SELINUX-RESTORECON`, `DSH-WEB-EADDRINUSE`, `CONCURRENT-STORE-CHECKPOINT`,
|
||||||
|
`MCP-BUNDLE-LAYER`, `CUBES-TWO-TREES`. See `git log` entry `bd3e3c0`.
|
||||||
|
- Coverage is precise: the report marks exactly the findings it reviewed, so the next
|
||||||
|
run reports only new ones. The notes store is also the harness GUI's CUBE surface
|
||||||
|
(`mcp-cubenotes note_list`), so everything the agent writes is visible and reviewable.
|
||||||
|
|
||||||
|
## Usage
|
||||||
|
```
|
||||||
|
python3 tools/cube-notes-agent.py --max-turns 6 --report /path/report.txt
|
||||||
|
```
|
||||||
|
Runs single-instance via `flock`; a cron schedule (e.g. `0 6,18 * * *`) drives it.
|
||||||
@@ -0,0 +1,405 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Headless CUBE notes agent — NO control panel. Reads a task from the CUBE
|
||||||
|
notes (a project), uses the cube note tools (note_search/note_list/
|
||||||
|
project_context + read_file), and writes a report. Designed to be driven by
|
||||||
|
cron (e.g. build an error-log report every hour); the interval is the cron
|
||||||
|
schedule, so it is fully configurable without touching the agent.
|
||||||
|
|
||||||
|
Usage: python3 agent.py --project NAME [--max-turns N] [--report PATH]
|
||||||
|
"""
|
||||||
|
import argparse, json, os, subprocess, sys, time
|
||||||
|
import datetime as _dt
|
||||||
|
|
||||||
|
def _now_ts():
|
||||||
|
return _dt.datetime.now().strftime("%Y-%m-%d %H:%M")
|
||||||
|
|
||||||
|
MODEL = "/home/luulu/models/gemma-3-4b-it-Q4_K_M.gguf"
|
||||||
|
LLAMA = "/home/luulu/llama.cpp/build/bin/llama-server"
|
||||||
|
CUBE = "/home/CUBELinux/CUBELinux-2/target/release/cube"
|
||||||
|
PORT = 8080
|
||||||
|
BASE = f"http://127.0.0.1:{PORT}"
|
||||||
|
LOG = "/root/workspace/cube-agent/llama.log" if os.path.isdir("/root/workspace/cube-agent") else "/tmp/local-agent-llama.log"
|
||||||
|
TRACE_PROJECT = "agent-trace" # every action the agent takes is logged here (replay/tuning)
|
||||||
|
|
||||||
|
def env():
|
||||||
|
e = dict(os.environ)
|
||||||
|
e["CUBE_NOTES_DIR"] = "/root/.cubelinux-notes"
|
||||||
|
return e
|
||||||
|
|
||||||
|
def start_llama():
|
||||||
|
p = subprocess.Popen(
|
||||||
|
[LLAMA, "--model", MODEL, "--host", "127.0.0.1", "--port", str(PORT),
|
||||||
|
"--alias", "gemma-3-4b", "--ctx-size", "8192", "-np", "1",
|
||||||
|
"--temp", "0.2", "--repeat-penalty", "1.1"],
|
||||||
|
stdout=open(LOG, "w"), stderr=subprocess.STDOUT, env=env())
|
||||||
|
import requests
|
||||||
|
for _ in range(180):
|
||||||
|
try:
|
||||||
|
if requests.get(BASE + "/v1/models", timeout=2).status_code == 200:
|
||||||
|
break
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
time.sleep(2)
|
||||||
|
return p
|
||||||
|
|
||||||
|
def chat(messages, max_tokens=600):
|
||||||
|
import requests
|
||||||
|
r = requests.post(BASE + "/v1/chat/completions", json={
|
||||||
|
"model": "gemma-3-4b", "messages": messages,
|
||||||
|
"max_tokens": max_tokens, "temperature": 0.08}, timeout=600)
|
||||||
|
return r.json()["choices"][0]["message"]["content"]
|
||||||
|
|
||||||
|
def run_cube(args):
|
||||||
|
try:
|
||||||
|
r = subprocess.run([CUBE] + args, capture_output=True, text=True,
|
||||||
|
env=env(), timeout=30)
|
||||||
|
return (r.stdout or r.stderr).strip()
|
||||||
|
except Exception as e:
|
||||||
|
return f"error: {e}"
|
||||||
|
|
||||||
|
def log_action(step, text):
|
||||||
|
"""Write an agent action to the CUBE notes trace (category=action) so the
|
||||||
|
run can be replayed and fine-tuned."""
|
||||||
|
body = f"step {step}: {text}"[:1600]
|
||||||
|
subprocess.run([CUBE, "note", "add", "--project", TRACE_PROJECT,
|
||||||
|
"--cat", "action", body], capture_output=True, text=True, env=env())
|
||||||
|
|
||||||
|
def open_file(path):
|
||||||
|
try:
|
||||||
|
return open(path).read()[:2000]
|
||||||
|
except Exception as e:
|
||||||
|
return f"error: {e}"
|
||||||
|
|
||||||
|
# ---- incremental coverage (deterministic; offloaded from the small model) ----
|
||||||
|
import re as _re
|
||||||
|
COORD_RE = _re.compile(r'\b(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\b')
|
||||||
|
|
||||||
|
def _coords(text):
|
||||||
|
return set(COORD_RE.findall(text))
|
||||||
|
|
||||||
|
# Bookkeeping categories that are NOT error-log findings (agent trace, coverage
|
||||||
|
# markers, resume checkpoints, test fixtures, design/impl notes, misc).
|
||||||
|
NONLOG_CATS = {"action", "checkpoint", "resume", "task", "debug", "impl", "design", "misc"}
|
||||||
|
# A finding-like title: a SHORT-UPPERCASE-HYPHEN/UNDERSCORE token then ':' or ' — '
|
||||||
|
FLAG_RE = _re.compile(r'\s*([A-Z][A-Z0-9_-]{2,})\s*[:—-]')
|
||||||
|
ERR_KW = ("error", "failed", "failure", "denied", "rejected", "cannot", "cannot be",
|
||||||
|
"must ", "timeout", "times out", "not registered", "broken", "corrupt",
|
||||||
|
"integrity", "permission denied", "selinux", "restorecon", "orphan",
|
||||||
|
"eadrinuse", "shallow", "user_tmp_t", "mcp-client", "mcp plugin")
|
||||||
|
|
||||||
|
def _scan(cat, project=None):
|
||||||
|
"""Run `cube note list`, optionally filtered by category/project. With no cat
|
||||||
|
and no project this scans the WHOLE store (every session), so the agent
|
||||||
|
reviews ALL CUBE error logs, not one project or one category."""
|
||||||
|
args = ["note", "list"]
|
||||||
|
if cat:
|
||||||
|
args += ["--cat", cat]
|
||||||
|
if project:
|
||||||
|
args += ["--project", project]
|
||||||
|
return run_cube(args)
|
||||||
|
|
||||||
|
def _parse_note_line(line):
|
||||||
|
"""-> (coord, cat_or_None, subject) for a `note list` line, or None if the
|
||||||
|
line is not a note (e.g. the 'N note(s):' header)."""
|
||||||
|
m = _re.match(r'\s*(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\s+\[[^\]]+\]', line)
|
||||||
|
if not m:
|
||||||
|
return None
|
||||||
|
coord = m.group(1)
|
||||||
|
rest = line[m.end():].strip()
|
||||||
|
cat = None
|
||||||
|
cm = _re.match(r'\[([a-z]+)\]\s*', rest)
|
||||||
|
if cm:
|
||||||
|
cat = cm.group(1)
|
||||||
|
rest = rest[cm.end():].strip()
|
||||||
|
return coord, cat, rest
|
||||||
|
|
||||||
|
def _looks_like_finding(subj):
|
||||||
|
"""True if an un-categorized note looks like an error-log finding."""
|
||||||
|
if FLAG_RE.match(subj):
|
||||||
|
return True
|
||||||
|
low = subj.lower()
|
||||||
|
return any(k in low for k in ERR_KW)
|
||||||
|
|
||||||
|
def _log_findings(project=None):
|
||||||
|
"""[(coord, cat, subject)] for every finding-like note in the store, across ALL
|
||||||
|
categories and projects. This includes `finding` notes and any other note with a
|
||||||
|
FLAG-like / error title (the un-categorized logs), while excluding bookkeeping
|
||||||
|
categories (action / checkpoint / resume / task / debug / impl / design / misc)."""
|
||||||
|
out = []
|
||||||
|
for line in _scan(None, project).splitlines():
|
||||||
|
parsed = _parse_note_line(line)
|
||||||
|
if parsed is None:
|
||||||
|
continue
|
||||||
|
coord, cat, subj = parsed
|
||||||
|
if cat in NONLOG_CATS:
|
||||||
|
continue
|
||||||
|
if cat == "finding" or _looks_like_finding(subj):
|
||||||
|
out.append((coord, cat, subj))
|
||||||
|
return out
|
||||||
|
|
||||||
|
def _checkpoint_coords(project=None):
|
||||||
|
"""Own coord (leftmost) of every checkpoint note (optionally in a project)."""
|
||||||
|
coords = []
|
||||||
|
for line in _scan("checkpoint", project).splitlines():
|
||||||
|
m = _re.match(r'\s*(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})', line)
|
||||||
|
if m:
|
||||||
|
coords.append(m.group(1))
|
||||||
|
return coords
|
||||||
|
|
||||||
|
def _covered(project=None):
|
||||||
|
"""Covered = each checkpoint note's own coord + the coords in its FULL body.
|
||||||
|
(note list only shows the truncated subject, so use note show to read the body.)"""
|
||||||
|
out = set()
|
||||||
|
for c in _checkpoint_coords(project):
|
||||||
|
out.add(c)
|
||||||
|
out |= _coords(run_cube(["note", "show", c]))
|
||||||
|
return out
|
||||||
|
|
||||||
|
def uncovered_findings(project=None):
|
||||||
|
covered = _covered(project)
|
||||||
|
un = "".join(f"{coord} {subj}\n" for coord, cat, subj in _log_findings(project) if coord not in covered)
|
||||||
|
return un.strip() or "(no new findings)"
|
||||||
|
|
||||||
|
# Canonical issue key: collapse duplicate representations of the same root cause
|
||||||
|
# (e.g. FK1-SELINUX_RESTORECON vs SELINUX-RESTORECON) so the report dedupes by ISSUE,
|
||||||
|
# not just by coordinate. Prefixes (FK, FK#, FA, FB, FLAG, FIX) are stripped and
|
||||||
|
# separators normalized; a small alias map unifies differently-worded duplicates.
|
||||||
|
_ISSUE_PREFIX_RE = _re.compile(r'^(FK\d*|FA|FB|FLAG|FIX)[-_]?', _re.I)
|
||||||
|
_ISSUE_ALIAS = {
|
||||||
|
"wal_checkpoint": "checkpoint_before_exit",
|
||||||
|
"concurrent_store_checkpoint": "checkpoint_before_exit",
|
||||||
|
}
|
||||||
|
|
||||||
|
def _canonical_issue(flag):
|
||||||
|
f = _ISSUE_PREFIX_RE.sub('', flag)
|
||||||
|
f = _re.sub(r'[-_/\s]+', '_', f).strip('_').lower()
|
||||||
|
return _ISSUE_ALIAS.get(f, f)
|
||||||
|
|
||||||
|
def _uncovered_meta(project=None):
|
||||||
|
"""[(coord, flag, desc)] for the currently-UNCOVERED findings, DEDUPED BY ISSUE.
|
||||||
|
desc is the FULL note body; the most-detailed note per canonical issue is kept."""
|
||||||
|
covered = _covered(project)
|
||||||
|
best = {} # issue -> (coord, flag, desc, len)
|
||||||
|
for coord, cat, subj in _log_findings(project):
|
||||||
|
if coord in covered:
|
||||||
|
continue
|
||||||
|
show = run_cube(["note", "show", coord])
|
||||||
|
body_text = show.split("\n", 1)[1].strip() if "\n" in show else ""
|
||||||
|
if not body_text:
|
||||||
|
body_text = subj
|
||||||
|
flag, sep, rest = body_text.partition(":")
|
||||||
|
flag = flag.strip()
|
||||||
|
desc = rest.strip() if sep else body_text
|
||||||
|
issue = _canonical_issue(flag)
|
||||||
|
cur = best.get(issue)
|
||||||
|
if cur is None or len(desc) > cur[3]:
|
||||||
|
best[issue] = (coord, flag, desc, len(desc))
|
||||||
|
return [(c, f, d) for c, f, d, _ in best.values()]
|
||||||
|
|
||||||
|
# Snapshot of the uncovered findings captured when read_findings ran, so the
|
||||||
|
# report can be composed faithfully even after mark_covered has covered them.
|
||||||
|
CURRENT_META = []
|
||||||
|
|
||||||
|
def read_findings(project=None):
|
||||||
|
"""Deterministic, grounded input for the model: the currently-UNCOVERED CUBE
|
||||||
|
findings, numbered with their FLAG. The model replies with ONLY the FLAGs in
|
||||||
|
priority order; Python composes the report from the grounded text, so it is
|
||||||
|
impossible for any finding to be invented."""
|
||||||
|
global CURRENT_META
|
||||||
|
meta = _uncovered_meta(project)
|
||||||
|
CURRENT_META = meta
|
||||||
|
if not meta:
|
||||||
|
return "(no new findings)"
|
||||||
|
lines = []
|
||||||
|
for i, (coord, flag, desc) in enumerate(meta, 1):
|
||||||
|
descf = (desc[:120] + "…") if len(desc) > 120 else desc
|
||||||
|
lines.append(f"{i}. {flag}: {descf} (coord {coord})")
|
||||||
|
return ("Uncovered CUBE findings (use ONLY these; do not invent, add, or guess):\n"
|
||||||
|
+ "\n".join(lines)
|
||||||
|
+ "\n\nReply with the finding FLAGs in priority order, most critical first, "
|
||||||
|
"comma-separated, using exactly the FLAGs above and nothing else — for example:\n"
|
||||||
|
+ ", ".join(f for _, f, _ in meta)
|
||||||
|
+ "\nThen call mark_covered, then reply done (no report body needed).")
|
||||||
|
|
||||||
|
_SEV_CRIT = ["data integrity", "security", "tls", "cert", "denied", "cannot",
|
||||||
|
"corrupt", "lost", "inaccessible", "integrity", "breach"]
|
||||||
|
_SEV_HIGH = ["timeout", "persist", "reload", "shallow", "must", "fail", "broken"]
|
||||||
|
|
||||||
|
def _severity(flag, desc):
|
||||||
|
"""Deterministic severity label from the finding's FLAG + description. Used to
|
||||||
|
give the report a meaningful priority order when the model provides no ranking."""
|
||||||
|
t = (flag + " " + desc).lower()
|
||||||
|
if any(k in t for k in _SEV_CRIT):
|
||||||
|
return "HIGH"
|
||||||
|
if any(k in t for k in _SEV_HIGH):
|
||||||
|
return "MEDIUM"
|
||||||
|
return "LOW"
|
||||||
|
|
||||||
|
def compose_report(project=None, model_text=""):
|
||||||
|
"""Build the grounded, prioritized report. The model supplies ONLY the priority
|
||||||
|
ORDER of the FLAGs (honored when present); every finding's text is taken verbatim
|
||||||
|
from CUBE, so any hallucinated finding is impossible. Unknown/duplicate flags are
|
||||||
|
dropped; flags the model did not rank are ordered by a deterministic severity
|
||||||
|
heuristic (HIGH → MEDIUM → LOW), then CUBE order. Uses the snapshot from read_findings."""
|
||||||
|
meta = CURRENT_META or _uncovered_meta(project)
|
||||||
|
if not meta:
|
||||||
|
return "ERROR LOG REPORT (ALL)\n\nNo new findings since the last checkpoint."
|
||||||
|
text_low = model_text.lower()
|
||||||
|
listed = {}
|
||||||
|
for coord, flag, desc in meta:
|
||||||
|
listed.setdefault(flag.lower(), (coord, flag, desc))
|
||||||
|
# Model's ranking first (by first-flag-appearance order in its reply).
|
||||||
|
positions = []
|
||||||
|
for key in listed:
|
||||||
|
idx = text_low.find(key)
|
||||||
|
if idx >= 0:
|
||||||
|
positions.append((idx, key))
|
||||||
|
positions.sort()
|
||||||
|
order = [key for _, key in positions]
|
||||||
|
remaining = [key for key in listed if key not in order]
|
||||||
|
sev_rank = {"HIGH": 0, "MEDIUM": 1, "LOW": 2}
|
||||||
|
remaining.sort(key=lambda k: sev_rank.get(_severity(listed[k][1], listed[k][2]), 3))
|
||||||
|
order += remaining
|
||||||
|
scope = project if project else "ALL"
|
||||||
|
lines = [f"ERROR LOG REPORT (scope: {scope})",
|
||||||
|
f"Generated: {_now_ts()}", ""]
|
||||||
|
for rank, key in enumerate(order, 1):
|
||||||
|
coord, flag, desc = listed[key]
|
||||||
|
sev = _severity(flag, desc)
|
||||||
|
lines.append(f"{rank}. [{sev}] {flag}: {desc if desc else '(no description)'}")
|
||||||
|
lines.append(f" coord: {coord}")
|
||||||
|
return "\n".join(lines)
|
||||||
|
|
||||||
|
def mark_covered(project=None, text=""):
|
||||||
|
covered = _covered(project)
|
||||||
|
coords = [c for c, cat, subj in _log_findings(project) if c not in covered]
|
||||||
|
if not coords:
|
||||||
|
return "nothing to cover"
|
||||||
|
body = "covered coords:\n" + "\n".join(coords)
|
||||||
|
args = [CUBE, "note", "add", "--cat", "checkpoint"]
|
||||||
|
if project:
|
||||||
|
args += ["--project", project]
|
||||||
|
args.append(body[:1400])
|
||||||
|
subprocess.run(args, capture_output=True, text=True, env=env())
|
||||||
|
return "marked covered"
|
||||||
|
|
||||||
|
def tool(kw, args, project):
|
||||||
|
if kw == "read_findings":
|
||||||
|
return read_findings(project)
|
||||||
|
if kw == "mark_covered":
|
||||||
|
return mark_covered(project, args)
|
||||||
|
if kw == "note_search":
|
||||||
|
return run_cube(["note", "search", args])
|
||||||
|
if kw == "note_list":
|
||||||
|
return run_cube(["note", "list"] + (args.split() if args else []))
|
||||||
|
if kw == "project_context":
|
||||||
|
return run_cube(["project", "context", args])
|
||||||
|
if kw == "read_file":
|
||||||
|
return open_file(args)
|
||||||
|
return f"unknown tool: {kw}"
|
||||||
|
|
||||||
|
ACTIONS = ("read_findings", "mark_covered", "note_search",
|
||||||
|
"note_list", "project_context", "read_file", "done")
|
||||||
|
|
||||||
|
def parse_action(text):
|
||||||
|
"""Parse an action word out of the model's reply. Tolerant of markdown code
|
||||||
|
fences, ```tool_code``` wrappers, and prose preamble (the small gemma model
|
||||||
|
tends to wrap calls like this). Scans lines and stops once it sees "done",
|
||||||
|
so the trailing report is never misread as another tool call."""
|
||||||
|
best = (None, "")
|
||||||
|
for raw in text.strip().splitlines():
|
||||||
|
s = raw.replace("`", "").replace("tool_code", " ").strip()
|
||||||
|
low = s.lower()
|
||||||
|
for kw in ACTIONS:
|
||||||
|
idx = low.find(kw)
|
||||||
|
if idx >= 0:
|
||||||
|
best = (kw, s[idx + len(kw):].strip())
|
||||||
|
break
|
||||||
|
if best[0] == "done":
|
||||||
|
break
|
||||||
|
return best
|
||||||
|
|
||||||
|
def main():
|
||||||
|
ap = argparse.ArgumentParser()
|
||||||
|
ap.add_argument("--project", default=None,
|
||||||
|
help="Restrict the report to one CUBE project (default: ALL finding notes across the store)")
|
||||||
|
ap.add_argument("--max-turns", type=int, default=6)
|
||||||
|
ap.add_argument("--report", default="/root/workspace/cube-agent/report.txt")
|
||||||
|
a = ap.parse_args()
|
||||||
|
|
||||||
|
p = start_llama()
|
||||||
|
scope = a.project if a.project else "ALL"
|
||||||
|
# Ground the report in CUBE up front: snapshot the uncovered findings and hand
|
||||||
|
# them to the model so it only has to RANK them (it never writes finding text).
|
||||||
|
findings_text = read_findings(a.project)
|
||||||
|
if findings_text == "(no new findings)":
|
||||||
|
initial_context = "There are no uncovered findings to report right now."
|
||||||
|
task = "Reply done (no report body needed)."
|
||||||
|
else:
|
||||||
|
initial_context = findings_text
|
||||||
|
task = ("Reply with the finding FLAGs in priority order (most critical first), "
|
||||||
|
"comma-separated, using exactly the FLAGs below and nothing else — then reply done.")
|
||||||
|
sys_prompt = (
|
||||||
|
f"You are a headless error-log reviewer for the CUBE notes store (scope: {scope}).\n"
|
||||||
|
"The findings are provided below. Order them by priority/severity, most critical first.\n"
|
||||||
|
"Reply with the FLAGs in that order, comma-separated, using exactly those FLAGs and nothing else.\n"
|
||||||
|
"Then reply: done. "
|
||||||
|
"Your ranked FLAG list sets the priority; the report body is assembled for you, so do not invent findings."
|
||||||
|
)
|
||||||
|
messages = [
|
||||||
|
{"role": "system", "content": sys_prompt},
|
||||||
|
{"role": "user", "content": f"{task}\n\n{initial_context}"},
|
||||||
|
]
|
||||||
|
trace = []
|
||||||
|
report_text = ""
|
||||||
|
last_assistant = ""
|
||||||
|
for turn in range(a.max_turns):
|
||||||
|
try:
|
||||||
|
text = chat(messages)
|
||||||
|
except Exception as e:
|
||||||
|
trace.append(f"chat error: {e}")
|
||||||
|
break
|
||||||
|
kw, args = parse_action(text)
|
||||||
|
if kw == "done":
|
||||||
|
# The model supplies ONLY the priority order of the FLAGs; Python
|
||||||
|
# composes the grounded report (impossible to invent a finding).
|
||||||
|
di = text.lower().find("done")
|
||||||
|
model_text = text[di + 4:].strip() if di >= 0 else last_assistant
|
||||||
|
report_text = compose_report(a.project, model_text)
|
||||||
|
log_action(turn, f"DONE\n\n~flags: {model_text[:400]}")
|
||||||
|
break
|
||||||
|
if kw:
|
||||||
|
result = tool(kw, args, a.project)
|
||||||
|
trace.append(f"turn{turn}: {kw}({args})")
|
||||||
|
log_action(turn, f"{kw}({args})\n\n{result[:1400]}")
|
||||||
|
messages.append({"role": "assistant", "content": text})
|
||||||
|
messages.append({"role": "user",
|
||||||
|
"content": f"Tool result:\n{result}\n\nContinue — output exactly one "
|
||||||
|
"action line (read_findings / mark_covered / note_search / "
|
||||||
|
"note_list / project_context / read_file / done)."})
|
||||||
|
else:
|
||||||
|
log_action(turn, f"(no action) {text[:300]}")
|
||||||
|
messages.append({"role": "assistant", "content": text})
|
||||||
|
messages.append({"role": "user",
|
||||||
|
"content": "Output exactly ONE action line starting with a tool name."})
|
||||||
|
trace.append(f"turn{turn}: (no action)")
|
||||||
|
last_assistant = text
|
||||||
|
if not report_text:
|
||||||
|
# Run ended without an explicit "done" — still produce a grounded report.
|
||||||
|
report_text = compose_report(a.project, last_assistant)
|
||||||
|
# Deterministic coverage: mark all uncovered findings covered (including any
|
||||||
|
# deduped-away duplicates) regardless of whether the model did it, so the next
|
||||||
|
# run is genuinely incremental. No-op when there are no uncovered findings.
|
||||||
|
mark_covered(a.project)
|
||||||
|
with open(a.report, "w") as f:
|
||||||
|
f.write(report_text)
|
||||||
|
os.makedirs(os.path.dirname(a.report), exist_ok=True) if os.path.dirname(a.report) else None
|
||||||
|
print("AGENT_DONE max_turns=", a.max_turns, "report=", a.report,
|
||||||
|
"bytes=", os.path.getsize(a.report) if os.path.exists(a.report) else 0)
|
||||||
|
try:
|
||||||
|
p.terminate()
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
Reference in New Issue
Block a user