118 lines
6.9 KiB
Markdown
118 lines
6.9 KiB
Markdown
# RESUME POINT — cubefs FUSE <-> cube-server daemon durability wiring
|
|
|
|
Saved: 2026-08-13 (user paused session; pick up from here).
|
|
|
|
## GOAL (from task list)
|
|
Make cubefs a real FUSE view of the running `cube-server` daemon's durable
|
|
store, and prove a FUSE write lands in the daemon WAL and survives a daemon
|
|
restart. (Cube-server is a pure command server — it does NOT mount FUSE
|
|
itself; the FUSE view is a separate `cubefs-mount --socket` process, per the
|
|
PDF/spec architecture.)
|
|
|
|
## WHAT IS DONE (compiles + tests green)
|
|
1. Raw-coordinate command family added to `cubesys/src/commands.rs` dispatch:
|
|
`rawget <c> <z> <y> <x>`, `rawput <c> <z> <y> <x> <hex>`,
|
|
`rawdel <c> <z> <y> <x>`, `rawkeys`, `rawscan <c> [z] [y]`.
|
|
Helpers added: `parse_u8`, `hex_encode`, `hex_decode`.
|
|
(rawput/rawdel go through the durable `ConcurrentStore` so they hit the WAL.)
|
|
2. `cubefs/src/backend.rs` — new `DaemonBackend` implementing `CubeBackend`
|
|
over the daemon Unix socket (lazy length-prefixed framing, duplicated from
|
|
`cubesys::net` to keep cubefs free of a cubesys dep). Registered in
|
|
`cubefs/src/lib.rs` (`pub mod backend; pub use backend::DaemonBackend;`).
|
|
`DaemonBackend` is Clone (clones share socket path, reconnect lazily).
|
|
3. `cubefs/src/bin/cubefs_mount.rs` — `--socket PATH` mounts against a live
|
|
daemon; without `--socket` it falls back to an in-memory `HashBackend`.
|
|
4. `cargo build --workspace` GREEN. `cargo test --workspace` GREEN.
|
|
(Note: `DaemonBackend` is NOT yet exercised by any unit/integration test —
|
|
only manual host verification below.)
|
|
|
|
## WHAT IS PROVEN (host-level, same kernel/code path as the VM)
|
|
A. Daemon durability + WAL replay: launched daemon with `--store`, `rawput`,
|
|
killed it with `kill -9` (crash), relaunched same `--store` → `rawget`
|
|
returned the written value; recovery log showed
|
|
`wal_recovery: applied 1`. So the DAEMON half of the durability story works.
|
|
B. Direct `rawput` from a Python client checkpoints to `store.json` (4-byte
|
|
record appeared after the checkpoint interval). Confirmed daemon is alive,
|
|
reachable, and durable on the host.
|
|
|
|
## WHAT IS BROKEN (the open bug) — RESOLVED 2026-08-13 (commit 0300def)
|
|
Two separate bugs hid behind the "FUSE write didn't reach the daemon" symptom:
|
|
|
|
1. `--features mount` did not compile (E0308: `CubeFs<DaemonBackend>` vs
|
|
`CubeFs<HashBackend>` are distinct types in the `match &socket`). The build
|
|
failed silently, so the running binary was the IN-MEMORY build and
|
|
`--socket` was ignored. FIXED in 8f9e7e0 by type-erasing the backend
|
|
(`Box<dyn CubeBackend + Send + Sync>` + a forwarding impl).
|
|
|
|
2. Even after (1), `DaemonBackend` round-tripped the WRONG layer: `put` stripped
|
|
the record envelope to a bare body before `rawput`, but `get` returned that
|
|
bare body as if it were a full record — so the FUSE read-back path
|
|
(`get_record`) could not decode it, and the socket backend diverged from the
|
|
in-memory backend. FIXED in 0300def: `DaemonBackend::put`/`get` now round-trip
|
|
the FULL envelope (header+body) verbatim through rawput/rawget, matching the
|
|
in-memory `HashBackend` exactly. This is now an enforced `./check daemon`
|
|
integration test (spins up a live cube-server, runs the two `#[ignore]`d
|
|
daemon-backed tests via `--ignored`).
|
|
|
|
## GATE STATUS (2026-08-13, final)
|
|
- `./check` (fmt+test+clippy -D warnings) GREEN.
|
|
- `./check daemon` GREEN — socket-backed record round-trip against a real daemon.
|
|
- `./check mount` GREEN — NOW runs the regression suite against BOTH the
|
|
in-memory `--seed` mount AND a daemon-backed `--socket` mount (with its own
|
|
fresh daemon), plus a `durable across restart` assertion. 57/57 pass. This is
|
|
the regression guard exercising the real kernel VFS path for BOTH backends.
|
|
|
|
## COMPLETED THIS SESSION (chronological)
|
|
- Fixed `DaemonBackend` envelope round-trip (commit 0300def).
|
|
- Fixed `--features mount` compile break (commit 8f9e7e0).
|
|
- Enforced both via `./check daemon` (the two #[ignore]d integration tests).
|
|
- Host: `./check` + `./check daemon` + `./check mount` all GREEN (mount now
|
|
covers the daemon-backed path too).
|
|
- VM (cubelinux-vm.qcow2, was pre-fix): rebuilt cube-server + cubefs-mount from
|
|
the fixed source; rewired cubefs.service to `--socket /run/cube/cube.sock`
|
|
with `Requires=cube-server.service`; fixed TWO real deployment bugs found only
|
|
because the new gate exercised the socket path:
|
|
(a) systemd ordering cycle (cube-server Wanted cubefs, cubefs Required
|
|
cube-server) — broke the cycle by dropping the cubefs dependency from
|
|
cube-server.service.
|
|
(b) `ExecStartPost chmod 666` raced the daemon's socket creation under the
|
|
restart loop and failed the unit — made it tolerant+delayed.
|
|
Proved a FUSE write survives a daemon restart (WAL persistence) on the VM.
|
|
Injected SSH key for cube+root to drive the VM. Snapshot `fixed-durable-20260813`
|
|
captures the working durable stack.
|
|
- `./check` mount stage hardened (commit da3aa3a): daemon-backed leg + durability
|
|
assertion; `run_fs_tests` is now backend-agnostic (self-seeds its fixture).
|
|
|
|
## NEXT STEPS (remaining, low priority)
|
|
1. DONE 2026-08-13 — `build_vm.sh` now rebuilds cube binaries with
|
|
`--features cubefs/mount` into a clean staging dir before copying, and aborts
|
|
if `cubefs-mount` is missing. A from-scratch image bake can no longer silently
|
|
omit the FUSE adapter. (Prior to this the script copied from host
|
|
`target/debug`, so a fresh host with no `mount` build would ship a broken image.)
|
|
2. DONE 2026-08-13 (commit f40b448) — `DaemonBackend::put` failures now surface as
|
|
FUSE EIO via `put_checked` instead of eprintln-only.
|
|
3. NOTE: VM is not auto-started on host boot. To bring the cold stack up:
|
|
`bash /root/run_vm_window.sh` (owned by luulu; shows the GTK window) — or
|
|
`launch_vm_detached.sh` for headless. The `fixed-durable-20260813` snapshot
|
|
captures the working durable stack.
|
|
4. OPTIONAL (cosmetic): visually launch the "CUBE Shell" desktop launcher to
|
|
confirm end-to-end at the file level. Already verified wired:
|
|
cube-term -> cubec -> /run/cube/cube.sock.
|
|
|
|
## KEY FILES
|
|
- /home/CUBELinux/CUBELinux-2/cubesys/src/commands.rs (raw* commands + helpers)
|
|
- /home/CUBELinux/CUBELinux-2/cubefs/src/backend.rs (DaemonBackend — FIX HERE)
|
|
- /home/CUBELinux/CUBELinux-2/cubefs/src/lib.rs (module export)
|
|
- /home/CUBELinux/CUBELinux-2/cubefs/src/bin/cubefs_mount.rs (--socket wiring)
|
|
- /home/CUBELinux/CUBELinux-2/cubestore/src/lib.rs (CubeStore::put_raw L358)
|
|
- /home/CUBELinux/CUBELinux-2/cubefs/src/vfs.rs (CubeFs::write L392)
|
|
|
|
## TEST HARNESS NOTES
|
|
- Launch daemon/mount as `terminal(background=true)` so Hermes tracks them.
|
|
- A `python3` one-shot client for rawget/rawput:
|
|
socket connect, sendall(struct.pack('<I',len(payload))+payload),
|
|
read 4-byte len then body.
|
|
- Daemon flags used:
|
|
`--socket S --store STORE.json --recovery-log REC.ndjson --allow-anonymous
|
|
--checkpoint-ms 400 --wal-fsync-ms 30`
|