cubelinux-2/RESUME-cubefs-daemon.md

RESUME POINT — cubefs FUSE <-> cube-server daemon durability wiring

Saved: 2026-08-13 (user paused session; pick up from here).

GOAL (from task list)

Make cubefs a real FUSE view of the running cube-server daemon's durable store, and prove a FUSE write lands in the daemon WAL and survives a daemon restart. (Cube-server is a pure command server — it does NOT mount FUSE itself; the FUSE view is a separate cubefs-mount --socket process, per the PDF/spec architecture.)

WHAT IS DONE (compiles + tests green)

1. Raw-coordinate command family added to cubesys/src/commands.rs dispatch: rawget <c> <z> <y> <x>, rawput <c> <z> <y> <x> <hex>, rawdel <c> <z> <y> <x>, rawkeys, rawscan <c> [z] [y]. Helpers added: parse_u8, hex_encode, hex_decode. (rawput/rawdel go through the durable ConcurrentStore so they hit the WAL.)
2. cubefs/src/backend.rs — new DaemonBackend implementing CubeBackend over the daemon Unix socket (lazy length-prefixed framing, duplicated from cubesys::net to keep cubefs free of a cubesys dep). Registered in cubefs/src/lib.rs (pub mod backend; pub use backend::DaemonBackend;). DaemonBackend is Clone (clones share socket path, reconnect lazily).
3. cubefs/src/bin/cubefs_mount.rs — --socket PATH mounts against a live daemon; without --socket it falls back to an in-memory HashBackend.
4. cargo build --workspace GREEN. cargo test --workspace GREEN. (Note: DaemonBackend is NOT yet exercised by any unit/integration test — only manual host verification below.)

WHAT IS PROVEN (host-level, same kernel/code path as the VM)

A. Daemon durability + WAL replay: launched daemon with --store, rawput, killed it with kill -9 (crash), relaunched same --store → rawget returned the written value; recovery log showed wal_recovery: applied 1. So the DAEMON half of the durability story works. B. Direct rawput from a Python client checkpoints to store.json (4-byte record appeared after the checkpoint interval). Confirmed daemon is alive, reachable, and durable on the host.

WHAT IS BROKEN (the open bug) — RESOLVED 2026-08-13 (commit 0300def)

Two separate bugs hid behind the "FUSE write didn't reach the daemon" symptom:

1. --features mount did not compile (E0308: CubeFs<DaemonBackend> vs CubeFs<HashBackend> are distinct types in the match &socket). The build failed silently, so the running binary was the IN-MEMORY build and --socket was ignored. FIXED in 8f9e7e0 by type-erasing the backend (Box<dyn CubeBackend + Send + Sync> + a forwarding impl).

2. Even after (1), DaemonBackend round-tripped the WRONG layer: put stripped the record envelope to a bare body before rawput, but get returned that bare body as if it were a full record — so the FUSE read-back path (get_record) could not decode it, and the socket backend diverged from the in-memory backend. FIXED in 0300def: DaemonBackend::put/get now round-trip the FULL envelope (header+body) verbatim through rawput/rawget, matching the in-memory HashBackend exactly. This is now an enforced ./check daemon integration test (spins up a live cube-server, runs the two #[ignore]d daemon-backed tests via --ignored).

GATE STATUS (2026-08-13, final)

• ./check (fmt+test+clippy -D warnings) GREEN.
• ./check daemon GREEN — socket-backed record round-trip against a real daemon.
• ./check mount GREEN — NOW runs the regression suite against BOTH the in-memory --seed mount AND a daemon-backed --socket mount (with its own fresh daemon), plus a durable across restart assertion. 57/57 pass. This is the regression guard exercising the real kernel VFS path for BOTH backends.

COMPLETED THIS SESSION (chronological)

• Fixed DaemonBackend envelope round-trip (commit 0300def).
• Fixed --features mount compile break (commit 8f9e7e0).
• Enforced both via ./check daemon (the two #[ignore]d integration tests).
• Host: ./check + ./check daemon + ./check mount all GREEN (mount now covers the daemon-backed path too).
• VM (cubelinux-vm.qcow2, was pre-fix): rebuilt cube-server + cubefs-mount from the fixed source; rewired cubefs.service to --socket /run/cube/cube.sock with Requires=cube-server.service; fixed TWO real deployment bugs found only because the new gate exercised the socket path: (a) systemd ordering cycle (cube-server Wanted cubefs, cubefs Required cube-server) — broke the cycle by dropping the cubefs dependency from cube-server.service. (b) ExecStartPost chmod 666 raced the daemon's socket creation under the restart loop and failed the unit — made it tolerant+delayed. Proved a FUSE write survives a daemon restart (WAL persistence) on the VM. Injected SSH key for cube+root to drive the VM. Snapshot fixed-durable-20260813 captures the working durable stack.
• ./check mount stage hardened (commit da3aa3a): daemon-backed leg + durability assertion; run_fs_tests is now backend-agnostic (self-seeds its fixture).

NEXT STEPS (remaining, low priority)

1. DONE 2026-08-13 — build_vm.sh now rebuilds cube binaries with --features cubefs/mount into a clean staging dir before copying, and aborts if cubefs-mount is missing. A from-scratch image bake can no longer silently omit the FUSE adapter. (Prior to this the script copied from host target/debug, so a fresh host with no mount build would ship a broken image.)
2. DONE 2026-08-13 (commit f40b448) — DaemonBackend::put failures now surface as FUSE EIO via put_checked instead of eprintln-only.
3. NOTE: VM is not auto-started on host boot. To bring the cold stack up: bash /root/run_vm_window.sh (owned by luulu; shows the GTK window) — or launch_vm_detached.sh for headless. The fixed-durable-20260813 snapshot captures the working durable stack.
4. OPTIONAL (cosmetic): visually launch the "CUBE Shell" desktop launcher to confirm end-to-end at the file level. Already verified wired: cube-term -> cubec -> /run/cube/cube.sock.

KEY FILES

• /home/CUBELinux/CUBELinux-2/cubesys/src/commands.rs (raw* commands + helpers)
• /home/CUBELinux/CUBELinux-2/cubefs/src/backend.rs (DaemonBackend — FIX HERE)
• /home/CUBELinux/CUBELinux-2/cubefs/src/lib.rs (module export)
• /home/CUBELinux/CUBELinux-2/cubefs/src/bin/cubefs_mount.rs (--socket wiring)
• /home/CUBELinux/CUBELinux-2/cubestore/src/lib.rs (CubeStore::put_raw L358)
• /home/CUBELinux/CUBELinux-2/cubefs/src/vfs.rs (CubeFs::write L392)

TEST HARNESS NOTES

• Launch daemon/mount as terminal(background=true) so Hermes tracks them.
• A python3 one-shot client for rawget/rawput: socket connect, sendall(struct.pack('<I',len(payload))+payload), read 4-byte len then body.
• Daemon flags used: --socket S --store STORE.json --recovery-log REC.ndjson --allow-anonymous --checkpoint-ms 400 --wal-fsync-ms 30