Until now the kernel could append to a log but needed a *userspace* checkpoint to
reclaim it — the dependency the write-authority decision was meant to remove.
This is the fold.
A store device is no longer an image at offset 0. It is:
[control copy A][control copy B][slot 0][slot 1][log]
with the control block saying which slot is live and how much log is in use, and
two copies each carrying a generation and a checksum. Two slots buy atomicity:
the folded image is written to the slot nothing is reading and flushed, and only
then does the control change — one small write to the copy that is *not* current.
At every instant there is either the old image with a log that still describes the
mutations since it, or the new image with an empty log. A crash in the middle of
the copy leaves the previous store intact, which is the entire reason for two
slots. Folding into the only copy of an image is not atomic, so `sync` refuses on
a bare image rather than pretending.
`sync` is a real operation, not a test hook: a caller that wants the log reclaimed
— a shutdown, a snapshot, a handover — is entitled to ask.
Gate (kernel/verify-kernel-checkpoint.sh): the kernel writes four mutations and
then folds its own log. The image it writes is compared with the one userspace
writes from the same mutations byte for byte, not by digest — and it is identical,
with the log empty afterwards.
Three bugs came out of building this, all caught by the gates rather than by
inspection:
- the refactor that extracted the merge left the image walk *duplicated* inside
the digest, re-adding image records after the log's entries — which gave them a
newer sequence number and quietly resurrected records the log had deleted;
- a v1 image has no extent, so it has no log either, and the new layout code was
demanding both;
- an unwritten log region is empty, not broken, and the first append was refusing
it. The append gate's harness was also counting a *refused* write as accepted,
which is how the second one hid.
All six gates pass: three image reads (v1 curated, v1 snapshot at 35,318 records,
v2 store), the log replay, the append with its SIGKILL durability test, and the
checkpoint compared byte for byte.
0.2 read the store. This appends to it: a mutation through /dev/cubelinux is
written to the log and fsynced *before* the write is accepted, which is the
kernel's version of the contract cube_duratest.py proves for the daemon.
- the mutation comes in as the argument block the coordinate interface will
pass: op(1) | space(32) | x(8) | y(8) | z(8) | len(4) | value[len]. The kernel
Morton-encodes the point itself — a key written here has to be the key a
userspace reader decodes, and that is a thing the gate would catch if it were
merely similar;
- the append lands after the log's valid prefix, so a torn tail is overwritten
rather than appended to, exactly as the userspace log does it;
- a log region that has never been written is zeros, not a log: the header is
written first, the way the userspace log creates its file;
- the entry's CRC covers space, key, length and value, computed the same way.
The write is the byte plane — bytes at a coordinate, no header written beside
them — because that is what a differential comparison against userspace's
`cell put` can be exact about. The header tier sits above this.
Gate (kernel/verify-kernel-append.sh), four mutations including an empty value
and a record in a second space:
reference : bytes=500 records=6 value_bytes=94 fnv1a64=6b679d39597a62b3
kernel : bytes=500 records=6 value_bytes=94 fnv1a64=6b679d39597a62b3
folded : bytes=500 records=6 value_bytes=94 fnv1a64=6b679d39597a62b3
survived : bytes=500 records=6 value_bytes=94 fnv1a64=6b679d39597a62b3
The third line is the one that matters: userspace *reads the log the kernel
wrote*, folds it, and lands on the same store. Without it the first two would
only show the kernel agreeing with itself. The fourth is a SIGKILL of the VM
with no shutdown and therefore no flush for us.
Not yet: the kernel folds nothing itself, so it still depends on a userspace
checkpoint to reclaim its log. That is the next step, and it has its own gate.
A store on a device is [image][log]: the image is a checkpoint, the log is the
mutations since, in the framing cube-store/src/wal.rs writes (op | crc32 | space
| key | len | value, prefix-trusting recovery). The kernel now reads both and
reports the store they *describe* — the image with the log applied, in the order
a checkpoint would write it.
- the log's offset is fixed by geometry (the first 4 KiB boundary at or after the
image's declared extent), so no superblock is needed to find it;
- recovery is prefix-trusting, exactly as userspace does it: the first entry that
is short, mis-framed or fails its CRC ends the log;
- merging is by coordinate with the newest write winning, which is what makes an
overwrite an overwrite and a delete a delete;
- the digest is taken over the surviving records in sorted order, and `bytes`
reports the size a checkpoint of this store would produce — so the number can
be compared with the image userspace actually writes, not merely eyeballed.
Gate (kernel/verify-log-replay.sh), on a pair carrying an overwrite, an
insertion and a deletion:
reference : bytes=377 records=4 value_bytes=99 fnv1a64=a38c1db74b00966f
kernel : bytes=377 records=4 value_bytes=99 fnv1a64=a38c1db74b00966f
log entries=6 applied over the image
It failed the first time, which is the point: the coalescing kept the earliest
write of a coordinate rather than the latest, resurrecting records the log had
deleted — six records where the fold produced four. A differential gate is what
turns that class of error into a red line instead of a corrupted store.
The image-only gates still pass unchanged (v1 curated, v1 snapshot 35,318
records, v2 store): with no log to apply the digest is the image's own order, so
v1 readers and every existing tool keep seeing exactly what they saw before.
The store's write path is decided (append a log, fold it into the image at a
checkpoint; DESIGN-cubelinux-write-path.md), and that decision forces the log to
sit right after the image on the device. v1 had no extent and no count — it
walked records until it met zero padding — so a v1 reader would have walked
straight into the log's header and parsed it as a record.
v2 states the image's byte extent and its record count, and this teaches the
kernel reader both:
- v1: walk from a 6-byte header until the trailing zeros, as before.
- v2: walk from a 22-byte header, stop exactly at the declared count, and never
read past the declared extent. A zero frame inside the count is a record, not
padding — which is the ambiguity v1 could not resolve.
- a v2 header whose extent does not cover the header itself is refused rather
than guessed at, and a count that is not met counts as an error instead of
quietly returning a shorter list.
Gate, unchanged in method: the kernel's digest of /dev/vda must equal
cube-image's digest of the same bytes. Passing on all three:
v1 curated 11 records fnv1a64=161113085b1573b2
v1 snapshot 35,318 records fnv1a64=5e20f98455387b08
v2 store 4 records fnv1a64=20ecadb5cdc9c994
The first CUBE code in the kernel, and deliberately only a reader: the write
authority has not moved yet, and PLAN-kernel-cubelinux.md records both that
decision and the hazard that makes the order matter — a kernel writing while a
userspace daemon still holds the same image loses one of the two writers' work,
silently. A reader cannot do that.
- drivers/cube/: a Rust module exposing /dev/cubelinux. Reading it reads the
pinned image from the block device through the kernel's own file layer
(filp_open + kernel_read — the path this kernel version binds for Rust, and
the reason no C helper was needed), parses the records, and returns one line:
digest curve=0 bytes=8400896 records=35318 value_bytes=6139148 fnv1a64=5e20f98455387b08 errors=0
The work happens on read, not at init, so there is no initcall ordering to get
wrong against the block driver that provides the device.
- The format is restated in the kernel (32-byte space, 24-byte key, 8-byte LE
length, value), including the two rules the userspace parser documents: a value
that runs past the buffer is a truncated record, and an all-zero frame ends the
records only when every remaining byte is zero — the rule that keeps a real
record at the origin from being read as padding.
- The digest is the point. A record count alone lets two different images agree;
folding the bytes in means the kernel and userspace are *compared* rather than
assumed to agree. `cube-image digest` prints the same line in the same field
order, and the QEMU gate fails if they differ by a byte.
Gate, on both images:
curated 11 records, 4,096 bytes, fnv1a64=161113085b1573b2 — match
snapshot 35,318 records, 8,400,896 bytes, fnv1a64=5e20f98455387b08 — match
The tree carries CONFIG_CUBELINUX_STORE=y on top of defconfig + RUST; a tree
without it boots and simply has no /dev/cubelinux.
CUBELinux build