Enumeration was the one operation the interface did not have, and the one a
capability interface owes an explanation for: a listing is the opposite of
"knowing a coordinate is the authorisation to use it". So the walk is bounded
and explicit. A caller names the space, holds a cursor, and gets as many whole
records as fit in the buffer it offered, packed `key(24) | value_len(u32) | value`
in the store's own order. SPACES walks the distinct spaces that hold a record,
one per call; range is that walk with the region as a filter, applied by the
caller rather than by a second operation in the kernel.
Its own argument block, versioned by its own size: SYSCALL_DEFINE2 peeks `size`
and routes — 80 bytes is the coordinate block, 64 is this one. An interface that
cannot grow has to be replaced, and this one grows by being given a new block.
The end of a walk is the cursor alone. A batch holds as many whole records as
fit, so it is FULL only when a record lands on the boundary and most batches
come back short; a caller that reads "the buffer was not filled" as "the space
is exhausted" truncates its listing to the first batch and cannot tell. ENUM
answers a finished walk with no records and the cursor unmoved; SPACES answers
it with -ENOENT, because the space after the last one is not there. That rule is
in the uapi header because it is a contract detail, not an implementation one.
The kernel caches no merged index: each call walks the records in order, asks
the log — small, and the only thing that can override one — what the winner is,
and skips what the cursor has already covered. O(records) a call, tens of
milliseconds for the live store, which is worth more than a cache every write
would have to invalidate.
The store's sort no longer lives on the kernel stack (heapsort, see the parent
commit's gate), and the release is now 6.19.3-cubelinux0.6 instead of
CUBELinux.0.6 — depmod/mkinitramfs reject a version whose first character is
not numeric, so the literal name cannot be uname -r. The box's own kernel uses
the same shape (6.19.3-cube+); this is the same concession.
'bytes=' reported how much had been read, so the same records disagreed between a
file (16,537,532) and a device slot holding them (67,108,864). The gates compare
this line, so it has to mean the same thing on both sides: it is now the header plus
the records — the length the store actually is.
Found by moving the machine's real store into a device store and comparing.
The write path was proven but unreachable: its operations lived behind a device
node. This is the interface the decision chose (DESIGN-cube-interface.md) — one
syscall number, an opcode, and a versioned argument block, with operations that
are the verbs the command language already defines: put, get, del, sync.
long cube(unsigned int op, struct cube_args __user *args)
`size` comes first and is checked, because syscall numbers are permanent and an
interface that cannot grow would have to be replaced. A coordinate is the space and
its three axes; nothing here resolves a name and nothing enumerates.
Split deliberately: the entry point, the user copies and the argument validation are
in C (cube_syscall.c) because `SYSCALL_DEFINE*` is a C macro this kernel has no Rust
equivalent for; everything that touches the store's bytes is in Rust, which passes
the coordinate to the format code as its parts so that Morton encoding stays in the
one module that must get it exactly right. A read that does not fit returns the size
it needs rather than truncating — a short read would be worse than an error.
Number 548: the x86_64 table says numbers 548 and above are available for
non-x32 use.
Gate (kernel/verify-syscall.sh): a static client in the initramfs does four writes
(including an empty value and a second space), reads one back *through the same
interface*, and folds with `sync`. Three different failures are separated — the
calls failing (a broken ABI), a read not returning what a write stored (a wrong key
encoding or index), and the folded image differing (a wrong format, order or merge).
put 7,0,0 ok (21 bytes)
put 8,0,0 ok (0 bytes)
put 9,0,0 ok (28 bytes)
put 1,2,3 ok (13 bytes)
get 7,0,0 21 bytes: the kernel wrote this
sync ok
and the image left on the device is byte-identical to the one userspace writes from
the same mutations, with the log empty. The interface's store is the same store.
All nine gates pass on 0.6.
The write path's last unproven claim was the one everything else rests on:
replay is prefix-trusting. Building the gate for it found that the claim was
false as implemented — in *both* implementations, and in the same way.
The log walkers advanced their offset past an entry's frame and only then checked
the checksum. A corrupted entry therefore never moved the *replay* point (it was
discarded, so the store looked right) but it did move the *append* point. The
consequences, in order of how bad they are:
- the kernel appended **after** the tear, burying the corruption inside a log that
then looked well-formed — the exact opposite of the documented rule;
- the userspace log truncated to a prefix that still contained the corrupt frame,
so every later append started past it and the corruption stayed in the log
forever;
- and because each append then read back the same wrong prefix, every append wrote
to the same offset and overwrote its predecessor — four mutations in, one on disk.
The fix is one line of ordering in three places: compute the frame's end, check the
length and the checksum, and only then move the offset. An entry that did not
validate may not move the point that says where the log ends.
Gate (kernel/verify-torn-tail.sh), on a store with an overwrite, an insertion and a
deletion whose last entry has been torn by zeroing its tail in place:
whole : 272 bytes, 3 records (the delete applied — the contrast)
reference : 349 bytes, 4 records (userspace, torn bytes: the delete discarded)
kernel : 349 bytes, 4 records
appended : 667 bytes, 8 records (the kernel appended over the tear)
expected : 667 bytes, 8 records (userspace: torn tail dropped, then the same four)
Building the gate also corrected the gate itself: tearing a log by *truncating the
device* is not a torn log, it is a smaller device — the capacity shrinks, writes
past the end vanish into the page cache with no error anywhere, and the test
measures an artifact. A torn write leaves the device the same size and corrupts
bytes in place.
All seven gates pass on 0.5.
Until now the kernel could append to a log but needed a *userspace* checkpoint to
reclaim it — the dependency the write-authority decision was meant to remove.
This is the fold.
A store device is no longer an image at offset 0. It is:
[control copy A][control copy B][slot 0][slot 1][log]
with the control block saying which slot is live and how much log is in use, and
two copies each carrying a generation and a checksum. Two slots buy atomicity:
the folded image is written to the slot nothing is reading and flushed, and only
then does the control change — one small write to the copy that is *not* current.
At every instant there is either the old image with a log that still describes the
mutations since it, or the new image with an empty log. A crash in the middle of
the copy leaves the previous store intact, which is the entire reason for two
slots. Folding into the only copy of an image is not atomic, so `sync` refuses on
a bare image rather than pretending.
`sync` is a real operation, not a test hook: a caller that wants the log reclaimed
— a shutdown, a snapshot, a handover — is entitled to ask.
Gate (kernel/verify-kernel-checkpoint.sh): the kernel writes four mutations and
then folds its own log. The image it writes is compared with the one userspace
writes from the same mutations byte for byte, not by digest — and it is identical,
with the log empty afterwards.
Three bugs came out of building this, all caught by the gates rather than by
inspection:
- the refactor that extracted the merge left the image walk *duplicated* inside
the digest, re-adding image records after the log's entries — which gave them a
newer sequence number and quietly resurrected records the log had deleted;
- a v1 image has no extent, so it has no log either, and the new layout code was
demanding both;
- an unwritten log region is empty, not broken, and the first append was refusing
it. The append gate's harness was also counting a *refused* write as accepted,
which is how the second one hid.
All six gates pass: three image reads (v1 curated, v1 snapshot at 35,318 records,
v2 store), the log replay, the append with its SIGKILL durability test, and the
checkpoint compared byte for byte.
0.2 read the store. This appends to it: a mutation through /dev/cubelinux is
written to the log and fsynced *before* the write is accepted, which is the
kernel's version of the contract cube_duratest.py proves for the daemon.
- the mutation comes in as the argument block the coordinate interface will
pass: op(1) | space(32) | x(8) | y(8) | z(8) | len(4) | value[len]. The kernel
Morton-encodes the point itself — a key written here has to be the key a
userspace reader decodes, and that is a thing the gate would catch if it were
merely similar;
- the append lands after the log's valid prefix, so a torn tail is overwritten
rather than appended to, exactly as the userspace log does it;
- a log region that has never been written is zeros, not a log: the header is
written first, the way the userspace log creates its file;
- the entry's CRC covers space, key, length and value, computed the same way.
The write is the byte plane — bytes at a coordinate, no header written beside
them — because that is what a differential comparison against userspace's
`cell put` can be exact about. The header tier sits above this.
Gate (kernel/verify-kernel-append.sh), four mutations including an empty value
and a record in a second space:
reference : bytes=500 records=6 value_bytes=94 fnv1a64=6b679d39597a62b3
kernel : bytes=500 records=6 value_bytes=94 fnv1a64=6b679d39597a62b3
folded : bytes=500 records=6 value_bytes=94 fnv1a64=6b679d39597a62b3
survived : bytes=500 records=6 value_bytes=94 fnv1a64=6b679d39597a62b3
The third line is the one that matters: userspace *reads the log the kernel
wrote*, folds it, and lands on the same store. Without it the first two would
only show the kernel agreeing with itself. The fourth is a SIGKILL of the VM
with no shutdown and therefore no flush for us.
Not yet: the kernel folds nothing itself, so it still depends on a userspace
checkpoint to reclaim its log. That is the next step, and it has its own gate.
A store on a device is [image][log]: the image is a checkpoint, the log is the
mutations since, in the framing cube-store/src/wal.rs writes (op | crc32 | space
| key | len | value, prefix-trusting recovery). The kernel now reads both and
reports the store they *describe* — the image with the log applied, in the order
a checkpoint would write it.
- the log's offset is fixed by geometry (the first 4 KiB boundary at or after the
image's declared extent), so no superblock is needed to find it;
- recovery is prefix-trusting, exactly as userspace does it: the first entry that
is short, mis-framed or fails its CRC ends the log;
- merging is by coordinate with the newest write winning, which is what makes an
overwrite an overwrite and a delete a delete;
- the digest is taken over the surviving records in sorted order, and `bytes`
reports the size a checkpoint of this store would produce — so the number can
be compared with the image userspace actually writes, not merely eyeballed.
Gate (kernel/verify-log-replay.sh), on a pair carrying an overwrite, an
insertion and a deletion:
reference : bytes=377 records=4 value_bytes=99 fnv1a64=a38c1db74b00966f
kernel : bytes=377 records=4 value_bytes=99 fnv1a64=a38c1db74b00966f
log entries=6 applied over the image
It failed the first time, which is the point: the coalescing kept the earliest
write of a coordinate rather than the latest, resurrecting records the log had
deleted — six records where the fold produced four. A differential gate is what
turns that class of error into a red line instead of a corrupted store.
The image-only gates still pass unchanged (v1 curated, v1 snapshot 35,318
records, v2 store): with no log to apply the digest is the image's own order, so
v1 readers and every existing tool keep seeing exactly what they saw before.
The store's write path is decided (append a log, fold it into the image at a
checkpoint; DESIGN-cubelinux-write-path.md), and that decision forces the log to
sit right after the image on the device. v1 had no extent and no count — it
walked records until it met zero padding — so a v1 reader would have walked
straight into the log's header and parsed it as a record.
v2 states the image's byte extent and its record count, and this teaches the
kernel reader both:
- v1: walk from a 6-byte header until the trailing zeros, as before.
- v2: walk from a 22-byte header, stop exactly at the declared count, and never
read past the declared extent. A zero frame inside the count is a record, not
padding — which is the ambiguity v1 could not resolve.
- a v2 header whose extent does not cover the header itself is refused rather
than guessed at, and a count that is not met counts as an error instead of
quietly returning a shorter list.
Gate, unchanged in method: the kernel's digest of /dev/vda must equal
cube-image's digest of the same bytes. Passing on all three:
v1 curated 11 records fnv1a64=161113085b1573b2
v1 snapshot 35,318 records fnv1a64=5e20f98455387b08
v2 store 4 records fnv1a64=20ecadb5cdc9c994
The first CUBE code in the kernel, and deliberately only a reader: the write
authority has not moved yet, and PLAN-kernel-cubelinux.md records both that
decision and the hazard that makes the order matter — a kernel writing while a
userspace daemon still holds the same image loses one of the two writers' work,
silently. A reader cannot do that.
- drivers/cube/: a Rust module exposing /dev/cubelinux. Reading it reads the
pinned image from the block device through the kernel's own file layer
(filp_open + kernel_read — the path this kernel version binds for Rust, and
the reason no C helper was needed), parses the records, and returns one line:
digest curve=0 bytes=8400896 records=35318 value_bytes=6139148 fnv1a64=5e20f98455387b08 errors=0
The work happens on read, not at init, so there is no initcall ordering to get
wrong against the block driver that provides the device.
- The format is restated in the kernel (32-byte space, 24-byte key, 8-byte LE
length, value), including the two rules the userspace parser documents: a value
that runs past the buffer is a truncated record, and an all-zero frame ends the
records only when every remaining byte is zero — the rule that keeps a real
record at the origin from being read as padding.
- The digest is the point. A record count alone lets two different images agree;
folding the bytes in means the kernel and userspace are *compared* rather than
assumed to agree. `cube-image digest` prints the same line in the same field
order, and the QEMU gate fails if they differ by a byte.
Gate, on both images:
curated 11 records, 4,096 bytes, fnv1a64=161113085b1573b2 — match
snapshot 35,318 records, 8,400,896 bytes, fnv1a64=5e20f98455387b08 — match
The tree carries CONFIG_CUBELINUX_STORE=y on top of defconfig + RUST; a tree
without it boots and simply has no /dev/cubelinux.
surface-camera-build