087c6b0e607b9d014429f11393e33eb6d38e38b6
7
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 surface-camera-build
|
087c6b0e60 |
cubelinux: the kernel records its own boot
The second half of "the OS stores itself". The store was already the kernel's; what was missing was the kernel *saying* something of its own rather than a client doing it. At its first write of a boot the driver now appends one record describing the boot it is having — its own version banner, the wall-clock time, and the store device it resolved — to a reserved space, through the same append path every other mutation uses. Three things had to be decided, and each had a wrong answer that looked right: WHERE IT HOOKS. "At store init" does not exist and must not be invented: there is no init-time open, deliberately, so there is no ordering to get wrong against the block driver that provides the device. An __initcall appending a record would reintroduce exactly that ordering problem. The moment is the FIRST WRITE, which needs no ordering at all and is the semantically right one — the kernel records itself when it becomes the writer. A boot in which the kernel only reads writes no record, which is honest rather than a gap. WHICH SPACE. 0xFD was the first choice, following the convention that a reserved space is one repeated byte. 0xFD is the OS KEYSTORE — the space the kill switch exists to destroy — and writing boot records into it would have been a serious bug. Only the userspace name table catches this (format_space in crates/cube-command) because the kernel keeps no table of space names, so the table is now written down where the constant is: 0x00 root, 0xFF edges, 0xFE portal, 0xFD keystore, 0xFC boot. The record lives at (0,0,0) in 0xFC: one record, the current boot. WHAT IT SAYS. boot=<epoch seconds> device=<resolved path> kernel=<the version banner>, banner last and unquoted so everything after the final = is the kernel's own words rather than a field this code parsed. Raw epoch seconds rather than a date: rendering a calendar date in the kernel is date arithmetic, and a caller with a clock can do it without a kernel bug being the reason a timestamp is wrong. The banner comes from linux_banner and the time from ktime_get_real_ts64. WHY IT IS OFF BY DEFAULT. Not caution, but an invariant. The gates' method is that the store the kernel produces is comparable, byte for byte, with the store userspace produces from the same mutations; a record the kernel injects that the caller never asked for would turn those comparisons into non-comparisons. So it is cube_boot_record=1 on the kernel command line, parsed in C beside cube_store= for the reason that parameter is in C (this kernel's Rust cannot express a string parameter), and kernel/verify-boot-record.sh is the gate that turns it on. A failure to record is logged and never propagated: the record is worth having and is not a precondition for the caller's write. It is attempted once per boot rather than retried per write, because a store that will not take it will not take it later, and one warning is information where a stream of them is noise. |
||
|
surface-camera-build
|
78540b5687 |
cubelinux: the fold reads the layout it writes
Found on the box, minutes after the live store was folded for the first time: the second fold answered -EINVAL. `build_merged` parsed the packed layout — `space | key | len | value`, repeated — so it could read a store that had never been folded and nothing else. The first fold reads packed and writes addressed; every fold after that reads addressed, which is what a store does for the rest of its life. As written, a store could be folded exactly once and then never again — the log would grow until it filled. No gate folded twice, which is why it got this far: verify-kernel-checkpoint.sh, verify-frontend.sh and verify-enum-cost.sh each folded a packed store once. The guest's bench folds twice now, and verify-enum-cost.sh insists on the second one — "a store can be folded once and then never again, which is not a layout" — so the case is covered rather than remembered. The v3 branch reads through the shared format's own arithmetic: the space table gives each space's index range, the index gives each key and the value's place, and the log is applied over the result exactly as before. The packed path is untouched. Verified: verify-enum-cost.sh (which now folds twice and still measures 1.0x for a 40x store) and verify-kernel-checkpoint.sh (the image a fold writes holds exactly what userspace holds). |
||
|
surface-camera-build
|
d569df710d |
cubelinux: the store's format is one file, and the kernel and userspace both include it
Two build systems cannot share a crate: the kernel's Rust build compiles what is in its own module
tree, and a cargo crate is not that. So they share a *file* — `drivers/cube/cube_format.rs`, which
the driver declares with `mod` and `crates/cube-format` includes by path. There is no copy to drift
from, which is the only arrangement that cannot go stale. What happened when v3 landed and userspace
did not is the argument: an hour of `unsupported-version`, one gate red, and two implementations of
one format each believing itself.
The file holds what both sides must agree about, and nothing else — pure functions over slices, no
allocation, no I/O, no logging:
* the constants every reader and writer derives its arithmetic from,
* the header, in all three versions, with refusal rather than guessing: a reader that invents an
extent can read somebody else's bytes,
* the v3 space table and index, and the three equalities a reader computes its addresses from,
* the packed record and the log entry, the two framings a fold and a walk have to parse alike,
* CRC-32, FNV-1a, and the digest line both sides print.
The driver's copies of the constants are now aliases of the shared ones, and the two functions that
*validate* a header — `parse_header` and the v3 geometry — delegate to it, because validation is
where a second description gets believed. The readers stay where they are: this driver streams from
a file with its own buffers while userspace already holds the whole image, and that difference is
real rather than duplicated.
Nothing changed in behaviour, which is the point: verify-enum.sh, verify-enum-cost.sh,
verify-frontend.sh, verify-kernel-checkpoint.sh and verify-kernel-append.sh all pass, and the shared
crate's own tests include a kernel-written image — ten records, three spaces, 110 value bytes — read
through its arithmetic.
|
||
|
surface-camera-build
|
1db3abce93 |
cubelinux: v3 — the image carries the addresses, so a coordinate is a place
A packed list (`space | key | len | value`, repeated) cannot answer "where is this coordinate":
record N's offset is the sum of every record before it. The key gives ORDER, and order is not an
address, so no coordinate could be turned into a place and not even a binary search was possible —
the middle record's offset is just as unknowable. Every lookup walked, every batch of a listing
walked again, and a read of one record cost as much as the store (measured: 110 ms per get, 119 ms
per `spaces`, 1,206 ms for a 6,127-record listing).
v3 puts the addresses in the image:
[header 46] magic, version, curve, extents, counts
[space table: space_count x 48] space | first index | records
[index: record_count x 40] key | value offset | value length
[values: packed, in index order]
A lookup is now a binary search over arithmetic addresses (`index_off + i * 40`); a listing starts
at its space's first index entry and streams, reading a span of values per batch; asking which
spaces exist is the space table; a spatial range is a contiguous run of index entries. An index
entry is 40 bytes against the packed frame's 64, so the image also gets smaller.
Measured by `verify-enum-cost.sh` between a 500-record store and a 20,000-record one:
listing the same 10-record space: 3.22 ms -> 3.27 ms (1.0x, store grew 40x)
asking which spaces exist: 3.25 ms -> 3.08 ms (0.9x)
reading a coordinate that is there: 1.07 ms -> 1.25 ms (1.2x)
reading one that is not: 1.19 ms -> 1.22 ms (1.0x)
Nothing scales with the store any more, which is the premise: knowing a coordinate is what lets you
reach it.
Two bugs found on the way, both of the kind only a real image shows. The index was written with a
4-byte value length (`Entry` carries it in a u32) while the header's arithmetic and both readers
assumed 8 — every entry four bytes short, the last of them overlapping the values. And the digest's
`bytes` figure added the layout's own header length, so two layouts of one store digested
differently; it is now the records' logical size, which is the same number either way.
Verified: verify-kernel-append.sh, verify-kernel-checkpoint.sh, verify-enum.sh, verify-enum-cost.sh
and verify-frontend.sh all pass, and the workspace tests pass — including a new fixture test in
cube-store-raw that reads a 700-byte image the kernel itself wrote, because a hand-built image
cannot catch a misunderstanding shared by the builder.
|
||
|
surface-camera-build
|
56665a303e |
cubelinux: read the store where it lies, instead of rebuilding it per call
Every read began by reading the whole device into kernel memory, parsing every
record of every space into a merged pool, and heapsorting the lot — then throwing
all of it away. So a `get` of one record cost as much as listing the store, every
batch of a listing paid it again, and the coordinate bought nothing mechanically:
knowing where a record is did not make reaching it any cheaper. Measured on the
workhorse: one `get` 110 ms, one `spaces` call 119 ms, one 6,127-record listing
1,206 ms.
The records are already in the order the coordinate computes — a checkpoint writes
them by (space, key) — so the kernel now walks them where they lie: zero-copy
values, no pool, no sort, and the log (small, and the only thing that can override
the image) consulted as an overlay. It reads the image and the log window rather
than the whole device, which is four images wide.
* `get` — the log's newest word on the coordinate, else the image walked to it.
* `enum` — one merge pass over two sorted sequences: the space's image records
and its log edits.
* `spaces` — from a table of where each space starts, cached by the control
block's generation. That table is one entry per space, not per record, and a
checkpoint is the only thing that invalidates it.
* `put`/`del`/`sync` — unchanged: a write is a log append, and the fold still
writes the whole sorted store.
Verified: verify-enum.sh (a 3,006-record listing identical to userspace's, record
for record) and verify-frontend.sh (socket -> front-end -> cube(2), including a
listing larger than one batch and the store the front-end leaves being
digest-identical to userspace's) both pass.
Not yet what it should be: listing a 10-record space in a 20,000-record store
still costs about six times what it costs in a 500-record one, so the space table
is not taking effect as intended and the lookup is not yet independent of store
size. That belongs in the image's layout — a coordinate cannot be turned into a
byte offset in a variable-length packed list, because a record's position is the
sum of every value before it — and the fix is a format one, not a caching one.
|
||
|
surface-camera-build
|
b9a7b8d8f3 |
cubelinux: cube(2) walks the store — CUBE_OP_ENUM and CUBE_OP_SPACES
Enumeration was the one operation the interface did not have, and the one a capability interface owes an explanation for: a listing is the opposite of "knowing a coordinate is the authorisation to use it". So the walk is bounded and explicit. A caller names the space, holds a cursor, and gets as many whole records as fit in the buffer it offered, packed `key(24) | value_len(u32) | value` in the store's own order. SPACES walks the distinct spaces that hold a record, one per call; range is that walk with the region as a filter, applied by the caller rather than by a second operation in the kernel. Its own argument block, versioned by its own size: SYSCALL_DEFINE2 peeks `size` and routes — 80 bytes is the coordinate block, 64 is this one. An interface that cannot grow has to be replaced, and this one grows by being given a new block. The end of a walk is the cursor alone. A batch holds as many whole records as fit, so it is FULL only when a record lands on the boundary and most batches come back short; a caller that reads "the buffer was not filled" as "the space is exhausted" truncates its listing to the first batch and cannot tell. ENUM answers a finished walk with no records and the cursor unmoved; SPACES answers it with -ENOENT, because the space after the last one is not there. That rule is in the uapi header because it is a contract detail, not an implementation one. The kernel caches no merged index: each call walks the records in order, asks the log — small, and the only thing that can override one — what the winner is, and skips what the cursor has already covered. O(records) a call, tens of milliseconds for the live store, which is worth more than a cache every write would have to invalidate. |
||
|
surface-camera-build
|
ed5ff764ba |
cubelinux: the release starts with a digit, because module tooling demands it
The store's sort no longer lives on the kernel stack (heapsort, see the parent commit's gate), and the release is now 6.19.3-cubelinux0.6 instead of CUBELinux.0.6 — depmod/mkinitramfs reject a version whose first character is not numeric, so the literal name cannot be uname -r. The box's own kernel uses the same shape (6.19.3-cube+); this is the same concession. |