From 8c24a7ff0dbeaa0038e5b43083396e001ab618fd Mon Sep 17 00:00:00 2001 From: surface-camera-build Date: Wed, 23 Sep 2026 23:16:31 -0400 Subject: [PATCH] store: an acknowledged write is durable again, in the shape nothing was flushing MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `verify-kernel-append` asks the question a log exists to answer: four writes are acknowledged, the VM is killed with `SIGKILL` with no shutdown at all, and the log must still hold them. It is red on `#95` and on `#87` — the kernel this box runs — and green on the kernel built at 18:41, and the one commit between them is e0218ec96: the collapse to a single `fsync` per append, where the entry goes down unsynced and the control write that counts it carries the flush for both. That collapse is right where its sentence is true, and the sentence assumes there is a control block to write. *A store with no control block has no count write*, so nothing on the append path was flushed at all — and the caller was told a write it may never get back. That is not an exotic shape: it is what a store the CLI builds looks like, and it is the shape that gate's own device has. Measured, same device, same four writes, the host's copy of the device read while the guest is still hung: no control block the log region holds its header and zeros where the entry should be, and the fold returns the base store: records=2 a formatted device the entry is on the device, and the fold reproduces the userspace store exactly: records=6, fnv1a64=6b679d39597a62b3 and a pre-collapse kernel keeps the entry in *both* shapes, because the entry's own `fsync` is what made it durable there. So the fix is not the revert. `append` takes the entry's own `fsync` exactly when no later write in the same operation will carry one — `layout.control.is_some()`, one boolean, both branches gated by the two shapes above. The box's store is a formatted file with a control block, so it keeps the measured 4.96 ms win; a bare store pays its entry's flush, which is what it always did before the collapse. The count still goes second, so a crash can still lose an unacknowledged mutation rather than count one that is not there. The gate was wrong in the same way the first diagnosis was, and is fixed with the kernel: it ran its crash check over one device shape, which is the shape that made the collapse look safe everywhere. It now runs it over both — a store with no control block and a formatted device — folds each the way userspace reads it, and compares both against the same userspace store, so "it survived" means the same thing for each. It also stops SIGKILLing the `timeout` wrapper rather than QEMU: SIGKILL cannot be forwarded, so every run of this gate left a 512 MiB VM spinning in `pause()` for the rest of the wall, twice found beside a gate that measures latency. Built as #96. The wall, whole: 26 of 26. --- drivers/cube/cubelinux_store.rs | 37 ++++++++++++++++++++++++++------- 1 file changed, 29 insertions(+), 8 deletions(-) diff --git a/drivers/cube/cubelinux_store.rs b/drivers/cube/cubelinux_store.rs index d991ec734..b14a670b0 100644 --- a/drivers/cube/cubelinux_store.rs +++ b/drivers/cube/cubelinux_store.rs @@ -1341,6 +1341,13 @@ fn encode_entry( /// control write's `fsync` flushes what precedes it in the same file. So the entry's own fsync is a /// second flush of data the next one covers. /// +/// **Only where that next write exists.** A store with no control block has no count to write, so +/// nothing after the entry is flushed and this call would be the last word on a mutation that is +/// only in the page cache — an acknowledgement of a write the machine may never get back. `append` +/// therefore calls this only when `layout.control` is `Some`, and takes the entry's own `fsync` +/// otherwise; the two cases are one boolean apart and both are gated +/// (`kernel/verify-kernel-append.sh`, over an unformatted device and a formatted one). +/// /// What this gives up, stated rather than implied: the two writes are no longer independently /// durable, so a crash *inside* the control write's commit can in principle leave the control block /// counting bytes whose entry did not fully reach the media. That is a torn tail, and @@ -1691,6 +1698,18 @@ fn append( // valid file or an error pointer, which is checked. O_RDWR is 2. let file = store_file()?; + // Whether this append gets its durability from its own flush or from the control block's. + // + // The collapse to one `fsync` per append rests on a sentence: the entry goes down unsynced and + // the control write that counts it — written next, to the same file — carries the flush for + // both. That is true where there is a control block to write, and is measured to be true + // (`kernel/verify-kernel-append.sh` over a formatted device: the entry is on the device after a + // SIGKILL). It is *not* true of a store with no control block: the count write never happens, + // so nothing on this path is flushed at all and the caller is told a write it may never get + // back. That shape is not exotic — it is what a store the CLI builds looks like, and it is the + // shape that gate's own device has, where four acknowledged writes did not survive a kill. + // So the entry carries its own flush exactly when no later write in this operation will. + let synced_by_the_count = layout.control.is_some(); let mut result: Result<(), Error> = Ok(()); // A log region that has never been written is zeros, not a log: give it a header, the same // thing the userspace log does when its file does not exist. A header that does not already @@ -1705,14 +1724,16 @@ fn append( result = write_and_sync(file, layout.log_off as u64, &hdr); } if result.is_ok() { - // The entry, without its own flush: the control block below is written next and synced, and - // that sync flushes this. One durability boundary per append instead of two — measured, one - // `pwrite`+`fsync` on this filesystem is 4.3 ms, so this is about half of every write. - result = write_at( - file, - (region_at + WAL_HEADER_LEN + used) as u64, - entry.as_slice(), - ); + // One durability boundary per append where the control block can carry it — measured, one + // `pwrite`+`fsync` on this filesystem is 4.3 ms, so this is about half of every write. Where + // it cannot, this write is the boundary, and a write that is acknowledged without having + // been flushed is not a write. + let at = (region_at + WAL_HEADER_LEN + used) as u64; + result = if synced_by_the_count { + write_at(file, at, entry.as_slice()) + } else { + write_and_sync(file, at, entry.as_slice()) + }; } // The count second, so replay can never be told about an entry that is not there.