From 35fcb6c7fe135687237c01a88177689d66ddb733 Mon Sep 17 00:00:00 2001 From: CUBELinux Date: Fri, 25 Sep 2026 03:35:52 -0400 Subject: [PATCH] store: a fold kept a deletion only when the record had no class MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A fold turns the log's effects into the image's by keeping, per coordinate, the *last* entry in the merged order — that is how "the later write wins" is implemented. The order came from a derive on `Entry`, which compares fields in declaration order, and `flags` was declared before `seq`. So the tie-break was not arrival at all: it was the class mask. A `del` is written with no class (0). A **sealed** record is written with `0x0800`. For the same coordinate that put the removal *before* the record it removed, so the record won and the fold wrote it back — a deleted record returning from the fold, with its old value, on any store whose writes are sealed. Found on the box on 2026-09-25: put → present, del → "(not found)", `cube-fold` → "folded the log into the image" (generation 2647 → 2648, log emptied), get → the value, back. It reproduced through a sealed front-end and an unsealed one, and the live store still holds the test records it resurrected. Why it hid: an unsealed store writes both entries with flags 0, the tie falls through to `seq`, and the order is right — which is exactly what the guest gate did before this, so the gate agreed with a bug it could not see. The shape that fails is the shape the machine uses. `Entry` now orders by `(space, key, seq, …)` explicitly, with the reason written beside it, because "the fields happen to be declared in this order" is what went wrong. The userspace store cannot have this bug and does not: it applies the log into a `HashMap`, where a removal *is* the removal of the key, so no ordering decides anything. --- drivers/cube/cubelinux_store.rs | 34 ++++++++++++++++++++++++++++++++- 1 file changed, 33 insertions(+), 1 deletion(-) diff --git a/drivers/cube/cubelinux_store.rs b/drivers/cube/cubelinux_store.rs index a930ab37f..a19b6b7cc 100644 --- a/drivers/cube/cubelinux_store.rs +++ b/drivers/cube/cubelinux_store.rs @@ -938,7 +938,7 @@ fn resolve_layout(b: &[u8]) -> Result { /// One record on its way into the merged store. Fixed size, so a `KVVec` of these sorts /// in place; values live in a pool beside them and are referenced by offset. -#[derive(Clone, Copy, PartialEq, Eq, PartialOrd, Ord)] +#[derive(Clone, Copy, PartialEq, Eq)] struct Entry { space: [u8; 32], key: [u8; 24], @@ -952,6 +952,38 @@ struct Entry { deleted: bool, } +/// Order by `(space, key, seq, …)` — **`seq` before `flags`**, which is not the order the fields are +/// declared in and cannot be left to a derive. +/// +/// A fold keeps, for each coordinate, the *last* entry in this order: that is how "the later write +/// wins" is implemented. With `flags` compared first, "later" is not what the order means: a `del` +/// carries no class (0) and a **sealed** record carries `0x0800`, so the removal sorted *before* the +/// record it removed and the record won — a deleted record came back from the fold, with its old +/// value, on a store whose writes are sealed. Found on the box on 2026-09-25 by `put` → `del` → +/// `cube-fold` → the record returning; the guest gate missed it because unsealed writes share one +/// flag value, which is why the gate now deletes a *classed* record. +/// +/// `flags` stays in the order, after `seq`: the sort also has to be total and deterministic, and two +/// entries can share a coordinate and a sequence only if they came from the same source. +impl Ord for Entry { + fn cmp(&self, other: &Self) -> core::cmp::Ordering { + self.space + .cmp(&other.space) + .then_with(|| self.key.cmp(&other.key)) + .then_with(|| self.seq.cmp(&other.seq)) + .then_with(|| self.flags.cmp(&other.flags)) + .then_with(|| self.value_off.cmp(&other.value_off)) + .then_with(|| self.value_len.cmp(&other.value_len)) + .then_with(|| self.deleted.cmp(&other.deleted)) + } +} + +impl PartialOrd for Entry { + fn partial_cmp(&self, other: &Self) -> Option { + Some(self.cmp(other)) + } +} + /// The store as a sorted set of records, which is what a checkpoint writes and what the /// digest is taken over. struct Merged {