surface-camera-build
56665a303e
Every read began by reading the whole device into kernel memory, parsing every
record of every space into a merged pool, and heapsorting the lot — then throwing
all of it away. So a `get` of one record cost as much as listing the store, every
batch of a listing paid it again, and the coordinate bought nothing mechanically:
knowing where a record is did not make reaching it any cheaper. Measured on the
workhorse: one `get` 110 ms, one `spaces` call 119 ms, one 6,127-record listing
1,206 ms.
The records are already in the order the coordinate computes — a checkpoint writes
them by (space, key) — so the kernel now walks them where they lie: zero-copy
values, no pool, no sort, and the log (small, and the only thing that can override
the image) consulted as an overlay. It reads the image and the log window rather
than the whole device, which is four images wide.
* `get` — the log's newest word on the coordinate, else the image walked to it.
* `enum` — one merge pass over two sorted sequences: the space's image records
and its log edits.
* `spaces` — from a table of where each space starts, cached by the control
block's generation. That table is one entry per space, not per record, and a
checkpoint is the only thing that invalidates it.
* `put`/`del`/`sync` — unchanged: a write is a log append, and the fold still
writes the whole sorted store.
Verified: verify-enum.sh (a 3,006-record listing identical to userspace's, record
for record) and verify-frontend.sh (socket -> front-end -> cube(2), including a
listing larger than one batch and the store the front-end leaves being
digest-identical to userspace's) both pass.
Not yet what it should be: listing a 10-record space in a 20,000-record store
still costs about six times what it costs in a 500-record one, so the space table
is not taking effect as intended and the lookup is not yet independent of store
size. That belongs in the image's layout — a coordinate cannot be turned into a
byte offset in a variable-length packed list, because a record's position is the
sum of every value before it — and the fix is a format one, not a caching one.
2456 lines
90 KiB
Rust
2456 lines
90 KiB
Rust
// SPDX-License-Identifier: GPL-2.0
|
|
|
|
//! CUBELinux store reader — the kernel reading CUBE coordinates off a block device.
|
|
//!
|
|
//! The store is addressed by coordinate, and the pinned image's layout is:
|
|
//!
|
|
//! ```text
|
|
//! [MAGIC "CUBE" 4][version 1][curve tag]
|
|
//! records: [SpaceId 32][Key 24][value length u64 LE][value]
|
|
//! ```
|
|
//!
|
|
//! A store on a device is that image followed by a **write-ahead log** — the mutations since
|
|
//! the last checkpoint, in the same `op | crc32 | space | key | len | value` framing
|
|
//! `cube-store/src/wal.rs` writes. The log is a delta: the image is authoritative and
|
|
//! self-contained, and a reader that ignores the log loses only the mutations recorded after
|
|
//! the last checkpoint. `DESIGN-cubelinux-write-path.md` is why the write path looks like
|
|
//! this at all.
|
|
//!
|
|
//! This module reads both and reports the store they *describe* — the image with the log
|
|
//! applied: the record count, the total value bytes, and an FNV-1a digest over every
|
|
//! `(space, key, length, value)` in the order a checkpoint would write them. The digest exists so the kernel and userspace can be *compared*
|
|
//! rather than assumed to agree — `cube-image digest <image>` prints the same line in the
|
|
//! same field order, and the QEMU gate fails if they differ by a byte.
|
|
//!
|
|
//! # Why a reader, and only a reader
|
|
//!
|
|
//! The store's write authority has not moved yet. `PLAN-kernel-cubelinux.md` records the
|
|
//! decision (the kernel owns the store) and the hazard that makes the order matter: a
|
|
//! kernel that writes while a userspace daemon still holds the same image in memory loses
|
|
//! one of the two writers' work, silently. Until that is settled, a reader is the correct
|
|
//! amount of authority for the kernel to hold.
|
|
//!
|
|
//! # Interface
|
|
//!
|
|
//! `/dev/cubelinux` — reading it performs the read (through the kernel's own file layer)
|
|
//! and returns one line. Nothing happens at module init, so there is no ordering to get
|
|
//! wrong against the block driver that provides the device.
|
|
//!
|
|
//! ```text
|
|
//! digest curve=0 bytes=1141 records=11 value_bytes=431 fnv1a64=161113085b1573b2 errors=0
|
|
//! ```
|
|
|
|
use core::fmt::{self, Write};
|
|
|
|
use kernel::{
|
|
alloc::AllocError,
|
|
bindings, c_str,
|
|
device::Device,
|
|
fs::{File, Kiocb},
|
|
iov::{IovIterDest, IovIterSource},
|
|
miscdevice::{MiscDevice, MiscDeviceOptions, MiscDeviceRegistration},
|
|
prelude::*,
|
|
sync::aref::ARef,
|
|
};
|
|
|
|
module! {
|
|
type: CubeStoreModule,
|
|
name: "cubelinux_store",
|
|
authors: ["CUBE OS"],
|
|
description: "CUBELinux store reader (coordinates in the kernel)",
|
|
license: "GPL",
|
|
}
|
|
|
|
/// The layout, restated here because the kernel cannot depend on the userspace crates.
|
|
/// `crates/cube-store-raw` is the source of truth; the digest comparison is what keeps
|
|
/// this copy honest.
|
|
const MAGIC: &[u8; 4] = b"CUBE";
|
|
/// The original format: magic, version, curve, then records until zero padding.
|
|
const VERSION_V1: u8 = 1;
|
|
/// The current format: the same, plus the image's byte extent and record count.
|
|
const VERSION: u8 = 2;
|
|
const HEADER_LEN_V1: usize = 6;
|
|
const HEADER_LEN_V2: usize = 6 + 8 + 8;
|
|
const SPACE_ID_LEN: usize = 32;
|
|
const RAW_KEY_LEN: usize = 24;
|
|
const RECORD_FIXED: usize = SPACE_ID_LEN + RAW_KEY_LEN + 8;
|
|
|
|
// The device the store lives on, as the `cube_store=` boot parameter resolved it.
|
|
//
|
|
// This was a constant — *"for the boot gate one fixed name is honest and has one less way to
|
|
// be wrong"* — and that was true while a virtual machine was the only place this driver ran.
|
|
// It runs on the box now, the box's device is not `/dev/vda`, and a constant cannot be both.
|
|
// So the box names its device on the kernel command line:
|
|
//
|
|
// cube_store=/dev/nvme0n1p2
|
|
//
|
|
// The parameter is declared in C (`cube_syscall.c`) because this kernel's Rust can express only
|
|
// *integer* module parameters — `rust/kernel/module_param.rs` implements `ModuleParam` through
|
|
// `ParseInt` and nothing else — and a path is not an integer. It is a `__setup` parameter rather
|
|
// than a module parameter for the naming reason recorded in that file.
|
|
//
|
|
// (A `///` here would be an unused doc comment: it would attach to the `extern` block, which
|
|
// documents nothing.)
|
|
extern "C" {
|
|
/// Returns a pointer to a static, NUL-terminated buffer holding the path.
|
|
fn cubelinux_store_device() -> *const core::ffi::c_char;
|
|
}
|
|
|
|
/// The store's path, NUL-terminated, owned by the C side for the life of the kernel.
|
|
fn store_device() -> *const u8 {
|
|
// SAFETY: `cubelinux_store_device` returns a pointer to a static buffer that the module
|
|
// parameter filled during boot and that nothing writes afterwards, so it stays valid and
|
|
// NUL-terminated for as long as this runs.
|
|
unsafe { cubelinux_store_device().cast::<u8>() }
|
|
}
|
|
|
|
/// Refuse to pull an unbounded device into memory: the guard against a wrong device name
|
|
/// turning a read into an allocation storm. It is sized for a *store device*, not for an image.
|
|
///
|
|
/// `Control::layout` gives each of the two slots a quarter of the device, so a device must be at
|
|
/// least four times the image it holds — and a *sealed* image is larger than the plaintext one,
|
|
/// because every record gains an envelope. A sealed 18.8 MiB store therefore needs a ~76 MiB
|
|
/// device, which the original 64 MiB cap refused. 128 MiB leaves room for both to grow while
|
|
/// still being a bound: pointing this at a whole 119 GiB disk is still refused.
|
|
const MAX_BYTES: usize = 128 * 1024 * 1024;
|
|
|
|
/// The log's own magic, distinct from the image's so a reader that opens the wrong one
|
|
/// cannot mistake it for the other.
|
|
const WAL_MAGIC: [u8; 4] = *b"CUBW";
|
|
const WAL_VERSION: u8 = 1;
|
|
/// Log header: magic(4) + version(1) + curve tag(1).
|
|
const WAL_HEADER_LEN: usize = 6;
|
|
/// Log entry, before the value: op(1) + crc32(4) + space(32) + key(24) + len(4).
|
|
const ENTRY_FIXED: usize = 1 + 4 + 32 + 24 + 4;
|
|
/// The log begins at the first 4 KiB boundary at or after the image. Fixed by geometry so
|
|
/// no superblock is needed to find it, and stated by the v2 header's extent.
|
|
///
|
|
/// That is the *bare* layout: an image on a device with a log after it. A device that is
|
|
/// meant to be checkpointed has a control block instead (see `Control`), because folding a
|
|
/// log into the only copy of an image is not atomic and a crash in the middle of it loses the
|
|
/// store.
|
|
const LOG_ALIGN: usize = 4096;
|
|
|
|
/// Control block magic, distinct from the image's `CUBE` and the log's `CUBW`.
|
|
const CTL_MAGIC: [u8; 4] = *b"CUBS";
|
|
const CTL_VERSION: u8 = 1;
|
|
/// One control copy, and where the two live.
|
|
const CTL_COPY_LEN: usize = 2048;
|
|
const CTL_COPY_A: usize = 0;
|
|
const CTL_COPY_B: usize = 2048;
|
|
/// Where the two image slots begin.
|
|
const CTL_DATA_OFF: u64 = 4096;
|
|
/// Bytes of a control copy covered by its checksum.
|
|
const CTL_SUMMED: usize = 44;
|
|
|
|
/// The one mutation this device accepts so far. The coordinate interface's opcodes will be
|
|
/// the language's verbs; this is the first of them.
|
|
const OP_PUT: u8 = 1;
|
|
/// Fold the log into the image.
|
|
const OP_SYNC: u8 = 3;
|
|
|
|
/// FNV-1a offset basis.
|
|
const FNV_OFFSET: u64 = 0xcbf2_9ce4_8422_2325;
|
|
|
|
/// A line of output, built in place. No allocation: the read path formats into a fixed
|
|
/// buffer, so it cannot fail for want of memory while holding a file open.
|
|
struct Line {
|
|
buf: [u8; 256],
|
|
len: usize,
|
|
}
|
|
|
|
impl Line {
|
|
fn new() -> Self {
|
|
Line {
|
|
buf: [0; 256],
|
|
len: 0,
|
|
}
|
|
}
|
|
|
|
fn as_bytes(&self) -> &[u8] {
|
|
&self.buf[..self.len]
|
|
}
|
|
}
|
|
|
|
impl Write for Line {
|
|
fn write_str(&mut self, s: &str) -> fmt::Result {
|
|
let bytes = s.as_bytes();
|
|
if self.len + bytes.len() > self.buf.len() {
|
|
return Err(fmt::Error);
|
|
}
|
|
self.buf[self.len..self.len + bytes.len()].copy_from_slice(bytes);
|
|
self.len += bytes.len();
|
|
Ok(())
|
|
}
|
|
}
|
|
|
|
/// FNV-1a 64 over a run of bytes, continuing from `h`.
|
|
fn fnv1a64(bytes: &[u8], mut h: u64) -> u64 {
|
|
for b in bytes {
|
|
h ^= *b as u64;
|
|
h = h.wrapping_mul(0x0000_0100_0000_01b3);
|
|
}
|
|
h
|
|
}
|
|
|
|
/// Read the whole store image through the kernel's file layer.
|
|
///
|
|
/// `kernel_read` on the block device rather than a raw bio: it is the path this kernel
|
|
/// version exposes to Rust, it goes through the page cache the way any other read does, and
|
|
/// it needs no C helper. If the store ever has to be read before the VFS is up (a root
|
|
/// filesystem, say), that is the moment to reach for the block layer directly.
|
|
fn read_image(image: &mut KVVec<u8>) -> Result<()> {
|
|
// O_RDONLY is 0 in Linux; filp_open takes the raw flags word.
|
|
// SAFETY: store_device() is a NUL-terminated C string that the module parameter filled at
|
|
// boot, and filp_open either returns a valid `struct file *` or an error pointer, which is
|
|
// checked below.
|
|
let file = unsafe { bindings::filp_open(store_device(), 0, 0) };
|
|
let file = kernel::error::from_err_ptr(file)?;
|
|
if file.is_null() {
|
|
return Err(EINVAL);
|
|
}
|
|
|
|
let mut pos: bindings::loff_t = 0;
|
|
let mut chunk = KVVec::<u8>::with_capacity(4096, GFP_KERNEL)?;
|
|
chunk.extend_from_slice(&[0u8; 4096][..], GFP_KERNEL)?;
|
|
let mut result = Ok(());
|
|
|
|
loop {
|
|
// SAFETY: `file` is a live `struct file *` from filp_open; `chunk` is a 4096-byte
|
|
// kernel buffer we own; `pos` is a valid loff_t. kernel_read reads at most
|
|
// `chunk.len()` bytes into the buffer and does not retain either pointer.
|
|
let n = unsafe {
|
|
bindings::kernel_read(
|
|
file,
|
|
chunk.as_mut_ptr().cast::<core::ffi::c_void>(),
|
|
chunk.len(),
|
|
&mut pos,
|
|
)
|
|
};
|
|
if n < 0 {
|
|
result = Err(Error::from_errno(n as i32));
|
|
break;
|
|
}
|
|
if n == 0 {
|
|
break; // end of device
|
|
}
|
|
let n = n as usize;
|
|
if let Err(e) = image.extend_from_slice(&chunk[..n], GFP_KERNEL) {
|
|
result = Err(e.into());
|
|
break;
|
|
}
|
|
if image.len() >= MAX_BYTES {
|
|
break;
|
|
}
|
|
}
|
|
|
|
// SAFETY: `file` came from filp_open and has not been closed; the owner argument is
|
|
// only meaningful for locks that no one holds here.
|
|
unsafe { bindings::filp_close(file, core::ptr::null_mut()) };
|
|
|
|
// Less than the shortest header is not an image at all; `digest` reports the rest.
|
|
if result.is_ok() && image.len() < HEADER_LEN_V1 {
|
|
return Err(EINVAL);
|
|
}
|
|
result
|
|
}
|
|
|
|
/// Walk the records of a store image, digesting each one, exactly as
|
|
/// `cube-store-raw::iter_records` + `cube-image digest` do.
|
|
///
|
|
/// The two rules that are easy to get wrong, both from the userspace parser: a record whose
|
|
/// value runs past the buffer is a truncated record and an error, and an all-zero frame is
|
|
/// *end of records* only when every remaining byte is also zero — otherwise a real record
|
|
/// at the origin, followed by padding, would be read as an empty one.
|
|
/// What a header says, in both formats.
|
|
struct Header {
|
|
version: u8,
|
|
curve: u8,
|
|
image_bytes: Option<u64>,
|
|
record_count: Option<u64>,
|
|
}
|
|
|
|
/// Read the header, or say what is wrong with it.
|
|
fn parse_header(image: &[u8]) -> Result<Header, &'static str> {
|
|
if image.len() < 5 || &image[0..4] != MAGIC {
|
|
return Err("not-a-store");
|
|
}
|
|
let version = image[4];
|
|
let curve = image[5];
|
|
match version {
|
|
VERSION_V1 => Ok(Header {
|
|
version,
|
|
curve,
|
|
image_bytes: None,
|
|
record_count: None,
|
|
}),
|
|
VERSION => {
|
|
if image.len() < HEADER_LEN_V2 {
|
|
return Err("short-v2-header");
|
|
}
|
|
let mut word = [0u8; 8];
|
|
word.copy_from_slice(&image[6..14]);
|
|
let image_bytes = u64::from_le_bytes(word);
|
|
word.copy_from_slice(&image[14..22]);
|
|
let record_count = u64::from_le_bytes(word);
|
|
// A header that does not cover itself is corrupt, and guessing at the extent
|
|
// of an image means possibly reading somebody else's bytes.
|
|
if (image_bytes as usize) < HEADER_LEN_V2 {
|
|
return Err("bad-extent");
|
|
}
|
|
Ok(Header {
|
|
version,
|
|
curve,
|
|
image_bytes: Some(image_bytes),
|
|
record_count: Some(record_count),
|
|
})
|
|
}
|
|
_ => Err("unsupported-version"),
|
|
}
|
|
}
|
|
|
|
fn digest(image: &[u8]) -> Line {
|
|
let mut line = Line::new();
|
|
|
|
// Where the image is, and which log belongs to it: a store device says so in its control
|
|
// block, a bare image says so in its own header.
|
|
let layout = match resolve_layout(image) {
|
|
Ok(l) => l,
|
|
Err(what) => {
|
|
let _ = write!(line, "error={what}");
|
|
return line;
|
|
}
|
|
};
|
|
let live = &image[layout.image_off..];
|
|
let header = match parse_header(live) {
|
|
Ok(h) => h,
|
|
Err(what) => {
|
|
let _ = write!(line, "error={what}");
|
|
return line;
|
|
}
|
|
};
|
|
let end = match header.image_bytes {
|
|
Some(n) => core::cmp::min(n as usize, live.len()),
|
|
None => live.len(),
|
|
};
|
|
|
|
// With a v2 image, the store is the image *plus* whatever log follows it. The order the
|
|
// digest is taken in is then the order a checkpoint would write: sorted by coordinate,
|
|
// later writes winning. Without a log there is nothing to merge, and the image's own
|
|
// order is the answer — which is also what keeps v1 images readable and unchanged.
|
|
if header.image_bytes.is_some() {
|
|
let log = log_window(image, &layout);
|
|
match merged_digest(&live[..end], log, &header) {
|
|
Ok(line) => return line,
|
|
// A log that exists but cannot be read is worth saying out loud: the digest
|
|
// alone would look like a shorter store.
|
|
Err(what) if what != "no-log" => {
|
|
let mut line = Line::new();
|
|
let _ = write!(line, "error={what}");
|
|
return line;
|
|
}
|
|
Err(_) => {}
|
|
}
|
|
}
|
|
|
|
let mut h = FNV_OFFSET;
|
|
let mut count: u64 = 0;
|
|
let mut value_bytes: u64 = 0;
|
|
let mut errors: u64 = 0;
|
|
let header_len = if header.version == VERSION_V1 {
|
|
HEADER_LEN_V1
|
|
} else {
|
|
HEADER_LEN_V2
|
|
} as u64;
|
|
let mut off = layout.image_off + header_len as usize;
|
|
let mut remaining = header.record_count;
|
|
let end = layout.image_off + end;
|
|
|
|
while off + RECORD_FIXED <= end {
|
|
if remaining == Some(0) {
|
|
break;
|
|
}
|
|
let frame = &image[off..off + RECORD_FIXED];
|
|
// Padding is not a record — but only if it is padding all the way down, and only
|
|
// in v1, where nothing else says where the records stop. A v2 image states its
|
|
// count, so a zero frame inside that count is a record like any other.
|
|
if remaining.is_none()
|
|
&& frame.iter().all(|b| *b == 0)
|
|
&& image[off..].iter().all(|b| *b == 0)
|
|
{
|
|
break;
|
|
}
|
|
|
|
let space = &image[off..off + SPACE_ID_LEN];
|
|
let key = &image[off + SPACE_ID_LEN..off + SPACE_ID_LEN + RAW_KEY_LEN];
|
|
let mut len_bytes = [0u8; 8];
|
|
len_bytes.copy_from_slice(&image[off + SPACE_ID_LEN + RAW_KEY_LEN..off + RECORD_FIXED]);
|
|
let value_len = u64::from_le_bytes(len_bytes) as usize;
|
|
|
|
let value_at = off + RECORD_FIXED;
|
|
// A promised record that is not there is a truncated image, and saying so beats
|
|
// returning a shorter list that looks complete.
|
|
if value_at + value_len > end {
|
|
errors += 1;
|
|
break;
|
|
}
|
|
let value = &image[value_at..value_at + value_len];
|
|
|
|
h = fnv1a64(space, h);
|
|
h = fnv1a64(key, h);
|
|
h = fnv1a64(&(value_len as u64).to_le_bytes(), h);
|
|
h = fnv1a64(value, h);
|
|
count += 1;
|
|
value_bytes += value_len as u64;
|
|
off = value_at + value_len;
|
|
if let Some(n) = remaining.as_mut() {
|
|
*n -= 1;
|
|
}
|
|
}
|
|
// The count was a promise; not meeting it is an error the caller must see.
|
|
if let Some(n) = remaining {
|
|
if n > 0 {
|
|
errors += 1;
|
|
}
|
|
}
|
|
|
|
// Field order matches cube-image's `digest` line so the two diff directly.
|
|
let _ = write!(
|
|
line,
|
|
"digest version={} curve={} bytes={} records={} value_bytes={} fnv1a64={:016x} errors={}",
|
|
header.version,
|
|
header.curve,
|
|
// `bytes` is a property of the store, not of how it was read: the header plus the
|
|
// records, whatever padding the reader happened to see. Reporting what was read made
|
|
// the same records disagree between a file and a device — and the gates compare this
|
|
// line, so it has to mean the same thing on both sides.
|
|
header_len + count * RECORD_FIXED as u64 + value_bytes,
|
|
count,
|
|
value_bytes,
|
|
h,
|
|
errors
|
|
);
|
|
line
|
|
}
|
|
|
|
/// What a store device's control block says: where the two image slots and the log are,
|
|
/// which slot is live, and how much of the log is in use.
|
|
#[derive(Clone, Copy)]
|
|
struct Control {
|
|
curve: u8,
|
|
slot_bytes: u64,
|
|
log_capacity: u64,
|
|
log_used: u64,
|
|
generation: u64,
|
|
active_slot: u8,
|
|
}
|
|
|
|
impl Control {
|
|
/// Parse a control copy. `None` when it is not one, or its checksum does not hold — a
|
|
/// torn update in the middle of a checkpoint is exactly what that checksum is for.
|
|
fn decode(b: &[u8]) -> Option<Control> {
|
|
if b.len() < CTL_SUMMED + 4 || &b[0..4] != CTL_MAGIC {
|
|
return None;
|
|
}
|
|
let word = |at: usize| -> u64 {
|
|
let mut w = [0u8; 8];
|
|
w.copy_from_slice(&b[at..at + 8]);
|
|
u64::from_le_bytes(w)
|
|
};
|
|
let mut c = [0u8; 4];
|
|
c.copy_from_slice(&b[44..48]);
|
|
if u32::from_le_bytes(c) != crc32(&b[..CTL_SUMMED]) {
|
|
return None;
|
|
}
|
|
let control = Control {
|
|
curve: b[5],
|
|
slot_bytes: word(8),
|
|
log_capacity: word(16),
|
|
log_used: word(24),
|
|
generation: word(32),
|
|
active_slot: b[40],
|
|
};
|
|
if b[4] != CTL_VERSION || control.active_slot > 1 || control.slot_bytes == 0 {
|
|
return None;
|
|
}
|
|
Some(control)
|
|
}
|
|
|
|
/// Encode into a control copy, checksum included.
|
|
fn encode(&self, b: &mut [u8]) {
|
|
for byte in b.iter_mut() {
|
|
*byte = 0;
|
|
}
|
|
b[0..4].copy_from_slice(&CTL_MAGIC);
|
|
b[4] = CTL_VERSION;
|
|
b[5] = self.curve;
|
|
b[8..16].copy_from_slice(&self.slot_bytes.to_le_bytes());
|
|
b[16..24].copy_from_slice(&self.log_capacity.to_le_bytes());
|
|
b[24..32].copy_from_slice(&self.log_used.to_le_bytes());
|
|
b[32..40].copy_from_slice(&self.generation.to_le_bytes());
|
|
b[40] = self.active_slot;
|
|
let crc = crc32(&b[..CTL_SUMMED]);
|
|
b[44..48].copy_from_slice(&crc.to_le_bytes());
|
|
}
|
|
|
|
fn image_off(&self) -> u64 {
|
|
CTL_DATA_OFF + self.active_slot as u64 * self.slot_bytes
|
|
}
|
|
|
|
fn spare_off(&self) -> u64 {
|
|
CTL_DATA_OFF + (1 - self.active_slot) as u64 * self.slot_bytes
|
|
}
|
|
|
|
fn log_off(&self) -> u64 {
|
|
CTL_DATA_OFF + 2 * self.slot_bytes
|
|
}
|
|
}
|
|
|
|
/// The valid control block: the copy with the higher generation. A reader that finds one
|
|
/// copy torn uses the other, which is the entire reason there are two.
|
|
fn read_control(b: &[u8]) -> Option<Control> {
|
|
if b.len() < CTL_COPY_B + CTL_COPY_LEN {
|
|
return None;
|
|
}
|
|
let a = Control::decode(&b[CTL_COPY_A..CTL_COPY_A + CTL_COPY_LEN]);
|
|
let b_copy = Control::decode(&b[CTL_COPY_B..CTL_COPY_B + CTL_COPY_LEN]);
|
|
match (a, b_copy) {
|
|
(Some(x), Some(y)) => Some(if x.generation >= y.generation { x } else { y }),
|
|
(Some(x), None) => Some(x),
|
|
(None, Some(y)) => Some(y),
|
|
(None, None) => None,
|
|
}
|
|
}
|
|
|
|
/// Where the image is and what log belongs to it.
|
|
struct Layout {
|
|
/// Byte offset of the live image.
|
|
image_off: usize,
|
|
/// Byte offset of its log.
|
|
log_off: usize,
|
|
/// Bytes of log in use.
|
|
log_used: usize,
|
|
/// The control block, when this is a store device rather than a bare image.
|
|
control: Option<Control>,
|
|
}
|
|
|
|
/// Work out the layout: a store device when the control block is there, a bare image when it
|
|
/// is not. Both are readable; only the store layout can be checkpointed atomically, which is
|
|
/// why `sync` refuses on the other.
|
|
fn resolve_layout(b: &[u8]) -> Result<Layout, &'static str> {
|
|
if b.len() >= CTL_COPY_B + CTL_COPY_LEN && &b[0..4] == CTL_MAGIC {
|
|
let control = read_control(b).ok_or("control-block-corrupt")?;
|
|
let log_off = control.log_off() as usize;
|
|
let used = control.log_used as usize;
|
|
if log_off + used > b.len() || used > control.log_capacity as usize {
|
|
return Err("log-outside-device");
|
|
}
|
|
return Ok(Layout {
|
|
image_off: control.image_off() as usize,
|
|
log_off,
|
|
log_used: used,
|
|
control: Some(control),
|
|
});
|
|
}
|
|
// A bare image. A v1 image has no extent and therefore no log — it is the whole device
|
|
// as far as this module is concerned, and it is read the way it always was.
|
|
let header = parse_header(b)?;
|
|
let log_off = match header.image_bytes {
|
|
Some(n) => log_offset(n),
|
|
None => b.len(),
|
|
};
|
|
Ok(Layout {
|
|
image_off: 0,
|
|
log_off,
|
|
// A bare device records no log length, so the log is bounded by the buffer and by
|
|
// the entries that parse: `apply_log` stops at the first one that does not.
|
|
log_used: b.len().saturating_sub(log_off),
|
|
control: None,
|
|
})
|
|
}
|
|
|
|
/// One record on its way into the merged store. Fixed size, so a `KVVec` of these sorts
|
|
/// in place; values live in a pool beside them and are referenced by offset.
|
|
#[derive(Clone, Copy, PartialEq, Eq, PartialOrd, Ord)]
|
|
struct Entry {
|
|
space: [u8; 32],
|
|
key: [u8; 24],
|
|
/// Order in which this entry arrived. Breaks ties between the same coordinate, so the
|
|
/// later write wins after sorting.
|
|
seq: u32,
|
|
value_off: u32,
|
|
value_len: u32,
|
|
deleted: bool,
|
|
}
|
|
|
|
/// The store as a sorted set of records, which is what a checkpoint writes and what the
|
|
/// digest is taken over.
|
|
struct Merged {
|
|
pool: KVVec<u8>,
|
|
entries: KVVec<Entry>,
|
|
seq: u32,
|
|
/// Log entries that were seen but not applied, because they were torn or malformed.
|
|
dropped: u64,
|
|
}
|
|
|
|
impl Merged {
|
|
fn new() -> Result<Self, AllocError> {
|
|
Ok(Merged {
|
|
pool: KVVec::new(),
|
|
entries: KVVec::new(),
|
|
seq: 0,
|
|
dropped: 0,
|
|
})
|
|
}
|
|
|
|
/// Add a record. `deleted` marks a removal, which is kept in the list so it can
|
|
/// override an older value for the same coordinate.
|
|
fn add(&mut self, space: &[u8], key: &[u8], value: &[u8], deleted: bool) -> Result<(), AllocError> {
|
|
let mut sp = [0u8; 32];
|
|
sp.copy_from_slice(&space[..32]);
|
|
let mut k = [0u8; 24];
|
|
k.copy_from_slice(&key[..24]);
|
|
let value_off = self.pool.len() as u32;
|
|
self.pool.extend_from_slice(value, GFP_KERNEL)?;
|
|
let seq = self.seq;
|
|
self.seq += 1;
|
|
self.entries.push(
|
|
Entry {
|
|
space: sp,
|
|
key: k,
|
|
seq,
|
|
value_off,
|
|
value_len: value.len() as u32,
|
|
deleted,
|
|
},
|
|
GFP_KERNEL,
|
|
)?;
|
|
Ok(())
|
|
}
|
|
|
|
/// Put the entries in the order a checkpoint would write them: by space, then by key,
|
|
/// with the later write of a coordinate last — so a group's final entry decides it.
|
|
///
|
|
/// This is a heapsort, and the reason is a panic rather than a preference.
|
|
/// `slice::sort_unstable` allocates about 3.5 KiB of *kernel stack* per recursion level in
|
|
/// this kernel's Rust, so a store with a couple of hundred records overflowed the 16 KiB
|
|
/// kernel stack and took the machine down — `BUG: TASK stack guard page was hit`, with the
|
|
/// instruction pointer inside
|
|
/// `core::slice::sort::unstable::quicksort::<cubelinux_store::Entry>` and `sub rsp, 0xdd8` in
|
|
/// the function's first bytes. A heapsort has a constant frame and allocates nothing, so the
|
|
/// size of a store cannot decide whether reading it is safe, and the sort cannot fail for want
|
|
/// of memory in the middle of a read.
|
|
///
|
|
/// It is not stable, which costs nothing here: `seq` is part of the ordering key, so the order
|
|
/// is total and two entries never compare equal.
|
|
fn sort_entries(&mut self) {
|
|
let entries = self.entries.as_mut_slice();
|
|
let len = entries.len();
|
|
if len < 2 {
|
|
return;
|
|
}
|
|
// Build a max-heap, then repeatedly move the maximum to the end.
|
|
let mut start = len / 2;
|
|
while start > 0 {
|
|
start -= 1;
|
|
sift_down(entries, start, len);
|
|
}
|
|
let mut end = len;
|
|
while end > 1 {
|
|
end -= 1;
|
|
entries.swap(0, end);
|
|
sift_down(entries, 0, end);
|
|
}
|
|
}
|
|
|
|
fn value(&self, e: &Entry) -> &[u8] {
|
|
let off = e.value_off as usize;
|
|
&self.pool.as_slice()[off..off + e.value_len as usize]
|
|
}
|
|
}
|
|
|
|
/// Restore the heap property below `root` over `entries[..end]`.
|
|
///
|
|
/// Iterative on purpose: the whole point of the heapsort above is that nothing here grows with the
|
|
/// size of the store.
|
|
fn sift_down(entries: &mut [Entry], mut root: usize, end: usize) {
|
|
loop {
|
|
let left = 2 * root + 1;
|
|
if left >= end {
|
|
return;
|
|
}
|
|
let mut largest = left;
|
|
if left + 1 < end && entries[left] < entries[left + 1] {
|
|
largest = left + 1;
|
|
}
|
|
if entries[root] < entries[largest] {
|
|
entries.swap(root, largest);
|
|
root = largest;
|
|
} else {
|
|
return;
|
|
}
|
|
}
|
|
}
|
|
|
|
/// Read the log that follows the image, if there is one, and apply its entries.
|
|
///
|
|
/// The log is a delta on the image, so its entries override image records for the same
|
|
/// coordinate. Recovery is prefix-trusting, exactly as userspace does it: entries are
|
|
/// replayed from the start and the first one that is short, mis-framed or fails its
|
|
/// checksum ends the log. A fault in the middle cannot be told from a torn tail without a
|
|
/// second copy, so the log stops there and reports what it dropped instead of guessing.
|
|
fn apply_log(log: &[u8], merged: &mut Merged) -> Result<Option<u64>, &'static str> {
|
|
if log.len() < WAL_HEADER_LEN || &log[0..4] != WAL_MAGIC {
|
|
return Ok(None);
|
|
}
|
|
if log[4] != WAL_VERSION {
|
|
return Err("unsupported-log-version");
|
|
}
|
|
|
|
let mut applied: u64 = 0;
|
|
let mut off = WAL_HEADER_LEN;
|
|
while off + ENTRY_FIXED <= log.len() {
|
|
let start = off;
|
|
let op = log[off];
|
|
if op != 1 && op != 2 {
|
|
break;
|
|
}
|
|
let mut word = [0u8; 4];
|
|
word.copy_from_slice(&log[off + 1..off + 5]);
|
|
let crc = u32::from_le_bytes(word);
|
|
let space = &log[off + 5..off + 37];
|
|
let key = &log[off + 37..off + 61];
|
|
word.copy_from_slice(&log[off + 61..off + 65]);
|
|
let len = u32::from_le_bytes(word) as usize;
|
|
let frame_end = start + ENTRY_FIXED + len;
|
|
if frame_end > log.len() {
|
|
break;
|
|
}
|
|
// The checksum covers space, key, length and value, so a corrupted entry is
|
|
// stopped at rather than applied.
|
|
if crc32(&log[start + 5..frame_end]) != crc {
|
|
break;
|
|
}
|
|
let value = &log[start + ENTRY_FIXED..frame_end];
|
|
off = frame_end;
|
|
merged
|
|
.add(space, key, value, op == 2)
|
|
.map_err(|_| "out-of-memory")?;
|
|
applied += 1;
|
|
}
|
|
|
|
merged.dropped = (log.len() - off) as u64;
|
|
Ok(Some(applied))
|
|
}
|
|
|
|
/// CRC-32 (IEEE 802.3), bitwise — the same polynomial and the same coverage as
|
|
/// `cube-store/src/wal.rs`, because a log written on either side must validate on both.
|
|
fn crc32(bytes: &[u8]) -> u32 {
|
|
let mut crc = 0xFFFF_FFFFu32;
|
|
for byte in bytes {
|
|
crc ^= *byte as u32;
|
|
for _ in 0..8 {
|
|
let mask = (crc & 1).wrapping_neg();
|
|
crc = (crc >> 1) ^ (0xEDB8_8320 & mask);
|
|
}
|
|
}
|
|
!crc
|
|
}
|
|
|
|
/// The log window that belongs to the live image.
|
|
fn log_window<'a>(device: &'a [u8], layout: &Layout) -> &'a [u8] {
|
|
let end = core::cmp::min(
|
|
layout.log_off + WAL_HEADER_LEN + layout.log_used,
|
|
device.len(),
|
|
);
|
|
if layout.log_off <= end {
|
|
&device[layout.log_off..end]
|
|
} else {
|
|
&[]
|
|
}
|
|
}
|
|
|
|
/// The store the image and its log together describe: the image's records, then the log's
|
|
/// entries applied over them. This is what a checkpoint writes and what the digest is taken
|
|
/// over, so both go through here rather than each walking the bytes its own way.
|
|
fn build_merged(image: &[u8], log: &[u8], header: &Header) -> Result<(Merged, u64), &'static str> {
|
|
let mut merged = Merged::new().map_err(|_| "out-of-memory")?;
|
|
|
|
let extent = header.image_bytes.ok_or("v1-has-no-extent")? as usize;
|
|
let extent = core::cmp::min(extent, image.len());
|
|
let mut off = if header.version == VERSION_V1 {
|
|
HEADER_LEN_V1
|
|
} else {
|
|
HEADER_LEN_V2
|
|
};
|
|
let mut seen: u64 = 0;
|
|
while off + RECORD_FIXED <= extent {
|
|
if let Some(c) = header.record_count {
|
|
if seen >= c {
|
|
break;
|
|
}
|
|
}
|
|
let space = &image[off..off + SPACE_ID_LEN];
|
|
let key = &image[off + SPACE_ID_LEN..off + SPACE_ID_LEN + RAW_KEY_LEN];
|
|
let mut word = [0u8; 8];
|
|
word.copy_from_slice(&image[off + SPACE_ID_LEN + RAW_KEY_LEN..off + RECORD_FIXED]);
|
|
let value_len = u64::from_le_bytes(word) as usize;
|
|
let value_at = off + RECORD_FIXED;
|
|
if value_at + value_len > extent {
|
|
return Err("truncated-image");
|
|
}
|
|
merged
|
|
.add(space, key, &image[value_at..value_at + value_len], false)
|
|
.map_err(|_| "out-of-memory")?;
|
|
seen += 1;
|
|
off = value_at + value_len;
|
|
}
|
|
|
|
let applied = match apply_log(log, &mut merged)? {
|
|
Some(n) => n,
|
|
None => 0,
|
|
};
|
|
Ok((merged, applied))
|
|
}
|
|
|
|
/// Digest the store the image and its log together describe.
|
|
///
|
|
/// `image` is the live image's bytes alone; `log` is exactly the window of log that belongs
|
|
/// to it. Returns `Err("no-log")` when there is none, so the caller can fall back to reading
|
|
/// the image by itself.
|
|
fn merged_digest(image: &[u8], log: &[u8], header: &Header) -> Result<Line, &'static str> {
|
|
if log.len() < WAL_HEADER_LEN || &log[0..4] != WAL_MAGIC {
|
|
return Err("no-log");
|
|
}
|
|
let (mut merged, applied) = build_merged(image, log, header)?;
|
|
|
|
merged.sort_entries();
|
|
let entries = merged.entries.as_slice();
|
|
let mut h = FNV_OFFSET;
|
|
let mut count: u64 = 0;
|
|
let mut value_bytes: u64 = 0;
|
|
let mut i = 0;
|
|
while i < entries.len() {
|
|
let head = &entries[i];
|
|
let mut last = i;
|
|
while last + 1 < entries.len()
|
|
&& entries[last + 1].space == head.space
|
|
&& entries[last + 1].key == head.key
|
|
{
|
|
last += 1;
|
|
}
|
|
let winner = &entries[last];
|
|
i = last + 1;
|
|
if winner.deleted {
|
|
continue;
|
|
}
|
|
let value = merged.value(winner);
|
|
h = fnv1a64(&winner.space, h);
|
|
h = fnv1a64(&winner.key, h);
|
|
h = fnv1a64(&(value.len() as u64).to_le_bytes(), h);
|
|
h = fnv1a64(value, h);
|
|
count += 1;
|
|
value_bytes += value.len() as u64;
|
|
}
|
|
|
|
// `bytes` is the size a checkpoint of this store would produce — the same number
|
|
// userspace reports for the image it writes from the same records.
|
|
let folded = HEADER_LEN_V2 as u64 + count * RECORD_FIXED as u64 + value_bytes;
|
|
let mut line = Line::new();
|
|
// The digest line is field-for-field what `cube-image digest` prints, so the two are
|
|
// compared by diff. What the log contributed goes on its own line: a padded log region
|
|
// makes a byte count meaningless, an entry count does not.
|
|
let _ = write!(
|
|
line,
|
|
"digest version={} curve={} bytes={} records={} value_bytes={} fnv1a64={:016x} errors=0\n",
|
|
header.version, header.curve, folded, count, value_bytes, h
|
|
);
|
|
let _ = write!(line, "log entries={} applied over the image\n", applied);
|
|
Ok(line)
|
|
}
|
|
|
|
/// Where a bare image's log starts, given the image's declared extent.
|
|
fn log_offset(image_bytes: u64) -> usize {
|
|
let n = image_bytes as usize;
|
|
(n + LOG_ALIGN - 1) & !(LOG_ALIGN - 1)
|
|
}
|
|
|
|
/// Length of a bare log's valid prefix: the header plus every entry that parses and passes
|
|
/// its checksum. This is the append position, and the reason a torn tail is overwritten
|
|
/// rather than appended to.
|
|
fn log_valid_len(log: &[u8]) -> Result<usize, &'static str> {
|
|
if log.len() < WAL_HEADER_LEN {
|
|
return Err("short-log");
|
|
}
|
|
if &log[0..4] != WAL_MAGIC {
|
|
return Err("no-log");
|
|
}
|
|
if log[4] != WAL_VERSION {
|
|
return Err("unsupported-log-version");
|
|
}
|
|
let mut off = WAL_HEADER_LEN;
|
|
while off + ENTRY_FIXED <= log.len() {
|
|
let start = off;
|
|
let op = log[off];
|
|
if op != 1 && op != 2 {
|
|
break;
|
|
}
|
|
let mut word = [0u8; 4];
|
|
word.copy_from_slice(&log[off + 1..off + 5]);
|
|
let crc = u32::from_le_bytes(word);
|
|
word.copy_from_slice(&log[off + 61..off + 65]);
|
|
let len = u32::from_le_bytes(word) as usize;
|
|
// Only a validated entry moves the append point. Advancing first and checking after
|
|
// counts a torn entry as part of the prefix, so the next append lands *after* the
|
|
// corruption and buries it — the opposite of the rule that a torn tail is overwritten.
|
|
let frame_end = start + ENTRY_FIXED + len;
|
|
if frame_end > log.len() {
|
|
break;
|
|
}
|
|
if crc32(&log[start + 5..frame_end]) != crc {
|
|
break;
|
|
}
|
|
off = frame_end;
|
|
}
|
|
Ok(off)
|
|
}
|
|
|
|
/// Morton-encode a point into the store's 24-byte key — the same interleaving
|
|
/// `cube-core`'s `Curve for Morton` does, because a key written here has to be the key a
|
|
/// userspace reader decodes. Bit `i` of each axis lands at bit `3i` of the 192-bit key,
|
|
/// counting from the least significant bit, which lives in the *last* byte.
|
|
fn morton_encode(x: u64, y: u64, z: u64) -> [u8; 24] {
|
|
let mut k = [0u8; 24];
|
|
let put = |n: usize, v: bool, k: &mut [u8; 24]| {
|
|
if !v {
|
|
return;
|
|
}
|
|
let byte = 23 - (n / 8);
|
|
k[byte] |= 1 << (n % 8);
|
|
};
|
|
for i in 0..64 {
|
|
put(3 * i, (x >> i) & 1 != 0, &mut k);
|
|
put(3 * i + 1, (y >> i) & 1 != 0, &mut k);
|
|
put(3 * i + 2, (z >> i) & 1 != 0, &mut k);
|
|
}
|
|
k
|
|
}
|
|
|
|
/// A mutation to append: the byte-plane write the store contract calls `put`.
|
|
struct Mutation {
|
|
space: [u8; 32],
|
|
key: [u8; 24],
|
|
value: KVVec<u8>,
|
|
}
|
|
|
|
/// Build a log entry: `op | crc32 | space | key | len | value`.
|
|
fn encode_entry(op: u8, space: &[u8; 32], key: &[u8; 24], value: &[u8]) -> Result<KVVec<u8>, AllocError> {
|
|
let total = ENTRY_FIXED + value.len();
|
|
let mut entry = KVVec::<u8>::with_capacity(total, GFP_KERNEL)?;
|
|
entry.extend_from_slice(&[op], GFP_KERNEL)?;
|
|
entry.extend_from_slice(&[0u8; 4][..], GFP_KERNEL)?;
|
|
entry.extend_from_slice(space, GFP_KERNEL)?;
|
|
entry.extend_from_slice(key, GFP_KERNEL)?;
|
|
entry.extend_from_slice(&(value.len() as u32).to_le_bytes(), GFP_KERNEL)?;
|
|
entry.extend_from_slice(value, GFP_KERNEL)?;
|
|
// The checksum covers everything after the crc field.
|
|
let crc = crc32(&entry.as_slice()[5..]);
|
|
entry.as_mut_slice()[1..5].copy_from_slice(&crc.to_le_bytes());
|
|
Ok(entry)
|
|
}
|
|
|
|
/// Write `bytes` at `off` and flush, or say why not. The one place the module writes.
|
|
fn write_and_sync(file: *mut bindings::file, off: u64, bytes: &[u8]) -> Result<(), Error> {
|
|
let mut at: bindings::loff_t = off as bindings::loff_t;
|
|
// SAFETY: the caller passes a live `struct file *`; `bytes` outlives the call; `at` is a
|
|
// valid loff_t.
|
|
let wrote = unsafe {
|
|
bindings::kernel_write(
|
|
file,
|
|
bytes.as_ptr().cast::<core::ffi::c_void>(),
|
|
bytes.len(),
|
|
&mut at,
|
|
)
|
|
};
|
|
if wrote < 0 {
|
|
return Err(Error::from_errno(wrote as i32));
|
|
}
|
|
if wrote as usize != bytes.len() {
|
|
return Err(EIO);
|
|
}
|
|
// Durability before acknowledgement: a short write that was never flushed is a write that
|
|
// did not happen.
|
|
// SAFETY: the caller passes a live, writable file; datasync is 0 (a full sync).
|
|
let rc = unsafe { bindings::vfs_fsync(file, 0) };
|
|
if rc < 0 {
|
|
return Err(Error::from_errno(rc));
|
|
}
|
|
Ok(())
|
|
}
|
|
|
|
/// The control copy an update should write: the older of the two, so the current one stays
|
|
/// readable throughout.
|
|
fn older_copy(generation: u64) -> usize {
|
|
if generation % 2 == 0 {
|
|
CTL_COPY_A
|
|
} else {
|
|
CTL_COPY_B
|
|
}
|
|
}
|
|
|
|
/// Append a mutation to the log and make it durable, returning the updated control block
|
|
/// when the layout has one.
|
|
///
|
|
/// The order is the durability contract: the entry is written and flushed to the device
|
|
/// *before* the caller is told it happened. A mutation this function has returned `Ok` for
|
|
/// survives a power cut; one it has not may or may not, which is exactly the boundary an
|
|
/// acknowledgement is supposed to mark.
|
|
///
|
|
/// On a store device the entry goes after the log bytes already in use, and the control
|
|
/// block's count is raised afterwards — entry first, count second, so a crash between them
|
|
/// loses an *unacknowledged* mutation rather than counting one that is not there. A count
|
|
/// that ran ahead would make replay read past valid data; a count that lags only forgets.
|
|
fn append(
|
|
image: &[u8],
|
|
layout: &Layout,
|
|
m: &Mutation,
|
|
op: u8,
|
|
) -> Result<Option<Control>, Error> {
|
|
let entry = encode_entry(op, &m.space, &m.key, m.value.as_slice())?;
|
|
|
|
// Where it goes: after the bytes in use on a store device, after the valid prefix on a
|
|
// bare image, where nothing records the length.
|
|
let region_at = core::cmp::min(layout.log_off, image.len());
|
|
let region = &image[region_at..];
|
|
let (used, capacity) = match layout.control {
|
|
// `log_used` counts entry bytes *after* the log's header, so the header is not
|
|
// something an append has to account for in the count.
|
|
Some(c) => (c.log_used as usize, c.log_capacity as usize),
|
|
None => {
|
|
// A region with no header is a log that has never been written; it is empty, not
|
|
// broken, and the header goes down before the first entry.
|
|
let used = match log_valid_len(region) {
|
|
Ok(valid) => valid - WAL_HEADER_LEN,
|
|
Err("no-log") | Err("short-log") => 0,
|
|
Err(what) => {
|
|
pr_err!("cubelinux: log region unreadable: {}\n", what);
|
|
return Err(EINVAL);
|
|
}
|
|
};
|
|
(used, region.len())
|
|
}
|
|
};
|
|
if WAL_HEADER_LEN + used + entry.len() > capacity {
|
|
pr_err!(
|
|
"cubelinux: the log is full ({} of {} bytes); checkpoint before appending\n",
|
|
used,
|
|
capacity
|
|
);
|
|
return Err(ENOSPC);
|
|
}
|
|
|
|
// SAFETY: store_device() is a NUL-terminated C string filled at boot; filp_open returns a
|
|
// valid file or an error pointer, which is checked. O_RDWR is 2.
|
|
let file = unsafe { bindings::filp_open(store_device(), 2, 0) };
|
|
let file = kernel::error::from_err_ptr(file)?;
|
|
if file.is_null() {
|
|
return Err(EINVAL);
|
|
}
|
|
|
|
let mut result: Result<(), Error> = Ok(());
|
|
// A log region that has never been written is zeros, not a log: give it a header, the
|
|
// same thing the userspace log does when its file does not exist.
|
|
// A log region with no header is not a log: give it one, whether this is a bare image or
|
|
// a store device, so a reader always finds the framing it expects.
|
|
if region.len() < WAL_HEADER_LEN || ®ion[0..4] != WAL_MAGIC {
|
|
let mut hdr = [0u8; WAL_HEADER_LEN];
|
|
hdr[0..4].copy_from_slice(&WAL_MAGIC);
|
|
hdr[4] = WAL_VERSION;
|
|
hdr[5] = 0; // morton
|
|
result = write_and_sync(file, layout.log_off as u64, &hdr);
|
|
}
|
|
if result.is_ok() {
|
|
result = write_and_sync(
|
|
file,
|
|
(region_at + WAL_HEADER_LEN + used) as u64,
|
|
entry.as_slice(),
|
|
);
|
|
}
|
|
|
|
// The count second, so replay can never be told about an entry that is not there.
|
|
let mut updated = None;
|
|
if result.is_ok() {
|
|
if let Some(c) = layout.control {
|
|
let next = Control {
|
|
log_used: c.log_used + entry.len() as u64,
|
|
generation: c.generation + 1,
|
|
..c
|
|
};
|
|
let mut buf = [0u8; CTL_COPY_LEN];
|
|
next.encode(&mut buf);
|
|
result = write_and_sync(file, older_copy(c.generation) as u64, &buf);
|
|
if result.is_ok() {
|
|
updated = Some(next);
|
|
}
|
|
}
|
|
}
|
|
|
|
// SAFETY: `file` came from filp_open and has not been closed.
|
|
unsafe { bindings::filp_close(file, core::ptr::null_mut()) };
|
|
result.map(|_| updated)
|
|
}
|
|
|
|
/// Serialize the merged store into the pinned v2 image.
|
|
///
|
|
/// The winners are collected first because the header states the image's extent and its
|
|
/// record count, and those are only known once the coalescing is done.
|
|
fn serialize_image(merged: &mut Merged, curve: u8) -> Result<KVVec<u8>, AllocError> {
|
|
merged.sort_entries();
|
|
let entries = merged.entries.as_slice();
|
|
|
|
let mut winners = KVVec::<u32>::new();
|
|
let mut values_len: u64 = 0;
|
|
let mut i = 0;
|
|
while i < entries.len() {
|
|
let head = &entries[i];
|
|
let mut last = i;
|
|
while last + 1 < entries.len()
|
|
&& entries[last + 1].space == head.space
|
|
&& entries[last + 1].key == head.key
|
|
{
|
|
last += 1;
|
|
}
|
|
if !entries[last].deleted {
|
|
winners.push(last as u32, GFP_KERNEL)?;
|
|
values_len += entries[last].value_len as u64;
|
|
}
|
|
i = last + 1;
|
|
}
|
|
|
|
let image_bytes = HEADER_LEN_V2 as u64 + winners.len() as u64 * RECORD_FIXED as u64 + values_len;
|
|
let mut out = KVVec::<u8>::with_capacity(image_bytes as usize, GFP_KERNEL)?;
|
|
out.extend_from_slice(MAGIC, GFP_KERNEL)?;
|
|
out.extend_from_slice(&[VERSION, curve], GFP_KERNEL)?;
|
|
out.extend_from_slice(&image_bytes.to_le_bytes(), GFP_KERNEL)?;
|
|
out.extend_from_slice(&(winners.len() as u64).to_le_bytes(), GFP_KERNEL)?;
|
|
|
|
for w in winners.as_slice() {
|
|
let e = &merged.entries.as_slice()[*w as usize];
|
|
let value = merged.value(e);
|
|
out.extend_from_slice(&e.space, GFP_KERNEL)?;
|
|
out.extend_from_slice(&e.key, GFP_KERNEL)?;
|
|
out.extend_from_slice(&(value.len() as u64).to_le_bytes(), GFP_KERNEL)?;
|
|
out.extend_from_slice(value, GFP_KERNEL)?;
|
|
}
|
|
Ok(out)
|
|
}
|
|
|
|
/// Fold the log into the image: write the merged store into the spare slot, then make it
|
|
/// live.
|
|
///
|
|
/// The order is what makes this atomic. The new image goes to the slot nothing is reading and
|
|
/// is flushed; only then does the control block change, and that change is one small write to
|
|
/// the control copy that is *not* current. At every instant there is either the old image with
|
|
/// a log that still describes the mutations since it, or the new image with an empty log. A
|
|
/// crash in the middle of the copy leaves the previous store intact, which is the entire
|
|
/// reason for two slots.
|
|
fn checkpoint(ctl: &Control, merged: &mut Merged) -> Result<u64, Error> {
|
|
let new_image = serialize_image(merged, ctl.curve)?;
|
|
if new_image.len() as u64 > ctl.slot_bytes {
|
|
pr_err!(
|
|
"cubelinux: checkpoint needs {} bytes but a slot holds {}\n",
|
|
new_image.len(),
|
|
ctl.slot_bytes
|
|
);
|
|
return Err(ENOSPC);
|
|
}
|
|
|
|
// SAFETY: as in `append`.
|
|
let file = unsafe { bindings::filp_open(store_device(), 2, 0) };
|
|
let file = kernel::error::from_err_ptr(file)?;
|
|
if file.is_null() {
|
|
return Err(EINVAL);
|
|
}
|
|
|
|
let mut result = write_and_sync(file, ctl.spare_off(), new_image.as_slice());
|
|
if result.is_ok() {
|
|
let next = Control {
|
|
log_used: 0,
|
|
generation: ctl.generation + 1,
|
|
active_slot: 1 - ctl.active_slot,
|
|
..*ctl
|
|
};
|
|
let mut buf = [0u8; CTL_COPY_LEN];
|
|
next.encode(&mut buf);
|
|
result = write_and_sync(file, older_copy(ctl.generation) as u64, &buf);
|
|
}
|
|
|
|
// SAFETY: `file` came from filp_open and has not been closed.
|
|
unsafe { bindings::filp_close(file, core::ptr::null_mut()) };
|
|
result.map(|_| new_image.len() as u64)
|
|
}
|
|
|
|
/// Everything a syscall needs to reach the store: read the device, resolve its layout, and
|
|
/// hand back what was asked for.
|
|
///
|
|
/// These are the `cube(2)` entry points as the Rust side exposes them. The C shim
|
|
/// (cube_syscall.c) owns the user copies and the argument validation; nothing here sees a
|
|
/// userspace pointer.
|
|
fn device_and_layout() -> Result<(KVVec<u8>, Layout), Error> {
|
|
let mut device = KVVec::<u8>::new();
|
|
read_image(&mut device)?;
|
|
let layout = resolve_layout(&device).map_err(|what| {
|
|
pr_err!("cubelinux: cannot resolve the store layout: {}\n", what);
|
|
EINVAL
|
|
})?;
|
|
Ok((device, layout))
|
|
}
|
|
|
|
/// The coordinate's key, and the space, as the store addresses them.
|
|
unsafe fn coord_key(space: *const u8, x: u64, y: u64, z: u64) -> ([u8; 32], [u8; 24]) {
|
|
let mut sp = [0u8; 32];
|
|
// SAFETY: the caller (the syscall shim) passes a pointer to 32 bytes it has already
|
|
// copied from userspace into kernel memory.
|
|
unsafe { core::ptr::copy_nonoverlapping(space, sp.as_mut_ptr(), 32) };
|
|
(sp, morton_encode(x, y, z))
|
|
}
|
|
|
|
/// `CUBE_OP_PUT`: store bytes at a coordinate.
|
|
///
|
|
/// # Safety
|
|
/// `space` must point to 32 readable bytes; `value` to `len` readable bytes when `len` is
|
|
/// non-zero.
|
|
#[unsafe(no_mangle)]
|
|
pub unsafe extern "C" fn cubelinux_kernel_put(
|
|
space: *const u8,
|
|
x: u64,
|
|
y: u64,
|
|
z: u64,
|
|
value: *const u8,
|
|
len: usize,
|
|
) -> i32 {
|
|
let (sp, key) = unsafe { coord_key(space, x, y, z) };
|
|
let mut bytes = KVVec::<u8>::new();
|
|
if len > 0 {
|
|
if let Err(_) = bytes.extend_from_slice(
|
|
// SAFETY: the shim guarantees `value` holds `len` bytes.
|
|
unsafe { core::slice::from_raw_parts(value, len) },
|
|
GFP_KERNEL,
|
|
) {
|
|
return -12; // -ENOMEM
|
|
}
|
|
}
|
|
let mutation = Mutation {
|
|
space: sp,
|
|
key,
|
|
value: bytes,
|
|
};
|
|
let (device, layout) = match device_and_layout() {
|
|
Ok(pair) => pair,
|
|
Err(e) => return -(e.to_errno() as i32),
|
|
};
|
|
match append(&device, &layout, &mutation, 1) {
|
|
Ok(_) => 0,
|
|
Err(e) => -(e.to_errno() as i32),
|
|
}
|
|
}
|
|
|
|
// ── Reading the store where it lies ─────────────────────────────────────────────────────
|
|
//
|
|
// A read used to start by reading the whole device into kernel memory, parsing every record into
|
|
// a merged pool, and heapsorting the lot — then throwing all of it away. That is why a `get` of
|
|
// one record cost as much as listing the store, and why the coordinate bought nothing
|
|
// mechanically: knowing where a record is did not make reaching it any cheaper.
|
|
//
|
|
// The records are already the index. A checkpoint writes them in `(space, key)` order, which is
|
|
// the order the coordinate itself computes, and the log that can override them is small. So the
|
|
// kernel walks the records where they lie — zero-copy, no pool, no sort — and consults the log as
|
|
// an overlay. What an operation costs is proportional to the records it actually touches.
|
|
|
|
/// How much is read at a time when a length is not known ahead of it.
|
|
const VIEW_CHUNK: usize = 256 * 1024;
|
|
|
|
/// Read exactly `len` bytes at `off`. A short read is an error, not a smaller view: the store says
|
|
/// how long its image is, and quietly reading less would turn a truncated store into a shorter one
|
|
/// that looks complete.
|
|
fn read_exact_at(
|
|
file: *mut bindings::file,
|
|
off: u64,
|
|
len: usize,
|
|
out: &mut KVVec<u8>,
|
|
scratch: &mut KVVec<u8>,
|
|
) -> Result<()> {
|
|
out.clear();
|
|
out.reserve(len, GFP_KERNEL)?;
|
|
let mut pos: bindings::loff_t = off as bindings::loff_t;
|
|
let mut left = len;
|
|
while left > 0 {
|
|
let want = core::cmp::min(left, scratch.len());
|
|
// SAFETY: `file` is a live `struct file *`; `scratch` is a kernel buffer we own; `pos` is
|
|
// a valid loff_t. kernel_read copies at most `want` bytes and retains neither pointer.
|
|
let n = unsafe {
|
|
bindings::kernel_read(file, scratch.as_mut_ptr().cast::<core::ffi::c_void>(), want, &mut pos)
|
|
};
|
|
if n < 0 {
|
|
return Err(Error::from_errno(n as i32));
|
|
}
|
|
if n == 0 {
|
|
return Err(EINVAL);
|
|
}
|
|
let n = n as usize;
|
|
out.extend_from_slice(&scratch.as_slice()[..n], GFP_KERNEL)?;
|
|
left -= n;
|
|
}
|
|
Ok(())
|
|
}
|
|
|
|
/// Read from `off` to the end, or to `cap` bytes, whichever comes first.
|
|
///
|
|
/// For a bare image, whose extent nothing states, and for a log whose length is implied.
|
|
fn read_to_end_at(
|
|
file: *mut bindings::file,
|
|
off: u64,
|
|
out: &mut KVVec<u8>,
|
|
scratch: &mut KVVec<u8>,
|
|
cap: usize,
|
|
) -> Result<()> {
|
|
out.clear();
|
|
let mut pos: bindings::loff_t = off as bindings::loff_t;
|
|
while out.len() < cap {
|
|
let want = core::cmp::min(scratch.len(), cap - out.len());
|
|
// SAFETY: as in `read_exact_at`.
|
|
let n = unsafe {
|
|
bindings::kernel_read(file, scratch.as_mut_ptr().cast::<core::ffi::c_void>(), want, &mut pos)
|
|
};
|
|
if n < 0 {
|
|
return Err(Error::from_errno(n as i32));
|
|
}
|
|
if n == 0 {
|
|
break;
|
|
}
|
|
let n = n as usize;
|
|
out.extend_from_slice(&scratch.as_slice()[..n], GFP_KERNEL)?;
|
|
}
|
|
Ok(())
|
|
}
|
|
|
|
/// The bytes one operation needs: the image, and the log beside it.
|
|
///
|
|
/// The device is four images wide plus its log; reading the device to read the image copies
|
|
/// several times what is used, on every call.
|
|
struct View {
|
|
image: KVVec<u8>,
|
|
log: KVVec<u8>,
|
|
header: Header,
|
|
/// The control block's generation, which a checkpoint changes — the key to what is cached
|
|
/// about this image. Zero for a bare image, which has no control block and does not change
|
|
/// underneath a reader.
|
|
generation: u64,
|
|
}
|
|
|
|
kernel::sync::global_lock! {
|
|
/// Where each space's records begin.
|
|
///
|
|
/// A walk has to start at its space. Records are variable-length, so "where does space S begin"
|
|
/// cannot be computed from the key — it can only be found by reading the image once. Finding it
|
|
/// once per checkpoint is what makes a listing cost what it returns, and the table is one entry
|
|
/// per space (tens of bytes), not one per record. It goes stale on exactly one event: a
|
|
/// checkpoint rewriting the image, which changes the control block's generation.
|
|
///
|
|
/// SAFETY: Initialized in the module initializer before first use.
|
|
unsafe(uninit) static SPACE_STARTS: Mutex<Option<SpaceStarts>> = None;
|
|
}
|
|
|
|
struct SpaceStarts {
|
|
generation: u64,
|
|
image_len: usize,
|
|
/// Each space's first record, in the image's order.
|
|
starts: KVVec<([u8; SPACE_ID_LEN], usize)>,
|
|
}
|
|
|
|
/// Where the live image's spaces begin, from the cache when it is current and by reading the image
|
|
/// when it is not.
|
|
fn space_starts(view: &View) -> Result<KVVec<([u8; SPACE_ID_LEN], usize)>> {
|
|
let mut guard = SPACE_STARTS.lock();
|
|
if let Some(cached) = guard.as_ref() {
|
|
if cached.generation == view.generation && cached.image_len == view.image().len() {
|
|
let mut out = KVVec::new();
|
|
out.extend_from_slice(cached.starts.as_slice(), GFP_KERNEL)?;
|
|
return Ok(out);
|
|
}
|
|
}
|
|
|
|
let mut starts = KVVec::new();
|
|
let mut records = Records::new(view.image(), &view.header);
|
|
let mut previous: Option<[u8; SPACE_ID_LEN]> = None;
|
|
loop {
|
|
let at = records.off;
|
|
match records.next() {
|
|
Some((space, _, _)) => {
|
|
if previous != Some(*space) {
|
|
starts.push((*space, at), GFP_KERNEL)?;
|
|
previous = Some(*space);
|
|
}
|
|
}
|
|
None => break,
|
|
}
|
|
}
|
|
|
|
let mut out = KVVec::new();
|
|
out.extend_from_slice(starts.as_slice(), GFP_KERNEL)?;
|
|
*guard = Some(SpaceStarts {
|
|
generation: view.generation,
|
|
image_len: view.image().len(),
|
|
starts,
|
|
});
|
|
Ok(out)
|
|
}
|
|
|
|
impl View {
|
|
fn image(&self) -> &[u8] {
|
|
self.image.as_slice()
|
|
}
|
|
|
|
fn log(&self) -> &[u8] {
|
|
self.log.as_slice()
|
|
}
|
|
|
|
/// Where one space's records begin, if the image holds it.
|
|
fn space_start(&self, space: &[u8; SPACE_ID_LEN]) -> Result<Option<usize>> {
|
|
let starts = space_starts(self)?;
|
|
for (candidate, at) in starts.as_slice() {
|
|
if candidate == space {
|
|
return Ok(Some(*at));
|
|
}
|
|
}
|
|
Ok(None)
|
|
}
|
|
}
|
|
|
|
/// Read the image and its log window — not the whole device.
|
|
fn read_view() -> Result<View> {
|
|
// O_RDONLY is 0 in Linux; filp_open takes the raw flags word.
|
|
// SAFETY: `store_device()` is a NUL-terminated C string the module parameter filled at boot,
|
|
// and filp_open returns a valid `struct file *` or an error pointer, checked here.
|
|
let file = unsafe { bindings::filp_open(store_device(), 0, 0) };
|
|
let file = kernel::error::from_err_ptr(file)?;
|
|
if file.is_null() {
|
|
return Err(EINVAL);
|
|
}
|
|
|
|
let view = read_view_from(file);
|
|
|
|
// SAFETY: `file` came from filp_open and has not been closed; the owner argument is only
|
|
// meaningful for locks that nobody holds here.
|
|
unsafe { bindings::filp_close(file, core::ptr::null_mut()) };
|
|
|
|
let view = view?;
|
|
if view.image.len() < HEADER_LEN_V1 {
|
|
return Err(EINVAL);
|
|
}
|
|
Ok(view)
|
|
}
|
|
|
|
fn read_view_from(file: *mut bindings::file) -> Result<View> {
|
|
let mut scratch = KVVec::<u8>::with_capacity(VIEW_CHUNK, GFP_KERNEL)?;
|
|
scratch.resize(VIEW_CHUNK, 0, GFP_KERNEL)?;
|
|
|
|
// The control block, or the image's own header when the device is a bare image.
|
|
let mut head = KVVec::<u8>::with_capacity(CTL_COPY_B + CTL_COPY_LEN, GFP_KERNEL)?;
|
|
read_exact_at(file, 0, CTL_COPY_B + CTL_COPY_LEN, &mut head, &mut scratch)?;
|
|
let is_device = &head.as_slice()[0..4] == CTL_MAGIC.as_slice();
|
|
|
|
let (image_off, log_off, log_used, generation) = if is_device {
|
|
let control = read_control(head.as_slice()).ok_or(EINVAL)?;
|
|
(
|
|
control.image_off() as usize,
|
|
control.log_off() as usize,
|
|
control.log_used as usize,
|
|
control.generation,
|
|
)
|
|
} else {
|
|
(0, 0, 0, 0)
|
|
};
|
|
|
|
// What the image says about itself. A device keeps its image at an offset, so its header is
|
|
// read from there; a bare image starts with it.
|
|
let mut header = if is_device {
|
|
let mut raw = KVVec::<u8>::with_capacity(HEADER_LEN_V2, GFP_KERNEL)?;
|
|
match read_exact_at(file, image_off as u64, HEADER_LEN_V2, &mut raw, &mut scratch) {
|
|
Ok(()) => parse_header(raw.as_slice()),
|
|
// A v1 image is shorter than a v2 header, and saying "not a store" about it would be
|
|
// wrong: read the six bytes it does have.
|
|
Err(_) => {
|
|
read_exact_at(file, image_off as u64, HEADER_LEN_V1, &mut raw, &mut scratch)?;
|
|
parse_header(raw.as_slice())
|
|
}
|
|
}
|
|
} else {
|
|
parse_header(head.as_slice())
|
|
}
|
|
.map_err(|what| {
|
|
pr_err!("cubelinux: {}\n", what);
|
|
EINVAL
|
|
})?;
|
|
|
|
let mut image = KVVec::<u8>::new();
|
|
let mut log = KVVec::<u8>::new();
|
|
match (is_device, header.image_bytes) {
|
|
(true, Some(bytes)) => {
|
|
let bytes = bytes as usize;
|
|
if bytes > MAX_BYTES {
|
|
return Err(EINVAL);
|
|
}
|
|
read_exact_at(file, image_off as u64, bytes, &mut image, &mut scratch)?;
|
|
}
|
|
// A v1 image states no extent, so it is read to the end — bounded, because a device that
|
|
// never ends must not become a read that never ends.
|
|
(true, None) => {
|
|
read_to_end_at(file, image_off as u64, &mut image, &mut scratch, MAX_BYTES)?;
|
|
}
|
|
(false, _) => {
|
|
read_to_end_at(file, 0, &mut image, &mut scratch, MAX_BYTES)?;
|
|
if header.image_bytes.is_none() {
|
|
header.image_bytes = Some(image.len() as u64);
|
|
}
|
|
}
|
|
}
|
|
|
|
if is_device {
|
|
if log_used > 0 {
|
|
let len = core::cmp::min(WAL_HEADER_LEN + log_used, MAX_BYTES);
|
|
read_exact_at(file, log_off as u64, len, &mut log, &mut scratch)?;
|
|
}
|
|
} else {
|
|
// A bare image's log starts after the image, aligned, and runs to the end.
|
|
let at = log_offset(image.len() as u64);
|
|
read_to_end_at(file, at as u64, &mut log, &mut scratch, MAX_BYTES)?;
|
|
}
|
|
|
|
Ok(View {
|
|
image,
|
|
log,
|
|
header,
|
|
generation,
|
|
})
|
|
}
|
|
|
|
/// Walk a store image's records, in the order the image holds them.
|
|
///
|
|
/// Zero-copy: the value a caller gets is a slice of the image already in memory, so listing a
|
|
/// store does not copy its values anywhere.
|
|
struct Records<'a> {
|
|
image: &'a [u8],
|
|
off: usize,
|
|
end: usize,
|
|
left: Option<u64>,
|
|
}
|
|
|
|
impl<'a> Records<'a> {
|
|
/// Begin at `off`, for a caller that has been told where a space starts.
|
|
///
|
|
/// The record count is dropped: it is a bound on the records ahead of the *first* record, and
|
|
/// starting anywhere else makes it a bound that no longer means anything. The image's extent
|
|
/// is the bound that still holds.
|
|
fn at(image: &'a [u8], header: &Header, off: usize) -> Self {
|
|
let mut records = Records::new(image, header);
|
|
if off > records.off && off <= records.end {
|
|
records.off = off;
|
|
records.left = None;
|
|
}
|
|
records
|
|
}
|
|
|
|
fn new(image: &'a [u8], header: &Header) -> Self {
|
|
let header_len = if header.version == VERSION_V1 {
|
|
HEADER_LEN_V1
|
|
} else {
|
|
HEADER_LEN_V2
|
|
};
|
|
let extent = match header.image_bytes {
|
|
Some(n) => core::cmp::min(n as usize, image.len()),
|
|
None => image.len(),
|
|
};
|
|
Records {
|
|
image,
|
|
off: core::cmp::min(header_len, extent),
|
|
end: extent,
|
|
left: header.record_count,
|
|
}
|
|
}
|
|
|
|
fn next(&mut self) -> Option<(&'a [u8; 32], &'a [u8; 24], &'a [u8])> {
|
|
if self.left == Some(0) || self.off + RECORD_FIXED > self.end {
|
|
return None;
|
|
}
|
|
let frame = &self.image[self.off..self.off + RECORD_FIXED];
|
|
// A v1 image states no count, so padding ends the walk — but only if every remaining byte
|
|
// is zero, because a record at the origin with an empty value is 64 zero bytes and is a
|
|
// record.
|
|
if self.left.is_none()
|
|
&& frame.iter().all(|b| *b == 0)
|
|
&& self.image[self.off..].iter().all(|b| *b == 0)
|
|
{
|
|
return None;
|
|
}
|
|
let mut word = [0u8; 8];
|
|
word.copy_from_slice(&frame[SPACE_ID_LEN + RAW_KEY_LEN..RECORD_FIXED]);
|
|
let value_len = u64::from_le_bytes(word) as usize;
|
|
let value_at = self.off + RECORD_FIXED;
|
|
if value_at + value_len > self.end {
|
|
return None;
|
|
}
|
|
let space: &[u8; SPACE_ID_LEN] = frame[..SPACE_ID_LEN].try_into().ok()?;
|
|
let key: &[u8; RAW_KEY_LEN] =
|
|
frame[SPACE_ID_LEN..SPACE_ID_LEN + RAW_KEY_LEN].try_into().ok()?;
|
|
let value = &self.image[value_at..value_at + value_len];
|
|
self.off = value_at + value_len;
|
|
if let Some(n) = self.left.as_mut() {
|
|
*n -= 1;
|
|
}
|
|
Some((space, key, value))
|
|
}
|
|
}
|
|
|
|
/// What the log says about one coordinate: written, or removed.
|
|
enum Effect<'a> {
|
|
Write(&'a [u8]),
|
|
Delete,
|
|
}
|
|
|
|
/// The log's newest word on `(space, key)`, if it says anything about it.
|
|
///
|
|
/// The log is the only thing that can override the image — it holds what has been written since
|
|
/// the last checkpoint — so this is what decides a read.
|
|
fn log_effect<'a>(log: &'a [u8], space: &[u8; SPACE_ID_LEN], key: &[u8; RAW_KEY_LEN]) -> Option<Effect<'a>> {
|
|
let mut entries = log_entries(log).ok()?;
|
|
let mut found = None;
|
|
while let Some((entry_space, entry_key, op, value_at, value_len)) = entries.next() {
|
|
if entry_space == space && entry_key == key {
|
|
found = Some(if op == 2 {
|
|
Effect::Delete
|
|
} else {
|
|
Effect::Write(&log[value_at..value_at + value_len])
|
|
});
|
|
}
|
|
}
|
|
found
|
|
}
|
|
|
|
/// One log entry, as the walk sees it: `(space, key, op, value)`.
|
|
struct LogEntries<'a> {
|
|
log: &'a [u8],
|
|
off: usize,
|
|
}
|
|
|
|
impl<'a> LogEntries<'a> {
|
|
/// The next entry: `(space, key, op, where its value starts, its value's length)`.
|
|
///
|
|
/// The value's *position* rather than a slice, because a caller collecting edits keeps them in
|
|
/// a fixed-size vector and a borrow of the log would tie that vector's type to the log's life.
|
|
fn next(&mut self) -> Option<(&'a [u8], &'a [u8], u8, usize, usize)> {
|
|
if self.off + ENTRY_FIXED > self.log.len() {
|
|
return None;
|
|
}
|
|
let start = self.off;
|
|
let op = self.log[start];
|
|
if op != 1 && op != 2 {
|
|
return None;
|
|
}
|
|
let mut word = [0u8; 4];
|
|
word.copy_from_slice(&self.log[start + 1..start + 5]);
|
|
let crc = u32::from_le_bytes(word);
|
|
let space = &self.log[start + 5..start + 37];
|
|
let key = &self.log[start + 37..start + 61];
|
|
word.copy_from_slice(&self.log[start + 61..start + 65]);
|
|
let len = u32::from_le_bytes(word) as usize;
|
|
let frame_end = start + ENTRY_FIXED + len;
|
|
if frame_end > self.log.len() {
|
|
return None;
|
|
}
|
|
// The checksum covers space, key, length and value, so a torn tail is stopped at rather
|
|
// than applied — the same rule the fold uses.
|
|
if crc32(&self.log[start + 5..frame_end]) != crc {
|
|
return None;
|
|
}
|
|
self.off = frame_end;
|
|
Some((space, key, op, start + ENTRY_FIXED, len))
|
|
}
|
|
}
|
|
|
|
/// The log's entries, when there is a valid log to read.
|
|
fn log_entries(log: &[u8]) -> Result<LogEntries<'_>, &'static str> {
|
|
if log.len() < WAL_HEADER_LEN || &log[0..4] != WAL_MAGIC {
|
|
return Err("no-log");
|
|
}
|
|
if log[4] != WAL_VERSION {
|
|
return Err("unsupported-log-version");
|
|
}
|
|
Ok(LogEntries {
|
|
log,
|
|
off: WAL_HEADER_LEN,
|
|
})
|
|
}
|
|
|
|
/// One space's override of the image, on its way to being merged with it.
|
|
///
|
|
/// The key and the value borrow the log rather than copying it, so a walker can hand out slices of
|
|
/// bytes that are already in memory — a listing never copies a value.
|
|
#[derive(Clone, Copy)]
|
|
struct LogEdit<'a> {
|
|
key: &'a [u8; RAW_KEY_LEN],
|
|
/// Position in the log, so the later of two edits to one key is the one that counts.
|
|
seq: u32,
|
|
value: &'a [u8],
|
|
deleted: bool,
|
|
}
|
|
|
|
/// Every edit the log holds for one space, in key order, later edits last.
|
|
///
|
|
/// The image and the log are both in key order, so one merge pass over them is a listing — and a
|
|
/// space is what a walk names, so only its own edits are collected.
|
|
fn log_edits<'a>(
|
|
log: &'a [u8],
|
|
space: &[u8; SPACE_ID_LEN],
|
|
out: &mut KVVec<LogEdit<'a>>,
|
|
) -> Result<()> {
|
|
out.clear();
|
|
let mut entries = match log_entries(log) {
|
|
Ok(e) => e,
|
|
Err(_) => return Ok(()),
|
|
};
|
|
let mut seq: u32 = 0;
|
|
while let Some((edit_space, key, op, value_at, value_len)) = entries.next() {
|
|
if edit_space == space {
|
|
let key: &[u8; RAW_KEY_LEN] = match key.try_into() {
|
|
Ok(k) => k,
|
|
Err(_) => break,
|
|
};
|
|
out.push(
|
|
LogEdit {
|
|
key,
|
|
seq,
|
|
value: &log[value_at..value_at + value_len],
|
|
deleted: op == 2,
|
|
},
|
|
GFP_KERNEL,
|
|
)?;
|
|
}
|
|
seq += 1;
|
|
}
|
|
sort_edits(out.as_mut_slice());
|
|
Ok(())
|
|
}
|
|
|
|
fn sort_edits(edits: &mut [LogEdit<'_>]) {
|
|
let len = edits.len();
|
|
if len < 2 {
|
|
return;
|
|
}
|
|
let mut start = len / 2;
|
|
while start > 0 {
|
|
start -= 1;
|
|
sift_down_edits(edits, start, len);
|
|
}
|
|
let mut end = len;
|
|
while end > 1 {
|
|
end -= 1;
|
|
edits.swap(0, end);
|
|
sift_down_edits(edits, 0, end);
|
|
}
|
|
}
|
|
|
|
/// Heapsort, for the same reason the entry sort is one: a constant stack frame, so the size of a
|
|
/// log cannot decide whether reading it is safe. `sort_unstable` is used nowhere in this module.
|
|
fn sift_down_edits(edits: &mut [LogEdit<'_>], mut root: usize, end: usize) {
|
|
loop {
|
|
let mut child = root * 2 + 1;
|
|
if child >= end {
|
|
return;
|
|
}
|
|
if child + 1 < end
|
|
&& (edits[child].key, edits[child].seq) < (edits[child + 1].key, edits[child + 1].seq)
|
|
{
|
|
child += 1;
|
|
}
|
|
if (edits[root].key, edits[root].seq) >= (edits[child].key, edits[child].seq) {
|
|
return;
|
|
}
|
|
edits.swap(root, child);
|
|
root = child;
|
|
}
|
|
}
|
|
|
|
/// The live records of one space: the image's records for it, with the log's edits applied, in key
|
|
/// order.
|
|
///
|
|
/// This is the whole of what a listing is — two sorted sequences and one pass — and it is why a
|
|
/// walk costs what it returns rather than what the store holds.
|
|
struct SpaceWalker<'a> {
|
|
records: Records<'a>,
|
|
edits: KVVec<LogEdit<'a>>,
|
|
edit_at: usize,
|
|
image_next: Option<(&'a [u8; RAW_KEY_LEN], &'a [u8])>,
|
|
wanted: [u8; SPACE_ID_LEN],
|
|
}
|
|
|
|
impl<'a> SpaceWalker<'a> {
|
|
fn new(view: &'a View, wanted: &[u8; SPACE_ID_LEN]) -> Result<Self> {
|
|
let mut edits = KVVec::<LogEdit<'a>>::new();
|
|
log_edits(view.log(), wanted, &mut edits)?;
|
|
// Start where the space starts: a listing of a small space in a large store must not read
|
|
// the records that are not in it.
|
|
let mut records = match view.space_start(wanted)? {
|
|
Some(at) => Records::at(view.image(), &view.header, at),
|
|
None => {
|
|
let mut empty = Records::new(view.image(), &view.header);
|
|
empty.off = empty.end; // nothing in the image: the log's edits are the space
|
|
empty
|
|
}
|
|
};
|
|
let image_next = next_in_space(&mut records, wanted);
|
|
Ok(SpaceWalker {
|
|
records,
|
|
edits,
|
|
edit_at: 0,
|
|
image_next,
|
|
wanted: *wanted,
|
|
})
|
|
}
|
|
|
|
/// The next live record, or nothing when the space is walked out.
|
|
fn next(&mut self) -> Option<(&'a [u8; RAW_KEY_LEN], &'a [u8])> {
|
|
loop {
|
|
// The entries of one key arrive in log order, so the last of a key's group is the
|
|
// newest word on it and the earlier ones are history.
|
|
while self.edit_at + 1 < self.edits.len()
|
|
&& self.edits[self.edit_at + 1].key == self.edits[self.edit_at].key
|
|
{
|
|
self.edit_at += 1;
|
|
}
|
|
let edit = self.edits.as_slice().get(self.edit_at).copied();
|
|
match (edit, self.image_next) {
|
|
// The log's word on a coordinate the image holds: it wins.
|
|
(Some(e), Some((key, _))) if e.key == key => {
|
|
self.edit_at += 1;
|
|
self.image_next = next_in_space(&mut self.records, &self.wanted);
|
|
if !e.deleted {
|
|
return Some((e.key, e.value));
|
|
}
|
|
}
|
|
// A record only the log holds, in its key's place.
|
|
(Some(e), Some((key, value))) if e.key < key => {
|
|
self.edit_at += 1;
|
|
self.image_next = Some((key, value));
|
|
if !e.deleted {
|
|
return Some((e.key, e.value));
|
|
}
|
|
}
|
|
// The image's own record, which the log says nothing about.
|
|
(_, Some((key, value))) => {
|
|
self.image_next = next_in_space(&mut self.records, &self.wanted);
|
|
return Some((key, value));
|
|
}
|
|
// The image is walked out; the rest of the log still has records to report.
|
|
(Some(e), None) => {
|
|
self.edit_at += 1;
|
|
if !e.deleted {
|
|
return Some((e.key, e.value));
|
|
}
|
|
}
|
|
(None, None) => return None,
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
/// Does this space hold at least one live record?
|
|
///
|
|
/// A space whose every record the log has removed is not a space to report, and a walk that stops
|
|
/// at the first record it finds is the cheapest way to know: it costs one record, not a listing.
|
|
fn space_has_records(view: &View, space: &[u8; SPACE_ID_LEN]) -> bool {
|
|
match SpaceWalker::new(view, space) {
|
|
Ok(mut walker) => walker.next().is_some(),
|
|
Err(_) => false,
|
|
}
|
|
}
|
|
|
|
/// The distinct spaces the log writes into, ascending. Small by construction: the log is what has
|
|
/// been written since the last checkpoint.
|
|
fn log_spaces(log: &[u8], out: &mut KVVec<[u8; SPACE_ID_LEN]>) -> Result<()> {
|
|
out.clear();
|
|
let mut entries = match log_entries(log) {
|
|
Ok(e) => e,
|
|
Err(_) => return Ok(()),
|
|
};
|
|
while let Some((space, _, _, _, _)) = entries.next() {
|
|
let mut s = [0u8; SPACE_ID_LEN];
|
|
s.copy_from_slice(&space[..SPACE_ID_LEN]);
|
|
if !out.as_slice().contains(&s) {
|
|
out.push(s, GFP_KERNEL)?;
|
|
}
|
|
}
|
|
sort_spaces(out.as_mut_slice());
|
|
Ok(())
|
|
}
|
|
|
|
/// Heapsort for a store's spaces. The log holds few, but `sort_unstable` is not used anywhere in
|
|
/// this module: it allocates about 3.5 KiB of kernel stack per level, which is how reading a
|
|
/// store once took the machine down.
|
|
fn sort_spaces(spaces: &mut [[u8; SPACE_ID_LEN]]) {
|
|
let len = spaces.len();
|
|
if len < 2 {
|
|
return;
|
|
}
|
|
let mut start = len / 2;
|
|
while start > 0 {
|
|
start -= 1;
|
|
sift_down_spaces(spaces, start, len);
|
|
}
|
|
let mut end = len;
|
|
while end > 1 {
|
|
end -= 1;
|
|
spaces.swap(0, end);
|
|
sift_down_spaces(spaces, 0, end);
|
|
}
|
|
}
|
|
|
|
fn sift_down_spaces(spaces: &mut [[u8; SPACE_ID_LEN]], mut root: usize, end: usize) {
|
|
loop {
|
|
let mut child = root * 2 + 1;
|
|
if child >= end {
|
|
return;
|
|
}
|
|
if child + 1 < end && spaces[child] < spaces[child + 1] {
|
|
child += 1;
|
|
}
|
|
if spaces[root] >= spaces[child] {
|
|
return;
|
|
}
|
|
spaces.swap(root, child);
|
|
root = child;
|
|
}
|
|
}
|
|
|
|
/// Hand the caller a value: its length, or -ENOENT when there is nothing there.
|
|
///
|
|
/// When the value does not fit in `len` the bytes are *not* copied and the length is returned
|
|
/// anyway, so the caller can size its buffer and ask again — a short read that silently truncated
|
|
/// would be worse than an error.
|
|
fn copy_out(value: &[u8], buf: *mut u8, len: usize) -> isize {
|
|
if value.len() > len {
|
|
return value.len() as isize;
|
|
}
|
|
if !value.is_empty() {
|
|
// SAFETY: the shim guarantees `buf` holds `len` bytes and `len >= value.len()`.
|
|
unsafe {
|
|
core::ptr::copy_nonoverlapping(value.as_ptr(), buf, value.len());
|
|
}
|
|
}
|
|
value.len() as isize
|
|
}
|
|
|
|
/// The next record of one space in an image walk, skipping the records of every other space.
|
|
fn next_in_space<'a>(
|
|
records: &mut Records<'a>,
|
|
wanted: &[u8; SPACE_ID_LEN],
|
|
) -> Option<(&'a [u8; RAW_KEY_LEN], &'a [u8])> {
|
|
while let Some((space, key, value)) = records.next() {
|
|
if space == wanted {
|
|
return Some((key, value));
|
|
}
|
|
}
|
|
None
|
|
}
|
|
|
|
/// Packs a walk's records into the caller's buffer, skipping what the cursor has covered.
|
|
struct Batch<'a> {
|
|
out: &'a mut [u8],
|
|
written: usize,
|
|
returned: u64,
|
|
seen: u64,
|
|
cursor: u64,
|
|
/// Set when a record did not fit: the size it needs, for the `-ERANGE` answer.
|
|
too_big: usize,
|
|
}
|
|
|
|
impl Batch<'_> {
|
|
/// Offer one record. `false` means nothing more will fit in this batch.
|
|
fn offer(&mut self, key: &[u8], value: &[u8]) -> bool {
|
|
self.seen += 1;
|
|
if self.seen <= self.cursor {
|
|
return true;
|
|
}
|
|
match pack_record(key, value, self.out, self.written) {
|
|
Some(n) => {
|
|
self.written += n;
|
|
self.returned += 1;
|
|
true
|
|
}
|
|
None => {
|
|
self.too_big = RAW_KEY_LEN + 4 + value.len();
|
|
false
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
/// `CUBE_OP_GET`: read a coordinate.
|
|
///
|
|
/// Returns the record's length, or a negative errno. When the value does not fit in `len` the
|
|
/// bytes are *not* copied and the length is returned anyway, so the caller can size its buffer
|
|
/// and ask again — a short read that silently truncated would be worse than an error.
|
|
///
|
|
/// The log has the last word, because it holds what has been written since the last checkpoint;
|
|
/// only if it says nothing about the coordinate does the image answer. Nothing is copied out of
|
|
/// either: the value the caller gets is a slice of bytes already in memory.
|
|
///
|
|
/// # Safety
|
|
/// `space` must point to 32 readable bytes; `buf` to `len` writable bytes.
|
|
#[unsafe(no_mangle)]
|
|
pub unsafe extern "C" fn cubelinux_kernel_get(
|
|
space: *const u8,
|
|
x: u64,
|
|
y: u64,
|
|
z: u64,
|
|
buf: *mut u8,
|
|
len: usize,
|
|
) -> isize {
|
|
let (sp, key) = unsafe { coord_key(space, x, y, z) };
|
|
let view = match read_view() {
|
|
Ok(v) => v,
|
|
Err(e) => return -(e.to_errno() as isize),
|
|
};
|
|
|
|
match log_effect(view.log(), &sp, &key) {
|
|
Some(Effect::Delete) => return -2, // -ENOENT
|
|
Some(Effect::Write(value)) => return copy_out(value, buf, len),
|
|
None => {}
|
|
}
|
|
|
|
// Walk the records where they lie, starting where this space's records start: the image is in
|
|
// `(space, key)` order, so the record for a coordinate is passed exactly once, and a coordinate
|
|
// in a late space does not pay for every record before it.
|
|
let at = match view.space_start(&sp) {
|
|
Ok(Some(at)) => at,
|
|
Ok(None) => return -2, // -ENOENT: the space is not in the image, and the log already spoke
|
|
Err(_) => return -12, // -ENOMEM
|
|
};
|
|
let mut records = Records::at(view.image(), &view.header, at);
|
|
while let Some((record_space, record_key, value)) = records.next() {
|
|
if record_space == &sp && record_key == &key {
|
|
return copy_out(value, buf, len);
|
|
}
|
|
}
|
|
-2 // -ENOENT
|
|
}
|
|
|
|
/// Pack one record the way the walk's uapi names it: `key(24) | value_len(u32, LE) | value`.
|
|
///
|
|
/// The space is not in the frame because the caller named it, and the order is the store's own
|
|
/// (space, then key), so a listing taken through the kernel and one taken in userspace are
|
|
/// byte-for-byte comparable — which is how this is tested.
|
|
fn pack_record(key: &[u8], value: &[u8], out: &mut [u8], at: usize) -> Option<usize> {
|
|
let need = RAW_KEY_LEN + 4 + value.len();
|
|
if at + need > out.len() {
|
|
return None;
|
|
}
|
|
out[at..at + RAW_KEY_LEN].copy_from_slice(&key[..RAW_KEY_LEN]);
|
|
out[at + RAW_KEY_LEN..at + RAW_KEY_LEN + 4]
|
|
.copy_from_slice(&(value.len() as u32).to_le_bytes());
|
|
out[at + RAW_KEY_LEN + 4..at + need].copy_from_slice(value);
|
|
Some(need)
|
|
}
|
|
|
|
/// `CUBE_OP_ENUM`: walk the records of one space into the caller's buffer, `cursor` records in.
|
|
///
|
|
/// The cursor is a COUNT OF RECORDS ALREADY RETURNED, not a position in the image. It is opaque to
|
|
/// the caller (pass back what you were given) and it survives an append, because the contract is
|
|
/// "the records I had not yet seen" rather than a snapshot. A record can be seen twice if the
|
|
/// image is rewritten underneath a walk; a caller that needs a snapshot takes one.
|
|
///
|
|
/// The walk walks. It used to read the whole device, parse every record of every space into a
|
|
/// merged pool and heapsort all of it, then discard the lot — so asking for one page of one space
|
|
/// cost the entire store, and every batch of a listing paid it again. Now the image is read once,
|
|
/// its records are walked where they lie in the order a checkpoint gave them, and the log is
|
|
/// consulted as an overlay. The cost is the records the walk touches.
|
|
///
|
|
/// `out_len` receives the bytes written, or with -ERANGE the size the first record that did not fit
|
|
/// would need — exactly as a read reports the size it wants.
|
|
///
|
|
/// # Safety
|
|
/// `space` must point to 32 readable bytes; `buf` must hold `cap` writable bytes; `out_len` and
|
|
/// `out_cursor` must each point to a writable `u64`.
|
|
#[unsafe(no_mangle)]
|
|
pub unsafe extern "C" fn cubelinux_kernel_enum(
|
|
space: *const u8,
|
|
cursor: u64,
|
|
buf: *mut u8,
|
|
cap: usize,
|
|
out_len: *mut u64,
|
|
out_cursor: *mut u64,
|
|
) -> i32 {
|
|
let mut wanted = [0u8; SPACE_ID_LEN];
|
|
// SAFETY: the caller guarantees 32 readable bytes at `space`.
|
|
unsafe { core::ptr::copy_nonoverlapping(space, wanted.as_mut_ptr(), SPACE_ID_LEN) };
|
|
|
|
let view = match read_view() {
|
|
Ok(v) => v,
|
|
Err(e) => return -(e.to_errno() as i32),
|
|
};
|
|
|
|
// The live records of this space: the image's, with the log's edits applied as an overlay.
|
|
// A space is what a walk names, so only its own edits are collected — and the log is what a
|
|
// checkpoint has not yet folded, so there are few.
|
|
let mut walker = match SpaceWalker::new(&view, &wanted) {
|
|
Ok(w) => w,
|
|
Err(_) => return -12, // -ENOMEM
|
|
};
|
|
|
|
// SAFETY: the shim guarantees `cap` writable bytes at `buf`.
|
|
let out = unsafe { core::slice::from_raw_parts_mut(buf, cap) };
|
|
let mut batch = Batch {
|
|
out,
|
|
written: 0,
|
|
returned: 0,
|
|
seen: 0,
|
|
cursor,
|
|
too_big: 0,
|
|
};
|
|
while let Some((key, value)) = walker.next() {
|
|
if !batch.offer(key, value) {
|
|
break;
|
|
}
|
|
}
|
|
|
|
// SAFETY: both out-pointers are writable under this function's contract.
|
|
unsafe {
|
|
if batch.too_big > 0 && batch.written == 0 {
|
|
// Not one whole record fits. Say how much it needs, the way a read does.
|
|
*out_len = batch.too_big as u64;
|
|
*out_cursor = cursor;
|
|
return -34; // -ERANGE
|
|
}
|
|
*out_len = batch.written as u64;
|
|
*out_cursor = cursor + batch.returned;
|
|
}
|
|
0
|
|
}
|
|
|
|
/// `CUBE_OP_SPACES`: the `cursor`-th distinct space that holds a record, or -ENOENT at the end.
|
|
///
|
|
/// An index rather than a count of records: a caller that wants a space's records walks it with
|
|
/// `CUBE_OP_ENUM` once it has learned the name. Entries are already in (space, key) order, so the
|
|
/// distinct spaces come out sorted.
|
|
///
|
|
/// # Safety
|
|
/// `space_out` must point to 32 writable bytes.
|
|
#[unsafe(no_mangle)]
|
|
pub unsafe extern "C" fn cubelinux_kernel_spaces(cursor: u64, space_out: *mut u8) -> i32 {
|
|
let view = match read_view() {
|
|
Ok(v) => v,
|
|
Err(e) => return -(e.to_errno() as i32),
|
|
};
|
|
|
|
// Candidates, in ascending order: the spaces the image's records run through (a checkpoint
|
|
// writes them in key order, and a space is the first half of the key), and any space the log
|
|
// writes into that the image has never seen. One pass over the image collects the first kind;
|
|
// the second kind is whatever the log holds, which is small.
|
|
let mut log_only = KVVec::<[u8; SPACE_ID_LEN]>::new();
|
|
if log_spaces(view.log(), &mut log_only).is_err() {
|
|
return -12; // -ENOMEM
|
|
}
|
|
// A space is only worth reporting if it still holds a record: one whose records the log has all
|
|
// removed is not a space anybody can list.
|
|
let emit = |space: &[u8; SPACE_ID_LEN], index: &mut u64| -> bool {
|
|
if !space_has_records(&view, space) {
|
|
return false;
|
|
}
|
|
if *index == cursor {
|
|
// SAFETY: the caller guarantees 32 writable bytes at `space_out`.
|
|
unsafe {
|
|
core::ptr::copy_nonoverlapping(space.as_ptr(), space_out, SPACE_ID_LEN);
|
|
}
|
|
return true;
|
|
}
|
|
*index += 1;
|
|
false
|
|
};
|
|
|
|
// The image's own spaces are the table of where they start — no pass over the records, because
|
|
// that pass already happened when the table was built.
|
|
let starts = match space_starts(&view) {
|
|
Ok(s) => s,
|
|
Err(_) => return -12, // -ENOMEM
|
|
};
|
|
|
|
let mut index: u64 = 0;
|
|
let mut log_at = 0usize;
|
|
for (space, _) in starts.as_slice() {
|
|
let space: [u8; SPACE_ID_LEN] = *space;
|
|
while log_at < log_only.len() && log_only.as_slice()[log_at] < space {
|
|
let candidate = log_only.as_slice()[log_at];
|
|
log_at += 1;
|
|
if emit(&candidate, &mut index) {
|
|
return 0;
|
|
}
|
|
}
|
|
// A space the log also writes into is the same space, not a second one.
|
|
if log_at < log_only.len() && log_only.as_slice()[log_at] == space {
|
|
log_at += 1;
|
|
}
|
|
if emit(&space, &mut index) {
|
|
return 0;
|
|
}
|
|
}
|
|
while log_at < log_only.len() {
|
|
let candidate = log_only.as_slice()[log_at];
|
|
log_at += 1;
|
|
if emit(&candidate, &mut index) {
|
|
return 0;
|
|
}
|
|
}
|
|
-2 // -ENOENT: no such space; the walk is finished
|
|
}
|
|
|
|
/// `CUBE_OP_DEL`: remove the record at a coordinate.
|
|
///
|
|
/// A removal is a log entry, not an erasure: nothing in an append-only store is rewritten in
|
|
/// place, and the checkpoint is what finally drops it.
|
|
///
|
|
/// # Safety
|
|
/// `space` must point to 32 readable bytes.
|
|
#[unsafe(no_mangle)]
|
|
pub unsafe extern "C" fn cubelinux_kernel_del(space: *const u8, x: u64, y: u64, z: u64) -> i32 {
|
|
let (sp, key) = unsafe { coord_key(space, x, y, z) };
|
|
let mutation = Mutation {
|
|
space: sp,
|
|
key,
|
|
value: KVVec::new(),
|
|
};
|
|
let (device, layout) = match device_and_layout() {
|
|
Ok(pair) => pair,
|
|
Err(e) => return -(e.to_errno() as i32),
|
|
};
|
|
// A delete is an entry with op=2; `append` writes op=1, so build it here from the same
|
|
// framing, and let the log's own reader be the judge of it.
|
|
match append(&device, &layout, &mutation, 2) {
|
|
Ok(_) => 0,
|
|
Err(e) => -(e.to_errno() as i32),
|
|
}
|
|
}
|
|
|
|
/// `CUBE_OP_SYNC`: fold the log into the image.
|
|
#[unsafe(no_mangle)]
|
|
pub extern "C" fn cubelinux_kernel_sync() -> i32 {
|
|
let (device, layout) = match device_and_layout() {
|
|
Ok(pair) => pair,
|
|
Err(e) => return -(e.to_errno() as i32),
|
|
};
|
|
let ctl = match layout.control {
|
|
Some(c) => c,
|
|
None => return -22, // -EINVAL: a bare image has no spare slot to fold into
|
|
};
|
|
let live = &device[layout.image_off..];
|
|
let header = match parse_header(live) {
|
|
Ok(h) => h,
|
|
Err(_) => return -22,
|
|
};
|
|
let window = log_window(&device, &layout);
|
|
let extent = core::cmp::min(
|
|
header.image_bytes.unwrap_or(live.len() as u64) as usize,
|
|
live.len(),
|
|
);
|
|
let mut merged = match build_merged(&live[..extent], window, &header) {
|
|
Ok((m, _)) => m,
|
|
Err(_) => return -22,
|
|
};
|
|
match checkpoint(&ctl, &mut merged) {
|
|
Ok(_) => 0,
|
|
Err(e) => -(e.to_errno() as i32),
|
|
}
|
|
}
|
|
|
|
/// The module's registration; holds the misc device for as long as the module lives.
|
|
#[pin_data]
|
|
struct CubeStoreModule {
|
|
#[pin]
|
|
_miscdev: MiscDeviceRegistration<CubeStore>,
|
|
}
|
|
|
|
impl kernel::InPlaceModule for CubeStoreModule {
|
|
fn init(_module: &'static ThisModule) -> impl PinInit<Self, Error> {
|
|
pr_info!("cubelinux: store reader registered at /dev/cubelinux\n");
|
|
// SAFETY: called exactly once, in the module initializer, before anything can take it.
|
|
unsafe { SPACE_STARTS.init() };
|
|
try_pin_init!(Self {
|
|
_miscdev <- MiscDeviceRegistration::register(MiscDeviceOptions {
|
|
name: c_str!("cubelinux"),
|
|
}),
|
|
})
|
|
}
|
|
}
|
|
|
|
#[pin_data]
|
|
struct CubeStore {
|
|
dev: ARef<Device>,
|
|
}
|
|
|
|
#[vtable]
|
|
impl MiscDevice for CubeStore {
|
|
type Ptr = Pin<KBox<Self>>;
|
|
|
|
fn open(_file: &File, misc: &MiscDeviceRegistration<Self>) -> Result<Pin<KBox<Self>>> {
|
|
KBox::try_pin_init(
|
|
try_pin_init! {
|
|
CubeStore { dev: ARef::from(misc.device()) }
|
|
},
|
|
GFP_KERNEL,
|
|
)
|
|
}
|
|
|
|
/// Write one operation, framed as the argument block the coordinate interface will pass.
|
|
///
|
|
/// ```text
|
|
/// put op=1 | space(32) | x(8) | y(8) | z(8) | len(4) | value[len]
|
|
/// sync op=3
|
|
/// ```
|
|
///
|
|
/// `put` is the byte plane — bytes at a coordinate, with no header written beside them —
|
|
/// which is what the store contract defines and what the differential gate compares
|
|
/// against userspace's `cell put`. The header tier sits above this.
|
|
///
|
|
/// `sync` folds the log into the image. It is a real operation rather than a test hook: a
|
|
/// caller that wants the log reclaimed — a shutdown, a snapshot, a handover — is entitled
|
|
/// to ask.
|
|
fn write_iter(kiocb: Kiocb<'_, Self::Ptr>, iov: &mut IovIterSource<'_>) -> Result<usize> {
|
|
let mut buf = KVVec::<u8>::new();
|
|
let len = iov.copy_from_iter_vec(&mut buf, GFP_KERNEL)?;
|
|
let bytes = buf.as_slice();
|
|
if bytes.is_empty() {
|
|
return Err(EINVAL);
|
|
}
|
|
let me = kiocb.file();
|
|
|
|
// Both operations need the device: its layout says where the image and log are.
|
|
let mut device = KVVec::<u8>::new();
|
|
read_image(&mut device)?;
|
|
let layout = match resolve_layout(&device) {
|
|
Ok(l) => l,
|
|
Err(what) => {
|
|
dev_err!(me.dev, "cubelinux: cannot resolve the store layout: {}\n", what);
|
|
return Err(EINVAL);
|
|
}
|
|
};
|
|
|
|
match bytes[0] {
|
|
OP_PUT => {
|
|
if bytes.len() < 1 + 32 + 24 + 4 {
|
|
return Err(EINVAL);
|
|
}
|
|
let mut space = [0u8; 32];
|
|
space.copy_from_slice(&bytes[1..33]);
|
|
let axis = |at: usize| -> u64 {
|
|
let mut w = [0u8; 8];
|
|
w.copy_from_slice(&bytes[at..at + 8]);
|
|
u64::from_le_bytes(w)
|
|
};
|
|
let (x, y, z) = (axis(33), axis(41), axis(49));
|
|
let mut w = [0u8; 4];
|
|
w.copy_from_slice(&bytes[57..61]);
|
|
let value_len = u32::from_le_bytes(w) as usize;
|
|
if bytes.len() < 61 + value_len {
|
|
return Err(EINVAL);
|
|
}
|
|
let mut value = KVVec::<u8>::new();
|
|
value.extend_from_slice(&bytes[61..61 + value_len], GFP_KERNEL)?;
|
|
let mutation = Mutation {
|
|
space,
|
|
key: morton_encode(x, y, z),
|
|
value,
|
|
};
|
|
match append(&device, &layout, &mutation, 1) {
|
|
Ok(_) => {
|
|
dev_info!(
|
|
me.dev,
|
|
"cubelinux: appended {} bytes for a {} byte value\n",
|
|
ENTRY_FIXED + value_len,
|
|
value_len
|
|
);
|
|
Ok(len)
|
|
}
|
|
Err(e) => {
|
|
dev_err!(me.dev, "cubelinux: append failed: {:?}\n", e);
|
|
Err(e)
|
|
}
|
|
}
|
|
}
|
|
OP_SYNC => {
|
|
let ctl = match layout.control {
|
|
Some(c) => c,
|
|
None => {
|
|
// A bare image has one slot, so folding into it is not atomic and a
|
|
// crash in the middle would lose the store. Refusing is the honest
|
|
// answer; the fix is to format the device as a store.
|
|
dev_err!(
|
|
me.dev,
|
|
"cubelinux: cannot checkpoint a bare image (no control block)\n"
|
|
);
|
|
return Err(EINVAL);
|
|
}
|
|
};
|
|
// The store the image and its log describe, ready to be written out.
|
|
let live = &device[layout.image_off..];
|
|
let header = match parse_header(live) {
|
|
Ok(h) => h,
|
|
Err(what) => {
|
|
dev_err!(me.dev, "cubelinux: {}\n", what);
|
|
return Err(EINVAL);
|
|
}
|
|
};
|
|
let window = log_window(&device, &layout);
|
|
let mut merged = match build_merged(live, window, &header) {
|
|
Ok(m) => m,
|
|
Err(what) => {
|
|
dev_err!(me.dev, "cubelinux: cannot build the store: {}\n", what);
|
|
return Err(EINVAL);
|
|
}
|
|
};
|
|
match checkpoint(&ctl, &mut merged.0) {
|
|
Ok(bytes) => {
|
|
dev_info!(me.dev, "cubelinux: checkpointed {} bytes\n", bytes);
|
|
Ok(len)
|
|
}
|
|
Err(e) => {
|
|
dev_err!(me.dev, "cubelinux: checkpoint failed: {:?}\n", e);
|
|
Err(e)
|
|
}
|
|
}
|
|
}
|
|
other => {
|
|
dev_err!(me.dev, "cubelinux: unknown operation {}\n", other);
|
|
Err(EINVAL)
|
|
}
|
|
}
|
|
}
|
|
|
|
fn read_iter(mut kiocb: Kiocb<'_, Self::Ptr>, iov: &mut IovIterDest<'_>) -> Result<usize> {
|
|
// The work happens here, on demand, rather than at init: by the time anything can
|
|
// open this device, the block driver that provides the store is certainly up.
|
|
let mut image = KVVec::<u8>::new();
|
|
let line = match read_image(&mut image) {
|
|
Ok(()) => digest(&image),
|
|
Err(e) => {
|
|
let mut line = Line::new();
|
|
let _ = write!(line, "error={:?}", e);
|
|
line
|
|
}
|
|
};
|
|
pr_info!("cubelinux: {}\n", core::str::from_utf8(line.as_bytes()).unwrap_or("<bad utf8>"));
|
|
|
|
let me = kiocb.file();
|
|
dev_info!(me.dev, "cubelinux: store read complete\n");
|
|
iov.simple_read_from_buffer(kiocb.ki_pos_mut(), line.as_bytes())
|
|
}
|
|
}
|