cubelinux-2/docs/task4-reader-writer-sharding.md

Task 4 — Reader/Writer Sharding: before / after

Plan: convert the single global Mutex<CubeStore> inside ConcurrentStore into an RwLock<CubeStore> so many reader threads run in parallel while writes stay serialized. Then demonstrate (via cube-bench, release build) that reads stop blocking each other.

What changed (Task 4 commit)

• cubesys/src/store.rs: ConcurrentStore.inner: Arc<Mutex<CubeStore>> -> Arc<RwLock<CubeStore>>. All GET-side methods (get_raw, get_record, keys, scan_prefix, linked_to, query_doc_type, read_snapshot) take the read side; all mutation paths (put_raw, delete_raw, put_record, associate, with_mut, checkpoint, and the durability helpers incremental / fold_delta_into_base) take the write side.
• The WAL, the checkpoint thread, and coordinate encoding are untouched — the change is purely the lock granularity, so durability/replay behavior is identical (proven by the existing durable_checkpoint_and_replay, wal_recovery_after_crash, incremental_checkpoint_delta_model tests).
• Added store::tests::concurrent_reads_dont_block_on_writer and a cube-bench Task 4 section (behind a bench feature flag on cubesys so the internals never ship in the daemon path).

Correctness result (verified)

• Reader-vs-reader exclusion is GONE: 8 concurrent reader threads + 1 writer thread all make progress; reads overlap an active writer instead of queuing behind it. No deadlock, no pathological stall.
• Durability tests still pass (./check green): the WAL boundary / delta model is unchanged.

Performance result (HONEST — see benchmarks below)

The RwLock delivers the correctness property the plan asked for (reads no longer serialize behind one another, and a writer no longer stalls readers from starting). It does NOT deliver a wall-clock speedup for short reads on this 8-core host, because std::sync::RwLock's per-lock atomic and the shared CubeStore cache line bounce when 8 readers contend on the same lock.

cargo run -p cube-bench --release (this host, warm in-memory backend):

Task 4 concurrency (RwLock, 8 readers x 400000 reads)
  reader solo                10.37 ms
  8 readers (RwLock)       300.58 ms
  8 readers (old Mutex)     82.95 ms   (serialized bound = 8 x solo)
  8 readers + writer        380.30 ms   (writer active 150ms; readers overlapped)
  aggregate read rate     10645.92 k reads/s
  NOTE: for ~15ns reads, RwLock overhead > parallel gain (per-lock atomics);
        the win is correctness (no reader-vs-reader exclusion) + writer overlap.

Task 4 concurrency, heavier reads (256B payload + ts, 30000 each)
  reader solo                 2.16 ms
  8 readers (RwLock)        23.25 ms
  8 readers (old Mutex)     17.31 ms   (serialized bound)
  RwLock vs Mutex ratio       0.74 x    (1.0 = parity; <1 means RwLock slower here)

Conclusion / recommendation

• Task 4 as specified (RwLock sharding) is correct and merged. It removes reader-vs-reader mutual exclusion and lets reads overlap writes — a real concurrency improvement in the API/correctness sense.
• If the goal is actual read-throughput scaling (not just non-exclusion), the single shared CubeStore under one RwLock is the bottleneck: all readers still serialize on the same lock word and contend on the same cache line. To scale reads you would need one of:
  1. a sharded store (split the coordinate space across N RwLock<CubeStore> shards, keyed by c/prefix) so concurrent reads land on different locks, or
  2. a lock-free / MVCC snapshot map (readers clone an Arc<Snapshot> and never touch a lock), or
  3. the existing read_snapshot() call pattern for bulk-read clients (VM, cubefs) so they take one consistent clone instead of per-call locks.
• None of those were required by Task 4 (which asked for RwLock sharding), so they are left as a follow-up decision rather than silently invented.