Hierarchical state diffs in hot DB (#6750)

This PR implements https://github.com/sigp/lighthouse/pull/5978 (tree-states) but on the hot DB. It allows Lighthouse to massively reduce its disk footprint during non-finality and overall I/O in all cases.

Closes https://github.com/sigp/lighthouse/issues/6580

Conga into https://github.com/sigp/lighthouse/pull/6744

### TODOs

- [x] Fix OOM in CI https://github.com/sigp/lighthouse/pull/7176
- [x] optimise store_hot_state to avoid storing a duplicate state if the summary already exists (should be safe from races now that pruning is cleaner)
- [x] mispelled: get_ancenstor_state_root
- [x] get_ancestor_state_root should use state summaries
- [x] Prevent split from changing during ancestor calc
- [x] Use same hierarchy for hot and cold

### TODO Good optimization for future PRs

- [ ] On the migration, if the latest hot snapshot is aligned with the cold snapshot migrate the diffs instead of the full states.
```
align slot  time
10485760    Nov-26-2024
12582912    Sep-14-2025
14680064    Jul-02-2026
```

### TODO Maybe things good to have

- [ ] Rename anchor_slot https://github.com/sigp/lighthouse/compare/tree-states-hot-rebase-oom...dapplion:lighthouse:tree-states-hot-anchor-slot-rename?expand=1
- [ ] Make anchor fields not public such that they must be mutated through a method. To prevent un-wanted changes of the anchor_slot

### NOTTODO

- [ ] Use fork-choice and a new method [`descendants_of_checkpoint`](ca2388e196 (diff-046fbdb517ca16b80e4464c2c824cf001a74a0a94ac0065e635768ac391062a8)) to filter only the state summaries that descend of finalized checkpoint]
This commit is contained in:
Lion - dapplion
2025-06-19 04:43:25 +02:00
committed by GitHub
parent 6786b9d12a
commit dd98534158
33 changed files with 2695 additions and 812 deletions

View File

@@ -2,7 +2,7 @@ use crate::{
errors::Error as DBError, get_key_for_col, hot_cold_store::BytesKey, ColumnIter, ColumnKeyIter,
DBColumn, Error, ItemStore, Key, KeyValueStore, KeyValueStoreOp,
};
use parking_lot::{Mutex, MutexGuard, RwLock};
use parking_lot::RwLock;
use std::collections::{BTreeMap, HashSet};
use std::marker::PhantomData;
use types::*;
@@ -12,7 +12,6 @@ type DBMap = BTreeMap<BytesKey, Vec<u8>>;
/// A thread-safe `BTreeMap` wrapper.
pub struct MemoryStore<E: EthSpec> {
db: RwLock<DBMap>,
transaction_mutex: Mutex<()>,
_phantom: PhantomData<E>,
}
@@ -21,7 +20,6 @@ impl<E: EthSpec> MemoryStore<E> {
pub fn open() -> Self {
Self {
db: RwLock::new(BTreeMap::new()),
transaction_mutex: Mutex::new(()),
_phantom: PhantomData,
}
}
@@ -107,10 +105,6 @@ impl<E: EthSpec> KeyValueStore<E> for MemoryStore<E> {
Box::new(self.iter_column(column).map(|res| res.map(|(k, _)| k)))
}
fn begin_rw_transaction(&self) -> MutexGuard<()> {
self.transaction_mutex.lock()
}
fn compact_column(&self, _column: DBColumn) -> Result<(), Error> {
Ok(())
}