Files
lighthouse/beacon_node/store/benches/hdiff.rs
Michael Sproul 9fdd53df56 Hierarchical state diffs (#5978)
* Start extracting freezer changes for tree-states

* Remove unused config args

* Add comments

* Remove unwraps

* Subjective more clear implementation

* Clean up hdiff

* Update xdelta3

* Tree states archive metrics (#6040)

* Add store cache size metrics

* Add compress timer metrics

* Add diff apply compute timer metrics

* Add diff buffer cache hit metrics

* Add hdiff buffer load times

* Add blocks replayed metric

* Move metrics to store

* Future proof some metrics

---------

Co-authored-by: Michael Sproul <michael@sigmaprime.io>

* Port and clean up forwards iterator changes

* Add and polish hierarchy-config flag

* Merge remote-tracking branch 'origin/unstable' into tree-states-archive

* Cleaner errors

* Fix beacon_chain test compilation

* Merge remote-tracking branch 'origin/unstable' into tree-states-archive

* Patch a few more freezer block roots

* Fix genesis block root bug

* Fix test failing due to pending updates

* Beacon chain tests passing

* Merge remote-tracking branch 'origin/unstable' into tree-states-archive

* Merge remote-tracking branch 'origin/unstable' into tree-states-archive

* Fix doc lint

* Implement DB schema upgrade for hierarchical state diffs (#6193)

* DB upgrade

* Add flag

* Delete RestorePointHash

* Update docs

* Update docs

* Implement hierarchical state diffs config migration (#6245)

* Implement hierarchical state diffs config migration

* Review PR

* Remove TODO

* Set CURRENT_SCHEMA_VERSION correctly

* Fix genesis state loading

* Re-delete some PartialBeaconState stuff

---------

Co-authored-by: Michael Sproul <michael@sigmaprime.io>

* Merge remote-tracking branch 'origin/unstable' into tree-states-archive

* Fix test compilation

* Update schema downgrade test

* Fix tests

* Fix null anchor migration

* Merge remote-tracking branch 'origin/unstable' into tree-states-archive

* Fix tree states upgrade migration (#6328)

* Towards crash safety

* Fix compilation

* Move cold summaries and state roots to new columns

* Rename StateRoots chunked field

* Update prune states

* Clean hdiff CLI flag and metrics

* Fix "staged reconstruction"

* Merge remote-tracking branch 'origin/unstable' into tree-states-archive

* Fix alloy issues

* Fix staged reconstruction logic

* Prevent weird slot drift

* Remove "allow" flag

* Update CLI help

* Remove FIXME about downgrade

* Merge remote-tracking branch 'origin/unstable' into tree-states-archive

* Remove some unnecessary error variants

* Fix new test

* Tree states archive - review comments and metrics (#6386)

* Review PR comments and metrics

* Comments

* Add anchor metrics

* drop prev comment

* Update metadata.rs

* Apply suggestions from code review

---------

Co-authored-by: Michael Sproul <micsproul@gmail.com>

* Update beacon_node/store/src/hot_cold_store.rs

Co-authored-by: Lion - dapplion <35266934+dapplion@users.noreply.github.com>

* Merge remote-tracking branch 'origin/unstable' into tree-states-archive

* Clarify comment and remove anchor_slot garbage

* Simplify database anchor (#6397)

* Simplify database anchor

* Update beacon_node/store/src/reconstruct.rs

* Add migration for anchor

* Fix and simplify light_client store tests

* Fix incompatible config test

* Merge remote-tracking branch 'origin/unstable' into tree-states-archive

* Merge remote-tracking branch 'origin/unstable' into tree-states-archive

* More metrics

* Merge remote-tracking branch 'origin/unstable' into tree-states-archive

* New historic state cache (#6475)

* New historic state cache

* Add more metrics

* State cache hit rate metrics

* Fix store metrics

* More logs and metrics

* Fix logger

* Ensure cached states have built caches :O

* Replay blocks in preference to diffing

* Two separate caches

* Distribute cache build time to next slot

* Re-plumb historic-state-cache flag

* Clean up metrics

* Update book

* Update beacon_node/store/src/hdiff.rs

Co-authored-by: Lion - dapplion <35266934+dapplion@users.noreply.github.com>

* Update beacon_node/store/src/historic_state_cache.rs

Co-authored-by: Lion - dapplion <35266934+dapplion@users.noreply.github.com>

---------

Co-authored-by: Lion - dapplion <35266934+dapplion@users.noreply.github.com>

* Update database docs

* Update diagram

* Merge remote-tracking branch 'origin/unstable' into tree-states-archive

* Update lockbud to work with bindgen/etc

* Correct pkg name for Debian

* Remove vestigial epochs_per_state_diff

* Merge remote-tracking branch 'origin/unstable' into tree-states-archive

* Markdown lint

* Merge remote-tracking branch 'origin/unstable' into tree-states-archive

* Address Jimmy's review comments

* Simplify ReplayFrom case

* Fix and document genesis_state_root

* Typo

Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>

* Merge branch 'unstable' into tree-states-archive

* Compute diff of validators list manually (#6556)

* Split hdiff computation

* Dedicated logic for historical roots and summaries

* Benchmark against real states

* Mutated source?

* Version the hdiff

* Add lighthouse DB config for hierarchy exponents

* Tidy up hierarchy exponents flag

* Apply suggestions from code review

Co-authored-by: Michael Sproul <micsproul@gmail.com>

* Address PR review

* Remove hardcoded paths in benchmarks

* Delete unused function in benches

* lint

---------

Co-authored-by: Michael Sproul <michael@sigmaprime.io>

* Test hdiff binary format stability (#6585)

* Merge remote-tracking branch 'origin/unstable' into tree-states-archive

* Add deprecation warning for SPRP

* Update xdelta to get rid of duplicate deps

* Document test
2024-11-18 01:51:44 +00:00

117 lines
3.5 KiB
Rust

use bls::PublicKeyBytes;
use criterion::{criterion_group, criterion_main, Criterion};
use rand::Rng;
use ssz::Decode;
use store::{
hdiff::{HDiff, HDiffBuffer},
StoreConfig,
};
use types::{BeaconState, Epoch, Eth1Data, EthSpec, MainnetEthSpec as E, Validator};
pub fn all_benches(c: &mut Criterion) {
let spec = E::default_spec();
let genesis_time = 0;
let eth1_data = Eth1Data::default();
let mut rng = rand::thread_rng();
let validator_mutations = 1000;
let validator_additions = 100;
for n in [1_000_000, 1_500_000, 2_000_000] {
let mut source_state = BeaconState::<E>::new(genesis_time, eth1_data.clone(), &spec);
for _ in 0..n {
append_validator(&mut source_state, &mut rng);
}
let mut target_state = source_state.clone();
// Change all balances
for i in 0..n {
let balance = target_state.balances_mut().get_mut(i).unwrap();
*balance += rng.gen_range(1..=1_000_000);
}
// And some validator records
for _ in 0..validator_mutations {
let index = rng.gen_range(1..n);
// TODO: Only change a few things, and not the pubkey
*target_state.validators_mut().get_mut(index).unwrap() = rand_validator(&mut rng);
}
for _ in 0..validator_additions {
append_validator(&mut target_state, &mut rng);
}
bench_against_states(
c,
source_state,
target_state,
&format!("n={n} v_mut={validator_mutations} v_add={validator_additions}"),
);
}
}
fn bench_against_states(
c: &mut Criterion,
source_state: BeaconState<E>,
target_state: BeaconState<E>,
id: &str,
) {
let slot_diff = target_state.slot() - source_state.slot();
let config = StoreConfig::default();
let source = HDiffBuffer::from_state(source_state);
let target = HDiffBuffer::from_state(target_state);
let diff = HDiff::compute(&source, &target, &config).unwrap();
println!(
"state slot diff {slot_diff} - diff size {id} {}",
diff.size()
);
c.bench_function(&format!("compute hdiff {id}"), |b| {
b.iter(|| {
HDiff::compute(&source, &target, &config).unwrap();
})
});
c.bench_function(&format!("apply hdiff {id}"), |b| {
b.iter(|| {
let mut source = source.clone();
diff.apply(&mut source, &config).unwrap();
})
});
}
fn rand_validator(mut rng: impl Rng) -> Validator {
let mut pubkey = [0u8; 48];
rng.fill_bytes(&mut pubkey);
let withdrawal_credentials: [u8; 32] = rng.gen();
Validator {
pubkey: PublicKeyBytes::from_ssz_bytes(&pubkey).unwrap(),
withdrawal_credentials: withdrawal_credentials.into(),
slashed: false,
effective_balance: 32_000_000_000,
activation_eligibility_epoch: Epoch::max_value(),
activation_epoch: Epoch::max_value(),
exit_epoch: Epoch::max_value(),
withdrawable_epoch: Epoch::max_value(),
}
}
fn append_validator(state: &mut BeaconState<E>, mut rng: impl Rng) {
state
.balances_mut()
.push(32_000_000_000 + rng.gen_range(1..=1_000_000_000))
.unwrap();
if let Ok(inactivity_scores) = state.inactivity_scores_mut() {
inactivity_scores.push(0).unwrap();
}
state
.validators_mut()
.push(rand_validator(&mut rng))
.unwrap();
}
criterion_group! {
name = benches;
config = Criterion::default().sample_size(10);
targets = all_benches
}
criterion_main!(benches);