gnosis/lighthouse
1
0
Fork 0
mirror of https://github.com/sigp/lighthouse.git synced 2026-03-20 21:34:46 +00:00
Code Issues Packages Projects Releases Wiki Activity

Separate execution payloads in the DB (#3157)

Browse Source 
## Proposed Changes

Reduce post-merge disk usage by not storing finalized execution payloads in Lighthouse's database.

⚠️ **This is achieved in a backwards-incompatible way for networks that have already merged** ⚠️. Kiln users and shadow fork enjoyers will be unable to downgrade after running the code from this PR. The upgrade migration may take several minutes to run, and can't be aborted after it begins.

The main changes are:

- New column in the database called `ExecPayload`, keyed by beacon block root.
- The `BeaconBlock` column now stores blinded blocks only.
- Lots of places that previously used full blocks now use blinded blocks, e.g. analytics APIs, block replay in the DB, etc.
- On finalization:
    - `prune_abanonded_forks` deletes non-canonical payloads whilst deleting non-canonical blocks.
    - `migrate_db` deletes finalized canonical payloads whilst deleting finalized states.
- Conversions between blinded and full blocks are implemented in a compositional way, duplicating some work from Sean's PR #3134.
- The execution layer has a new `get_payload_by_block_hash` method that reconstructs a payload using the EE's `eth_getBlockByHash` call.
   - I've tested manually that it works on Kiln, using Geth and Nethermind.
   - This isn't necessarily the most efficient method, and new engine APIs are being discussed to improve this: https://github.com/ethereum/execution-apis/pull/146.
   - We're depending on the `ethers` master branch, due to lots of recent changes. We're also using a workaround for https://github.com/gakonst/ethers-rs/issues/1134.
- Payload reconstruction is used in the HTTP API via `BeaconChain::get_block`, which is now `async`. Due to the `async` fn, the `blocking_json` wrapper has been removed.
- Payload reconstruction is used in network RPC to serve blocks-by-{root,range} responses. Here the `async` adjustment is messier, although I think I've managed to come up with a reasonable compromise: the handlers take the `SendOnDrop` by value so that they can drop it on _task completion_ (after the `fn` returns). Still, this is introducing disk reads onto core executor threads, which may have a negative performance impact (thoughts appreciated).

## Additional Info

- [x] For performance it would be great to remove the cloning of full blocks when converting them to blinded blocks to write to disk. I'm going to experiment with a `put_block` API that takes the block by value, breaks it into a blinded block and a payload, stores the blinded block, and then re-assembles the full block for the caller.
- [x] We should measure the latency of blocks-by-root and blocks-by-range responses.
- [x] We should add integration tests that stress the payload reconstruction (basic tests done, issue for more extensive tests: https://github.com/sigp/lighthouse/issues/3159)
- [x] We should (manually) test the schema v9 migration from several prior versions, particularly as blocks have changed on disk and some migrations rely on being able to load blocks.

Co-authored-by: Paul Hauner <paul@paulhauner.com>
This commit is contained in:
Michael Sproul
2022-05-12 00:42:17 +00:00
parent be59fd9af7
commit bcdd960ab1
62 changed files with 2009 additions and 392 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
beacon_node/beacon_chain/src/schema_change/migration_schema_v7.rs
View File

@@ -31,7 +31,7 @@ pub(crate) fn update_with_reinitialized_fork_choice<T: BeaconChainTypes>(
.finalized_checkpoint
.root;
let anchor_block = db
.get_block(&anchor_block_root)
.get_full_block_prior_to_v9(&anchor_block_root)
.map_err(|e| format!("{:?}", e))?
.ok_or_else(|| "Missing anchor beacon block".to_string())?;
let anchor_state = db

2
beacon_node/beacon_chain/src/schema_change/migration_schema_v8.rs
View File

@@ -34,7 +34,7 @@ pub fn update_fork_choice<T: BeaconChainTypes>(
// before schema v8 the cache would always miss on skipped slots.
for item in balances_cache.items {
// Drop any blocks that aren't found, they're presumably too old and this is only a cache.
if let Some(block) = db.get_block(&item.block_root)? {
if let Some(block) = db.get_full_block_prior_to_v9(&item.block_root)? {
fork_choice_store.balances_cache.items.push(CacheItemV8 {
block_root: item.block_root,
epoch: block.slot().epoch(T::EthSpec::slots_per_epoch()),

176
beacon_node/beacon_chain/src/schema_change/migration_schema_v9.rs Normal file
View File

@@ -0,0 +1,176 @@
use crate::beacon_chain::BeaconChainTypes;
use slog::{debug, error, info, Logger};
use slot_clock::SlotClock;
use std::sync::Arc;
use std::time::Duration;
use store::{DBColumn, Error, HotColdDB, KeyValueStore};
use types::{EthSpec, Hash256, Slot};
const OPS_PER_BLOCK_WRITE: usize = 2;
/// The slot clock isn't usually available before the database is initialized, so we construct a
/// temporary slot clock by reading the genesis state. It should always exist if the database is
/// initialized at a prior schema version, however we still handle the lack of genesis state
/// gracefully.
fn get_slot_clock<T: BeaconChainTypes>(
db: &HotColdDB<T::EthSpec, T::HotStore, T::ColdStore>,
log: &Logger,
) -> Result<Option<T::SlotClock>, Error> {
// At schema v8 the genesis block must be a *full* block (with payload). In all likeliness it
// actually has no payload.
let spec = db.get_chain_spec();
let genesis_block = if let Some(block) = db.get_full_block_prior_to_v9(&Hash256::zero())? {
block
} else {
error!(log, "Missing genesis block");
return Ok(None);
};
let genesis_state =
if let Some(state) = db.get_state(&genesis_block.state_root(), Some(Slot::new(0)))? {
state
} else {
error!(log, "Missing genesis state"; "state_root" => ?genesis_block.state_root());
return Ok(None);
};
Ok(Some(T::SlotClock::new(
spec.genesis_slot,
Duration::from_secs(genesis_state.genesis_time()),
Duration::from_secs(spec.seconds_per_slot),
)))
}
pub fn upgrade_to_v9<T: BeaconChainTypes>(
db: Arc<HotColdDB<T::EthSpec, T::HotStore, T::ColdStore>>,
log: Logger,
) -> Result<(), Error> {
// This upgrade is a no-op if the Bellatrix fork epoch has not already passed. This migration
// was implemented before the activation of Bellatrix on all networks except Kiln, so the only
// users who will need to wait for the slow copying migration are Kiln users.
let slot_clock = if let Some(slot_clock) = get_slot_clock::<T>(&db, &log)? {
slot_clock
} else {
error!(
log,
"Unable to complete migration because genesis state or genesis block is missing"
);
return Err(Error::SlotClockUnavailableForMigration);
};
let current_epoch = if let Some(slot) = slot_clock.now() {
slot.epoch(T::EthSpec::slots_per_epoch())
} else {
return Ok(());
};
let bellatrix_fork_epoch = if let Some(fork_epoch) = db.get_chain_spec().bellatrix_fork_epoch {
fork_epoch
} else {
info!(
log,
"Upgrading database schema to v9 (no-op)";
"info" => "To downgrade before the merge run `lighthouse db migrate`"
);
return Ok(());
};
if current_epoch >= bellatrix_fork_epoch {
info!(
log,
"Upgrading database schema to v9";
"info" => "This will take several minutes. Each block will be read from and \
re-written to the database. You may safely exit now (Ctrl-C) and resume \
the migration later. Downgrading is no longer possible."
);
for res in db.hot_db.iter_column_keys(DBColumn::BeaconBlock) {
let block_root = res?;
let block = match db.get_full_block_prior_to_v9(&block_root) {
// A pre-v9 block is present.
Ok(Some(block)) => block,
// A block is missing.
Ok(None) => return Err(Error::BlockNotFound(block_root)),
// There was an error reading a pre-v9 block. Try reading it as a post-v9 block.
Err(_) => {
if db.try_get_full_block(&block_root)?.is_some() {
// The block is present as a post-v9 block, assume that it was already
// correctly migrated.
continue;
} else {
// This scenario should not be encountered since a prior check has ensured
// that this block exists.
return Err(Error::V9MigrationFailure(block_root));
}
}
};
if block.message().execution_payload().is_ok() {
// Overwrite block with blinded block and store execution payload separately.
debug!(
log,
"Rewriting Bellatrix block";
"block_root" => ?block_root,
);
let mut kv_batch = Vec::with_capacity(OPS_PER_BLOCK_WRITE);
db.block_as_kv_store_ops(&block_root, block, &mut kv_batch)?;
db.hot_db.do_atomically(kv_batch)?;
}
}
} else {
info!(
log,
"Upgrading database schema to v9 (no-op)";
"info" => "To downgrade before the merge run `lighthouse db migrate`"
);
}
Ok(())
}
// This downgrade is conditional and will only succeed if the Bellatrix fork epoch hasn't been
// reached.
pub fn downgrade_from_v9<T: BeaconChainTypes>(
db: Arc<HotColdDB<T::EthSpec, T::HotStore, T::ColdStore>>,
log: Logger,
) -> Result<(), Error> {
let slot_clock = if let Some(slot_clock) = get_slot_clock::<T>(&db, &log)? {
slot_clock
} else {
error!(
log,
"Unable to complete migration because genesis state or genesis block is missing"
);
return Err(Error::SlotClockUnavailableForMigration);
};
let current_epoch = if let Some(slot) = slot_clock.now() {
slot.epoch(T::EthSpec::slots_per_epoch())
} else {
return Ok(());
};
let bellatrix_fork_epoch = if let Some(fork_epoch) = db.get_chain_spec().bellatrix_fork_epoch {
fork_epoch
} else {
info!(
log,
"Downgrading database schema from v9";
"info" => "You need to upgrade to v9 again before the merge"
);
return Ok(());
};
if current_epoch >= bellatrix_fork_epoch {
error!(
log,
"Downgrading from schema v9 after the Bellatrix fork epoch is not supported";
"current_epoch" => current_epoch,
"bellatrix_fork_epoch" => bellatrix_fork_epoch,
"reason" => "You need a v9 schema database to run on a merged version of Prater or \
mainnet. On Kiln, you have to re-sync",
);
Err(Error::ResyncRequiredForExecutionPayloadSeparation)
} else {
Ok(())
}
}