* some blob reprocessing work

* remove ForceBlockLookup

* reorder enum match arms in sync manager

* a lot more reprocessing work

* impl logic for triggerng blob lookups along with block lookups

* deal with rpc blobs in groups per block in the da checker. don't cache missing blob ids in the da checker.

* make single block lookup generic

* more work

* add delayed processing logic and combine some requests

* start fixing some compile errors

* fix compilation in main block lookup mod

* much work

* get things compiling

* parent blob lookups

* fix compile

* revert red/stevie changes

* fix up sync manager delay message logic

* add peer usefulness enum

* should remove lookup refactor

* consolidate retry error handling

* improve peer scoring during certain failures in parent lookups

* improve retry code

* drop parent lookup if either req has a peer disconnect during download

* refactor single block processed method

* processing peer refactor

* smol bugfix

* fix some todos

* fix lints

* fix lints

* fix compile in lookup tests

* fix lints

* fix lints

* fix existing block lookup tests

* renamings

* fix after merge

* cargo fmt

* compilation fix in beacon chain tests

* fix

* refactor lookup tests to work with multiple forks and response types

* make tests into macros

* wrap availability check error

* fix compile after merge

* add random blobs

* start fixing up lookup verify error handling

* some bug fixes and the start of deneb only tests

* make tests work for all forks

* track information about peer source

* error refactoring

* improve peer scoring

* fix test compilation

* make sure blobs are sent for processing after stream termination, delete copied tests

* add some tests and fix a bug

* smol bugfixes and moar tests

* add tests and fix some things

* compile after merge

* lots of refactoring

* retry on invalid block/blob

* merge unknown parent messages before current slot lookup

* get tests compiling

* penalize blob peer on invalid blobs

* Check disk on in-memory cache miss

* Update beacon_node/beacon_chain/src/data_availability_checker/overflow_lru_cache.rs

* Update beacon_node/network/src/sync/network_context.rs

Co-authored-by: Divma <26765164+divagant-martian@users.noreply.github.com>

* fix bug in matching blocks and blobs in range sync

* pr feedback

* fix conflicts

* upgrade logs from warn to crit when we receive incorrect response in range

* synced_and_connected_within_tolerance -> should_search_for_block

* remove todo

* add data gas used and update excess data gas to u64

* Fix Broken Overflow Tests

* payload verification with commitments

* fix merge conflicts

* restore payload file

* Restore payload file

* remove todo

* add max blob commitments per block

* c-kzg lib update

* Fix ef tests

* Abstract over minimal/mainnet spec in kzg crate

* Start integrating new KZG

* checkpoint sync without alignment

* checkpoint sync without alignment

* add import

* add import

* query for checkpoint state by slot rather than state root (teku doesn't serve by state root)

* query for checkpoint state by slot rather than state root (teku doesn't serve by state root)

* loosen check

* get state first and query by most recent block root

* Revert "loosen check"

This reverts commit 069d13dd63.

* get state first and query by most recent block root

* merge max blobs change

* simplify delay logic

* rename unknown parent sync message variants

* rename parameter, block_slot -> slot

* add some docs to the lookup module

* use interval instead of sleep

* drop request if blocks and blobs requests both return `None` for `Id`

* clean up `find_single_lookup` logic

* add lookup source enum

* clean up `find_single_lookup` logic

* add docs to find_single_lookup_request

* move LookupSource our of param where unnecessary

* remove unnecessary todo

* query for block by `state.latest_block_header.slot`

* fix lint

* fix merge transition ef tests

* fix test

* fix test

* fix observed  blob sidecars test

* Add some metrics (#33)

* fix protocol limits for blobs by root

* Update Engine API for 1:1 Structure Method

* make beacon chain tests to fix devnet 6 changes

* get ckzg working and fix some tests

* fix remaining tests

* fix lints

* Fix KZG linking issues

* remove unused dep

* lockfile

* test fixes

* remove dbgs

* remove unwrap

* cleanup tx generator

* small fixes

* fixing fixes

* more self reivew

* more self review

* refactor genesis header initialization

* refactor mock el instantiations

* fix compile

* fix network test, make sure they run for each fork

* pr feedback

* fix last test (hopefully)

---------

Co-authored-by: Pawan Dhananjay <pawandhananjay@gmail.com>
Co-authored-by: Mark Mackey <mark@sigmaprime.io>
Co-authored-by: Divma <26765164+divagant-martian@users.noreply.github.com>
Co-authored-by: Michael Sproul <michael@sigmaprime.io>
This commit is contained in:
realbigsean
2023-06-29 15:35:43 -04:00
committed by GitHub
parent 4c9fcf1e83
commit adbb62f7f3
69 changed files with 2114 additions and 1338 deletions

View File

@@ -6,24 +6,23 @@ use crate::{
},
ExecutionBlock, PayloadAttributes, PayloadId, PayloadStatusV1, PayloadStatusV1Status,
},
BlobsBundleV1, ExecutionBlockWithTransactions,
random_valid_tx, BlobsBundleV1, ExecutionBlockWithTransactions,
};
use kzg::{Kzg, BYTES_PER_BLOB, BYTES_PER_FIELD_ELEMENT, FIELD_ELEMENTS_PER_BLOB};
use rand::RngCore;
use kzg::Kzg;
use rand::thread_rng;
use serde::{Deserialize, Serialize};
use ssz::Encode;
use std::collections::HashMap;
use std::sync::Arc;
use tree_hash::TreeHash;
use tree_hash_derive::TreeHash;
use types::consts::deneb::BLOB_TX_TYPE;
use types::transaction::{BlobTransaction, EcdsaSignature, SignedBlobTransaction};
use types::{
Blob, EthSpec, ExecutionBlockHash, ExecutionPayload, ExecutionPayloadCapella,
ExecutionPayloadDeneb, ExecutionPayloadMerge, ForkName, Hash256, Transaction, Transactions,
Uint256,
BlobSidecar, ChainSpec, EthSpec, ExecutionBlockHash, ExecutionPayload, ExecutionPayloadCapella,
ExecutionPayloadDeneb, ExecutionPayloadHeader, ExecutionPayloadMerge, ForkName, Hash256,
Transactions, Uint256,
};
use super::DEFAULT_TERMINAL_BLOCK;
const GAS_LIMIT: u64 = 16384;
const GAS_USED: u64 = GAS_LIMIT - 1;
@@ -125,12 +124,12 @@ pub struct ExecutionBlockGenerator<T: EthSpec> {
* Post-merge fork triggers
*/
pub shanghai_time: Option<u64>, // withdrawals
pub deneb_time: Option<u64>, // 4844
pub cancun_time: Option<u64>, // deneb
/*
* deneb stuff
*/
pub blobs_bundles: HashMap<PayloadId, BlobsBundleV1<T>>,
pub kzg: Option<Arc<Kzg>>,
pub kzg: Option<Arc<Kzg<T::Kzg>>>,
}
impl<T: EthSpec> ExecutionBlockGenerator<T> {
@@ -139,8 +138,8 @@ impl<T: EthSpec> ExecutionBlockGenerator<T> {
terminal_block_number: u64,
terminal_block_hash: ExecutionBlockHash,
shanghai_time: Option<u64>,
deneb_time: Option<u64>,
kzg: Option<Kzg>,
cancun_time: Option<u64>,
kzg: Option<Kzg<T::Kzg>>,
) -> Self {
let mut gen = Self {
head_block: <_>::default(),
@@ -154,7 +153,7 @@ impl<T: EthSpec> ExecutionBlockGenerator<T> {
next_payload_id: 0,
payload_ids: <_>::default(),
shanghai_time,
deneb_time,
cancun_time,
blobs_bundles: <_>::default(),
kzg: kzg.map(Arc::new),
};
@@ -189,7 +188,7 @@ impl<T: EthSpec> ExecutionBlockGenerator<T> {
}
pub fn get_fork_at_timestamp(&self, timestamp: u64) -> ForkName {
match self.deneb_time {
match self.cancun_time {
Some(fork_time) if timestamp >= fork_time => ForkName::Deneb,
_ => match self.shanghai_time {
Some(fork_time) if timestamp >= fork_time => ForkName::Capella,
@@ -270,10 +269,15 @@ impl<T: EthSpec> ExecutionBlockGenerator<T> {
finalized_block_hash
));
}
let parent_hash = if block_number == 0 {
ExecutionBlockHash::zero()
let block = if block_number == 0 {
generate_genesis_block(self.terminal_total_difficulty, self.terminal_block_number)?
} else if let Some(block) = self.block_by_number(block_number - 1) {
block.block_hash()
generate_pow_block(
self.terminal_total_difficulty,
self.terminal_block_number,
block_number,
block.block_hash(),
)?
} else {
return Err(format!(
"parent with block number {} not found",
@@ -281,13 +285,6 @@ impl<T: EthSpec> ExecutionBlockGenerator<T> {
));
};
let block = generate_pow_block(
self.terminal_total_difficulty,
self.terminal_block_number,
block_number,
parent_hash,
)?;
// Insert block into block tree
self.insert_block(Block::PoW(block))?;
@@ -348,10 +345,10 @@ impl<T: EthSpec> ExecutionBlockGenerator<T> {
Ok(hash)
}
// This does not reject duplicate blocks inserted. This lets us re-use the same execution
// block generator for multiple beacon chains which is useful in testing.
pub fn insert_block(&mut self, block: Block<T>) -> Result<ExecutionBlockHash, String> {
if self.blocks.contains_key(&block.block_hash()) {
return Err(format!("{:?} is already known", block.block_hash()));
} else if block.parent_hash() != ExecutionBlockHash::zero()
if block.parent_hash() != ExecutionBlockHash::zero()
&& !self.blocks.contains_key(&block.parent_hash())
{
return Err(format!("parent block {:?} is unknown", block.parent_hash()));
@@ -410,8 +407,7 @@ impl<T: EthSpec> ExecutionBlockGenerator<T> {
}
pub fn get_blobs_bundle(&mut self, id: &PayloadId) -> Option<BlobsBundleV1<T>> {
// remove it to free memory
self.blobs_bundles.remove(id)
self.blobs_bundles.get(id).cloned()
}
pub fn new_payload(&mut self, payload: ExecutionPayload<T>) -> PayloadStatusV1 {
@@ -450,6 +446,14 @@ impl<T: EthSpec> ExecutionBlockGenerator<T> {
forkchoice_state: ForkchoiceState,
payload_attributes: Option<PayloadAttributes>,
) -> Result<JsonForkchoiceUpdatedV1Response, String> {
// This is meant to cover starting post-merge transition at genesis. Useful for
// testing Capella forks and later.
if let Some(genesis_pow_block) = self.block_by_number(0) {
if genesis_pow_block.block_hash() == forkchoice_state.head_block_hash {
self.terminal_block_hash = forkchoice_state.head_block_hash;
}
}
if let Some(payload) = self
.pending_payloads
.remove(&forkchoice_state.head_block_hash)
@@ -518,67 +522,61 @@ impl<T: EthSpec> ExecutionBlockGenerator<T> {
block_hash: ExecutionBlockHash::zero(),
transactions: vec![].into(),
}),
PayloadAttributes::V2(pa) => {
match self.get_fork_at_timestamp(pa.timestamp) {
ForkName::Merge => ExecutionPayload::Merge(ExecutionPayloadMerge {
parent_hash: forkchoice_state.head_block_hash,
fee_recipient: pa.suggested_fee_recipient,
receipts_root: Hash256::repeat_byte(42),
state_root: Hash256::repeat_byte(43),
logs_bloom: vec![0; 256].into(),
prev_randao: pa.prev_randao,
block_number: parent.block_number() + 1,
gas_limit: GAS_LIMIT,
gas_used: GAS_USED,
timestamp: pa.timestamp,
extra_data: "block gen was here".as_bytes().to_vec().into(),
base_fee_per_gas: Uint256::one(),
block_hash: ExecutionBlockHash::zero(),
transactions: vec![].into(),
}),
ForkName::Capella => {
ExecutionPayload::Capella(ExecutionPayloadCapella {
parent_hash: forkchoice_state.head_block_hash,
fee_recipient: pa.suggested_fee_recipient,
receipts_root: Hash256::repeat_byte(42),
state_root: Hash256::repeat_byte(43),
logs_bloom: vec![0; 256].into(),
prev_randao: pa.prev_randao,
block_number: parent.block_number() + 1,
gas_limit: GAS_LIMIT,
gas_used: GAS_USED,
timestamp: pa.timestamp,
extra_data: "block gen was here".as_bytes().to_vec().into(),
base_fee_per_gas: Uint256::one(),
block_hash: ExecutionBlockHash::zero(),
transactions: vec![].into(),
withdrawals: pa.withdrawals.clone().into(),
})
}
ForkName::Deneb => {
ExecutionPayload::Deneb(ExecutionPayloadDeneb {
parent_hash: forkchoice_state.head_block_hash,
fee_recipient: pa.suggested_fee_recipient,
receipts_root: Hash256::repeat_byte(42),
state_root: Hash256::repeat_byte(43),
logs_bloom: vec![0; 256].into(),
prev_randao: pa.prev_randao,
block_number: parent.block_number() + 1,
gas_limit: GAS_LIMIT,
gas_used: GAS_USED,
timestamp: pa.timestamp,
extra_data: "block gen was here".as_bytes().to_vec().into(),
base_fee_per_gas: Uint256::one(),
block_hash: ExecutionBlockHash::zero(),
transactions: vec![].into(),
withdrawals: pa.withdrawals.clone().into(),
// FIXME(deneb) maybe this should be set to something?
excess_data_gas: Uint256::one(),
})
}
_ => unreachable!(),
}
}
PayloadAttributes::V2(pa) => match self.get_fork_at_timestamp(pa.timestamp) {
ForkName::Merge => ExecutionPayload::Merge(ExecutionPayloadMerge {
parent_hash: forkchoice_state.head_block_hash,
fee_recipient: pa.suggested_fee_recipient,
receipts_root: Hash256::repeat_byte(42),
state_root: Hash256::repeat_byte(43),
logs_bloom: vec![0; 256].into(),
prev_randao: pa.prev_randao,
block_number: parent.block_number() + 1,
gas_limit: GAS_LIMIT,
gas_used: GAS_USED,
timestamp: pa.timestamp,
extra_data: "block gen was here".as_bytes().to_vec().into(),
base_fee_per_gas: Uint256::one(),
block_hash: ExecutionBlockHash::zero(),
transactions: vec![].into(),
}),
ForkName::Capella => ExecutionPayload::Capella(ExecutionPayloadCapella {
parent_hash: forkchoice_state.head_block_hash,
fee_recipient: pa.suggested_fee_recipient,
receipts_root: Hash256::repeat_byte(42),
state_root: Hash256::repeat_byte(43),
logs_bloom: vec![0; 256].into(),
prev_randao: pa.prev_randao,
block_number: parent.block_number() + 1,
gas_limit: GAS_LIMIT,
gas_used: GAS_USED,
timestamp: pa.timestamp,
extra_data: "block gen was here".as_bytes().to_vec().into(),
base_fee_per_gas: Uint256::one(),
block_hash: ExecutionBlockHash::zero(),
transactions: vec![].into(),
withdrawals: pa.withdrawals.clone().into(),
}),
ForkName::Deneb => ExecutionPayload::Deneb(ExecutionPayloadDeneb {
parent_hash: forkchoice_state.head_block_hash,
fee_recipient: pa.suggested_fee_recipient,
receipts_root: Hash256::repeat_byte(42),
state_root: Hash256::repeat_byte(43),
logs_bloom: vec![0; 256].into(),
prev_randao: pa.prev_randao,
block_number: parent.block_number() + 1,
gas_limit: GAS_LIMIT,
gas_used: GAS_USED,
timestamp: pa.timestamp,
extra_data: "block gen was here".as_bytes().to_vec().into(),
base_fee_per_gas: Uint256::one(),
block_hash: ExecutionBlockHash::zero(),
transactions: vec![].into(),
withdrawals: pa.withdrawals.clone().into(),
data_gas_used: 0,
excess_data_gas: 0,
}),
_ => unreachable!(),
},
};
match execution_payload.fork_name() {
@@ -631,61 +629,22 @@ impl<T: EthSpec> ExecutionBlockGenerator<T> {
pub fn generate_random_blobs<T: EthSpec>(
n_blobs: usize,
kzg: &Kzg,
kzg: &Kzg<T::Kzg>,
) -> Result<(BlobsBundleV1<T>, Transactions<T>), String> {
let mut bundle = BlobsBundleV1::<T>::default();
let mut transactions = vec![];
for blob_index in 0..n_blobs {
// fill a vector with random bytes
let mut blob_bytes = [0u8; BYTES_PER_BLOB];
rand::thread_rng().fill_bytes(&mut blob_bytes);
// Ensure that the blob is canonical by ensuring that
// each field element contained in the blob is < BLS_MODULUS
for i in 0..FIELD_ELEMENTS_PER_BLOB {
blob_bytes[i * BYTES_PER_FIELD_ELEMENT + BYTES_PER_FIELD_ELEMENT - 1] = 0;
}
let random_valid_sidecar = BlobSidecar::<T>::random_valid(&mut thread_rng(), kzg)?;
let blob = Blob::<T>::new(Vec::from(blob_bytes))
.map_err(|e| format!("error constructing random blob: {:?}", e))?;
let BlobSidecar {
blob,
kzg_commitment,
kzg_proof,
..
} = random_valid_sidecar;
let commitment = kzg
.blob_to_kzg_commitment(blob_bytes.into())
.map_err(|e| format!("error computing kzg commitment: {:?}", e))?;
let proof = kzg
.compute_blob_kzg_proof(blob_bytes.into(), commitment)
.map_err(|e| format!("error computing kzg proof: {:?}", e))?;
let versioned_hash = commitment.calculate_versioned_hash();
let blob_transaction = BlobTransaction {
chain_id: Default::default(),
nonce: 0,
max_priority_fee_per_gas: Default::default(),
max_fee_per_gas: Default::default(),
gas: 100000,
to: None,
value: Default::default(),
data: Default::default(),
access_list: Default::default(),
max_fee_per_data_gas: Default::default(),
versioned_hashes: vec![versioned_hash].into(),
};
let bad_signature = EcdsaSignature {
y_parity: false,
r: Uint256::from(0),
s: Uint256::from(0),
};
let signed_blob_transaction = SignedBlobTransaction {
message: blob_transaction,
signature: bad_signature,
};
// calculate transaction bytes
let tx_bytes = [BLOB_TX_TYPE]
.into_iter()
.chain(signed_blob_transaction.as_ssz_bytes().into_iter())
.collect::<Vec<_>>();
let tx = Transaction::<T::MaxBytesPerTransaction>::from(tx_bytes);
let tx = random_valid_tx::<T>()
.map_err(|e| format!("error creating valid tx SSZ bytes: {:?}", e))?;
transactions.push(tx);
bundle
@@ -694,11 +653,11 @@ pub fn generate_random_blobs<T: EthSpec>(
.map_err(|_| format!("blobs are full, blob index: {:?}", blob_index))?;
bundle
.commitments
.push(commitment)
.push(kzg_commitment)
.map_err(|_| format!("blobs are full, blob index: {:?}", blob_index))?;
bundle
.proofs
.push(proof)
.push(kzg_proof)
.map_err(|_| format!("blobs are full, blob index: {:?}", blob_index))?;
}
@@ -709,6 +668,51 @@ fn payload_id_from_u64(n: u64) -> PayloadId {
n.to_le_bytes()
}
pub fn generate_genesis_header<T: EthSpec>(
spec: &ChainSpec,
post_transition_merge: bool,
) -> Option<ExecutionPayloadHeader<T>> {
let genesis_fork = spec.fork_name_at_slot::<T>(spec.genesis_slot);
let genesis_block_hash =
generate_genesis_block(spec.terminal_total_difficulty, DEFAULT_TERMINAL_BLOCK)
.ok()
.map(|block| block.block_hash);
match genesis_fork {
ForkName::Base | ForkName::Altair => None,
ForkName::Merge => {
if post_transition_merge {
let mut header = ExecutionPayloadHeader::Merge(<_>::default());
*header.block_hash_mut() = genesis_block_hash.unwrap_or_default();
Some(header)
} else {
Some(ExecutionPayloadHeader::<T>::Merge(<_>::default()))
}
}
ForkName::Capella => {
let mut header = ExecutionPayloadHeader::Capella(<_>::default());
*header.block_hash_mut() = genesis_block_hash.unwrap_or_default();
Some(header)
}
ForkName::Deneb => {
let mut header = ExecutionPayloadHeader::Capella(<_>::default());
*header.block_hash_mut() = genesis_block_hash.unwrap_or_default();
Some(header)
}
}
}
pub fn generate_genesis_block(
terminal_total_difficulty: Uint256,
terminal_block_number: u64,
) -> Result<PoWBlock, String> {
generate_pow_block(
terminal_total_difficulty,
terminal_block_number,
0,
ExecutionBlockHash::zero(),
)
}
pub fn generate_pow_block(
terminal_total_difficulty: Uint256,
terminal_block_number: u64,

View File

@@ -43,12 +43,12 @@ impl<T: EthSpec> MockExecutionLayer<T> {
executor: TaskExecutor,
terminal_block: u64,
shanghai_time: Option<u64>,
deneb_time: Option<u64>,
cancun_time: Option<u64>,
builder_threshold: Option<u128>,
jwt_key: Option<JwtKey>,
spec: ChainSpec,
builder_url: Option<SensitiveUrl>,
kzg: Option<Kzg>,
kzg: Option<Kzg<T::Kzg>>,
) -> Self {
let handle = executor.handle().unwrap();
@@ -60,7 +60,7 @@ impl<T: EthSpec> MockExecutionLayer<T> {
terminal_block,
spec.terminal_block_hash,
shanghai_time,
deneb_time,
cancun_time,
kzg,
);
@@ -204,7 +204,7 @@ impl<T: EthSpec> MockExecutionLayer<T> {
Some(payload.clone())
);
let status = self.el.notify_new_payload(&payload).await.unwrap();
let status = self.el.notify_new_payload(&payload, None).await.unwrap();
assert_eq!(status, PayloadStatus::Valid);
// Use junk values for slot/head-root to ensure there is no payload supplied.

View File

@@ -25,7 +25,8 @@ use warp::{http::StatusCode, Filter, Rejection};
use crate::EngineCapabilities;
pub use execution_block_generator::{
generate_pow_block, generate_random_blobs, Block, ExecutionBlockGenerator,
generate_genesis_header, generate_pow_block, generate_random_blobs, Block,
ExecutionBlockGenerator,
};
pub use hook::Hook;
pub use mock_builder::{Context as MockBuilderContext, MockBuilder, Operation, TestingBuilder};
@@ -65,7 +66,7 @@ pub struct MockExecutionConfig {
pub terminal_block: u64,
pub terminal_block_hash: ExecutionBlockHash,
pub shanghai_time: Option<u64>,
pub deneb_time: Option<u64>,
pub cancun_time: Option<u64>,
}
impl Default for MockExecutionConfig {
@@ -77,7 +78,7 @@ impl Default for MockExecutionConfig {
terminal_block_hash: ExecutionBlockHash::zero(),
server_config: Config::default(),
shanghai_time: None,
deneb_time: None,
cancun_time: None,
}
}
}
@@ -106,7 +107,7 @@ impl<T: EthSpec> MockServer<T> {
pub fn new_with_config(
handle: &runtime::Handle,
config: MockExecutionConfig,
kzg: Option<Kzg>,
kzg: Option<Kzg<T::Kzg>>,
) -> Self {
let MockExecutionConfig {
jwt_key,
@@ -115,7 +116,7 @@ impl<T: EthSpec> MockServer<T> {
terminal_block_hash,
server_config,
shanghai_time,
deneb_time,
cancun_time,
} = config;
let last_echo_request = Arc::new(RwLock::new(None));
let preloaded_responses = Arc::new(Mutex::new(vec![]));
@@ -124,7 +125,7 @@ impl<T: EthSpec> MockServer<T> {
terminal_block,
terminal_block_hash,
shanghai_time,
deneb_time,
cancun_time,
kzg,
);
@@ -186,8 +187,8 @@ impl<T: EthSpec> MockServer<T> {
terminal_block: u64,
terminal_block_hash: ExecutionBlockHash,
shanghai_time: Option<u64>,
deneb_time: Option<u64>,
kzg: Option<Kzg>,
cancun_time: Option<u64>,
kzg: Option<Kzg<T::Kzg>>,
) -> Self {
Self::new_with_config(
handle,
@@ -198,7 +199,7 @@ impl<T: EthSpec> MockServer<T> {
terminal_block,
terminal_block_hash,
shanghai_time,
deneb_time,
cancun_time,
},
kzg,
)