Devnet6 (#4404)

* some blob reprocessing work * remove ForceBlockLookup * reorder enum match arms in sync manager * a lot more reprocessing work * impl logic for triggerng blob lookups along with block lookups * deal with rpc blobs in groups per block in the da checker. don't cache missing blob ids in the da checker. * make single block lookup generic * more work * add delayed processing logic and combine some requests * start fixing some compile errors * fix compilation in main block lookup mod * much work * get things compiling * parent blob lookups * fix compile * revert red/stevie changes * fix up sync manager delay message logic * add peer usefulness enum * should remove lookup refactor * consolidate retry error handling * improve peer scoring during certain failures in parent lookups * improve retry code * drop parent lookup if either req has a peer disconnect during download * refactor single block processed method * processing peer refactor * smol bugfix * fix some todos * fix lints * fix lints * fix compile in lookup tests * fix lints * fix lints * fix existing block lookup tests * renamings * fix after merge * cargo fmt * compilation fix in beacon chain tests * fix * refactor lookup tests to work with multiple forks and response types * make tests into macros * wrap availability check error * fix compile after merge * add random blobs * start fixing up lookup verify error handling * some bug fixes and the start of deneb only tests * make tests work for all forks * track information about peer source * error refactoring * improve peer scoring * fix test compilation * make sure blobs are sent for processing after stream termination, delete copied tests * add some tests and fix a bug * smol bugfixes and moar tests * add tests and fix some things * compile after merge * lots of refactoring * retry on invalid block/blob * merge unknown parent messages before current slot lookup * get tests compiling * penalize blob peer on invalid blobs * Check disk on in-memory cache miss * Update beacon_node/beacon_chain/src/data_availability_checker/overflow_lru_cache.rs * Update beacon_node/network/src/sync/network_context.rs Co-authored-by: Divma <26765164+divagant-martian@users.noreply.github.com> * fix bug in matching blocks and blobs in range sync * pr feedback * fix conflicts * upgrade logs from warn to crit when we receive incorrect response in range * synced_and_connected_within_tolerance -> should_search_for_block * remove todo * add data gas used and update excess data gas to u64 * Fix Broken Overflow Tests * payload verification with commitments * fix merge conflicts * restore payload file * Restore payload file * remove todo * add max blob commitments per block * c-kzg lib update * Fix ef tests * Abstract over minimal/mainnet spec in kzg crate * Start integrating new KZG * checkpoint sync without alignment * checkpoint sync without alignment * add import * add import * query for checkpoint state by slot rather than state root (teku doesn't serve by state root) * query for checkpoint state by slot rather than state root (teku doesn't serve by state root) * loosen check * get state first and query by most recent block root * Revert "loosen check" This reverts commit 069d13dd63. * get state first and query by most recent block root * merge max blobs change * simplify delay logic * rename unknown parent sync message variants * rename parameter, block_slot -> slot * add some docs to the lookup module * use interval instead of sleep * drop request if blocks and blobs requests both return `None` for `Id` * clean up `find_single_lookup` logic * add lookup source enum * clean up `find_single_lookup` logic * add docs to find_single_lookup_request * move LookupSource our of param where unnecessary * remove unnecessary todo * query for block by `state.latest_block_header.slot` * fix lint * fix merge transition ef tests * fix test * fix test * fix observed blob sidecars test * Add some metrics (#33) * fix protocol limits for blobs by root * Update Engine API for 1:1 Structure Method * make beacon chain tests to fix devnet 6 changes * get ckzg working and fix some tests * fix remaining tests * fix lints * Fix KZG linking issues * remove unused dep * lockfile * test fixes * remove dbgs * remove unwrap * cleanup tx generator * small fixes * fixing fixes * more self reivew * more self review * refactor genesis header initialization * refactor mock el instantiations * fix compile * fix network test, make sure they run for each fork * pr feedback * fix last test (hopefully) --------- Co-authored-by: Pawan Dhananjay <pawandhananjay@gmail.com> Co-authored-by: Mark Mackey <mark@sigmaprime.io> Co-authored-by: Divma <26765164+divagant-martian@users.noreply.github.com> Co-authored-by: Michael Sproul <michael@sigmaprime.io>
2026-03-09 19:51:47 +00:00 · 2023-06-29 15:35:43 -04:00
parent 4c9fcf1e83
commit adbb62f7f3
69 changed files with 2114 additions and 1338 deletions
--- a/testing/ef_tests/src/cases/kzg_blob_to_kzg_commitment.rs
+++ b/testing/ef_tests/src/cases/kzg_blob_to_kzg_commitment.rs
@@ -31,7 +31,7 @@ impl<E: EthSpec> Case for KZGBlobToKZGCommitment<E> {
    }

    fn result(&self, _case_index: usize, _fork_name: ForkName) -> Result<(), Error> {
-        let kzg = get_kzg()?;
+        let kzg = get_kzg::<E::Kzg>()?;

        let commitment = parse_blob::<E>(&self.input.blob).and_then(|blob| {
            blob_to_kzg_commitment::<E>(&kzg, blob).map_err(|e| {
--- a/testing/ef_tests/src/cases/kzg_compute_blob_kzg_proof.rs
+++ b/testing/ef_tests/src/cases/kzg_compute_blob_kzg_proof.rs
@@ -38,7 +38,7 @@ impl<E: EthSpec> Case for KZGComputeBlobKZGProof<E> {
            Ok((blob, commitment))
        };

-        let kzg = get_kzg()?;
+        let kzg = get_kzg::<E::Kzg>()?;
        let proof = parse_input(&self.input).and_then(|(blob, commitment)| {
            compute_blob_kzg_proof::<E>(&kzg, &blob, commitment)
                .map_err(|e| Error::InternalError(format!("Failed to compute kzg proof: {:?}", e)))
--- a/testing/ef_tests/src/cases/kzg_compute_kzg_proof.rs
+++ b/testing/ef_tests/src/cases/kzg_compute_kzg_proof.rs
@@ -45,7 +45,7 @@ impl<E: EthSpec> Case for KZGComputeKZGProof<E> {
            Ok((blob, z))
        };

-        let kzg = get_kzg()?;
+        let kzg = get_kzg::<E::Kzg>()?;
        let proof = parse_input(&self.input).and_then(|(blob, z)| {
            compute_kzg_proof::<E>(&kzg, blob, z)
                .map_err(|e| Error::InternalError(format!("Failed to compute kzg proof: {:?}", e)))
--- a/testing/ef_tests/src/cases/kzg_verify_blob_kzg_proof.rs
+++ b/testing/ef_tests/src/cases/kzg_verify_blob_kzg_proof.rs
@@ -1,15 +1,15 @@
 use super::*;
 use crate::case_result::compare_result;
 use beacon_chain::kzg_utils::validate_blob;
-use eth2_network_config::TRUSTED_SETUP;
-use kzg::{Kzg, KzgCommitment, KzgProof, TrustedSetup};
+use eth2_network_config::get_trusted_setup;
+use kzg::{Kzg, KzgCommitment, KzgPreset, KzgProof, TrustedSetup};
 use serde_derive::Deserialize;
 use std::convert::TryInto;
 use std::marker::PhantomData;
 use types::Blob;

-pub fn get_kzg() -> Result<Kzg, Error> {
-    let trusted_setup: TrustedSetup = serde_json::from_reader(TRUSTED_SETUP)
+pub fn get_kzg<P: KzgPreset>() -> Result<Kzg<P>, Error> {
+    let trusted_setup: TrustedSetup = serde_json::from_reader(get_trusted_setup::<P>())
        .map_err(|e| Error::InternalError(format!("Failed to initialize kzg: {:?}", e)))?;
    Kzg::new_from_trusted_setup(trusted_setup)
        .map_err(|e| Error::InternalError(format!("Failed to initialize kzg: {:?}", e)))
@@ -81,7 +81,7 @@ impl<E: EthSpec> Case for KZGVerifyBlobKZGProof<E> {
            Ok((blob, commitment, proof))
        };

-        let kzg = get_kzg()?;
+        let kzg = get_kzg::<E::Kzg>()?;
        let result = parse_input(&self.input).and_then(|(blob, commitment, proof)| {
            validate_blob::<E>(&kzg, blob, commitment, proof)
                .map_err(|e| Error::InternalError(format!("Failed to validate blob: {:?}", e)))
--- a/testing/ef_tests/src/cases/kzg_verify_blob_kzg_proof_batch.rs
+++ b/testing/ef_tests/src/cases/kzg_verify_blob_kzg_proof_batch.rs
@@ -51,7 +51,7 @@ impl<E: EthSpec> Case for KZGVerifyBlobKZGProofBatch<E> {
            Ok((commitments, blobs, proofs))
        };

-        let kzg = get_kzg()?;
+        let kzg = get_kzg::<E::Kzg>()?;
        let result = parse_input(&self.input).and_then(|(commitments, blobs, proofs)| {
            validate_blobs::<E>(&kzg, &commitments, &blobs, &proofs)
                .map_err(|e| Error::InternalError(format!("Failed to validate blobs: {:?}", e)))
--- a/testing/ef_tests/src/cases/kzg_verify_kzg_proof.rs
+++ b/testing/ef_tests/src/cases/kzg_verify_kzg_proof.rs
@@ -41,7 +41,7 @@ impl<E: EthSpec> Case for KZGVerifyKZGProof<E> {
            Ok((commitment, z, y, proof))
        };

-        let kzg = get_kzg()?;
+        let kzg = get_kzg::<E::Kzg>()?;
        let result = parse_input(&self.input).and_then(|(commitment, z, y, proof)| {
            verify_kzg_proof::<E>(&kzg, commitment, proof, z, y)
                .map_err(|e| Error::InternalError(format!("Failed to validate proof: {:?}", e)))
--- a/testing/ef_tests/src/cases/operations.rs
+++ b/testing/ef_tests/src/cases/operations.rs
@@ -4,6 +4,7 @@ use crate::case_result::compare_beacon_state_results_without_caches;
 use crate::decode::{ssz_decode_file, ssz_decode_file_with, ssz_decode_state, yaml_decode_file};
 use crate::testing_spec;
 use serde_derive::Deserialize;
+use ssz::Decode;
 use state_processing::{
    per_block_processing::{
        errors::BlockProcessingError,
@@ -19,7 +20,8 @@ use state_processing::{
 use std::fmt::Debug;
 use std::path::Path;
 use types::{
-    Attestation, AttesterSlashing, BeaconBlock, BeaconState, BlindedPayload, ChainSpec, Deposit,
+    Attestation, AttesterSlashing, BeaconBlock, BeaconBlockBody, BeaconBlockBodyCapella,
+    BeaconBlockBodyDeneb, BeaconBlockBodyMerge, BeaconState, BlindedPayload, ChainSpec, Deposit,
    EthSpec, ExecutionPayload, ForkName, FullPayload, ProposerSlashing, SignedBlsToExecutionChange,
    SignedVoluntaryExit, SyncAggregate,
 };
@@ -259,13 +261,13 @@ impl<E: EthSpec> Operation<E> for SyncAggregate<E> {
    }
 }

-impl<E: EthSpec> Operation<E> for FullPayload<E> {
+impl<E: EthSpec> Operation<E> for BeaconBlockBody<E, FullPayload<E>> {
    fn handler_name() -> String {
        "execution_payload".into()
    }

    fn filename() -> String {
-        "execution_payload.ssz_snappy".into()
+        "body.ssz_snappy".into()
    }

    fn is_enabled_for_fork(fork_name: ForkName) -> bool {
@@ -274,9 +276,13 @@ impl<E: EthSpec> Operation<E> for FullPayload<E> {

    fn decode(path: &Path, fork_name: ForkName, _spec: &ChainSpec) -> Result<Self, Error> {
        ssz_decode_file_with(path, |bytes| {
-            ExecutionPayload::from_ssz_bytes(bytes, fork_name)
+            Ok(match fork_name {
+                ForkName::Merge => BeaconBlockBody::Merge(<_>::from_ssz_bytes(bytes)?),
+                ForkName::Capella => BeaconBlockBody::Capella(<_>::from_ssz_bytes(bytes)?),
+                ForkName::Deneb => BeaconBlockBody::Deneb(<_>::from_ssz_bytes(bytes)?),
+                _ => panic!(),
+            })
        })
-        .map(Into::into)
    }

    fn apply_to(
@@ -296,13 +302,13 @@ impl<E: EthSpec> Operation<E> for FullPayload<E> {
        }
    }
 }
-impl<E: EthSpec> Operation<E> for BlindedPayload<E> {
+impl<E: EthSpec> Operation<E> for BeaconBlockBody<E, BlindedPayload<E>> {
    fn handler_name() -> String {
        "execution_payload".into()
    }

    fn filename() -> String {
-        "execution_payload.ssz_snappy".into()
+        "body.ssz_snappy".into()
    }

    fn is_enabled_for_fork(fork_name: ForkName) -> bool {
@@ -311,9 +317,22 @@ impl<E: EthSpec> Operation<E> for BlindedPayload<E> {

    fn decode(path: &Path, fork_name: ForkName, _spec: &ChainSpec) -> Result<Self, Error> {
        ssz_decode_file_with(path, |bytes| {
-            ExecutionPayload::from_ssz_bytes(bytes, fork_name)
+            Ok(match fork_name {
+                ForkName::Merge => {
+                    let inner = <BeaconBlockBodyMerge<E, FullPayload<E>>>::from_ssz_bytes(bytes)?;
+                    BeaconBlockBody::Merge(inner.clone_as_blinded())
+                }
+                ForkName::Capella => {
+                    let inner = <BeaconBlockBodyCapella<E, FullPayload<E>>>::from_ssz_bytes(bytes)?;
+                    BeaconBlockBody::Capella(inner.clone_as_blinded())
+                }
+                ForkName::Deneb => {
+                    let inner = <BeaconBlockBodyDeneb<E, FullPayload<E>>>::from_ssz_bytes(bytes)?;
+                    BeaconBlockBody::Deneb(inner.clone_as_blinded())
+                }
+                _ => panic!(),
+            })
        })
-        .map(Into::into)
    }

    fn apply_to(