Devnet6 (#4404)

* some blob reprocessing work

* remove ForceBlockLookup

* reorder enum match arms in sync manager

* a lot more reprocessing work

* impl logic for triggerng blob lookups along with block lookups

* deal with rpc blobs in groups per block in the da checker. don't cache missing blob ids in the da checker.

* make single block lookup generic

* more work

* add delayed processing logic and combine some requests

* start fixing some compile errors

* fix compilation in main block lookup mod

* much work

* get things compiling

* parent blob lookups

* fix compile

* revert red/stevie changes

* fix up sync manager delay message logic

* add peer usefulness enum

* should remove lookup refactor

* consolidate retry error handling

* improve peer scoring during certain failures in parent lookups

* improve retry code

* drop parent lookup if either req has a peer disconnect during download

* refactor single block processed method

* processing peer refactor

* smol bugfix

* fix some todos

* fix lints

* fix lints

* fix compile in lookup tests

* fix lints

* fix lints

* fix existing block lookup tests

* renamings

* fix after merge

* cargo fmt

* compilation fix in beacon chain tests

* fix

* refactor lookup tests to work with multiple forks and response types

* make tests into macros

* wrap availability check error

* fix compile after merge

* add random blobs

* start fixing up lookup verify error handling

* some bug fixes and the start of deneb only tests

* make tests work for all forks

* track information about peer source

* error refactoring

* improve peer scoring

* fix test compilation

* make sure blobs are sent for processing after stream termination, delete copied tests

* add some tests and fix a bug

* smol bugfixes and moar tests

* add tests and fix some things

* compile after merge

* lots of refactoring

* retry on invalid block/blob

* merge unknown parent messages before current slot lookup

* get tests compiling

* penalize blob peer on invalid blobs

* Check disk on in-memory cache miss

* Update beacon_node/beacon_chain/src/data_availability_checker/overflow_lru_cache.rs

* Update beacon_node/network/src/sync/network_context.rs

Co-authored-by: Divma <26765164+divagant-martian@users.noreply.github.com>

* fix bug in matching blocks and blobs in range sync

* pr feedback

* fix conflicts

* upgrade logs from warn to crit when we receive incorrect response in range

* synced_and_connected_within_tolerance -> should_search_for_block

* remove todo

* add data gas used and update excess data gas to u64

* Fix Broken Overflow Tests

* payload verification with commitments

* fix merge conflicts

* restore payload file

* Restore payload file

* remove todo

* add max blob commitments per block

* c-kzg lib update

* Fix ef tests

* Abstract over minimal/mainnet spec in kzg crate

* Start integrating new KZG

* checkpoint sync without alignment

* checkpoint sync without alignment

* add import

* add import

* query for checkpoint state by slot rather than state root (teku doesn't serve by state root)

* query for checkpoint state by slot rather than state root (teku doesn't serve by state root)

* loosen check

* get state first and query by most recent block root

* Revert "loosen check"

This reverts commit 069d13dd63.

* get state first and query by most recent block root

* merge max blobs change

* simplify delay logic

* rename unknown parent sync message variants

* rename parameter, block_slot -> slot

* add some docs to the lookup module

* use interval instead of sleep

* drop request if blocks and blobs requests both return `None` for `Id`

* clean up `find_single_lookup` logic

* add lookup source enum

* clean up `find_single_lookup` logic

* add docs to find_single_lookup_request

* move LookupSource our of param where unnecessary

* remove unnecessary todo

* query for block by `state.latest_block_header.slot`

* fix lint

* fix merge transition ef tests

* fix test

* fix test

* fix observed  blob sidecars test

* Add some metrics (#33)

* fix protocol limits for blobs by root

* Update Engine API for 1:1 Structure Method

* make beacon chain tests to fix devnet 6 changes

* get ckzg working and fix some tests

* fix remaining tests

* fix lints

* Fix KZG linking issues

* remove unused dep

* lockfile

* test fixes

* remove dbgs

* remove unwrap

* cleanup tx generator

* small fixes

* fixing fixes

* more self reivew

* more self review

* refactor genesis header initialization

* refactor mock el instantiations

* fix compile

* fix network test, make sure they run for each fork

* pr feedback

* fix last test (hopefully)

---------

Co-authored-by: Pawan Dhananjay <pawandhananjay@gmail.com>
Co-authored-by: Mark Mackey <mark@sigmaprime.io>
Co-authored-by: Divma <26765164+divagant-martian@users.noreply.github.com>
Co-authored-by: Michael Sproul <michael@sigmaprime.io>

consensus/state_processing/src/per_block_processing.rs

@@ -13,7 +13,6 @@ pub use self::verify_attester_slashing::{
 pub use self::verify_proposer_slashing::verify_proposer_slashing;
 pub use altair::sync_committee::process_sync_aggregate;
 pub use block_signature_verifier::{BlockSignatureVerifier, ParallelSignatureSets};
-pub use deneb::deneb::process_blob_kzg_commitments;
 pub use is_valid_indexed_attestation::is_valid_indexed_attestation;
 pub use process_operations::process_operations;
 pub use verify_attestation::{
@@ -163,11 +162,11 @@ pub fn per_block_processing<T: EthSpec, Payload: AbstractExecPayload<T>>(
     // `process_randao` as the former depends on the `randao_mix` computed with the reveal of the
     // previous block.
     if is_execution_enabled(state, block.body()) {
-        let payload = block.body().execution_payload()?;
+        let body = block.body();
         if state_processing_strategy == StateProcessingStrategy::Accurate {
-            process_withdrawals::<T, Payload>(state, payload, spec)?;
+            process_withdrawals::<T, Payload>(state, body.execution_payload()?, spec)?;
         }
-        process_execution_payload::<T, Payload>(state, payload, spec)?;
+        process_execution_payload::<T, Payload>(state, body, spec)?;
     }
 
     process_randao(state, block, verify_randao, ctxt, spec)?;
@@ -184,8 +183,6 @@ pub fn per_block_processing<T: EthSpec, Payload: AbstractExecPayload<T>>(
         )?;
     }
 
-    process_blob_kzg_commitments(block.body(), ctxt)?;
-
     Ok(())
 }
 
@@ -350,9 +347,10 @@ pub fn get_new_eth1_data<T: EthSpec>(
 pub fn partially_verify_execution_payload<T: EthSpec, Payload: AbstractExecPayload<T>>(
     state: &BeaconState<T>,
     block_slot: Slot,
-    payload: Payload::Ref<'_>,
+    body: BeaconBlockBodyRef<T, Payload>,
     spec: &ChainSpec,
 ) -> Result<(), BlockProcessingError> {
+    let payload = body.execution_payload()?;
     if is_merge_transition_complete(state) {
         block_verify!(
             payload.parent_hash() == state.latest_execution_payload_header()?.block_hash(),
@@ -379,6 +377,17 @@ pub fn partially_verify_execution_payload<T: EthSpec, Payload: AbstractExecPaylo
         }
     );
 
+    if let Ok(blob_commitments) = body.blob_kzg_commitments() {
+        // Verify commitments are under the limit.
+        block_verify!(
+            blob_commitments.len() <= T::max_blobs_per_block(),
+            BlockProcessingError::ExecutionInvalidBlobsLen {
+                max: T::max_blobs_per_block(),
+                actual: blob_commitments.len(),
+            }
+        );
+    }
+
     Ok(())
 }
 
@@ -391,11 +400,11 @@ pub fn partially_verify_execution_payload<T: EthSpec, Payload: AbstractExecPaylo
 /// https://github.com/ethereum/consensus-specs/blob/v1.1.5/specs/merge/beacon-chain.md#process_execution_payload
 pub fn process_execution_payload<T: EthSpec, Payload: AbstractExecPayload<T>>(
     state: &mut BeaconState<T>,
-    payload: Payload::Ref<'_>,
+    body: BeaconBlockBodyRef<T, Payload>,
     spec: &ChainSpec,
 ) -> Result<(), BlockProcessingError> {
-    partially_verify_execution_payload::<T, Payload>(state, state.slot(), payload, spec)?;
-
+    partially_verify_execution_payload::<T, Payload>(state, state.slot(), body, spec)?;
+    let payload = body.execution_payload()?;
     match state.latest_execution_payload_header_mut()? {
         ExecutionPayloadHeaderRefMut::Merge(header_mut) => {
             match payload.to_execution_payload_header() {
@@ -423,15 +432,19 @@ pub fn process_execution_payload<T: EthSpec, Payload: AbstractExecPayload<T>>(
 /// These functions will definitely be called before the merge. Their entire purpose is to check if
 /// the merge has happened or if we're on the transition block. Thus we don't want to propagate
 /// errors from the `BeaconState` being an earlier variant than `BeaconStateMerge` as we'd have to
-/// repeaetedly write code to treat these errors as false.
+/// repeatedly write code to treat these errors as false.
 /// https://github.com/ethereum/consensus-specs/blob/dev/specs/bellatrix/beacon-chain.md#is_merge_transition_complete
 pub fn is_merge_transition_complete<T: EthSpec>(state: &BeaconState<T>) -> bool {
-    // We must check defaultness against the payload header with 0x0 roots, as that's what's meant
-    // by `ExecutionPayloadHeader()` in the spec.
-    state
-        .latest_execution_payload_header()
-        .map(|header| !header.is_default_with_zero_roots())
-        .unwrap_or(false)
+    match state {
+        // We must check defaultness against the payload header with 0x0 roots, as that's what's meant
+        // by `ExecutionPayloadHeader()` in the spec.
+        BeaconState::Merge(_) => state
+            .latest_execution_payload_header()
+            .map(|header| !header.is_default_with_zero_roots())
+            .unwrap_or(false),
+        BeaconState::Deneb(_) | BeaconState::Capella(_) => true,
+        BeaconState::Base(_) | BeaconState::Altair(_) => false,
+    }
 }
 /// https://github.com/ethereum/consensus-specs/blob/dev/specs/bellatrix/beacon-chain.md#is_merge_transition_block
 pub fn is_merge_transition_block<T: EthSpec, Payload: AbstractExecPayload<T>>(