Devnet6 (#4404)

* some blob reprocessing work

* remove ForceBlockLookup

* reorder enum match arms in sync manager

* a lot more reprocessing work

* impl logic for triggerng blob lookups along with block lookups

* deal with rpc blobs in groups per block in the da checker. don't cache missing blob ids in the da checker.

* make single block lookup generic

* more work

* add delayed processing logic and combine some requests

* start fixing some compile errors

* fix compilation in main block lookup mod

* much work

* get things compiling

* parent blob lookups

* fix compile

* revert red/stevie changes

* fix up sync manager delay message logic

* add peer usefulness enum

* should remove lookup refactor

* consolidate retry error handling

* improve peer scoring during certain failures in parent lookups

* improve retry code

* drop parent lookup if either req has a peer disconnect during download

* refactor single block processed method

* processing peer refactor

* smol bugfix

* fix some todos

* fix lints

* fix lints

* fix compile in lookup tests

* fix lints

* fix lints

* fix existing block lookup tests

* renamings

* fix after merge

* cargo fmt

* compilation fix in beacon chain tests

* fix

* refactor lookup tests to work with multiple forks and response types

* make tests into macros

* wrap availability check error

* fix compile after merge

* add random blobs

* start fixing up lookup verify error handling

* some bug fixes and the start of deneb only tests

* make tests work for all forks

* track information about peer source

* error refactoring

* improve peer scoring

* fix test compilation

* make sure blobs are sent for processing after stream termination, delete copied tests

* add some tests and fix a bug

* smol bugfixes and moar tests

* add tests and fix some things

* compile after merge

* lots of refactoring

* retry on invalid block/blob

* merge unknown parent messages before current slot lookup

* get tests compiling

* penalize blob peer on invalid blobs

* Check disk on in-memory cache miss

* Update beacon_node/beacon_chain/src/data_availability_checker/overflow_lru_cache.rs

* Update beacon_node/network/src/sync/network_context.rs

Co-authored-by: Divma <26765164+divagant-martian@users.noreply.github.com>

* fix bug in matching blocks and blobs in range sync

* pr feedback

* fix conflicts

* upgrade logs from warn to crit when we receive incorrect response in range

* synced_and_connected_within_tolerance -> should_search_for_block

* remove todo

* add data gas used and update excess data gas to u64

* Fix Broken Overflow Tests

* payload verification with commitments

* fix merge conflicts

* restore payload file

* Restore payload file

* remove todo

* add max blob commitments per block

* c-kzg lib update

* Fix ef tests

* Abstract over minimal/mainnet spec in kzg crate

* Start integrating new KZG

* checkpoint sync without alignment

* checkpoint sync without alignment

* add import

* add import

* query for checkpoint state by slot rather than state root (teku doesn't serve by state root)

* query for checkpoint state by slot rather than state root (teku doesn't serve by state root)

* loosen check

* get state first and query by most recent block root

* Revert "loosen check"

This reverts commit 069d13dd63.

* get state first and query by most recent block root

* merge max blobs change

* simplify delay logic

* rename unknown parent sync message variants

* rename parameter, block_slot -> slot

* add some docs to the lookup module

* use interval instead of sleep

* drop request if blocks and blobs requests both return `None` for `Id`

* clean up `find_single_lookup` logic

* add lookup source enum

* clean up `find_single_lookup` logic

* add docs to find_single_lookup_request

* move LookupSource our of param where unnecessary

* remove unnecessary todo

* query for block by `state.latest_block_header.slot`

* fix lint

* fix merge transition ef tests

* fix test

* fix test

* fix observed  blob sidecars test

* Add some metrics (#33)

* fix protocol limits for blobs by root

* Update Engine API for 1:1 Structure Method

* make beacon chain tests to fix devnet 6 changes

* get ckzg working and fix some tests

* fix remaining tests

* fix lints

* Fix KZG linking issues

* remove unused dep

* lockfile

* test fixes

* remove dbgs

* remove unwrap

* cleanup tx generator

* small fixes

* fixing fixes

* more self reivew

* more self review

* refactor genesis header initialization

* refactor mock el instantiations

* fix compile

* fix network test, make sure they run for each fork

* pr feedback

* fix last test (hopefully)

---------

Co-authored-by: Pawan Dhananjay <pawandhananjay@gmail.com>
Co-authored-by: Mark Mackey <mark@sigmaprime.io>
Co-authored-by: Divma <26765164+divagant-martian@users.noreply.github.com>
Co-authored-by: Michael Sproul <michael@sigmaprime.io>

consensus/types/src/beacon_block_body.rs

@@ -9,7 +9,8 @@ use superstruct::superstruct;
 use test_random_derive::TestRandom;
 use tree_hash_derive::TreeHash;
 
-pub type KzgCommitments<T> = VariableList<KzgCommitment, <T as EthSpec>::MaxBlobsPerBlock>;
+pub type KzgCommitments<T> =
+    VariableList<KzgCommitment, <T as EthSpec>::MaxBlobCommitmentsPerBlock>;
 
 /// The body of a `BeaconChain` block, containing operations.
 ///

consensus/types/src/blob_sidecar.rs

@@ -2,7 +2,8 @@ use crate::test_utils::TestRandom;
 use crate::{Blob, ChainSpec, Domain, EthSpec, Fork, Hash256, SignedBlobSidecar, SignedRoot, Slot};
 use bls::SecretKey;
 use derivative::Derivative;
-use kzg::{KzgCommitment, KzgProof};
+use kzg::{Kzg, KzgCommitment, KzgPreset, KzgProof};
+use rand::Rng;
 use serde_derive::{Deserialize, Serialize};
 use ssz::Encode;
 use ssz_derive::{Decode, Encode};
@@ -93,6 +94,38 @@ impl<T: EthSpec> BlobSidecar<T> {
         Self::default()
     }
 
+    pub fn random_valid<R: Rng>(rng: &mut R, kzg: &Kzg<T::Kzg>) -> Result<Self, String> {
+        let mut blob_bytes = vec![0u8; T::Kzg::BYTES_PER_BLOB];
+        rng.fill_bytes(&mut blob_bytes);
+        // Ensure that the blob is canonical by ensuring that
+        // each field element contained in the blob is < BLS_MODULUS
+        for i in 0..T::Kzg::FIELD_ELEMENTS_PER_BLOB {
+            let Some(byte) = blob_bytes.get_mut(i.checked_mul(T::Kzg::BYTES_PER_FIELD_ELEMENT).ok_or("overflow".to_string())?)  else {
+                return Err(format!("blob byte index out of bounds: {:?}", i));
+            };
+            *byte = 0;
+        }
+
+        let blob = Blob::<T>::new(blob_bytes)
+            .map_err(|e| format!("error constructing random blob: {:?}", e))?;
+        let kzg_blob = T::blob_from_bytes(&blob).unwrap();
+
+        let commitment = kzg
+            .blob_to_kzg_commitment(kzg_blob.clone())
+            .map_err(|e| format!("error computing kzg commitment: {:?}", e))?;
+
+        let proof = kzg
+            .compute_blob_kzg_proof(kzg_blob, commitment)
+            .map_err(|e| format!("error computing kzg proof: {:?}", e))?;
+
+        Ok(Self {
+            blob,
+            kzg_commitment: commitment,
+            kzg_proof: proof,
+            ..Default::default()
+        })
+    }
+
     #[allow(clippy::integer_arithmetic)]
     pub fn max_size() -> usize {
         // Fixed part

consensus/types/src/consts.rs

@@ -34,7 +34,6 @@ pub mod deneb {
         .expect("should initialize BLS_MODULUS");
         pub static ref MIN_EPOCHS_FOR_BLOB_SIDECARS_REQUESTS: Epoch = Epoch::from(4096_u64);
     }
-    pub const BLOB_TX_TYPE: u8 = 3;
     pub const VERSIONED_HASH_VERSION_KZG: u8 = 1;
     pub const BLOB_SIDECAR_SUBNET_COUNT: u64 = 6;
 }

consensus/types/src/eth_spec.rs

@@ -1,5 +1,6 @@
 use crate::*;
 
+use kzg::{BlobTrait, KzgPreset, MainnetKzgPreset, MinimalKzgPreset};
 use safe_arith::SafeArith;
 use serde_derive::{Deserialize, Serialize};
 use ssz_types::typenum::{
@@ -51,6 +52,8 @@ impl fmt::Display for EthSpecId {
 pub trait EthSpec:
     'static + Default + Sync + Send + Clone + Debug + PartialEq + Eq + for<'a> arbitrary::Arbitrary<'a>
 {
+    type Kzg: KzgPreset;
+
     /*
      * Constants
      */
@@ -106,6 +109,7 @@ pub trait EthSpec:
      * New in Deneb
      */
     type MaxBlobsPerBlock: Unsigned + Clone + Sync + Send + Debug + PartialEq + Unpin;
+    type MaxBlobCommitmentsPerBlock: Unsigned + Clone + Sync + Send + Debug + PartialEq + Unpin;
     type FieldElementsPerBlob: Unsigned + Clone + Sync + Send + Debug + PartialEq;
     type BytesPerFieldElement: Unsigned + Clone + Sync + Send + Debug + PartialEq;
     /*
@@ -256,6 +260,10 @@ pub trait EthSpec:
         Self::MaxBlobsPerBlock::to_usize()
     }
 
+    fn blob_from_bytes(bytes: &[u8]) -> Result<<Self::Kzg as KzgPreset>::Blob, kzg::Error> {
+        <Self::Kzg as KzgPreset>::Blob::from_bytes(bytes)
+    }
+
     /// Returns the `BYTES_PER_BLOB` constant for this specification.
     fn bytes_per_blob() -> usize {
         Self::BytesPerBlob::to_usize()
@@ -275,6 +283,8 @@ macro_rules! params_from_eth_spec {
 pub struct MainnetEthSpec;
 
 impl EthSpec for MainnetEthSpec {
+    type Kzg = MainnetKzgPreset;
+
     type JustificationBitsLength = U4;
     type SubnetBitfieldLength = U64;
     type MaxValidatorsPerCommittee = U2048;
@@ -300,6 +310,7 @@ impl EthSpec for MainnetEthSpec {
     type MinGasLimit = U5000;
     type MaxExtraDataBytes = U32;
     type MaxBlobsPerBlock = U6;
+    type MaxBlobCommitmentsPerBlock = U4096;
     type BytesPerFieldElement = U32;
     type FieldElementsPerBlob = U4096;
     type BytesPerBlob = U131072;
@@ -323,6 +334,8 @@ impl EthSpec for MainnetEthSpec {
 pub struct MinimalEthSpec;
 
 impl EthSpec for MinimalEthSpec {
+    type Kzg = MinimalKzgPreset;
+
     type SlotsPerEpoch = U8;
     type EpochsPerEth1VotingPeriod = U4;
     type SlotsPerHistoricalRoot = U64;
@@ -335,6 +348,7 @@ impl EthSpec for MinimalEthSpec {
     type MaxWithdrawalsPerPayload = U4;
     type FieldElementsPerBlob = U4; //FIXME(sean) this is spec'd out currently but will likely change
     type BytesPerBlob = U128; //FIXME(sean) this is spec'd out currently but will likely change
+    type MaxBlobCommitmentsPerBlock = U16;
 
     params_from_eth_spec!(MainnetEthSpec {
         JustificationBitsLength,
@@ -374,6 +388,8 @@ impl EthSpec for MinimalEthSpec {
 pub struct GnosisEthSpec;
 
 impl EthSpec for GnosisEthSpec {
+    type Kzg = MainnetKzgPreset;
+
     type JustificationBitsLength = U4;
     type SubnetBitfieldLength = U64;
     type MaxValidatorsPerCommittee = U2048;
@@ -404,6 +420,7 @@ impl EthSpec for GnosisEthSpec {
     type MaxBlsToExecutionChanges = U16;
     type MaxWithdrawalsPerPayload = U8;
     type MaxBlobsPerBlock = U6;
+    type MaxBlobCommitmentsPerBlock = U4096;
     type FieldElementsPerBlob = U4096;
     type BytesPerFieldElement = U32;
     type BytesPerBlob = U131072;

consensus/types/src/execution_block_header.rs

@@ -26,6 +26,7 @@ use metastruct::metastruct;
 #[derive(Debug, Clone, PartialEq, Eq, Hash)]
 #[metastruct(mappings(map_execution_block_header_fields_except_withdrawals(exclude(
     withdrawals_root,
+    data_gas_used,
     excess_data_gas
 )),))]
 pub struct ExecutionBlockHeader {
@@ -46,7 +47,8 @@ pub struct ExecutionBlockHeader {
     pub nonce: Hash64,
     pub base_fee_per_gas: Uint256,
     pub withdrawals_root: Option<Hash256>,
-    pub excess_data_gas: Option<Uint256>,
+    pub data_gas_used: Option<u64>,
+    pub excess_data_gas: Option<u64>,
 }
 
 impl ExecutionBlockHeader {
@@ -55,7 +57,8 @@ impl ExecutionBlockHeader {
         rlp_empty_list_root: Hash256,
         rlp_transactions_root: Hash256,
         rlp_withdrawals_root: Option<Hash256>,
-        rlp_excess_data_gas: Option<Uint256>,
+        rlp_data_gas_used: Option<u64>,
+        rlp_excess_data_gas: Option<u64>,
     ) -> Self {
         // Most of these field mappings are defined in EIP-3675 except for `mixHash`, which is
         // defined in EIP-4399.
@@ -77,6 +80,7 @@ impl ExecutionBlockHeader {
             nonce: Hash64::zero(),
             base_fee_per_gas: payload.base_fee_per_gas(),
             withdrawals_root: rlp_withdrawals_root,
+            data_gas_used: rlp_data_gas_used,
             excess_data_gas: rlp_excess_data_gas,
         }
     }

consensus/types/src/execution_payload.rs

@@ -84,9 +84,13 @@ pub struct ExecutionPayload<T: EthSpec> {
     #[superstruct(only(Capella, Deneb))]
     pub withdrawals: Withdrawals<T>,
     #[superstruct(only(Deneb))]
-    #[serde(with = "serde_utils::quoted_u256")]
+    #[serde(with = "serde_utils::quoted_u64")]
     #[superstruct(getter(copy))]
-    pub excess_data_gas: Uint256,
+    pub data_gas_used: u64,
+    #[superstruct(only(Deneb))]
+    #[serde(with = "serde_utils::quoted_u64")]
+    #[superstruct(getter(copy))]
+    pub excess_data_gas: u64,
 }
 
 impl<'a, T: EthSpec> ExecutionPayloadRef<'a, T> {

consensus/types/src/execution_payload_header.rs

@@ -78,9 +78,13 @@ pub struct ExecutionPayloadHeader<T: EthSpec> {
     #[superstruct(getter(copy))]
     pub withdrawals_root: Hash256,
     #[superstruct(only(Deneb))]
-    #[serde(with = "serde_utils::quoted_u256")]
+    #[serde(with = "serde_utils::quoted_u64")]
     #[superstruct(getter(copy))]
-    pub excess_data_gas: Uint256,
+    pub data_gas_used: u64,
+    #[superstruct(only(Deneb))]
+    #[serde(with = "serde_utils::quoted_u64")]
+    #[superstruct(getter(copy))]
+    pub excess_data_gas: u64,
 }
 
 impl<T: EthSpec> ExecutionPayloadHeader<T> {
@@ -151,8 +155,8 @@ impl<T: EthSpec> ExecutionPayloadHeaderCapella<T> {
             block_hash: self.block_hash,
             transactions_root: self.transactions_root,
             withdrawals_root: self.withdrawals_root,
-            // TODO: verify if this is correct
-            excess_data_gas: Uint256::zero(),
+            data_gas_used: 0,
+            excess_data_gas: 0,
         }
     }
 }
@@ -217,6 +221,7 @@ impl<'a, T: EthSpec> From<&'a ExecutionPayloadDeneb<T>> for ExecutionPayloadHead
             block_hash: payload.block_hash,
             transactions_root: payload.transactions.tree_hash_root(),
             withdrawals_root: payload.withdrawals.tree_hash_root(),
+            data_gas_used: payload.data_gas_used,
             excess_data_gas: payload.excess_data_gas,
         }
     }

consensus/types/src/lib.rs

@@ -99,7 +99,6 @@ pub mod sqlite;
 
 pub mod blob_sidecar;
 pub mod signed_blob;
-pub mod transaction;
 
 use ethereum_types::{H160, H256};
 

consensus/types/src/transaction.rs

@@ -1,44 +0,0 @@
-use crate::{Hash256, Uint256, VersionedHash};
-use ethereum_types::Address;
-use ssz_derive::{Decode, Encode};
-use ssz_types::typenum::{U16777216, U4096};
-use ssz_types::VariableList;
-
-pub type MaxCalldataSize = U16777216;
-pub type MaxAccessListSize = U16777216;
-pub type MaxAccessListStorageKeys = U16777216;
-pub type MaxVersionedHashesListSize = U4096;
-
-#[derive(Debug, Clone, PartialEq, Encode, Decode)]
-pub struct SignedBlobTransaction {
-    pub message: BlobTransaction,
-    pub signature: EcdsaSignature,
-}
-
-#[derive(Debug, Clone, PartialEq, Encode, Decode)]
-pub struct BlobTransaction {
-    pub chain_id: Uint256,
-    pub nonce: u64,
-    pub max_priority_fee_per_gas: Uint256,
-    pub max_fee_per_gas: Uint256,
-    pub gas: u64,
-    pub to: Option<Address>,
-    pub value: Uint256,
-    pub data: VariableList<u8, MaxCalldataSize>,
-    pub access_list: VariableList<AccessTuple, MaxAccessListSize>,
-    pub max_fee_per_data_gas: Uint256,
-    pub versioned_hashes: VariableList<VersionedHash, MaxVersionedHashesListSize>,
-}
-
-#[derive(Debug, Clone, PartialEq, Encode, Decode)]
-pub struct AccessTuple {
-    pub address: Address,
-    pub storage_keys: VariableList<Hash256, MaxAccessListStorageKeys>,
-}
-
-#[derive(Debug, Clone, PartialEq, Encode, Decode)]
-pub struct EcdsaSignature {
-    pub y_parity: bool,
-    pub r: Uint256,
-    pub s: Uint256,
-}