Electra: Remaining Consensus Data Structures (#5712)

* Attestation superstruct changes for EIP 7549 (#5644) * update * experiment * superstruct changes * revert * superstruct changes * fix tests * indexed attestation * indexed attestation superstruct * updated TODOs * `superstruct` the `AttesterSlashing` (#5636) * `superstruct` Attester Fork Variants * Push a little further * Deal with Encode / Decode of AttesterSlashing * not so sure about this.. * Stop Encode/Decode Bounds from Propagating Out * Tons of Changes.. * More Conversions to AttestationRef * Add AsReference trait (#15) * Add AsReference trait * Fix some snafus * Got it Compiling! :D * Got Tests Building * Get beacon chain tests compiling --------- Co-authored-by: Michael Sproul <micsproul@gmail.com> * Merge remote-tracking branch 'upstream/unstable' into electra_attestation_changes * Make EF Tests Fork-Agnostic (#5713) * Finish EF Test Fork Agnostic (#5714) * Superstruct `AggregateAndProof` (#5715) * Upgrade `superstruct` to `0.8.0` * superstruct `AggregateAndProof` * Merge remote-tracking branch 'sigp/unstable' into electra_attestation_changes * cargo fmt * Merge pull request #5726 from realbigsean/electra_attestation_changes Merge unstable into Electra attestation changes * EIP7549 `get_attestation_indices` (#5657) * get attesting indices electra impl * fmt * get tests to pass * fmt * fix some beacon chain tests * fmt * fix slasher test * fmt got me again * fix more tests * fix tests * Some small changes (#5739) * cargo fmt (#5740) * Sketch op pool changes * fix get attesting indices (#5742) * fix get attesting indices * better errors * fix compile * only get committee index once * Ef test fixes (#5753) * attestation related ef test fixes * delete commented out stuff * Fix Aggregation Pool for Electra (#5754) * Fix Aggregation Pool for Electra * Remove Outdated Interface * fix ssz (#5755) * Get `electra_op_pool` up to date (#5756) * fix get attesting indices (#5742) * fix get attesting indices * better errors * fix compile * only get committee index once * Ef test fixes (#5753) * attestation related ef test fixes * delete commented out stuff * Fix Aggregation Pool for Electra (#5754) * Fix Aggregation Pool for Electra * Remove Outdated Interface * fix ssz (#5755) --------- Co-authored-by: realbigsean <sean@sigmaprime.io> * Revert "Get `electra_op_pool` up to date (#5756)" (#5757) This reverts commit ab9e58aa3d. * Merge branch 'electra_attestation_changes' of https://github.com/sigp/lighthouse into electra_op_pool * Compute on chain aggregate impl (#5752) * add compute_on_chain_agg impl to op pool changes * fmt * get op pool tests to pass * update the naive agg pool interface (#5760) * Fix bugs in cross-committee aggregation * Add comment to max cover optimisation * Fix assert * Merge pull request #5749 from sigp/electra_op_pool Optimise Electra op pool aggregation * update committee offset * Fix Electra Fork Choice Tests (#5764) * Subscribe to the correct subnets for electra attestations (#5782) * subscribe to the correct att subnets for electra * subscribe to the correct att subnets for electra * cargo fmt * fix slashing handling * Merge remote-tracking branch 'upstream/unstable' * Send unagg attestation based on fork * Publish all aggregates * just one more check bro plz.. * Merge pull request #5832 from ethDreamer/electra_attestation_changes_merge_unstable Merge `unstable` into `electra_attestation_changes` * Merge pull request #5835 from realbigsean/fix-validator-logic Fix validator logic * Merge pull request #5816 from realbigsean/electra-attestation-slashing-handling Electra slashing handling * Electra attestation changes rm decode impl (#5856) * Remove Crappy Decode impl for Attestation * Remove Inefficient Attestation Decode impl * Implement Schema Upgrade / Downgrade * Update beacon_node/beacon_chain/src/schema_change/migration_schema_v20.rs Co-authored-by: Michael Sproul <micsproul@gmail.com> --------- Co-authored-by: Michael Sproul <micsproul@gmail.com> * Fix failing attestation tests and misc electra attestation cleanup (#5810) * - get attestation related beacon chain tests to pass - observed attestations are now keyed off of data + committee index - rename op pool attestationref to compactattestationref - remove unwraps in agg pool and use options instead - cherry pick some changes from ef-tests-electra * cargo fmt * fix failing test * Revert dockerfile changes * make committee_index return option * function args shouldnt be a ref to attestation ref * fmt * fix dup imports --------- Co-authored-by: realbigsean <seananderson33@GMAIL.com> * fix some todos (#5817) * Merge branch 'unstable' of https://github.com/sigp/lighthouse into electra_attestation_changes * add consolidations to merkle calc for inclusion proof * Remove Duplicate KZG Commitment Merkle Proof Code (#5874) * Remove Duplicate KZG Commitment Merkle Proof Code * s/tree_lists/fields/ * Merge branch 'unstable' of https://github.com/sigp/lighthouse into electra_attestation_changes * fix compile * Fix slasher tests (#5906) * Fix electra tests * Add electra attestations to double vote tests * Update superstruct to 0.8 * Merge remote-tracking branch 'origin/unstable' into electra_attestation_changes * Small cleanup in slasher tests * Clean up Electra observed aggregates (#5929) * Use consistent key in observed_attestations * Remove unwraps from observed aggregates * Merge branch 'unstable' of https://github.com/sigp/lighthouse into electra_attestation_changes * De-dup attestation constructor logic * Remove unwraps in Attestation construction * Dedup match_attestation_data * Remove outdated TODO * Use ForkName Ord in fork-choice tests * Use ForkName Ord in BeaconBlockBody * Make to_electra not fallible * Remove TestRandom impl for IndexedAttestation * Remove IndexedAttestation faulty Decode impl * Drop TestRandom impl * Add PendingAttestationInElectra * Indexed att on disk (#35) * indexed att on disk * fix lints * Update slasher/src/migrate.rs Co-authored-by: ethDreamer <37123614+ethDreamer@users.noreply.github.com> --------- Co-authored-by: Lion - dapplion <35266934+dapplion@users.noreply.github.com> Co-authored-by: ethDreamer <37123614+ethDreamer@users.noreply.github.com> * add electra fork enabled fn to ForkName impl (#36) * add electra fork enabled fn to ForkName impl * remove inadvertent file * Update common/eth2/src/types.rs Co-authored-by: ethDreamer <37123614+ethDreamer@users.noreply.github.com> * Dedup attestation constructor logic in attester cache * Use if let Ok for committee_bits * Dedup Attestation constructor code * Diff reduction in tests * Fix beacon_chain tests * Diff reduction * Use Ord for ForkName in pubsub * Resolve into_attestation_and_indices todo * Remove stale TODO * Fix beacon_chain tests * Test spec invariant * Use electra_enabled in pubsub * Remove get_indexed_attestation_from_signed_aggregate * Use ok_or instead of if let else * committees are sorted * remove dup method `get_indexed_attestation_from_committees` * Merge pull request #5940 from dapplion/electra_attestation_changes_lionreview Electra attestations #5712 review * update default persisted op pool deserialization * ensure aggregate and proof uses serde untagged on ref * Fork aware ssz static attestation tests * Electra attestation changes from Lions review (#5971) * dedup/cleanup and remove unneeded hashset use * remove irrelevant TODOs * Merge branch 'unstable' of https://github.com/sigp/lighthouse into electra_attestation_changes * Electra attestation changes sean review (#5972) * instantiate empty bitlist in unreachable code * clean up error conversion * fork enabled bool cleanup * remove a couple todos * return bools instead of options in `aggregate` and use the result * delete commented out code * use map macros in simple transformations * remove signers_disjoint_from * get ef tests compiling * get ef tests compiling * update intentionally excluded files * Avoid changing slasher schema for Electra * Delete slasher schema v4 * Fix clippy * Fix compilation of beacon_chain tests * Update database.rs * Add electra lightclient types * Update slasher/src/database.rs * fix imports * Merge pull request #5980 from dapplion/electra-lightclient Add electra lightclient types * Merge pull request #5975 from michaelsproul/electra-slasher-no-migration Avoid changing slasher schema for Electra * Update beacon_node/beacon_chain/src/attestation_verification.rs * Update beacon_node/beacon_chain/src/attestation_verification.rs
2026-03-23 23:04:53 +00:00 · 2024-06-24 16:08:07 -05:00
parent 758b58c9e9
commit c52c598f69
118 changed files with 5076 additions and 1741 deletions
--- a/beacon_node/operation_pool/src/lib.rs
+++ b/beacon_node/operation_pool/src/lib.rs
@@ -11,11 +11,10 @@ mod sync_aggregate_id;

 pub use crate::bls_to_execution_changes::ReceivedPreCapella;
 pub use attestation::{earliest_attestation_validators, AttMaxCover};
-pub use attestation_storage::{AttestationRef, SplitAttestation};
+pub use attestation_storage::{CompactAttestationRef, SplitAttestation};
 pub use max_cover::MaxCover;
 pub use persistence::{
-    PersistedOperationPool, PersistedOperationPoolV12, PersistedOperationPoolV14,
-    PersistedOperationPoolV15, PersistedOperationPoolV5,
+    PersistedOperationPool, PersistedOperationPoolV15, PersistedOperationPoolV20,
 };
 pub use reward_cache::RewardCache;
 use state_processing::epoch_cache::is_epoch_cache_initialized;
@@ -228,7 +227,7 @@ impl<E: EthSpec> OperationPool<E> {
        state: &'a BeaconState<E>,
        reward_cache: &'a RewardCache,
        total_active_balance: u64,
-        validity_filter: impl FnMut(&AttestationRef<'a, E>) -> bool + Send,
+        validity_filter: impl FnMut(&CompactAttestationRef<'a, E>) -> bool + Send,
        spec: &'a ChainSpec,
    ) -> impl Iterator<Item = AttMaxCover<'a, E>> + Send {
        all_attestations
@@ -252,10 +251,11 @@ impl<E: EthSpec> OperationPool<E> {
    pub fn get_attestations(
        &self,
        state: &BeaconState<E>,
-        prev_epoch_validity_filter: impl for<'a> FnMut(&AttestationRef<'a, E>) -> bool + Send,
-        curr_epoch_validity_filter: impl for<'a> FnMut(&AttestationRef<'a, E>) -> bool + Send,
+        prev_epoch_validity_filter: impl for<'a> FnMut(&CompactAttestationRef<'a, E>) -> bool + Send,
+        curr_epoch_validity_filter: impl for<'a> FnMut(&CompactAttestationRef<'a, E>) -> bool + Send,
        spec: &ChainSpec,
    ) -> Result<Vec<Attestation<E>>, OpPoolError> {
+        let fork_name = state.fork_name_unchecked();
        if !matches!(state, BeaconState::Base(_)) {
            // Epoch cache must be initialized to fetch base reward values in the max cover `score`
            // function. Currently max cover ignores items on errors. If epoch cache is not
@@ -267,7 +267,6 @@ impl<E: EthSpec> OperationPool<E> {

        // Attestations for the current fork, which may be from the current or previous epoch.
        let (prev_epoch_key, curr_epoch_key) = CheckpointKey::keys_for_state(state);
-        let all_attestations = self.attestations.read();
        let total_active_balance = state
            .get_total_active_balance()
            .map_err(OpPoolError::GetAttestationsTotalBalanceError)?;
@@ -284,6 +283,16 @@ impl<E: EthSpec> OperationPool<E> {
        let mut num_prev_valid = 0_i64;
        let mut num_curr_valid = 0_i64;

+        // TODO(electra): Work out how to do this more elegantly. This is a bit of a hack.
+        let mut all_attestations = self.attestations.write();
+
+        if fork_name.electra_enabled() {
+            all_attestations.aggregate_across_committees(prev_epoch_key);
+            all_attestations.aggregate_across_committees(curr_epoch_key);
+        }
+
+        let all_attestations = parking_lot::RwLockWriteGuard::downgrade(all_attestations);
+
        let prev_epoch_att = self
            .get_valid_attestations_for_epoch(
                &prev_epoch_key,
@@ -307,6 +316,11 @@ impl<E: EthSpec> OperationPool<E> {
            )
            .inspect(|_| num_curr_valid += 1);

+        let curr_epoch_limit = if fork_name.electra_enabled() {
+            E::MaxAttestationsElectra::to_usize()
+        } else {
+            E::MaxAttestations::to_usize()
+        };
        let prev_epoch_limit = if let BeaconState::Base(base_state) = state {
            std::cmp::min(
                E::MaxPendingAttestations::to_usize()
@@ -314,7 +328,7 @@ impl<E: EthSpec> OperationPool<E> {
                E::MaxAttestations::to_usize(),
            )
        } else {
-            E::MaxAttestations::to_usize()
+            curr_epoch_limit
        };

        let (prev_cover, curr_cover) = rayon::join(
@@ -329,11 +343,7 @@ impl<E: EthSpec> OperationPool<E> {
            },
            move || {
                let _timer = metrics::start_timer(&metrics::ATTESTATION_CURR_EPOCH_PACKING_TIME);
-                maximum_cover(
-                    curr_epoch_att,
-                    E::MaxAttestations::to_usize(),
-                    "curr_epoch_attestations",
-                )
+                maximum_cover(curr_epoch_att, curr_epoch_limit, "curr_epoch_attestations")
            },
        );

@@ -343,7 +353,7 @@ impl<E: EthSpec> OperationPool<E> {
        Ok(max_cover::merge_solutions(
            curr_cover,
            prev_cover,
-            E::MaxAttestations::to_usize(),
+            curr_epoch_limit,
        ))
    }

@@ -428,7 +438,7 @@ impl<E: EthSpec> OperationPool<E> {

        let relevant_attester_slashings = reader.iter().flat_map(|slashing| {
            if slashing.signature_is_still_valid(&state.fork()) {
-                AttesterSlashingMaxCover::new(slashing.as_inner(), to_be_slashed, state)
+                AttesterSlashingMaxCover::new(slashing.as_inner().to_ref(), to_be_slashed, state)
            } else {
                None
            }
@@ -442,7 +452,7 @@ impl<E: EthSpec> OperationPool<E> {
        .into_iter()
        .map(|cover| {
            to_be_slashed.extend(cover.covering_set().keys());
-            cover.intermediate().clone()
+            AttesterSlashingMaxCover::convert_to_object(cover.intermediate())
        })
        .collect()
    }
@@ -463,16 +473,19 @@ impl<E: EthSpec> OperationPool<E> {
            // Check that the attestation's signature is still valid wrt the fork version.
            let signature_ok = slashing.signature_is_still_valid(&head_state.fork());
            // Slashings that don't slash any validators can also be dropped.
-            let slashing_ok =
-                get_slashable_indices_modular(head_state, slashing.as_inner(), |_, validator| {
+            let slashing_ok = get_slashable_indices_modular(
+                head_state,
+                slashing.as_inner().to_ref(),
+                |_, validator| {
                    // Declare that a validator is still slashable if they have not exited prior
                    // to the finalized epoch.
                    //
                    // We cannot check the `slashed` field since the `head` is not finalized and
                    // a fork could un-slash someone.
                    validator.exit_epoch > head_state.finalized_checkpoint().epoch
-                })
-                .map_or(false, |indices| !indices.is_empty());
+                },
+            )
+            .map_or(false, |indices| !indices.is_empty());

            signature_ok && slashing_ok
        });
@@ -891,7 +904,7 @@ mod release_tests {
        );

        for (atts, aggregate) in &attestations {
-            let att2 = aggregate.as_ref().unwrap().message.aggregate.clone();
+            let att2 = aggregate.as_ref().unwrap().message().aggregate().clone();

            let att1 = atts
                .into_iter()
@@ -899,7 +912,7 @@ mod release_tests {
                .take(2)
                .fold::<Option<Attestation<MainnetEthSpec>>, _>(None, |att, new_att| {
                    if let Some(mut a) = att {
-                        a.aggregate(&new_att);
+                        a.aggregate(new_att.to_ref());
                        Some(a)
                    } else {
                        Some(new_att.clone())
@@ -907,13 +920,13 @@ mod release_tests {
                })
                .unwrap();

-            let att1_indices = get_attesting_indices_from_state(&state, &att1).unwrap();
-            let att2_indices = get_attesting_indices_from_state(&state, &att2).unwrap();
+            let att1_indices = get_attesting_indices_from_state(&state, att1.to_ref()).unwrap();
+            let att2_indices = get_attesting_indices_from_state(&state, att2).unwrap();
            let att1_split = SplitAttestation::new(att1.clone(), att1_indices);
-            let att2_split = SplitAttestation::new(att2.clone(), att2_indices);
+            let att2_split = SplitAttestation::new(att2.clone_as_attestation(), att2_indices);

            assert_eq!(
-                att1.aggregation_bits.num_set_bits(),
+                att1.num_set_aggregation_bits(),
                earliest_attestation_validators(
                    &att1_split.as_ref(),
                    &state,
@@ -927,8 +940,8 @@ mod release_tests {
                .unwrap()
                .current_epoch_attestations
                .push(PendingAttestation {
-                    aggregation_bits: att1.aggregation_bits.clone(),
-                    data: att1.data.clone(),
+                    aggregation_bits: att1.aggregation_bits_base().unwrap().clone(),
+                    data: att1.data().clone(),
                    inclusion_delay: 0,
                    proposer_index: 0,
                })
@@ -981,7 +994,8 @@ mod release_tests {

        for (atts, _) in attestations {
            for (att, _) in atts {
-                let attesting_indices = get_attesting_indices_from_state(&state, &att).unwrap();
+                let attesting_indices =
+                    get_attesting_indices_from_state(&state, att.to_ref()).unwrap();
                op_pool.insert_attestation(att, attesting_indices).unwrap();
            }
        }
@@ -1007,7 +1021,7 @@ mod release_tests {

        let agg_att = &block_attestations[0];
        assert_eq!(
-            agg_att.aggregation_bits.num_set_bits(),
+            agg_att.num_set_aggregation_bits(),
            spec.target_committee_size as usize
        );

@@ -1050,12 +1064,15 @@ mod release_tests {
        );

        for (_, aggregate) in attestations {
-            let att = aggregate.unwrap().message.aggregate;
-            let attesting_indices = get_attesting_indices_from_state(&state, &att).unwrap();
+            let agg = aggregate.unwrap();
+            let att = agg.message().aggregate();
+            let attesting_indices = get_attesting_indices_from_state(&state, att).unwrap();
            op_pool
-                .insert_attestation(att.clone(), attesting_indices.clone())
+                .insert_attestation(att.clone_as_attestation(), attesting_indices.clone())
+                .unwrap();
+            op_pool
+                .insert_attestation(att.clone_as_attestation(), attesting_indices)
                .unwrap();
-            op_pool.insert_attestation(att, attesting_indices).unwrap();
        }

        assert_eq!(op_pool.num_attestations(), committees.len());
@@ -1104,7 +1121,7 @@ mod release_tests {
                        None,
                        |att, new_att| {
                            if let Some(mut a) = att {
-                                a.aggregate(new_att);
+                                a.aggregate(new_att.to_ref());
                                Some(a)
                            } else {
                                Some(new_att.clone())
@@ -1127,7 +1144,7 @@ mod release_tests {
                        None,
                        |att, new_att| {
                            if let Some(mut a) = att {
-                                a.aggregate(new_att);
+                                a.aggregate(new_att.to_ref());
                                Some(a)
                            } else {
                                Some(new_att.clone())
@@ -1139,7 +1156,8 @@ mod release_tests {
                .collect::<Vec<_>>();

            for att in aggs1.into_iter().chain(aggs2.into_iter()) {
-                let attesting_indices = get_attesting_indices_from_state(&state, &att).unwrap();
+                let attesting_indices =
+                    get_attesting_indices_from_state(&state, att.to_ref()).unwrap();
                op_pool.insert_attestation(att, attesting_indices).unwrap();
            }
        }
@@ -1203,7 +1221,7 @@ mod release_tests {
                        .fold::<Attestation<MainnetEthSpec>, _>(
                            att_0.clone(),
                            |mut att, new_att| {
-                                att.aggregate(new_att);
+                                att.aggregate(new_att.to_ref());
                                att
                            },
                        )
@@ -1211,7 +1229,8 @@ mod release_tests {
                .collect::<Vec<_>>();

            for att in aggs {
-                let attesting_indices = get_attesting_indices_from_state(&state, &att).unwrap();
+                let attesting_indices =
+                    get_attesting_indices_from_state(&state, att.to_ref()).unwrap();
                op_pool.insert_attestation(att, attesting_indices).unwrap();
            }
        };
@@ -1228,7 +1247,17 @@ mod release_tests {
        let num_big = target_committee_size / big_step_size;

        let stats = op_pool.attestation_stats();
-        assert_eq!(stats.num_attestation_data, committees.len());
+        let fork_name = state.fork_name_unchecked();
+
+        match fork_name {
+            ForkName::Electra => {
+                assert_eq!(stats.num_attestation_data, 1);
+            }
+            _ => {
+                assert_eq!(stats.num_attestation_data, committees.len());
+            }
+        };
+
        assert_eq!(
            stats.num_attestations,
            (num_small + num_big) * committees.len()
@@ -1239,11 +1268,25 @@ mod release_tests {
        let best_attestations = op_pool
            .get_attestations(&state, |_| true, |_| true, spec)
            .expect("should have best attestations");
-        assert_eq!(best_attestations.len(), max_attestations);
+        match fork_name {
+            ForkName::Electra => {
+                assert_eq!(best_attestations.len(), 8);
+            }
+            _ => {
+                assert_eq!(best_attestations.len(), max_attestations);
+            }
+        };

        // All the best attestations should be signed by at least `big_step_size` (4) validators.
        for att in &best_attestations {
-            assert!(att.aggregation_bits.num_set_bits() >= big_step_size);
+            match fork_name {
+                ForkName::Electra => {
+                    assert!(att.num_set_aggregation_bits() >= small_step_size);
+                }
+                _ => {
+                    assert!(att.num_set_aggregation_bits() >= big_step_size);
+                }
+            };
        }
    }

@@ -1298,7 +1341,7 @@ mod release_tests {
                        .fold::<Attestation<MainnetEthSpec>, _>(
                            att_0.clone(),
                            |mut att, new_att| {
-                                att.aggregate(new_att);
+                                att.aggregate(new_att.to_ref());
                                att
                            },
                        )
@@ -1306,7 +1349,8 @@ mod release_tests {
                .collect::<Vec<_>>();

            for att in aggs {
-                let attesting_indices = get_attesting_indices_from_state(&state, &att).unwrap();
+                let attesting_indices =
+                    get_attesting_indices_from_state(&state, att.to_ref()).unwrap();
                op_pool.insert_attestation(att, attesting_indices).unwrap();
            }
        };
@@ -1321,11 +1365,20 @@ mod release_tests {

        let num_small = target_committee_size / small_step_size;
        let num_big = target_committee_size / big_step_size;
+        let fork_name = state.fork_name_unchecked();
+
+        match fork_name {
+            ForkName::Electra => {
+                assert_eq!(op_pool.attestation_stats().num_attestation_data, 1);
+            }
+            _ => {
+                assert_eq!(
+                    op_pool.attestation_stats().num_attestation_data,
+                    committees.len()
+                );
+            }
+        };

-        assert_eq!(
-            op_pool.attestation_stats().num_attestation_data,
-            committees.len()
-        );
        assert_eq!(
            op_pool.num_attestations(),
            (num_small + num_big) * committees.len()
@@ -1336,7 +1389,15 @@ mod release_tests {
        let best_attestations = op_pool
            .get_attestations(&state, |_| true, |_| true, spec)
            .expect("should have valid best attestations");
-        assert_eq!(best_attestations.len(), max_attestations);
+
+        match fork_name {
+            ForkName::Electra => {
+                assert_eq!(best_attestations.len(), 8);
+            }
+            _ => {
+                assert_eq!(best_attestations.len(), max_attestations);
+            }
+        };

        let total_active_balance = state.get_total_active_balance().unwrap();

@@ -1349,7 +1410,7 @@ mod release_tests {
        reward_cache.update(&state).unwrap();

        for att in best_attestations {
-            let attesting_indices = get_attesting_indices_from_state(&state, &att).unwrap();
+            let attesting_indices = get_attesting_indices_from_state(&state, att.to_ref()).unwrap();
            let split_attestation = SplitAttestation::new(att, attesting_indices);
            let mut fresh_validators_rewards = AttMaxCover::new(
                split_attestation.as_ref(),