Electra: Remaining Consensus Data Structures (#5712)

* Attestation superstruct changes for EIP 7549 (#5644) * update * experiment * superstruct changes * revert * superstruct changes * fix tests * indexed attestation * indexed attestation superstruct * updated TODOs * `superstruct` the `AttesterSlashing` (#5636) * `superstruct` Attester Fork Variants * Push a little further * Deal with Encode / Decode of AttesterSlashing * not so sure about this.. * Stop Encode/Decode Bounds from Propagating Out * Tons of Changes.. * More Conversions to AttestationRef * Add AsReference trait (#15) * Add AsReference trait * Fix some snafus * Got it Compiling! :D * Got Tests Building * Get beacon chain tests compiling --------- Co-authored-by: Michael Sproul <micsproul@gmail.com> * Merge remote-tracking branch 'upstream/unstable' into electra_attestation_changes * Make EF Tests Fork-Agnostic (#5713) * Finish EF Test Fork Agnostic (#5714) * Superstruct `AggregateAndProof` (#5715) * Upgrade `superstruct` to `0.8.0` * superstruct `AggregateAndProof` * Merge remote-tracking branch 'sigp/unstable' into electra_attestation_changes * cargo fmt * Merge pull request #5726 from realbigsean/electra_attestation_changes Merge unstable into Electra attestation changes * EIP7549 `get_attestation_indices` (#5657) * get attesting indices electra impl * fmt * get tests to pass * fmt * fix some beacon chain tests * fmt * fix slasher test * fmt got me again * fix more tests * fix tests * Some small changes (#5739) * cargo fmt (#5740) * Sketch op pool changes * fix get attesting indices (#5742) * fix get attesting indices * better errors * fix compile * only get committee index once * Ef test fixes (#5753) * attestation related ef test fixes * delete commented out stuff * Fix Aggregation Pool for Electra (#5754) * Fix Aggregation Pool for Electra * Remove Outdated Interface * fix ssz (#5755) * Get `electra_op_pool` up to date (#5756) * fix get attesting indices (#5742) * fix get attesting indices * better errors * fix compile * only get committee index once * Ef test fixes (#5753) * attestation related ef test fixes * delete commented out stuff * Fix Aggregation Pool for Electra (#5754) * Fix Aggregation Pool for Electra * Remove Outdated Interface * fix ssz (#5755) --------- Co-authored-by: realbigsean <sean@sigmaprime.io> * Revert "Get `electra_op_pool` up to date (#5756)" (#5757) This reverts commit ab9e58aa3d. * Merge branch 'electra_attestation_changes' of https://github.com/sigp/lighthouse into electra_op_pool * Compute on chain aggregate impl (#5752) * add compute_on_chain_agg impl to op pool changes * fmt * get op pool tests to pass * update the naive agg pool interface (#5760) * Fix bugs in cross-committee aggregation * Add comment to max cover optimisation * Fix assert * Merge pull request #5749 from sigp/electra_op_pool Optimise Electra op pool aggregation * update committee offset * Fix Electra Fork Choice Tests (#5764) * Subscribe to the correct subnets for electra attestations (#5782) * subscribe to the correct att subnets for electra * subscribe to the correct att subnets for electra * cargo fmt * fix slashing handling * Merge remote-tracking branch 'upstream/unstable' * Send unagg attestation based on fork * Publish all aggregates * just one more check bro plz.. * Merge pull request #5832 from ethDreamer/electra_attestation_changes_merge_unstable Merge `unstable` into `electra_attestation_changes` * Merge pull request #5835 from realbigsean/fix-validator-logic Fix validator logic * Merge pull request #5816 from realbigsean/electra-attestation-slashing-handling Electra slashing handling * Electra attestation changes rm decode impl (#5856) * Remove Crappy Decode impl for Attestation * Remove Inefficient Attestation Decode impl * Implement Schema Upgrade / Downgrade * Update beacon_node/beacon_chain/src/schema_change/migration_schema_v20.rs Co-authored-by: Michael Sproul <micsproul@gmail.com> --------- Co-authored-by: Michael Sproul <micsproul@gmail.com> * Fix failing attestation tests and misc electra attestation cleanup (#5810) * - get attestation related beacon chain tests to pass - observed attestations are now keyed off of data + committee index - rename op pool attestationref to compactattestationref - remove unwraps in agg pool and use options instead - cherry pick some changes from ef-tests-electra * cargo fmt * fix failing test * Revert dockerfile changes * make committee_index return option * function args shouldnt be a ref to attestation ref * fmt * fix dup imports --------- Co-authored-by: realbigsean <seananderson33@GMAIL.com> * fix some todos (#5817) * Merge branch 'unstable' of https://github.com/sigp/lighthouse into electra_attestation_changes * add consolidations to merkle calc for inclusion proof * Remove Duplicate KZG Commitment Merkle Proof Code (#5874) * Remove Duplicate KZG Commitment Merkle Proof Code * s/tree_lists/fields/ * Merge branch 'unstable' of https://github.com/sigp/lighthouse into electra_attestation_changes * fix compile * Fix slasher tests (#5906) * Fix electra tests * Add electra attestations to double vote tests * Update superstruct to 0.8 * Merge remote-tracking branch 'origin/unstable' into electra_attestation_changes * Small cleanup in slasher tests * Clean up Electra observed aggregates (#5929) * Use consistent key in observed_attestations * Remove unwraps from observed aggregates * Merge branch 'unstable' of https://github.com/sigp/lighthouse into electra_attestation_changes * De-dup attestation constructor logic * Remove unwraps in Attestation construction * Dedup match_attestation_data * Remove outdated TODO * Use ForkName Ord in fork-choice tests * Use ForkName Ord in BeaconBlockBody * Make to_electra not fallible * Remove TestRandom impl for IndexedAttestation * Remove IndexedAttestation faulty Decode impl * Drop TestRandom impl * Add PendingAttestationInElectra * Indexed att on disk (#35) * indexed att on disk * fix lints * Update slasher/src/migrate.rs Co-authored-by: ethDreamer <37123614+ethDreamer@users.noreply.github.com> --------- Co-authored-by: Lion - dapplion <35266934+dapplion@users.noreply.github.com> Co-authored-by: ethDreamer <37123614+ethDreamer@users.noreply.github.com> * add electra fork enabled fn to ForkName impl (#36) * add electra fork enabled fn to ForkName impl * remove inadvertent file * Update common/eth2/src/types.rs Co-authored-by: ethDreamer <37123614+ethDreamer@users.noreply.github.com> * Dedup attestation constructor logic in attester cache * Use if let Ok for committee_bits * Dedup Attestation constructor code * Diff reduction in tests * Fix beacon_chain tests * Diff reduction * Use Ord for ForkName in pubsub * Resolve into_attestation_and_indices todo * Remove stale TODO * Fix beacon_chain tests * Test spec invariant * Use electra_enabled in pubsub * Remove get_indexed_attestation_from_signed_aggregate * Use ok_or instead of if let else * committees are sorted * remove dup method `get_indexed_attestation_from_committees` * Merge pull request #5940 from dapplion/electra_attestation_changes_lionreview Electra attestations #5712 review * update default persisted op pool deserialization * ensure aggregate and proof uses serde untagged on ref * Fork aware ssz static attestation tests * Electra attestation changes from Lions review (#5971) * dedup/cleanup and remove unneeded hashset use * remove irrelevant TODOs * Merge branch 'unstable' of https://github.com/sigp/lighthouse into electra_attestation_changes * Electra attestation changes sean review (#5972) * instantiate empty bitlist in unreachable code * clean up error conversion * fork enabled bool cleanup * remove a couple todos * return bools instead of options in `aggregate` and use the result * delete commented out code * use map macros in simple transformations * remove signers_disjoint_from * get ef tests compiling * get ef tests compiling * update intentionally excluded files * Avoid changing slasher schema for Electra * Delete slasher schema v4 * Fix clippy * Fix compilation of beacon_chain tests * Update database.rs * Add electra lightclient types * Update slasher/src/database.rs * fix imports * Merge pull request #5980 from dapplion/electra-lightclient Add electra lightclient types * Merge pull request #5975 from michaelsproul/electra-slasher-no-migration Avoid changing slasher schema for Electra * Update beacon_node/beacon_chain/src/attestation_verification.rs * Update beacon_node/beacon_chain/src/attestation_verification.rs
2026-07-01 20:04:41 +00:00 · 2024-06-24 16:08:07 -05:00
parent 758b58c9e9
commit c52c598f69
118 changed files with 5076 additions and 1741 deletions
--- a/beacon_node/beacon_chain/src/observed_aggregates.rs
+++ b/beacon_node/beacon_chain/src/observed_aggregates.rs
@@ -6,22 +6,33 @@ use ssz_types::{BitList, BitVector};
 use std::collections::HashMap;
 use std::marker::PhantomData;
 use tree_hash::TreeHash;
+use tree_hash_derive::TreeHash;
 use types::consts::altair::{
    SYNC_COMMITTEE_SUBNET_COUNT, TARGET_AGGREGATORS_PER_SYNC_SUBCOMMITTEE,
 };
 use types::slot_data::SlotData;
-use types::{Attestation, EthSpec, Hash256, Slot, SyncCommitteeContribution};
+use types::{
+    Attestation, AttestationData, AttestationRef, EthSpec, Hash256, Slot, SyncCommitteeContribution,
+};

 pub type ObservedSyncContributions<E> = ObservedAggregates<
    SyncCommitteeContribution<E>,
    E,
    BitVector<<E as types::EthSpec>::SyncSubcommitteeSize>,
 >;
-pub type ObservedAggregateAttestations<E> = ObservedAggregates<
-    Attestation<E>,
-    E,
-    BitList<<E as types::EthSpec>::MaxValidatorsPerCommittee>,
->;
+pub type ObservedAggregateAttestations<E> =
+    ObservedAggregates<Attestation<E>, E, BitList<<E as types::EthSpec>::MaxValidatorsPerSlot>>;
+
+/// Attestation data augmented with committee index
+///
+/// This is hashed and used to key the map of observed aggregate attestations. This is important
+/// post-Electra where the attestation data committee index is 0 and we want to avoid accidentally
+/// comparing aggregation bits for *different* committees.
+#[derive(TreeHash)]
+pub struct ObservedAttestationKey {
+    pub committee_index: u64,
+    pub attestation_data: AttestationData,
+}

 /// A trait use to associate capacity constants with the type being stored in `ObservedAggregates`.
 pub trait Consts {
@@ -95,35 +106,60 @@ pub trait SubsetItem {

    /// Returns the item that gets stored in `ObservedAggregates` for later subset
    /// comparison with incoming aggregates.
-    fn get_item(&self) -> Self::Item;
+    fn get_item(&self) -> Result<Self::Item, Error>;

    /// Returns a unique value that keys the object to the item that is being stored
    /// in `ObservedAggregates`.
-    fn root(&self) -> Hash256;
+    fn root(&self) -> Result<Hash256, Error>;
 }

-impl<E: EthSpec> SubsetItem for Attestation<E> {
-    type Item = BitList<E::MaxValidatorsPerCommittee>;
+impl<'a, E: EthSpec> SubsetItem for AttestationRef<'a, E> {
+    type Item = BitList<E::MaxValidatorsPerSlot>;
    fn is_subset(&self, other: &Self::Item) -> bool {
-        self.aggregation_bits.is_subset(other)
+        match self {
+            Self::Base(att) => {
+                if let Ok(extended_aggregation_bits) = att.extend_aggregation_bits() {
+                    return extended_aggregation_bits.is_subset(other);
+                }
+                false
+            }
+            Self::Electra(att) => att.aggregation_bits.is_subset(other),
+        }
    }

    fn is_superset(&self, other: &Self::Item) -> bool {
-        other.is_subset(&self.aggregation_bits)
+        match self {
+            Self::Base(att) => {
+                if let Ok(extended_aggregation_bits) = att.extend_aggregation_bits() {
+                    return other.is_subset(&extended_aggregation_bits);
+                }
+                false
+            }
+            Self::Electra(att) => other.is_subset(&att.aggregation_bits),
+        }
    }

    /// Returns the sync contribution aggregation bits.
-    fn get_item(&self) -> Self::Item {
-        self.aggregation_bits.clone()
+    fn get_item(&self) -> Result<Self::Item, Error> {
+        match self {
+            Self::Base(att) => att
+                .extend_aggregation_bits()
+                .map_err(|_| Error::GetItemError),
+            Self::Electra(att) => Ok(att.aggregation_bits.clone()),
+        }
    }

-    /// Returns the hash tree root of the attestation data.
-    fn root(&self) -> Hash256 {
-        self.data.tree_hash_root()
+    /// Returns the hash tree root of the attestation data augmented with the committee index.
+    fn root(&self) -> Result<Hash256, Error> {
+        Ok(ObservedAttestationKey {
+            committee_index: self.committee_index().ok_or(Error::RootError)?,
+            attestation_data: self.data().clone(),
+        }
+        .tree_hash_root())
    }
 }

-impl<E: EthSpec> SubsetItem for SyncCommitteeContribution<E> {
+impl<'a, E: EthSpec> SubsetItem for &'a SyncCommitteeContribution<E> {
    type Item = BitVector<E::SyncSubcommitteeSize>;
    fn is_subset(&self, other: &Self::Item) -> bool {
        self.aggregation_bits.is_subset(other)
@@ -134,19 +170,19 @@ impl<E: EthSpec> SubsetItem for SyncCommitteeContribution<E> {
    }

    /// Returns the sync contribution aggregation bits.
-    fn get_item(&self) -> Self::Item {
-        self.aggregation_bits.clone()
+    fn get_item(&self) -> Result<Self::Item, Error> {
+        Ok(self.aggregation_bits.clone())
    }

    /// Returns the hash tree root of the root, slot and subcommittee index
    /// of the sync contribution.
-    fn root(&self) -> Hash256 {
-        SyncCommitteeData {
+    fn root(&self) -> Result<Hash256, Error> {
+        Ok(SyncCommitteeData {
            root: self.beacon_block_root,
            slot: self.slot,
            subcommittee_index: self.subcommittee_index,
        }
-        .tree_hash_root()
+        .tree_hash_root())
    }
 }

@@ -173,6 +209,8 @@ pub enum Error {
        expected: Slot,
        attestation: Slot,
    },
+    GetItemError,
+    RootError,
 }

 /// A `HashMap` that contains entries related to some `Slot`.
@@ -196,7 +234,7 @@ impl<I> SlotHashSet<I> {
    /// Store the items in self so future observations recognise its existence.
    pub fn observe_item<S: SlotData + SubsetItem<Item = I>>(
        &mut self,
-        item: &S,
+        item: S,
        root: Hash256,
    ) -> Result<ObserveOutcome, Error> {
        if item.get_slot() != self.slot {
@@ -215,7 +253,7 @@ impl<I> SlotHashSet<I> {
                // If true, we replace the new item with its existing subset. This allows us
                // to hold fewer items in the list.
                } else if item.is_superset(existing) {
-                    *existing = item.get_item();
+                    *existing = item.get_item()?;
                    return Ok(ObserveOutcome::New);
                }
            }
@@ -233,7 +271,7 @@ impl<I> SlotHashSet<I> {
            return Err(Error::ReachedMaxObservationsPerSlot(self.max_capacity));
        }

-        let item = item.get_item();
+        let item = item.get_item()?;
        self.map.entry(root).or_default().push(item);
        Ok(ObserveOutcome::New)
    }
@@ -242,7 +280,7 @@ impl<I> SlotHashSet<I> {
    /// the given root and slot.
    pub fn is_known_subset<S: SlotData + SubsetItem<Item = I>>(
        &self,
-        item: &S,
+        item: S,
        root: Hash256,
    ) -> Result<bool, Error> {
        if item.get_slot() != self.slot {
@@ -264,16 +302,43 @@ impl<I> SlotHashSet<I> {
    }
 }

+/// Trait for observable items that can be observed from their reference type.
+///
+/// This is used to make observations for `Attestation`s from `AttestationRef`s.
+pub trait AsReference {
+    type Reference<'a>
+    where
+        Self: 'a;
+
+    fn as_reference(&self) -> Self::Reference<'_>;
+}
+
+impl<E: EthSpec> AsReference for Attestation<E> {
+    type Reference<'a> = AttestationRef<'a, E>;
+
+    fn as_reference(&self) -> AttestationRef<'_, E> {
+        self.to_ref()
+    }
+}
+
+impl<E: EthSpec> AsReference for SyncCommitteeContribution<E> {
+    type Reference<'a> = &'a Self;
+
+    fn as_reference(&self) -> &Self {
+        self
+    }
+}
+
 /// Stores the roots of objects for some number of `Slots`, so we can determine if
 /// these have previously been seen on the network.
-pub struct ObservedAggregates<T: SlotData + Consts, E: EthSpec, I> {
+pub struct ObservedAggregates<T: Consts + AsReference, E: EthSpec, I> {
    lowest_permissible_slot: Slot,
    sets: Vec<SlotHashSet<I>>,
    _phantom_spec: PhantomData<E>,
    _phantom_tree_hash: PhantomData<T>,
 }

-impl<T: SlotData + Consts, E: EthSpec, I> Default for ObservedAggregates<T, E, I> {
+impl<T: Consts + AsReference, E: EthSpec, I> Default for ObservedAggregates<T, E, I> {
    fn default() -> Self {
        Self {
            lowest_permissible_slot: Slot::new(0),
@@ -284,17 +349,22 @@ impl<T: SlotData + Consts, E: EthSpec, I> Default for ObservedAggregates<T, E, I
    }
 }

-impl<T: SlotData + Consts + SubsetItem<Item = I>, E: EthSpec, I> ObservedAggregates<T, E, I> {
+impl<T, E, I> ObservedAggregates<T, E, I>
+where
+    T: Consts + AsReference,
+    E: EthSpec,
+    for<'a> T::Reference<'a>: SubsetItem<Item = I> + SlotData,
+{
    /// Store `item` in `self` keyed at `root`.
    ///
    /// `root` must equal `item.root::<SubsetItem>()`.
    pub fn observe_item(
        &mut self,
-        item: &T,
+        item: T::Reference<'_>,
        root_opt: Option<Hash256>,
    ) -> Result<ObserveOutcome, Error> {
        let index = self.get_set_index(item.get_slot())?;
-        let root = root_opt.unwrap_or_else(|| item.root());
+        let root = root_opt.map_or_else(|| item.root(), Ok)?;

        self.sets
            .get_mut(index)
@@ -307,7 +377,11 @@ impl<T: SlotData + Consts + SubsetItem<Item = I>, E: EthSpec, I> ObservedAggrega
    ///
    /// `root` must equal `item.root::<SubsetItem>()`.
    #[allow(clippy::wrong_self_convention)]
-    pub fn is_known_subset(&mut self, item: &T, root: Hash256) -> Result<bool, Error> {
+    pub fn is_known_subset(
+        &mut self,
+        item: T::Reference<'_>,
+        root: Hash256,
+    ) -> Result<bool, Error> {
        let index = self.get_set_index(item.get_slot())?;

        self.sets
@@ -399,14 +473,15 @@ impl<T: SlotData + Consts + SubsetItem<Item = I>, E: EthSpec, I> ObservedAggrega
 #[cfg(not(debug_assertions))]
 mod tests {
    use super::*;
-    use types::{test_utils::test_random_instance, Hash256};
+    use types::{test_utils::test_random_instance, AttestationBase, Hash256};

    type E = types::MainnetEthSpec;

    fn get_attestation(slot: Slot, beacon_block_root: u64) -> Attestation<E> {
-        let mut a: Attestation<E> = test_random_instance();
-        a.data.slot = slot;
-        a.data.beacon_block_root = Hash256::from_low_u64_be(beacon_block_root);
+        let a: AttestationBase<E> = test_random_instance();
+        let mut a = Attestation::Base(a);
+        a.data_mut().slot = slot;
+        a.data_mut().beacon_block_root = Hash256::from_low_u64_be(beacon_block_root);
        a
    }

@@ -432,12 +507,15 @@ mod tests {

                    for a in &items {
                        assert_eq!(
-                            store.is_known_subset(a, a.root()),
+                            store.is_known_subset(
+                                a.as_reference(),
+                                a.as_reference().root().unwrap()
+                            ),
                            Ok(false),
                            "should indicate an unknown attestation is unknown"
                        );
                        assert_eq!(
-                            store.observe_item(a, None),
+                            store.observe_item(a.as_reference(), None),
                            Ok(ObserveOutcome::New),
                            "should observe new attestation"
                        );
@@ -445,12 +523,18 @@ mod tests {

                    for a in &items {
                        assert_eq!(
-                            store.is_known_subset(a, a.root()),
+                            store.is_known_subset(
+                                a.as_reference(),
+                                a.as_reference().root().unwrap()
+                            ),
                            Ok(true),
                            "should indicate a known attestation is known"
                        );
                        assert_eq!(
-                            store.observe_item(a, Some(a.root())),
+                            store.observe_item(
+                                a.as_reference(),
+                                Some(a.as_reference().root().unwrap())
+                            ),
                            Ok(ObserveOutcome::Subset),
                            "should acknowledge an existing attestation"
                        );