Electra: Remaining Consensus Data Structures (#5712)

* Attestation superstruct changes for EIP 7549 (#5644) * update * experiment * superstruct changes * revert * superstruct changes * fix tests * indexed attestation * indexed attestation superstruct * updated TODOs * `superstruct` the `AttesterSlashing` (#5636) * `superstruct` Attester Fork Variants * Push a little further * Deal with Encode / Decode of AttesterSlashing * not so sure about this.. * Stop Encode/Decode Bounds from Propagating Out * Tons of Changes.. * More Conversions to AttestationRef * Add AsReference trait (#15) * Add AsReference trait * Fix some snafus * Got it Compiling! :D * Got Tests Building * Get beacon chain tests compiling --------- Co-authored-by: Michael Sproul <micsproul@gmail.com> * Merge remote-tracking branch 'upstream/unstable' into electra_attestation_changes * Make EF Tests Fork-Agnostic (#5713) * Finish EF Test Fork Agnostic (#5714) * Superstruct `AggregateAndProof` (#5715) * Upgrade `superstruct` to `0.8.0` * superstruct `AggregateAndProof` * Merge remote-tracking branch 'sigp/unstable' into electra_attestation_changes * cargo fmt * Merge pull request #5726 from realbigsean/electra_attestation_changes Merge unstable into Electra attestation changes * EIP7549 `get_attestation_indices` (#5657) * get attesting indices electra impl * fmt * get tests to pass * fmt * fix some beacon chain tests * fmt * fix slasher test * fmt got me again * fix more tests * fix tests * Some small changes (#5739) * cargo fmt (#5740) * Sketch op pool changes * fix get attesting indices (#5742) * fix get attesting indices * better errors * fix compile * only get committee index once * Ef test fixes (#5753) * attestation related ef test fixes * delete commented out stuff * Fix Aggregation Pool for Electra (#5754) * Fix Aggregation Pool for Electra * Remove Outdated Interface * fix ssz (#5755) * Get `electra_op_pool` up to date (#5756) * fix get attesting indices (#5742) * fix get attesting indices * better errors * fix compile * only get committee index once * Ef test fixes (#5753) * attestation related ef test fixes * delete commented out stuff * Fix Aggregation Pool for Electra (#5754) * Fix Aggregation Pool for Electra * Remove Outdated Interface * fix ssz (#5755) --------- Co-authored-by: realbigsean <sean@sigmaprime.io> * Revert "Get `electra_op_pool` up to date (#5756)" (#5757) This reverts commit ab9e58aa3d. * Merge branch 'electra_attestation_changes' of https://github.com/sigp/lighthouse into electra_op_pool * Compute on chain aggregate impl (#5752) * add compute_on_chain_agg impl to op pool changes * fmt * get op pool tests to pass * update the naive agg pool interface (#5760) * Fix bugs in cross-committee aggregation * Add comment to max cover optimisation * Fix assert * Merge pull request #5749 from sigp/electra_op_pool Optimise Electra op pool aggregation * update committee offset * Fix Electra Fork Choice Tests (#5764) * Subscribe to the correct subnets for electra attestations (#5782) * subscribe to the correct att subnets for electra * subscribe to the correct att subnets for electra * cargo fmt * fix slashing handling * Merge remote-tracking branch 'upstream/unstable' * Send unagg attestation based on fork * Publish all aggregates * just one more check bro plz.. * Merge pull request #5832 from ethDreamer/electra_attestation_changes_merge_unstable Merge `unstable` into `electra_attestation_changes` * Merge pull request #5835 from realbigsean/fix-validator-logic Fix validator logic * Merge pull request #5816 from realbigsean/electra-attestation-slashing-handling Electra slashing handling * Electra attestation changes rm decode impl (#5856) * Remove Crappy Decode impl for Attestation * Remove Inefficient Attestation Decode impl * Implement Schema Upgrade / Downgrade * Update beacon_node/beacon_chain/src/schema_change/migration_schema_v20.rs Co-authored-by: Michael Sproul <micsproul@gmail.com> --------- Co-authored-by: Michael Sproul <micsproul@gmail.com> * Fix failing attestation tests and misc electra attestation cleanup (#5810) * - get attestation related beacon chain tests to pass - observed attestations are now keyed off of data + committee index - rename op pool attestationref to compactattestationref - remove unwraps in agg pool and use options instead - cherry pick some changes from ef-tests-electra * cargo fmt * fix failing test * Revert dockerfile changes * make committee_index return option * function args shouldnt be a ref to attestation ref * fmt * fix dup imports --------- Co-authored-by: realbigsean <seananderson33@GMAIL.com> * fix some todos (#5817) * Merge branch 'unstable' of https://github.com/sigp/lighthouse into electra_attestation_changes * add consolidations to merkle calc for inclusion proof * Remove Duplicate KZG Commitment Merkle Proof Code (#5874) * Remove Duplicate KZG Commitment Merkle Proof Code * s/tree_lists/fields/ * Merge branch 'unstable' of https://github.com/sigp/lighthouse into electra_attestation_changes * fix compile * Fix slasher tests (#5906) * Fix electra tests * Add electra attestations to double vote tests * Update superstruct to 0.8 * Merge remote-tracking branch 'origin/unstable' into electra_attestation_changes * Small cleanup in slasher tests * Clean up Electra observed aggregates (#5929) * Use consistent key in observed_attestations * Remove unwraps from observed aggregates * Merge branch 'unstable' of https://github.com/sigp/lighthouse into electra_attestation_changes * De-dup attestation constructor logic * Remove unwraps in Attestation construction * Dedup match_attestation_data * Remove outdated TODO * Use ForkName Ord in fork-choice tests * Use ForkName Ord in BeaconBlockBody * Make to_electra not fallible * Remove TestRandom impl for IndexedAttestation * Remove IndexedAttestation faulty Decode impl * Drop TestRandom impl * Add PendingAttestationInElectra * Indexed att on disk (#35) * indexed att on disk * fix lints * Update slasher/src/migrate.rs Co-authored-by: ethDreamer <37123614+ethDreamer@users.noreply.github.com> --------- Co-authored-by: Lion - dapplion <35266934+dapplion@users.noreply.github.com> Co-authored-by: ethDreamer <37123614+ethDreamer@users.noreply.github.com> * add electra fork enabled fn to ForkName impl (#36) * add electra fork enabled fn to ForkName impl * remove inadvertent file * Update common/eth2/src/types.rs Co-authored-by: ethDreamer <37123614+ethDreamer@users.noreply.github.com> * Dedup attestation constructor logic in attester cache * Use if let Ok for committee_bits * Dedup Attestation constructor code * Diff reduction in tests * Fix beacon_chain tests * Diff reduction * Use Ord for ForkName in pubsub * Resolve into_attestation_and_indices todo * Remove stale TODO * Fix beacon_chain tests * Test spec invariant * Use electra_enabled in pubsub * Remove get_indexed_attestation_from_signed_aggregate * Use ok_or instead of if let else * committees are sorted * remove dup method `get_indexed_attestation_from_committees` * Merge pull request #5940 from dapplion/electra_attestation_changes_lionreview Electra attestations #5712 review * update default persisted op pool deserialization * ensure aggregate and proof uses serde untagged on ref * Fork aware ssz static attestation tests * Electra attestation changes from Lions review (#5971) * dedup/cleanup and remove unneeded hashset use * remove irrelevant TODOs * Merge branch 'unstable' of https://github.com/sigp/lighthouse into electra_attestation_changes * Electra attestation changes sean review (#5972) * instantiate empty bitlist in unreachable code * clean up error conversion * fork enabled bool cleanup * remove a couple todos * return bools instead of options in `aggregate` and use the result * delete commented out code * use map macros in simple transformations * remove signers_disjoint_from * get ef tests compiling * get ef tests compiling * update intentionally excluded files * Avoid changing slasher schema for Electra * Delete slasher schema v4 * Fix clippy * Fix compilation of beacon_chain tests * Update database.rs * Add electra lightclient types * Update slasher/src/database.rs * fix imports * Merge pull request #5980 from dapplion/electra-lightclient Add electra lightclient types * Merge pull request #5975 from michaelsproul/electra-slasher-no-migration Avoid changing slasher schema for Electra * Update beacon_node/beacon_chain/src/attestation_verification.rs * Update beacon_node/beacon_chain/src/attestation_verification.rs
2026-04-18 13:28:33 +00:00 · 2024-06-24 16:08:07 -05:00
parent 758b58c9e9
commit c52c598f69
118 changed files with 5076 additions and 1741 deletions
--- a/slasher/src/database.rs
+++ b/slasher/src/database.rs
@@ -13,12 +13,15 @@ use parking_lot::Mutex;
 use serde::de::DeserializeOwned;
 use slog::{info, Logger};
 use ssz::{Decode, Encode};
+use ssz_derive::{Decode, Encode};
 use std::borrow::{Borrow, Cow};
 use std::marker::PhantomData;
 use std::sync::Arc;
 use tree_hash::TreeHash;
 use types::{
-    Epoch, EthSpec, Hash256, IndexedAttestation, ProposerSlashing, SignedBeaconBlockHeader, Slot,
+    AggregateSignature, AttestationData, ChainSpec, Epoch, EthSpec, Hash256, IndexedAttestation,
+    IndexedAttestationBase, IndexedAttestationElectra, ProposerSlashing, SignedBeaconBlockHeader,
+    Slot, VariableList,
 };

 /// Current database schema version, to check compatibility of on-disk DB with software.
@@ -69,6 +72,7 @@ pub struct SlasherDB<E: EthSpec> {
    /// LRU cache mapping indexed attestation IDs to their attestation data roots.
    attestation_root_cache: Mutex<LruCache<IndexedAttestationId, Hash256>>,
    pub(crate) config: Arc<Config>,
+    pub(crate) spec: Arc<ChainSpec>,
    _phantom: PhantomData<E>,
 }

@@ -235,6 +239,43 @@ impl AsRef<[u8]> for IndexedAttestationId {
    }
 }

+/// Indexed attestation that abstracts over Phase0 and Electra variants by using a plain `Vec` for
+/// the attesting indices.
+///
+/// This allows us to avoid rewriting the entire indexed attestation database at Electra, which
+/// saves a lot of execution time. The bytes that it encodes to are the same as the bytes that a
+/// regular IndexedAttestation encodes to, because SSZ doesn't care about the length-bound.
+#[derive(Debug, PartialEq, Decode, Encode)]
+pub struct IndexedAttestationOnDisk {
+    attesting_indices: Vec<u64>,
+    data: AttestationData,
+    signature: AggregateSignature,
+}
+
+impl IndexedAttestationOnDisk {
+    fn into_indexed_attestation<E: EthSpec>(
+        self,
+        spec: &ChainSpec,
+    ) -> Result<IndexedAttestation<E>, Error> {
+        let fork_at_target_epoch = spec.fork_name_at_epoch(self.data.target.epoch);
+        if fork_at_target_epoch.electra_enabled() {
+            let attesting_indices = VariableList::new(self.attesting_indices)?;
+            Ok(IndexedAttestation::Electra(IndexedAttestationElectra {
+                attesting_indices,
+                data: self.data,
+                signature: self.signature,
+            }))
+        } else {
+            let attesting_indices = VariableList::new(self.attesting_indices)?;
+            Ok(IndexedAttestation::Base(IndexedAttestationBase {
+                attesting_indices,
+                data: self.data,
+                signature: self.signature,
+            }))
+        }
+    }
+}
+
 /// Bincode deserialization specialised to `Cow<[u8]>`.
 fn bincode_deserialize<T: DeserializeOwned>(bytes: Cow<[u8]>) -> Result<T, Error> {
    Ok(bincode::deserialize(bytes.borrow())?)
@@ -245,7 +286,7 @@ fn ssz_decode<T: Decode>(bytes: Cow<[u8]>) -> Result<T, Error> {
 }

 impl<E: EthSpec> SlasherDB<E> {
-    pub fn open(config: Arc<Config>, log: Logger) -> Result<Self, Error> {
+    pub fn open(config: Arc<Config>, spec: Arc<ChainSpec>, log: Logger) -> Result<Self, Error> {
        info!(log, "Opening slasher database"; "backend" => %config.backend);

        std::fs::create_dir_all(&config.database_path)?;
@@ -268,6 +309,7 @@ impl<E: EthSpec> SlasherDB<E> {
            databases,
            attestation_root_cache,
            config,
+            spec,
            _phantom: PhantomData,
        };

@@ -438,7 +480,7 @@ impl<E: EthSpec> SlasherDB<E> {
    ) -> Result<u64, Error> {
        // Look-up ID by hash.
        let id_key = IndexedAttestationIdKey::new(
-            indexed_attestation.data.target.epoch,
+            indexed_attestation.data().target.epoch,
            indexed_attestation_hash,
        );

@@ -457,6 +499,7 @@ impl<E: EthSpec> SlasherDB<E> {
        };

        let attestation_key = IndexedAttestationId::new(indexed_att_id);
+        // IndexedAttestationOnDisk and IndexedAttestation have compatible encodings.
        let data = indexed_attestation.as_ssz_bytes();

        cursor.put(attestation_key.as_ref(), &data)?;
@@ -481,7 +524,8 @@ impl<E: EthSpec> SlasherDB<E> {
            .ok_or(Error::MissingIndexedAttestation {
                id: indexed_attestation_id.as_u64(),
            })?;
-        ssz_decode(bytes)
+        let indexed_attestation_on_disk: IndexedAttestationOnDisk = ssz_decode(bytes)?;
+        indexed_attestation_on_disk.into_indexed_attestation(&self.spec)
    }

    fn get_attestation_data_root(
@@ -500,7 +544,7 @@ impl<E: EthSpec> SlasherDB<E> {

        // Otherwise, load the indexed attestation, compute the root and cache it.
        let indexed_attestation = self.get_indexed_attestation(txn, indexed_id)?;
-        let attestation_data_root = indexed_attestation.data.tree_hash_root();
+        let attestation_data_root = indexed_attestation.data().tree_hash_root();

        cache.put(indexed_id, attestation_data_root);

@@ -536,7 +580,7 @@ impl<E: EthSpec> SlasherDB<E> {
        indexed_attestation_id: IndexedAttestationId,
    ) -> Result<AttesterSlashingStatus<E>, Error> {
        // See if there's an existing attestation for this attester.
-        let target_epoch = attestation.data.target.epoch;
+        let target_epoch = attestation.data().target.epoch;

        let prev_max_target = self.get_attester_max_target(validator_index, txn)?;

@@ -771,3 +815,93 @@ impl<E: EthSpec> SlasherDB<E> {
        Ok(())
    }
 }
+
+#[cfg(test)]
+mod test {
+    use super::*;
+    use types::{Checkpoint, ForkName, MainnetEthSpec, Unsigned};
+
+    type E = MainnetEthSpec;
+
+    fn indexed_attestation_on_disk_roundtrip_test(
+        spec: &ChainSpec,
+        make_attestation: fn(
+            Vec<u64>,
+            AttestationData,
+            AggregateSignature,
+        ) -> IndexedAttestation<E>,
+        committee_len: u64,
+    ) {
+        let attestation_data = AttestationData {
+            slot: Slot::new(1000),
+            index: 0,
+            beacon_block_root: Hash256::repeat_byte(0xaa),
+            source: Checkpoint {
+                epoch: Epoch::new(0),
+                root: Hash256::repeat_byte(0xbb),
+            },
+            target: Checkpoint {
+                epoch: Epoch::new(31),
+                root: Hash256::repeat_byte(0xcc),
+            },
+        };
+
+        let attesting_indices = (0..committee_len).collect::<Vec<_>>();
+        let signature = AggregateSignature::infinity();
+
+        let fork_attestation = make_attestation(
+            attesting_indices.clone(),
+            attestation_data.clone(),
+            signature.clone(),
+        );
+
+        let on_disk = IndexedAttestationOnDisk {
+            attesting_indices,
+            data: attestation_data,
+            signature,
+        };
+        let encoded = on_disk.as_ssz_bytes();
+        assert_eq!(encoded, fork_attestation.as_ssz_bytes());
+
+        let decoded_on_disk = IndexedAttestationOnDisk::from_ssz_bytes(&encoded).unwrap();
+        assert_eq!(decoded_on_disk, on_disk);
+
+        let decoded = on_disk.into_indexed_attestation(spec).unwrap();
+        assert_eq!(decoded, fork_attestation);
+    }
+
+    /// Check that `IndexedAttestationOnDisk` and `IndexedAttestation` have compatible encodings.
+    #[test]
+    fn indexed_attestation_on_disk_roundtrip_base() {
+        let spec = ForkName::Base.make_genesis_spec(E::default_spec());
+        let make_attestation = |attesting_indices, data, signature| {
+            IndexedAttestation::<E>::Base(IndexedAttestationBase {
+                attesting_indices: VariableList::new(attesting_indices).unwrap(),
+                data,
+                signature,
+            })
+        };
+        indexed_attestation_on_disk_roundtrip_test(
+            &spec,
+            make_attestation,
+            <E as EthSpec>::MaxValidatorsPerCommittee::to_u64(),
+        )
+    }
+
+    #[test]
+    fn indexed_attestation_on_disk_roundtrip_electra() {
+        let spec = ForkName::Electra.make_genesis_spec(E::default_spec());
+        let make_attestation = |attesting_indices, data, signature| {
+            IndexedAttestation::<E>::Electra(IndexedAttestationElectra {
+                attesting_indices: VariableList::new(attesting_indices).unwrap(),
+                data,
+                signature,
+            })
+        };
+        indexed_attestation_on_disk_roundtrip_test(
+            &spec,
+            make_attestation,
+            <E as EthSpec>::MaxValidatorsPerSlot::to_u64(),
+        )
+    }
+}