Electra: Remaining Consensus Data Structures (#5712)

* Attestation superstruct changes for EIP 7549 (#5644)

* update

* experiment

* superstruct changes

* revert

* superstruct changes

* fix tests

* indexed attestation

* indexed attestation superstruct

* updated TODOs

* `superstruct` the `AttesterSlashing` (#5636)

* `superstruct` Attester Fork Variants

* Push a little further

* Deal with Encode / Decode of AttesterSlashing

* not so sure about this..

* Stop Encode/Decode Bounds from Propagating Out

* Tons of Changes..

* More Conversions to AttestationRef

* Add AsReference trait (#15)

* Add AsReference trait

* Fix some snafus

* Got it Compiling! :D

* Got Tests Building

* Get beacon chain tests compiling

---------

Co-authored-by: Michael Sproul <micsproul@gmail.com>

* Merge remote-tracking branch 'upstream/unstable' into electra_attestation_changes

* Make EF Tests Fork-Agnostic (#5713)

* Finish EF Test Fork Agnostic (#5714)

* Superstruct `AggregateAndProof` (#5715)

* Upgrade `superstruct` to `0.8.0`

* superstruct `AggregateAndProof`

* Merge remote-tracking branch 'sigp/unstable' into electra_attestation_changes

* cargo fmt

* Merge pull request #5726 from realbigsean/electra_attestation_changes

Merge unstable into Electra attestation changes

* EIP7549 `get_attestation_indices` (#5657)

* get attesting indices electra impl

* fmt

* get tests to pass

* fmt

* fix some beacon chain tests

* fmt

* fix slasher test

* fmt got me again

* fix more tests

* fix tests

* Some small changes (#5739)

* cargo fmt (#5740)

* Sketch op pool changes

* fix get attesting indices (#5742)

* fix get attesting indices

* better errors

* fix compile

* only get committee index once

* Ef test fixes (#5753)

* attestation related ef test fixes

* delete commented out stuff

* Fix Aggregation Pool for Electra (#5754)

* Fix Aggregation Pool for Electra

* Remove Outdated Interface

* fix ssz (#5755)

* Get `electra_op_pool` up to date (#5756)

* fix get attesting indices (#5742)

* fix get attesting indices

* better errors

* fix compile

* only get committee index once

* Ef test fixes (#5753)

* attestation related ef test fixes

* delete commented out stuff

* Fix Aggregation Pool for Electra (#5754)

* Fix Aggregation Pool for Electra

* Remove Outdated Interface

* fix ssz (#5755)

---------

Co-authored-by: realbigsean <sean@sigmaprime.io>

* Revert "Get `electra_op_pool` up to date (#5756)" (#5757)

This reverts commit ab9e58aa3d.

* Merge branch 'electra_attestation_changes' of https://github.com/sigp/lighthouse into electra_op_pool

* Compute on chain aggregate impl (#5752)

* add compute_on_chain_agg impl to op pool changes

* fmt

* get op pool tests to pass

* update the naive agg pool interface (#5760)

* Fix bugs in cross-committee aggregation

* Add comment to max cover optimisation

* Fix assert

* Merge pull request #5749 from sigp/electra_op_pool

Optimise Electra op pool aggregation

* update committee offset

* Fix Electra Fork Choice Tests (#5764)

* Subscribe to the correct subnets for electra attestations (#5782)

* subscribe to the correct att subnets for electra

* subscribe to the correct att subnets for electra

* cargo fmt

* fix slashing handling

* Merge remote-tracking branch 'upstream/unstable'

* Send unagg attestation based on fork

* Publish all aggregates

* just one more check bro plz..

* Merge pull request #5832 from ethDreamer/electra_attestation_changes_merge_unstable

Merge `unstable` into `electra_attestation_changes`

* Merge pull request #5835 from realbigsean/fix-validator-logic

Fix validator logic

* Merge pull request #5816 from realbigsean/electra-attestation-slashing-handling

Electra slashing handling

* Electra attestation changes rm decode impl (#5856)

* Remove Crappy Decode impl for Attestation

* Remove Inefficient Attestation Decode impl

* Implement Schema Upgrade / Downgrade

* Update beacon_node/beacon_chain/src/schema_change/migration_schema_v20.rs

Co-authored-by: Michael Sproul <micsproul@gmail.com>

---------

Co-authored-by: Michael Sproul <micsproul@gmail.com>

* Fix failing attestation tests and misc electra attestation cleanup (#5810)

* - get attestation related beacon chain tests to pass
- observed attestations are now keyed off of data + committee index
- rename op pool attestationref to compactattestationref
- remove unwraps in agg pool and use options instead
- cherry pick some changes from ef-tests-electra

* cargo fmt

* fix failing test

* Revert dockerfile changes

* make committee_index return option

* function args shouldnt be a ref to attestation ref

* fmt

* fix dup imports

---------

Co-authored-by: realbigsean <seananderson33@GMAIL.com>

* fix some todos (#5817)

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into electra_attestation_changes

* add consolidations to merkle calc for inclusion proof

* Remove Duplicate KZG Commitment Merkle Proof Code (#5874)

* Remove Duplicate KZG Commitment Merkle Proof Code

* s/tree_lists/fields/

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into electra_attestation_changes

* fix compile

* Fix slasher tests (#5906)

* Fix electra tests

* Add electra attestations to double vote tests

* Update superstruct to 0.8

* Merge remote-tracking branch 'origin/unstable' into electra_attestation_changes

* Small cleanup in slasher tests

* Clean up Electra observed aggregates (#5929)

* Use consistent key in observed_attestations

* Remove unwraps from observed aggregates

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into electra_attestation_changes

* De-dup attestation constructor logic

* Remove unwraps in Attestation construction

* Dedup match_attestation_data

* Remove outdated TODO

* Use ForkName Ord in fork-choice tests

* Use ForkName Ord in BeaconBlockBody

* Make to_electra not fallible

* Remove TestRandom impl for IndexedAttestation

* Remove IndexedAttestation faulty Decode impl

* Drop TestRandom impl

* Add PendingAttestationInElectra

* Indexed att on disk (#35)

* indexed att on disk

* fix lints

* Update slasher/src/migrate.rs

Co-authored-by: ethDreamer <37123614+ethDreamer@users.noreply.github.com>

---------

Co-authored-by: Lion - dapplion <35266934+dapplion@users.noreply.github.com>
Co-authored-by: ethDreamer <37123614+ethDreamer@users.noreply.github.com>

* add electra fork enabled fn to ForkName impl (#36)

* add electra fork enabled fn to ForkName impl

* remove inadvertent file

* Update common/eth2/src/types.rs

Co-authored-by: ethDreamer <37123614+ethDreamer@users.noreply.github.com>

* Dedup attestation constructor logic in attester cache

* Use if let Ok for committee_bits

* Dedup Attestation constructor code

* Diff reduction in tests

* Fix beacon_chain tests

* Diff reduction

* Use Ord for ForkName in pubsub

* Resolve into_attestation_and_indices todo

* Remove stale TODO

* Fix beacon_chain tests

* Test spec invariant

* Use electra_enabled in pubsub

* Remove get_indexed_attestation_from_signed_aggregate

* Use ok_or instead of if let else

* committees are sorted

* remove dup method `get_indexed_attestation_from_committees`

* Merge pull request #5940 from dapplion/electra_attestation_changes_lionreview

Electra attestations #5712 review

* update default persisted op pool deserialization

* ensure aggregate and proof uses serde untagged on ref

* Fork aware ssz static attestation tests

* Electra attestation changes from Lions review (#5971)

* dedup/cleanup and remove unneeded hashset use

* remove irrelevant TODOs

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into electra_attestation_changes

* Electra attestation changes sean review (#5972)

* instantiate empty bitlist in unreachable code

* clean up error conversion

* fork enabled bool cleanup

* remove a couple todos

* return bools instead of options in `aggregate` and use the result

* delete commented out code

* use map macros in simple transformations

* remove signers_disjoint_from

* get ef tests compiling

* get ef tests compiling

* update intentionally excluded files

* Avoid changing slasher schema for Electra

* Delete slasher schema v4

* Fix clippy

* Fix compilation of beacon_chain tests

* Update database.rs

* Add electra lightclient types

* Update slasher/src/database.rs

* fix imports

* Merge pull request #5980 from dapplion/electra-lightclient

Add electra lightclient types

* Merge pull request #5975 from michaelsproul/electra-slasher-no-migration

Avoid changing slasher schema for Electra

* Update beacon_node/beacon_chain/src/attestation_verification.rs

* Update beacon_node/beacon_chain/src/attestation_verification.rs

consensus/state_processing/src/verify_operation.rs

@@ -13,29 +13,132 @@ use ssz::{Decode, Encode};
 use ssz_derive::{Decode, Encode};
 use std::marker::PhantomData;
 use types::{
-    AttesterSlashing, BeaconState, ChainSpec, Epoch, EthSpec, Fork, ForkVersion, ProposerSlashing,
+    AttesterSlashing, AttesterSlashingBase, AttesterSlashingOnDisk, AttesterSlashingRefOnDisk,
+};
+use types::{
+    BeaconState, ChainSpec, Epoch, EthSpec, Fork, ForkVersion, ProposerSlashing,
     SignedBlsToExecutionChange, SignedVoluntaryExit,
 };
 
 const MAX_FORKS_VERIFIED_AGAINST: usize = 2;
 
+pub trait TransformPersist {
+    type Persistable: Encode + Decode;
+    type PersistableRef<'a>: Encode
+    where
+        Self: 'a;
+
+    /// Returns a reference to the object in a form that implements `Encode`
+    fn as_persistable_ref(&self) -> Self::PersistableRef<'_>;
+
+    /// Converts the object back into its original form.
+    fn from_persistable(persistable: Self::Persistable) -> Self;
+}
+
 /// Wrapper around an operation type that acts as proof that its signature has been checked.
 ///
 /// The inner `op` field is private, meaning instances of this type can only be constructed
 /// by calling `validate`.
-#[derive(Derivative, Debug, Clone, Encode, Decode)]
+#[derive(Derivative, Debug, Clone)]
 #[derivative(
     PartialEq,
     Eq,
-    Hash(bound = "T: Encode + Decode + std::hash::Hash, E: EthSpec")
+    Hash(bound = "T: TransformPersist + std::hash::Hash, E: EthSpec")
 )]
-pub struct SigVerifiedOp<T: Encode + Decode, E: EthSpec> {
+pub struct SigVerifiedOp<T: TransformPersist, E: EthSpec> {
     op: T,
     verified_against: VerifiedAgainst,
-    #[ssz(skip_serializing, skip_deserializing)]
     _phantom: PhantomData<E>,
 }
 
+impl<T: TransformPersist, E: EthSpec> Encode for SigVerifiedOp<T, E> {
+    fn is_ssz_fixed_len() -> bool {
+        <T::Persistable as Encode>::is_ssz_fixed_len()
+            && <VerifiedAgainst as Encode>::is_ssz_fixed_len()
+    }
+
+    #[allow(clippy::expect_used)]
+    fn ssz_fixed_len() -> usize {
+        if <Self as Encode>::is_ssz_fixed_len() {
+            <T::Persistable as Encode>::ssz_fixed_len()
+                .checked_add(<VerifiedAgainst as Encode>::ssz_fixed_len())
+                .expect("encode ssz_fixed_len length overflow")
+        } else {
+            ssz::BYTES_PER_LENGTH_OFFSET
+        }
+    }
+
+    #[allow(clippy::expect_used)]
+    fn ssz_bytes_len(&self) -> usize {
+        if <Self as Encode>::is_ssz_fixed_len() {
+            <Self as Encode>::ssz_fixed_len()
+        } else {
+            let persistable = self.op.as_persistable_ref();
+            persistable
+                .ssz_bytes_len()
+                .checked_add(self.verified_against.ssz_bytes_len())
+                .expect("ssz_bytes_len length overflow")
+        }
+    }
+
+    fn ssz_append(&self, buf: &mut Vec<u8>) {
+        let mut encoder = ssz::SszEncoder::container(buf, <Self as Encode>::ssz_fixed_len());
+        let persistable = self.op.as_persistable_ref();
+        encoder.append(&persistable);
+        encoder.append(&self.verified_against);
+        encoder.finalize();
+    }
+}
+
+impl<T: TransformPersist, E: EthSpec> Decode for SigVerifiedOp<T, E> {
+    fn is_ssz_fixed_len() -> bool {
+        <T::Persistable as Decode>::is_ssz_fixed_len()
+            && <VerifiedAgainst as Decode>::is_ssz_fixed_len()
+    }
+
+    #[allow(clippy::expect_used)]
+    fn ssz_fixed_len() -> usize {
+        if <Self as Decode>::is_ssz_fixed_len() {
+            <T::Persistable as Decode>::ssz_fixed_len()
+                .checked_add(<VerifiedAgainst as Decode>::ssz_fixed_len())
+                .expect("decode ssz_fixed_len length overflow")
+        } else {
+            ssz::BYTES_PER_LENGTH_OFFSET
+        }
+    }
+
+    fn from_ssz_bytes(bytes: &[u8]) -> Result<Self, ssz::DecodeError> {
+        let mut builder = ssz::SszDecoderBuilder::new(bytes);
+
+        // Register types based on whether they are fixed or variable length
+        if <T::Persistable as Decode>::is_ssz_fixed_len() {
+            builder.register_type::<T::Persistable>()?;
+        } else {
+            builder.register_anonymous_variable_length_item()?;
+        }
+
+        if <VerifiedAgainst as Decode>::is_ssz_fixed_len() {
+            builder.register_type::<VerifiedAgainst>()?;
+        } else {
+            builder.register_anonymous_variable_length_item()?;
+        }
+
+        let mut decoder = builder.build()?;
+        // Decode each component
+        let persistable: T::Persistable = decoder.decode_next()?;
+        let verified_against: VerifiedAgainst = decoder.decode_next()?;
+
+        // Use TransformPersist to convert persistable back into the original type
+        let op = T::from_persistable(persistable);
+
+        Ok(SigVerifiedOp {
+            op,
+            verified_against,
+            _phantom: PhantomData,
+        })
+    }
+}
+
 /// Information about the fork versions that this message was verified against.
 ///
 /// In general it is not safe to assume that a `SigVerifiedOp` constructed at some point in the past
@@ -109,7 +212,7 @@ where
 }
 
 /// Trait for operations that can be verified and transformed into a `SigVerifiedOp`.
-pub trait VerifyOperation<E: EthSpec>: Encode + Decode + Sized {
+pub trait VerifyOperation<E: EthSpec>: TransformPersist + Sized {
     type Error;
 
     fn validate(
@@ -152,15 +255,15 @@ impl<E: EthSpec> VerifyOperation<E> for AttesterSlashing<E> {
         state: &BeaconState<E>,
         spec: &ChainSpec,
     ) -> Result<SigVerifiedOp<Self, E>, Self::Error> {
-        verify_attester_slashing(state, &self, VerifySignatures::True, spec)?;
+        verify_attester_slashing(state, self.to_ref(), VerifySignatures::True, spec)?;
         Ok(SigVerifiedOp::new(self, state))
     }
 
     #[allow(clippy::arithmetic_side_effects)]
     fn verification_epochs(&self) -> SmallVec<[Epoch; MAX_FORKS_VERIFIED_AGAINST]> {
         smallvec![
-            self.attestation_1.data.target.epoch,
-            self.attestation_2.data.target.epoch
+            self.attestation_1().data().target.epoch,
+            self.attestation_2().data().target.epoch
         ]
     }
 }
@@ -237,3 +340,98 @@ impl<E: EthSpec> VerifyOperationAt<E> for SignedVoluntaryExit {
         Ok(SigVerifiedOp::new(self, state))
     }
 }
+
+impl TransformPersist for SignedVoluntaryExit {
+    type Persistable = Self;
+    type PersistableRef<'a> = &'a Self;
+
+    fn as_persistable_ref(&self) -> Self::PersistableRef<'_> {
+        self
+    }
+
+    fn from_persistable(persistable: Self::Persistable) -> Self {
+        persistable
+    }
+}
+
+impl<E: EthSpec> TransformPersist for AttesterSlashing<E> {
+    type Persistable = AttesterSlashingOnDisk<E>;
+    type PersistableRef<'a> = AttesterSlashingRefOnDisk<'a, E>;
+
+    fn as_persistable_ref(&self) -> Self::PersistableRef<'_> {
+        self.to_ref().into()
+    }
+
+    fn from_persistable(persistable: Self::Persistable) -> Self {
+        persistable.into()
+    }
+}
+
+// TODO: Remove this once we no longer support DB schema version 17
+impl<E: EthSpec> TransformPersist for types::AttesterSlashingBase<E> {
+    type Persistable = Self;
+    type PersistableRef<'a> = &'a Self;
+
+    fn as_persistable_ref(&self) -> Self::PersistableRef<'_> {
+        self
+    }
+
+    fn from_persistable(persistable: Self::Persistable) -> Self {
+        persistable
+    }
+}
+// TODO: Remove this once we no longer support DB schema version 17
+impl<E: EthSpec> From<SigVerifiedOp<AttesterSlashingBase<E>, E>>
+    for SigVerifiedOp<AttesterSlashing<E>, E>
+{
+    fn from(base: SigVerifiedOp<AttesterSlashingBase<E>, E>) -> Self {
+        SigVerifiedOp {
+            op: AttesterSlashing::Base(base.op),
+            verified_against: base.verified_against,
+            _phantom: PhantomData,
+        }
+    }
+}
+// TODO: Remove this once we no longer support DB schema version 17
+impl<E: EthSpec> TryFrom<SigVerifiedOp<AttesterSlashing<E>, E>>
+    for SigVerifiedOp<AttesterSlashingBase<E>, E>
+{
+    type Error = String;
+
+    fn try_from(slashing: SigVerifiedOp<AttesterSlashing<E>, E>) -> Result<Self, Self::Error> {
+        match slashing.op {
+            AttesterSlashing::Base(base) => Ok(SigVerifiedOp {
+                op: base,
+                verified_against: slashing.verified_against,
+                _phantom: PhantomData,
+            }),
+            AttesterSlashing::Electra(_) => Err("non-base attester slashing".to_string()),
+        }
+    }
+}
+
+impl TransformPersist for ProposerSlashing {
+    type Persistable = Self;
+    type PersistableRef<'a> = &'a Self;
+
+    fn as_persistable_ref(&self) -> Self::PersistableRef<'_> {
+        self
+    }
+
+    fn from_persistable(persistable: Self::Persistable) -> Self {
+        persistable
+    }
+}
+
+impl TransformPersist for SignedBlsToExecutionChange {
+    type Persistable = Self;
+    type PersistableRef<'a> = &'a Self;
+
+    fn as_persistable_ref(&self) -> Self::PersistableRef<'_> {
+        self
+    }
+
+    fn from_persistable(persistable: Self::Persistable) -> Self {
+        persistable
+    }
+}