Electra: Remaining Consensus Data Structures (#5712)

* Attestation superstruct changes for EIP 7549 (#5644)

* update

* experiment

* superstruct changes

* revert

* superstruct changes

* fix tests

* indexed attestation

* indexed attestation superstruct

* updated TODOs

* `superstruct` the `AttesterSlashing` (#5636)

* `superstruct` Attester Fork Variants

* Push a little further

* Deal with Encode / Decode of AttesterSlashing

* not so sure about this..

* Stop Encode/Decode Bounds from Propagating Out

* Tons of Changes..

* More Conversions to AttestationRef

* Add AsReference trait (#15)

* Add AsReference trait

* Fix some snafus

* Got it Compiling! :D

* Got Tests Building

* Get beacon chain tests compiling

---------

Co-authored-by: Michael Sproul <micsproul@gmail.com>

* Merge remote-tracking branch 'upstream/unstable' into electra_attestation_changes

* Make EF Tests Fork-Agnostic (#5713)

* Finish EF Test Fork Agnostic (#5714)

* Superstruct `AggregateAndProof` (#5715)

* Upgrade `superstruct` to `0.8.0`

* superstruct `AggregateAndProof`

* Merge remote-tracking branch 'sigp/unstable' into electra_attestation_changes

* cargo fmt

* Merge pull request #5726 from realbigsean/electra_attestation_changes

Merge unstable into Electra attestation changes

* EIP7549 `get_attestation_indices` (#5657)

* get attesting indices electra impl

* fmt

* get tests to pass

* fmt

* fix some beacon chain tests

* fmt

* fix slasher test

* fmt got me again

* fix more tests

* fix tests

* Some small changes (#5739)

* cargo fmt (#5740)

* Sketch op pool changes

* fix get attesting indices (#5742)

* fix get attesting indices

* better errors

* fix compile

* only get committee index once

* Ef test fixes (#5753)

* attestation related ef test fixes

* delete commented out stuff

* Fix Aggregation Pool for Electra (#5754)

* Fix Aggregation Pool for Electra

* Remove Outdated Interface

* fix ssz (#5755)

* Get `electra_op_pool` up to date (#5756)

* fix get attesting indices (#5742)

* fix get attesting indices

* better errors

* fix compile

* only get committee index once

* Ef test fixes (#5753)

* attestation related ef test fixes

* delete commented out stuff

* Fix Aggregation Pool for Electra (#5754)

* Fix Aggregation Pool for Electra

* Remove Outdated Interface

* fix ssz (#5755)

---------

Co-authored-by: realbigsean <sean@sigmaprime.io>

* Revert "Get `electra_op_pool` up to date (#5756)" (#5757)

This reverts commit ab9e58aa3d.

* Merge branch 'electra_attestation_changes' of https://github.com/sigp/lighthouse into electra_op_pool

* Compute on chain aggregate impl (#5752)

* add compute_on_chain_agg impl to op pool changes

* fmt

* get op pool tests to pass

* update the naive agg pool interface (#5760)

* Fix bugs in cross-committee aggregation

* Add comment to max cover optimisation

* Fix assert

* Merge pull request #5749 from sigp/electra_op_pool

Optimise Electra op pool aggregation

* update committee offset

* Fix Electra Fork Choice Tests (#5764)

* Subscribe to the correct subnets for electra attestations (#5782)

* subscribe to the correct att subnets for electra

* subscribe to the correct att subnets for electra

* cargo fmt

* fix slashing handling

* Merge remote-tracking branch 'upstream/unstable'

* Send unagg attestation based on fork

* Publish all aggregates

* just one more check bro plz..

* Merge pull request #5832 from ethDreamer/electra_attestation_changes_merge_unstable

Merge `unstable` into `electra_attestation_changes`

* Merge pull request #5835 from realbigsean/fix-validator-logic

Fix validator logic

* Merge pull request #5816 from realbigsean/electra-attestation-slashing-handling

Electra slashing handling

* Electra attestation changes rm decode impl (#5856)

* Remove Crappy Decode impl for Attestation

* Remove Inefficient Attestation Decode impl

* Implement Schema Upgrade / Downgrade

* Update beacon_node/beacon_chain/src/schema_change/migration_schema_v20.rs

Co-authored-by: Michael Sproul <micsproul@gmail.com>

---------

Co-authored-by: Michael Sproul <micsproul@gmail.com>

* Fix failing attestation tests and misc electra attestation cleanup (#5810)

* - get attestation related beacon chain tests to pass
- observed attestations are now keyed off of data + committee index
- rename op pool attestationref to compactattestationref
- remove unwraps in agg pool and use options instead
- cherry pick some changes from ef-tests-electra

* cargo fmt

* fix failing test

* Revert dockerfile changes

* make committee_index return option

* function args shouldnt be a ref to attestation ref

* fmt

* fix dup imports

---------

Co-authored-by: realbigsean <seananderson33@GMAIL.com>

* fix some todos (#5817)

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into electra_attestation_changes

* add consolidations to merkle calc for inclusion proof

* Remove Duplicate KZG Commitment Merkle Proof Code (#5874)

* Remove Duplicate KZG Commitment Merkle Proof Code

* s/tree_lists/fields/

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into electra_attestation_changes

* fix compile

* Fix slasher tests (#5906)

* Fix electra tests

* Add electra attestations to double vote tests

* Update superstruct to 0.8

* Merge remote-tracking branch 'origin/unstable' into electra_attestation_changes

* Small cleanup in slasher tests

* Clean up Electra observed aggregates (#5929)

* Use consistent key in observed_attestations

* Remove unwraps from observed aggregates

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into electra_attestation_changes

* De-dup attestation constructor logic

* Remove unwraps in Attestation construction

* Dedup match_attestation_data

* Remove outdated TODO

* Use ForkName Ord in fork-choice tests

* Use ForkName Ord in BeaconBlockBody

* Make to_electra not fallible

* Remove TestRandom impl for IndexedAttestation

* Remove IndexedAttestation faulty Decode impl

* Drop TestRandom impl

* Add PendingAttestationInElectra

* Indexed att on disk (#35)

* indexed att on disk

* fix lints

* Update slasher/src/migrate.rs

Co-authored-by: ethDreamer <37123614+ethDreamer@users.noreply.github.com>

---------

Co-authored-by: Lion - dapplion <35266934+dapplion@users.noreply.github.com>
Co-authored-by: ethDreamer <37123614+ethDreamer@users.noreply.github.com>

* add electra fork enabled fn to ForkName impl (#36)

* add electra fork enabled fn to ForkName impl

* remove inadvertent file

* Update common/eth2/src/types.rs

Co-authored-by: ethDreamer <37123614+ethDreamer@users.noreply.github.com>

* Dedup attestation constructor logic in attester cache

* Use if let Ok for committee_bits

* Dedup Attestation constructor code

* Diff reduction in tests

* Fix beacon_chain tests

* Diff reduction

* Use Ord for ForkName in pubsub

* Resolve into_attestation_and_indices todo

* Remove stale TODO

* Fix beacon_chain tests

* Test spec invariant

* Use electra_enabled in pubsub

* Remove get_indexed_attestation_from_signed_aggregate

* Use ok_or instead of if let else

* committees are sorted

* remove dup method `get_indexed_attestation_from_committees`

* Merge pull request #5940 from dapplion/electra_attestation_changes_lionreview

Electra attestations #5712 review

* update default persisted op pool deserialization

* ensure aggregate and proof uses serde untagged on ref

* Fork aware ssz static attestation tests

* Electra attestation changes from Lions review (#5971)

* dedup/cleanup and remove unneeded hashset use

* remove irrelevant TODOs

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into electra_attestation_changes

* Electra attestation changes sean review (#5972)

* instantiate empty bitlist in unreachable code

* clean up error conversion

* fork enabled bool cleanup

* remove a couple todos

* return bools instead of options in `aggregate` and use the result

* delete commented out code

* use map macros in simple transformations

* remove signers_disjoint_from

* get ef tests compiling

* get ef tests compiling

* update intentionally excluded files

* Avoid changing slasher schema for Electra

* Delete slasher schema v4

* Fix clippy

* Fix compilation of beacon_chain tests

* Update database.rs

* Add electra lightclient types

* Update slasher/src/database.rs

* fix imports

* Merge pull request #5980 from dapplion/electra-lightclient

Add electra lightclient types

* Merge pull request #5975 from michaelsproul/electra-slasher-no-migration

Avoid changing slasher schema for Electra

* Update beacon_node/beacon_chain/src/attestation_verification.rs

* Update beacon_node/beacon_chain/src/attestation_verification.rs
This commit is contained in:
ethDreamer
2024-06-24 16:08:07 -05:00
committed by GitHub
parent 758b58c9e9
commit c52c598f69
118 changed files with 5076 additions and 1741 deletions

View File

@@ -58,6 +58,7 @@ use store::{config::StoreConfig, HotColdDB, ItemStore, LevelDB, MemoryStore};
use task_executor::TaskExecutor;
use task_executor::{test_utils::TestRuntime, ShutdownReason};
use tree_hash::TreeHash;
use types::indexed_attestation::IndexedAttestationBase;
use types::payload::BlockProductionVersion;
pub use types::test_utils::generate_deterministic_keypairs;
use types::test_utils::TestRandom;
@@ -1031,20 +1032,18 @@ where
*state.get_block_root(target_slot)?
};
Ok(Attestation {
aggregation_bits: BitList::with_capacity(committee_len)?,
data: AttestationData {
slot,
index,
beacon_block_root,
source: state.current_justified_checkpoint(),
target: Checkpoint {
epoch,
root: target_root,
},
Ok(Attestation::empty_for_signing(
index,
committee_len,
slot,
beacon_block_root,
state.current_justified_checkpoint(),
Checkpoint {
epoch,
root: target_root,
},
signature: AggregateSignature::empty(),
})
&self.spec,
)?)
}
/// A list of attestations for each committee for the given slot.
@@ -1119,17 +1118,24 @@ where
)
.unwrap();
attestation.aggregation_bits.set(i, true).unwrap();
match attestation {
Attestation::Base(ref mut att) => {
att.aggregation_bits.set(i, true).unwrap()
}
Attestation::Electra(ref mut att) => {
att.aggregation_bits.set(i, true).unwrap()
}
}
attestation.signature = {
*attestation.signature_mut() = {
let domain = self.spec.get_domain(
attestation.data.target.epoch,
attestation.data().target.epoch,
Domain::BeaconAttester,
&fork,
state.genesis_validators_root(),
);
let message = attestation.data.signing_root(domain);
let message = attestation.data().signing_root(domain);
let mut agg_sig = AggregateSignature::infinity();
@@ -1140,8 +1146,8 @@ where
agg_sig
};
let subnet_id = SubnetId::compute_subnet_for_attestation_data::<E>(
&attestation.data,
let subnet_id = SubnetId::compute_subnet_for_attestation::<E>(
attestation.to_ref(),
committee_count,
&self.chain.spec,
)
@@ -1315,7 +1321,10 @@ where
// If there are any attestations in this committee, create an aggregate.
if let Some((attestation, _)) = committee_attestations.first() {
let bc = state
.get_beacon_committee(attestation.data.slot, attestation.data.index)
.get_beacon_committee(
attestation.data().slot,
attestation.committee_index().unwrap(),
)
.unwrap();
// Find an aggregator if one exists. Return `None` if there are no
@@ -1342,25 +1351,35 @@ where
})
.copied()?;
let fork_name = self.spec.fork_name_at_slot::<E>(slot);
let aggregate = if fork_name.electra_enabled() {
self.chain.get_aggregated_attestation_electra(
slot,
&attestation.data().tree_hash_root(),
bc.index,
)
} else {
self.chain
.get_aggregated_attestation_base(attestation.data())
}
.unwrap()
.unwrap_or_else(|| {
committee_attestations.iter().skip(1).fold(
attestation.clone(),
|mut agg, (att, _)| {
agg.aggregate(att.to_ref());
agg
},
)
});
// If the chain is able to produce an aggregate, use that. Otherwise, build an
// aggregate locally.
let aggregate = self
.chain
.get_aggregated_attestation(&attestation.data)
.unwrap()
.unwrap_or_else(|| {
committee_attestations.iter().skip(1).fold(
attestation.clone(),
|mut agg, (att, _)| {
agg.aggregate(att);
agg
},
)
});
let signed_aggregate = SignedAggregateAndProof::from_aggregate(
aggregator_index as u64,
aggregate,
aggregate.to_ref(),
None,
&self.validator_keypairs[aggregator_index].sk,
&fork,
@@ -1486,50 +1505,89 @@ where
) -> AttesterSlashing<E> {
let fork = self.chain.canonical_head.cached_head().head_fork();
let mut attestation_1 = IndexedAttestation {
attesting_indices: VariableList::new(validator_indices).unwrap(),
data: AttestationData {
slot: Slot::new(0),
index: 0,
beacon_block_root: Hash256::zero(),
target: Checkpoint {
root: Hash256::zero(),
epoch: target1.unwrap_or(fork.epoch),
},
source: Checkpoint {
root: Hash256::zero(),
epoch: source1.unwrap_or(Epoch::new(0)),
},
let fork_name = self.spec.fork_name_at_slot::<E>(Slot::new(0));
let data = AttestationData {
slot: Slot::new(0),
index: 0,
beacon_block_root: Hash256::zero(),
target: Checkpoint {
root: Hash256::zero(),
epoch: target1.unwrap_or(fork.epoch),
},
signature: AggregateSignature::infinity(),
source: Checkpoint {
root: Hash256::zero(),
epoch: source1.unwrap_or(Epoch::new(0)),
},
};
let mut attestation_1 = if fork_name.electra_enabled() {
IndexedAttestation::Electra(IndexedAttestationElectra {
attesting_indices: VariableList::new(validator_indices).unwrap(),
data,
signature: AggregateSignature::infinity(),
})
} else {
IndexedAttestation::Base(IndexedAttestationBase {
attesting_indices: VariableList::new(validator_indices).unwrap(),
data,
signature: AggregateSignature::infinity(),
})
};
let mut attestation_2 = attestation_1.clone();
attestation_2.data.index += 1;
attestation_2.data.source.epoch = source2.unwrap_or(Epoch::new(0));
attestation_2.data.target.epoch = target2.unwrap_or(fork.epoch);
attestation_2.data_mut().index += 1;
attestation_2.data_mut().source.epoch = source2.unwrap_or(Epoch::new(0));
attestation_2.data_mut().target.epoch = target2.unwrap_or(fork.epoch);
for attestation in &mut [&mut attestation_1, &mut attestation_2] {
for &i in &attestation.attesting_indices {
let sk = &self.validator_keypairs[i as usize].sk;
match attestation {
IndexedAttestation::Base(attestation) => {
for i in attestation.attesting_indices.iter() {
let sk = &self.validator_keypairs[*i as usize].sk;
let genesis_validators_root = self.chain.genesis_validators_root;
let genesis_validators_root = self.chain.genesis_validators_root;
let domain = self.chain.spec.get_domain(
attestation.data.target.epoch,
Domain::BeaconAttester,
&fork,
genesis_validators_root,
);
let message = attestation.data.signing_root(domain);
let domain = self.chain.spec.get_domain(
attestation.data.target.epoch,
Domain::BeaconAttester,
&fork,
genesis_validators_root,
);
let message = attestation.data.signing_root(domain);
attestation.signature.add_assign(&sk.sign(message));
attestation.signature.add_assign(&sk.sign(message));
}
}
IndexedAttestation::Electra(attestation) => {
for i in attestation.attesting_indices.iter() {
let sk = &self.validator_keypairs[*i as usize].sk;
let genesis_validators_root = self.chain.genesis_validators_root;
let domain = self.chain.spec.get_domain(
attestation.data.target.epoch,
Domain::BeaconAttester,
&fork,
genesis_validators_root,
);
let message = attestation.data.signing_root(domain);
attestation.signature.add_assign(&sk.sign(message));
}
}
}
}
AttesterSlashing {
attestation_1,
attestation_2,
if fork_name.electra_enabled() {
AttesterSlashing::Electra(AttesterSlashingElectra {
attestation_1: attestation_1.as_electra().unwrap().clone(),
attestation_2: attestation_2.as_electra().unwrap().clone(),
})
} else {
AttesterSlashing::Base(AttesterSlashingBase {
attestation_1: attestation_1.as_base().unwrap().clone(),
attestation_2: attestation_2.as_base().unwrap().clone(),
})
}
}
@@ -1538,6 +1596,8 @@ where
validator_indices_1: Vec<u64>,
validator_indices_2: Vec<u64>,
) -> AttesterSlashing<E> {
let fork_name = self.spec.fork_name_at_slot::<E>(Slot::new(0));
let data = AttestationData {
slot: Slot::new(0),
index: 0,
@@ -1552,42 +1612,94 @@ where
},
};
let mut attestation_1 = IndexedAttestation {
attesting_indices: VariableList::new(validator_indices_1).unwrap(),
data: data.clone(),
signature: AggregateSignature::infinity(),
let (mut attestation_1, mut attestation_2) = if fork_name.electra_enabled() {
let attestation_1 = IndexedAttestationElectra {
attesting_indices: VariableList::new(validator_indices_1).unwrap(),
data: data.clone(),
signature: AggregateSignature::infinity(),
};
let attestation_2 = IndexedAttestationElectra {
attesting_indices: VariableList::new(validator_indices_2).unwrap(),
data,
signature: AggregateSignature::infinity(),
};
(
IndexedAttestation::Electra(attestation_1),
IndexedAttestation::Electra(attestation_2),
)
} else {
let attestation_1 = IndexedAttestationBase {
attesting_indices: VariableList::new(validator_indices_1).unwrap(),
data: data.clone(),
signature: AggregateSignature::infinity(),
};
let attestation_2 = IndexedAttestationBase {
attesting_indices: VariableList::new(validator_indices_2).unwrap(),
data,
signature: AggregateSignature::infinity(),
};
(
IndexedAttestation::Base(attestation_1),
IndexedAttestation::Base(attestation_2),
)
};
let mut attestation_2 = IndexedAttestation {
attesting_indices: VariableList::new(validator_indices_2).unwrap(),
data,
signature: AggregateSignature::infinity(),
};
attestation_2.data.index += 1;
attestation_2.data_mut().index += 1;
let fork = self.chain.canonical_head.cached_head().head_fork();
for attestation in &mut [&mut attestation_1, &mut attestation_2] {
for &i in &attestation.attesting_indices {
let sk = &self.validator_keypairs[i as usize].sk;
match attestation {
IndexedAttestation::Base(attestation) => {
for i in attestation.attesting_indices.iter() {
let sk = &self.validator_keypairs[*i as usize].sk;
let genesis_validators_root = self.chain.genesis_validators_root;
let genesis_validators_root = self.chain.genesis_validators_root;
let domain = self.chain.spec.get_domain(
attestation.data.target.epoch,
Domain::BeaconAttester,
&fork,
genesis_validators_root,
);
let message = attestation.data.signing_root(domain);
let domain = self.chain.spec.get_domain(
attestation.data.target.epoch,
Domain::BeaconAttester,
&fork,
genesis_validators_root,
);
let message = attestation.data.signing_root(domain);
attestation.signature.add_assign(&sk.sign(message));
attestation.signature.add_assign(&sk.sign(message));
}
}
IndexedAttestation::Electra(attestation) => {
for i in attestation.attesting_indices.iter() {
let sk = &self.validator_keypairs[*i as usize].sk;
let genesis_validators_root = self.chain.genesis_validators_root;
let domain = self.chain.spec.get_domain(
attestation.data.target.epoch,
Domain::BeaconAttester,
&fork,
genesis_validators_root,
);
let message = attestation.data.signing_root(domain);
attestation.signature.add_assign(&sk.sign(message));
}
}
}
}
AttesterSlashing {
attestation_1,
attestation_2,
if fork_name.electra_enabled() {
AttesterSlashing::Electra(AttesterSlashingElectra {
attestation_1: attestation_1.as_electra().unwrap().clone(),
attestation_2: attestation_2.as_electra().unwrap().clone(),
})
} else {
AttesterSlashing::Base(AttesterSlashingBase {
attestation_1: attestation_1.as_base().unwrap().clone(),
attestation_2: attestation_2.as_base().unwrap().clone(),
})
}
}
@@ -2380,6 +2492,7 @@ where
AttestationStrategy::AllValidators => self.get_all_validators(),
AttestationStrategy::SomeValidators(vals) => vals,
};
let state_root = state.update_tree_hash_cache().unwrap();
let (_, _, last_produced_block_hash, _) = self
.add_attested_blocks_at_slots_with_sync(
@@ -2506,6 +2619,7 @@ pub fn generate_rand_block_and_blobs<E: EthSpec>(
rng: &mut impl Rng,
) -> (SignedBeaconBlock<E, FullPayload<E>>, Vec<BlobSidecar<E>>) {
let inner = map_fork_name!(fork_name, BeaconBlock, <_>::random_for_test(rng));
let mut block = SignedBeaconBlock::from_block(inner, types::Signature::random_for_test(rng));
let mut blob_sidecars = vec![];
@@ -2542,7 +2656,6 @@ pub fn generate_rand_block_and_blobs<E: EthSpec>(
};
let (bundle, transactions) =
execution_layer::test_utils::generate_blobs::<E>(num_blobs).unwrap();
payload.execution_payload.transactions = <_>::default();
for tx in Vec::from(transactions) {
payload.execution_payload.transactions.push(tx).unwrap();