Process network attestations (#966)

* Start updating types

* WIP

* Signature hacking

* Existing EF tests passing with fake_crypto

* Updates

* Delete outdated API spec

* The refactor continues

* It compiles

* WIP test fixes

* All release tests passing bar genesis state parsing

* Update and test YamlConfig

* Update to spec v0.10 compatible BLS

* Updates to BLS EF tests

* Add EF test for AggregateVerify

And delete unused hash2curve tests for uncompressed points

* Update EF tests to v0.10.1

* Use optional block root correctly in block proc

* Use genesis fork in deposit domain. All tests pass

* Cargo fmt

* Fast aggregate verify test

* Update REST API docs

* Cargo fmt

* Fix unused import

* Bump spec tags to v0.10.1

* Add `seconds_per_eth1_block` to chainspec

* Update to timestamp based eth1 voting scheme

* Return None from `get_votes_to_consider` if block cache is empty

* Handle overflows in `is_candidate_block`

* Revert to failing tests

* Fix eth1 data sets test

* Choose default vote according to spec

* Fix collect_valid_votes tests

* Fix `get_votes_to_consider` to choose all eligible blocks

* Uncomment winning_vote tests

* Add comments; remove unused code

* Reduce seconds_per_eth1_block for simulation

* Addressed review comments

* Add test for default vote case

* Fix logs

* Remove unused functions

* Meter default eth1 votes

* Fix comments

* Address review comments; remove unused dependency

* Add first attempt at attestation proc. re-write

* Add version 2 of attestation processing

* Minor fixes

* Add validator pubkey cache

* Make get_indexed_attestation take a committee

* Link signature processing into new attn verification

* First working version

* Ensure pubkey cache is updated

* Add more metrics, slight optimizations

* Clone committee cache during attestation processing

* Update shuffling cache during block processing

* Remove old commented-out code

* Fix shuffling cache insert bug

* Used indexed attestation in fork choice

* Restructure attn processing, add metrics

* Add more detailed metrics

* Tidy, fix failing tests

* Fix failing tests, tidy

* Disable/delete two outdated tests

* Add new Pubkeys struct to signature_sets

* Refactor with functional approach

* Update beacon chain

* Remove decompressed member from pubkey bytes

* Add hashmap for indices lookup

* Add state cache, remove store cache

* Only build the head committee cache

* Change `get_attesting_indices` to use Vec

* Fix failing test

* Tidy

* Add pubkey cache persistence file

* Add more comments

* Integrate persistence file into builder

* Add pubkey cache tests

* Add data_dir to beacon chain builder

* Remove Option in pubkey cache persistence file

* Ensure consistency between datadir/data_dir

* Fix failing network test

* Tidy

* Fix todos

* Improve tests

* Fix compile error

* Fix compile error from merge

* Split up block processing metrics

* Tidy

* Refactor get_pubkey_from_state

* Remove commented-out code

* Rename state_cache -> checkpoint_cache

* Rename Checkpoint -> Snapshot

* Tidy, add comments

* Tidy up find_head function

* Change some checkpoint -> snapshot

* Add tests

* Expose max_len

* Remove dead code

* Tidy

* Fix bug

* Add sync-speed metric

* Add first attempt at VerifiableBlock

* Start integrating into beacon chain

* Integrate VerifiableBlock

* Rename VerifableBlock -> PartialBlockVerification

* Add start of typed methods

* Add progress

* Add further progress

* Rename structs

* Add full block verification to block_processing.rs

* Further beacon chain integration

* Update checks for gossip

* Add todo

* Start adding segement verification

* Add passing chain segement test

* Initial integration with batch sync

* Minor changes

* Tidy, add more error checking

* Start adding chain_segment tests

* Finish invalid signature tests

* Include single and gossip verified blocks in tests

* Add gossip verification tests

* Start adding docs

* Finish adding comments to block_processing.rs

* Rename block_processing.rs -> block_verification

* Start removing old block processing code

* Fixes beacon_chain compilation

* Fix project-wide compile errors

* Remove old code

* Fix bug with beacon proposer index

* Fix shim for BlockProcessingError

* Only process one epoch at a time

* Fix loop in chain segment processing

* Add caching for state.eth1_data_votes

* Add BeaconChain::validator_pubkey

* Revert "Add caching for state.eth1_data_votes"

This reverts commit cd73dcd643.

* Allow for clock disparity

* Ensure errors are returned during batch processing

* Add block gossip verification

* Connect attestation processing to beacon chain

* Optimistically subscribe to subnets on the same slot

Co-authored-by: Michael Sproul <michael@sigmaprime.io>
Co-authored-by: Michael Sproul <micsproul@gmail.com>
Co-authored-by: pawan <pawandhananjay@gmail.com>
Co-authored-by: Paul Hauner <paul@paulhauner.com>

beacon_node/beacon_chain/src/beacon_chain.rs

@@ -2,7 +2,7 @@ use crate::block_verification::{
     check_block_relevancy, get_block_root, signature_verify_chain_segment, BlockError,
     FullyVerifiedBlock, GossipVerifiedBlock, IntoFullyVerifiedBlock,
 };
-use crate::errors::{AttestationDropReason, BeaconChainError as Error, BlockProductionError};
+use crate::errors::{BeaconChainError as Error, BlockProductionError};
 use crate::eth1_chain::{Eth1Chain, Eth1ChainBackend};
 use crate::events::{EventHandler, EventKind};
 use crate::fork_choice::{Error as ForkChoiceError, ForkChoice};
@@ -1203,90 +1203,6 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
         }
     }
 
-    /// Check that the `aggregator_index` in an aggregate attestation is as it should be.
-    // TODO: Check for optimisation/relevance
-    fn check_attestation_aggregator(
-        &self,
-        signed_aggregate_and_proof: &SignedAggregateAndProof<T::EthSpec>,
-        indexed_attestation: &IndexedAttestation<T::EthSpec>,
-        state: &BeaconState<T::EthSpec>,
-    ) -> Result<(), AttestationDropReason> {
-        let aggregate_and_proof = &signed_aggregate_and_proof.message;
-        let attestation = &aggregate_and_proof.aggregate;
-
-        // Check that aggregator index is part of the committee attesting (quick).
-        if !indexed_attestation
-            .attesting_indices
-            .contains(&aggregate_and_proof.aggregator_index)
-        {
-            return Err(AttestationDropReason::AggregatorNotInAttestingIndices);
-        }
-        // Check that the aggregator is allowed to be aggregating (medium, one hash).
-        else if !state
-            .is_aggregator(
-                attestation.data.slot,
-                attestation.data.index,
-                &aggregate_and_proof.selection_proof,
-                &self.spec,
-            )
-            .unwrap_or(false)
-        {
-            return Err(AttestationDropReason::AggregatorNotSelected);
-        }
-        // Check that the signature is valid and the aggregator's selection proof is valid (slow-ish). Two sig verifications
-        if let Ok(Some(pubkey)) =
-            self.validator_pubkey(aggregate_and_proof.aggregator_index as usize)
-        {
-            if !signed_aggregate_and_proof.is_valid(&pubkey, &state.fork, &self.spec) {
-                Err(AttestationDropReason::AggregatorSignatureInvalid)
-            } else {
-                Ok(())
-            }
-        } else {
-            Err(AttestationDropReason::AggregatorNotInAttestingIndices)
-        }
-    }
-
-    /// Check that an attestation's slot doesn't make it ineligible for gossip.
-    fn check_attestation_slot_for_gossip(
-        &self,
-        attestation: &Attestation<T::EthSpec>,
-    ) -> Result<(), AttestationDropReason> {
-        // `now_low_slot` is the slot of the current time minus MAXIMUM_GOSSIP_CLOCK_DISPARITY
-        // `now_high_slot` is the slot of the current time plus MAXIMUM_GOSSIP_CLOCK_DISPARITY
-        let (now_low_slot, now_high_slot) = self
-            .slot_clock
-            .now_duration()
-            .and_then(|now| {
-                let maximum_clock_disparity =
-                    Duration::from_millis(self.spec.maximum_gossip_clock_disparity_millis);
-                let now_low_duration = now.checked_sub(maximum_clock_disparity)?;
-                let now_high_duration = now.checked_add(maximum_clock_disparity)?;
-                Some((
-                    self.slot_clock.slot_of(now_low_duration)?,
-                    self.slot_clock.slot_of(now_high_duration)?,
-                ))
-            })
-            .ok_or_else(|| AttestationDropReason::SlotClockError)?;
-
-        let min_slot = attestation.data.slot;
-        let max_slot = min_slot + self.spec.attestation_propagation_slot_range;
-
-        if now_high_slot < min_slot {
-            Err(AttestationDropReason::TooNew {
-                attestation_slot: min_slot,
-                now: now_high_slot,
-            })
-        } else if now_low_slot > max_slot {
-            Err(AttestationDropReason::TooOld {
-                attestation_slot: min_slot,
-                now: now_low_slot,
-            })
-        } else {
-            Ok(())
-        }
-    }
-
     /// Accept some exit and queue it for inclusion in an appropriate block.
     pub fn process_voluntary_exit(
         &self,

beacon_node/beacon_chain/src/block_verification.rs

@@ -209,7 +209,7 @@ pub fn signature_verify_chain_segment<T: BeaconChainTypes>(
 /// A wrapper around a `SignedBeaconBlock` that indicates it has been approved for re-gossiping on
 /// the p2p network.
 pub struct GossipVerifiedBlock<T: BeaconChainTypes> {
-    block: SignedBeaconBlock<T::EthSpec>,
+    pub block: SignedBeaconBlock<T::EthSpec>,
     block_root: Hash256,
     parent: BeaconSnapshot<T::EthSpec>,
 }

beacon_node/beacon_chain/src/errors.rs

@@ -94,27 +94,3 @@ easy_from_to!(BlockProcessingError, BlockProductionError);
 easy_from_to!(BeaconStateError, BlockProductionError);
 easy_from_to!(SlotProcessingError, BlockProductionError);
 easy_from_to!(Eth1ChainError, BlockProductionError);
-
-/// A reason for not propagating an attestation (single or aggregate).
-#[derive(Debug, PartialEq)]
-pub enum AttestationDropReason {
-    SlotClockError,
-    TooNew { attestation_slot: Slot, now: Slot },
-    TooOld { attestation_slot: Slot, now: Slot },
-    NoValidationState(BeaconChainError),
-    BlockUnknown(Hash256),
-    BadIndexedAttestation(AttestationValidationError),
-    AggregatorNotInAttestingIndices,
-    AggregatorNotSelected,
-    AggregatorSignatureInvalid,
-    SignatureInvalid,
-}
-
-/// A reason for not propagating a block.
-#[derive(Debug, PartialEq)]
-pub enum BlockDropReason {
-    SlotClockError,
-    TooNew { block_slot: Slot, now: Slot },
-    // FIXME(sproul): add detail here
-    ValidationFailure,
-}

beacon_node/beacon_chain/src/lib.rs

@@ -25,7 +25,7 @@ pub use self::beacon_chain::{
 };
 pub use self::beacon_snapshot::BeaconSnapshot;
 pub use self::errors::{BeaconChainError, BlockProductionError};
-pub use block_verification::{BlockError, BlockProcessingOutcome};
+pub use block_verification::{BlockError, BlockProcessingOutcome, GossipVerifiedBlock};
 pub use eth1_chain::{Eth1Chain, Eth1ChainBackend};
 pub use events::EventHandler;
 pub use fork_choice::ForkChoice;