Update to frozen spec ❄️ (v0.8.1) (#444)

* types: first updates for v0.8

* state_processing: epoch processing v0.8.0

* state_processing: block processing v0.8.0

* tree_hash_derive: support generics in SignedRoot

* types v0.8: update to use ssz_types

* state_processing v0.8: use ssz_types

* ssz_types: add bitwise methods and from_elem

* types: fix v0.8 FIXMEs

* ssz_types: add bitfield shift_up

* ssz_types: iterators and DerefMut for VariableList

* types,state_processing: use VariableList

* ssz_types: fix BitVector Decode impl

Fixed a typo in the implementation of ssz::Decode for BitVector, which caused it
to be considered variable length!

* types: fix test modules for v0.8 update

* types: remove slow type-level arithmetic

* state_processing: fix tests for v0.8

* op_pool: update for v0.8

* ssz_types: Bitfield difference length-independent

Allow computing the difference of two bitfields of different lengths.

* Implement compact committee support

* epoch_processing: committee & active index roots

* state_processing: genesis state builder v0.8

* state_processing: implement v0.8.1

* Further improve tree_hash

* Strip examples, tests from cached_tree_hash

* Update TreeHash, un-impl CachedTreeHash

* Update bitfield TreeHash, un-impl CachedTreeHash

* Update FixedLenVec TreeHash, unimpl CachedTreeHash

* Update update tree_hash_derive for new TreeHash

* Fix TreeHash, un-impl CachedTreeHash for ssz_types

* Remove fixed_len_vec, ssz benches

SSZ benches relied upon fixed_len_vec -- it is easier to just delete
them and rebuild them later (when necessary)

* Remove boolean_bitfield crate

* Fix fake_crypto BLS compile errors

* Update ef_tests for new v.8 type params

* Update ef_tests submodule to v0.8.1 tag

* Make fixes to support parsing ssz ef_tests

* `compact_committee...` to `compact_committees...`

* Derive more traits for `CompactCommittee`

* Flip bitfield byte-endianness

* Fix tree_hash for bitfields

* Modify CLI output for ef_tests

* Bump ssz crate version

* Update ssz_types doc comment

* Del cached tree hash tests from ssz_static tests

* Tidy SSZ dependencies

* Rename ssz_types crate to eth2_ssz_types

* validator_client: update for v0.8

* ssz_types: update union/difference for bit order swap

* beacon_node: update for v0.8, EthSpec

* types: disable cached tree hash, update min spec

* state_processing: fix slot bug in committee update

* tests: temporarily disable fork choice harness test

See #447

* committee cache: prevent out-of-bounds access

In the case where we tried to access the committee of a shard that didn't have a committee in the
current epoch, we were accessing elements beyond the end of the shuffling vector and panicking! This
commit adds a check to make the failure safe and explicit.

* fix bug in get_indexed_attestation and simplify

There was a bug in our implementation of get_indexed_attestation whereby
incorrect "committee indices" were used to index into the custody bitfield. The
bug was only observable in the case where some bits of the custody bitfield were
set to 1. The implementation has been simplified to remove the bug, and a test
added.

* state_proc: workaround for compact committees bug

https://github.com/ethereum/eth2.0-specs/issues/1315

* v0.8: updates to make the EF tests pass

* Remove redundant max operation checks.
* Always supply both messages when checking attestation signatures -- allowing
  verification of an attestation with no signatures.
* Swap the order of the fork and domain constant in `get_domain`, to match
  the spec.

* rustfmt

* ef_tests: add new epoch processing tests

* Integrate v0.8 into master (compiles)

* Remove unused crates, fix clippy lints

* Replace v0.6.3 tags w/ v0.8.1

* Remove old comment

* Ensure lmd ghost tests only run in release

* Update readme

validator_client/src/attestation_producer/beacon_node_attestation.rs

@@ -1,6 +1,6 @@
 //TODO: generalise these enums to the crate
 use crate::block_producer::{BeaconNodeError, PublishOutcome};
-use types::{Attestation, AttestationData, Slot};
+use types::{Attestation, AttestationData, EthSpec, Slot};
 
 /// Defines the methods required to produce and publish attestations on a Beacon Node. Abstracts the
 /// actual beacon node.
@@ -16,8 +16,8 @@ pub trait BeaconNodeAttestation: Send + Sync {
     /// Request that the node publishes a attestation.
     ///
     /// Returns `true` if the publish was successful.
-    fn publish_attestation(
+    fn publish_attestation<T: EthSpec>(
         &self,
-        attestation: Attestation,
+        attestation: Attestation<T>,
     ) -> Result<PublishOutcome, BeaconNodeError>;
 }

validator_client/src/attestation_producer/grpc.rs

@@ -6,7 +6,7 @@ use ssz::{Decode, Encode};
 use protos::services::{
     Attestation as GrpcAttestation, ProduceAttestationDataRequest, PublishAttestationRequest,
 };
-use types::{Attestation, AttestationData, Slot};
+use types::{Attestation, AttestationData, EthSpec, Slot};
 
 impl BeaconNodeAttestation for AttestationServiceClient {
     fn produce_attestation_data(
@@ -28,9 +28,9 @@ impl BeaconNodeAttestation for AttestationServiceClient {
         Ok(attestation_data)
     }
 
-    fn publish_attestation(
+    fn publish_attestation<T: EthSpec>(
         &self,
-        attestation: Attestation,
+        attestation: Attestation<T>,
     ) -> Result<PublishOutcome, BeaconNodeError> {
         let mut req = PublishAttestationRequest::new();
 

validator_client/src/attestation_producer/mod.rs

@@ -2,16 +2,17 @@ mod beacon_node_attestation;
 mod grpc;
 
 use std::sync::Arc;
-use types::{ChainSpec, Domain, Fork};
+use types::{ChainSpec, Domain, EthSpec, Fork};
 //TODO: Move these higher up in the crate
 use super::block_producer::{BeaconNodeError, PublishOutcome, ValidatorEvent};
 use crate::signer::Signer;
 use beacon_node_attestation::BeaconNodeAttestation;
+use core::marker::PhantomData;
 use slog::{error, info, warn};
 use tree_hash::TreeHash;
 use types::{
     AggregateSignature, Attestation, AttestationData, AttestationDataAndCustodyBit,
-    AttestationDuty, Bitfield,
+    AttestationDuty, BitList,
 };
 
 //TODO: Group these errors at a crate level
@@ -28,7 +29,7 @@ impl From<BeaconNodeError> for Error {
 
 /// This struct contains the logic for requesting and signing beacon attestations for a validator. The
 /// validator can abstractly sign via the Signer trait object.
-pub struct AttestationProducer<'a, B: BeaconNodeAttestation, S: Signer> {
+pub struct AttestationProducer<'a, B: BeaconNodeAttestation, S: Signer, E: EthSpec> {
     /// The current fork.
     pub fork: Fork,
     /// The attestation duty to perform.
@@ -41,9 +42,11 @@ pub struct AttestationProducer<'a, B: BeaconNodeAttestation, S: Signer> {
     pub signer: &'a S,
     /// Used for calculating epoch.
     pub slots_per_epoch: u64,
+    /// Mere vessel for E.
+    pub _phantom: PhantomData<E>,
 }
 
-impl<'a, B: BeaconNodeAttestation, S: Signer> AttestationProducer<'a, B, S> {
+impl<'a, B: BeaconNodeAttestation, S: Signer, E: EthSpec> AttestationProducer<'a, B, S, E> {
     /// Handle outputs and results from attestation production.
     pub fn handle_produce_attestation(&mut self, log: slog::Logger) {
         match self.produce_attestation() {
@@ -116,7 +119,7 @@ impl<'a, B: BeaconNodeAttestation, S: Signer> AttestationProducer<'a, B, S> {
         attestation: AttestationData,
         duties: AttestationDuty,
         domain: u64,
-    ) -> Option<Attestation> {
+    ) -> Option<Attestation<E>> {
         self.store_produce(&attestation);
 
         // build the aggregate signature
@@ -134,14 +137,14 @@ impl<'a, B: BeaconNodeAttestation, S: Signer> AttestationProducer<'a, B, S> {
             agg_sig
         };
 
-        let mut aggregation_bitfield = Bitfield::with_capacity(duties.committee_len);
-        let custody_bitfield = Bitfield::with_capacity(duties.committee_len);
-        aggregation_bitfield.set(duties.committee_index, true);
+        let mut aggregation_bits = BitList::with_capacity(duties.committee_len).ok()?;
+        let custody_bits = BitList::with_capacity(duties.committee_len).ok()?;
+        aggregation_bits.set(duties.committee_index, true).ok()?;
 
         Some(Attestation {
-            aggregation_bitfield,
+            aggregation_bits,
             data: attestation,
-            custody_bitfield,
+            custody_bits,
             signature: aggregate_signature,
         })
     }

validator_client/src/block_producer/beacon_node_block.rs

@@ -1,4 +1,4 @@
-use types::{BeaconBlock, Signature, Slot};
+use types::{BeaconBlock, EthSpec, Signature, Slot};
 #[derive(Debug, PartialEq, Clone)]
 pub enum BeaconNodeError {
     RemoteFailure(String),
@@ -18,14 +18,17 @@ pub trait BeaconNodeBlock: Send + Sync {
     /// Request that the node produces a block.
     ///
     /// Returns Ok(None) if the Beacon Node is unable to produce at the given slot.
-    fn produce_beacon_block(
+    fn produce_beacon_block<T: EthSpec>(
         &self,
         slot: Slot,
         randao_reveal: &Signature,
-    ) -> Result<Option<BeaconBlock>, BeaconNodeError>;
+    ) -> Result<Option<BeaconBlock<T>>, BeaconNodeError>;
 
     /// Request that the node publishes a block.
     ///
     /// Returns `true` if the publish was successful.
-    fn publish_beacon_block(&self, block: BeaconBlock) -> Result<PublishOutcome, BeaconNodeError>;
+    fn publish_beacon_block<T: EthSpec>(
+        &self,
+        block: BeaconBlock<T>,
+    ) -> Result<PublishOutcome, BeaconNodeError>;
 }

validator_client/src/block_producer/grpc.rs

@@ -5,7 +5,7 @@ use protos::services::{
 use protos::services_grpc::BeaconBlockServiceClient;
 use ssz::{Decode, Encode};
 use std::sync::Arc;
-use types::{BeaconBlock, Signature, Slot};
+use types::{BeaconBlock, EthSpec, Signature, Slot};
 
 //TODO: Remove this new type. Do not need to wrap
 /// A newtype designed to wrap the gRPC-generated service so the `BeaconNode` trait may be
@@ -25,11 +25,11 @@ impl BeaconNodeBlock for BeaconBlockGrpcClient {
     ///
     /// Returns `None` if it is not possible to produce at the supplied slot. For example, if the
     /// BN is unable to find a parent block.
-    fn produce_beacon_block(
+    fn produce_beacon_block<T: EthSpec>(
         &self,
         slot: Slot,
         randao_reveal: &Signature,
-    ) -> Result<Option<BeaconBlock>, BeaconNodeError> {
+    ) -> Result<Option<BeaconBlock<T>>, BeaconNodeError> {
         // request a beacon block from the node
         let mut req = ProduceBeaconBlockRequest::new();
         req.set_slot(slot.as_u64());
@@ -59,7 +59,10 @@ impl BeaconNodeBlock for BeaconBlockGrpcClient {
     ///
     /// Generally, this will be called after a `produce_beacon_block` call with a block that has
     /// been completed (signed) by the validator client.
-    fn publish_beacon_block(&self, block: BeaconBlock) -> Result<PublishOutcome, BeaconNodeError> {
+    fn publish_beacon_block<T: EthSpec>(
+        &self,
+        block: BeaconBlock<T>,
+    ) -> Result<PublishOutcome, BeaconNodeError> {
         let mut req = PublishBeaconBlockRequest::new();
 
         let ssz = block.as_ssz_bytes();

validator_client/src/block_producer/mod.rs

@@ -5,10 +5,11 @@ use self::beacon_node_block::BeaconNodeBlock;
 pub use self::beacon_node_block::{BeaconNodeError, PublishOutcome};
 pub use self::grpc::BeaconBlockGrpcClient;
 use crate::signer::Signer;
+use core::marker::PhantomData;
 use slog::{error, info, warn};
 use std::sync::Arc;
 use tree_hash::{SignedRoot, TreeHash};
-use types::{BeaconBlock, ChainSpec, Domain, Fork, Slot};
+use types::{BeaconBlock, ChainSpec, Domain, EthSpec, Fork, Slot};
 
 #[derive(Debug, PartialEq)]
 pub enum Error {
@@ -37,7 +38,7 @@ pub enum ValidatorEvent {
 
 /// This struct contains the logic for requesting and signing beacon blocks for a validator. The
 /// validator can abstractly sign via the Signer trait object.
-pub struct BlockProducer<'a, B: BeaconNodeBlock, S: Signer> {
+pub struct BlockProducer<'a, B: BeaconNodeBlock, S: Signer, E: EthSpec> {
     /// The current fork.
     pub fork: Fork,
     /// The current slot to produce a block for.
@@ -50,9 +51,11 @@ pub struct BlockProducer<'a, B: BeaconNodeBlock, S: Signer> {
     pub signer: &'a S,
     /// Used for calculating epoch.
     pub slots_per_epoch: u64,
+    /// Mere vessel for E.
+    pub _phantom: PhantomData<E>,
 }
 
-impl<'a, B: BeaconNodeBlock, S: Signer> BlockProducer<'a, B, S> {
+impl<'a, B: BeaconNodeBlock, S: Signer, E: EthSpec> BlockProducer<'a, B, S, E> {
     /// Handle outputs and results from block production.
     pub fn handle_produce_block(&mut self, log: slog::Logger) {
         match self.produce_block() {
@@ -123,7 +126,7 @@ impl<'a, B: BeaconNodeBlock, S: Signer> BlockProducer<'a, B, S> {
     ///
     /// Important: this function will not check to ensure the block is not slashable. This must be
     /// done upstream.
-    fn sign_block(&mut self, mut block: BeaconBlock, domain: u64) -> Option<BeaconBlock> {
+    fn sign_block(&mut self, mut block: BeaconBlock<E>, domain: u64) -> Option<BeaconBlock<E>> {
         self.store_produce(&block);
 
         match self.signer.sign_message(&block.signed_root()[..], domain) {
@@ -140,7 +143,7 @@ impl<'a, B: BeaconNodeBlock, S: Signer> BlockProducer<'a, B, S> {
     /// !!! UNSAFE !!!
     ///
     /// Important: this function is presently stubbed-out. It provides ZERO SAFETY.
-    fn safe_to_produce(&self, _block: &BeaconBlock) -> bool {
+    fn safe_to_produce(&self, _block: &BeaconBlock<E>) -> bool {
         // TODO: ensure the producer doesn't produce slashable blocks.
         // https://github.com/sigp/lighthouse/issues/160
         true
@@ -151,7 +154,7 @@ impl<'a, B: BeaconNodeBlock, S: Signer> BlockProducer<'a, B, S> {
     /// !!! UNSAFE !!!
     ///
     /// Important: this function is presently stubbed-out. It provides ZERO SAFETY.
-    fn store_produce(&mut self, _block: &BeaconBlock) {
+    fn store_produce(&mut self, _block: &BeaconBlock<E>) {
         // TODO: record this block production to prevent future slashings.
         // https://github.com/sigp/lighthouse/issues/160
     }

validator_client/src/main.rs

@@ -110,7 +110,7 @@ fn main() {
                 }
             };
             default_dir.push(DEFAULT_DATA_DIR);
-            PathBuf::from(default_dir)
+            default_dir
         }
     };
 
@@ -203,12 +203,12 @@ fn main() {
     );
 
     let result = match eth2_config.spec_constants.as_str() {
-        "mainnet" => ValidatorService::<ValidatorServiceClient, Keypair>::start::<MainnetEthSpec>(
+        "mainnet" => ValidatorService::<ValidatorServiceClient, Keypair, MainnetEthSpec>::start(
             client_config,
             eth2_config,
             log.clone(),
         ),
-        "minimal" => ValidatorService::<ValidatorServiceClient, Keypair>::start::<MinimalEthSpec>(
+        "minimal" => ValidatorService::<ValidatorServiceClient, Keypair, MinimalEthSpec>::start(
             client_config,
             eth2_config,
             log.clone(),

validator_client/src/service.rs

@@ -25,6 +25,7 @@ use protos::services_grpc::{
 };
 use slog::{error, info, warn};
 use slot_clock::{SlotClock, SystemTimeSlotClock};
+use std::marker::PhantomData;
 use std::sync::Arc;
 use std::sync::RwLock;
 use std::time::{Duration, Instant, SystemTime};
@@ -41,7 +42,7 @@ const TIME_DELAY_FROM_SLOT: Duration = Duration::from_millis(200);
 /// The validator service. This is the main thread that executes and maintains validator
 /// duties.
 //TODO: Generalize the BeaconNode types to use testing
-pub struct Service<B: BeaconNodeDuties + 'static, S: Signer + 'static> {
+pub struct Service<B: BeaconNodeDuties + 'static, S: Signer + 'static, E: EthSpec> {
     /// The node's current fork version we are processing on.
     fork: Fork,
     /// The slot clock for this service.
@@ -60,18 +61,19 @@ pub struct Service<B: BeaconNodeDuties + 'static, S: Signer + 'static> {
     attestation_client: Arc<AttestationServiceClient>,
     /// The validator client logger.
     log: slog::Logger,
+    _phantom: PhantomData<E>,
 }
 
-impl<B: BeaconNodeDuties + 'static, S: Signer + 'static> Service<B, S> {
+impl<B: BeaconNodeDuties + 'static, S: Signer + 'static, E: EthSpec> Service<B, S, E> {
     ///  Initial connection to the beacon node to determine its properties.
     ///
     ///  This tries to connect to a beacon node. Once connected, it initialised the gRPC clients
     ///  and returns an instance of the service.
-    fn initialize_service<T: EthSpec>(
+    fn initialize_service(
         client_config: ValidatorConfig,
         eth2_config: Eth2Config,
         log: slog::Logger,
-    ) -> error_chain::Result<Service<ValidatorServiceClient, Keypair>> {
+    ) -> error_chain::Result<Service<ValidatorServiceClient, Keypair, E>> {
         // initialise the beacon node client to check for a connection
 
         let env = Arc::new(EnvBuilder::new().build());
@@ -180,7 +182,7 @@ impl<B: BeaconNodeDuties + 'static, S: Signer + 'static> Service<B, S> {
             }
         };
 
-        let slots_per_epoch = T::slots_per_epoch();
+        let slots_per_epoch = E::slots_per_epoch();
 
         // TODO: keypairs are randomly generated; they should be loaded from a file or generated.
         // https://github.com/sigp/lighthouse/issues/160
@@ -212,18 +214,19 @@ impl<B: BeaconNodeDuties + 'static, S: Signer + 'static> Service<B, S> {
             beacon_block_client,
             attestation_client,
             log,
+            _phantom: PhantomData,
         })
     }
 
     /// Initialise the service then run the core thread.
     // TODO: Improve handling of generic BeaconNode types, to stub grpcClient
-    pub fn start<T: EthSpec>(
+    pub fn start(
         client_config: ValidatorConfig,
         eth2_config: Eth2Config,
         log: slog::Logger,
     ) -> error_chain::Result<()> {
         // connect to the node and retrieve its properties and initialize the gRPC clients
-        let mut service = Service::<ValidatorServiceClient, Keypair>::initialize_service::<T>(
+        let mut service = Service::<ValidatorServiceClient, Keypair, E>::initialize_service(
             client_config,
             eth2_config,
             log,
@@ -351,6 +354,7 @@ impl<B: BeaconNodeDuties + 'static, S: Signer + 'static> Service<B, S> {
                             beacon_node,
                             signer,
                             slots_per_epoch,
+                            _phantom: PhantomData::<E>,
                         };
                         block_producer.handle_produce_block(log);
                     });
@@ -374,6 +378,7 @@ impl<B: BeaconNodeDuties + 'static, S: Signer + 'static> Service<B, S> {
                             beacon_node,
                             signer,
                             slots_per_epoch,
+                            _phantom: PhantomData::<E>,
                         };
                         attestation_producer.handle_produce_attestation(log);
                     });