Prepare for public testnet (#628)

* Update to spec v0.9.0

* Update to v0.9.1

* Bump spec tags for v0.9.1

* Formatting, fix CI failures

* Resolve accidental KeyPair merge conflict

* Document new BeaconState functions

* Add `validator` changes from `validator-to-rest`

* Add initial (failing) REST api tests

* Fix signature parsing

* Add more tests

* Refactor http router

* Add working tests for publish beacon block

* Add validator duties tests

* Move account_manager under `lighthouse` binary

* Unify logfile handling in `environment` crate.

* Fix incorrect cache drops in `advance_caches`

* Update fork choice for v0.9.1

* Add `deposit_contract` crate

* Add progress on validator onboarding

* Add unfinished attesation code

* Update account manager CLI

* Write eth1 data file as hex string

* Integrate ValidatorDirectory with validator_client

* Move ValidatorDirectory into validator_client

* Clean up some FIXMEs

* Add beacon_chain_sim

* Fix a few docs/logs

* Expand `beacon_chain_sim`

* Fix spec for `beacon_chain_sim

* More testing for api

* Start work on attestation endpoint

* Reject empty attestations

* Allow attestations to genesis block

* Add working tests for `rest_api` validator endpoint

* Remove grpc from beacon_node

* Start heavy refactor of validator client

- Block production is working

* Prune old validator client files

* Start works on attestation service

* Add attestation service to validator client

* Use full pubkey for validator directories

* Add validator duties post endpoint

* Use par_iter for keypair generation

* Use bulk duties request in validator client

* Add version http endpoint tests

* Add interop keys and startup wait

* Ensure a prompt exit

* Add duties pruning

* Fix compile error in beacon node tests

* Add github workflow

* Modify rust.yaml

* Modify gitlab actions

* Add to CI file

* Add sudo to CI npm install

* Move cargo fmt to own job in tests

* Fix cargo fmt in CI

* Add rustup update before cargo fmt

* Change name of CI job

* Make other CI jobs require cargo fmt

* Add CI badge

* Remove gitlab and travis files

* Add different http timeout for debug

* Update docker file, use makefile in CI

* Use make in the dockerfile, skip the test

* Use the makefile for debug GI test

* Update book

* Tidy grpc and misc things

* Apply discv5 fixes

* Address other minor issues

* Fix warnings

* Attempt fix for addr parsing

* Tidy validator config, CLIs

* Tidy comments

* Tidy signing, reduce ForkService duplication

* Fail if skipping too many slots

* Set default recent genesis time to 0

* Add custom http timeout to validator

* Fix compile bug in node_test_rig

* Remove old bootstrap flag from val CLI

* Update docs

* Tidy val client

* Change val client log levels

* Add comments, more validity checks

* Fix compile error, add comments

* Undo changes to eth2-libp2p/src

* Reduce duplication of keypair generation

* Add more logging for validator duties

* Fix beacon_chain_sim, nitpicks

* Fix compile error, minor nits

* Update to use v0.9.2 version of deposit contract

* Add efforts to automate eth1 testnet deployment

* Fix lcli testnet deployer

* Modify bn CLI to parse eth2_testnet_dir

* Progress with account_manager deposit tools

* Make account manager submit deposits

* Add password option for submitting deposits

* Allow custom deposit amount

* Add long names to lcli clap

* Add password option to lcli deploy command

* Fix minor bugs whilst testing

* Address Michael's comments

* Add refund-deposit-contract to lcli

* Use time instead of skip count for denying long skips

* Improve logging for eth1

* Fix bug with validator services exiting on error

* Drop the block cache after genesis

* Modify eth1 testnet config

* Improve eth1 logging

* Make validator wait until genesis time

* Fix bug in eth1 voting

* Add more logging to eth1 voting

* Handle errors in eth1 http module

* Set SECONDS_PER_DAY to sensible minimum

* Shorten delay before testnet start

* Ensure eth1 block is produced without any votes

* Improve eth1 logging

* Fix broken tests in eth1

* Tidy code in rest_api

* Fix failing test in deposit_contract

* Make CLI args more consistent

* Change validator/duties endpoint

* Add time-based skip slot limiting

* Add new error type missed in previous commit

* Add log when waiting for genesis

* Refactor beacon node CLI

* Remove unused dep

* Add lcli eth1-genesis command

* Fix bug in master merge

* Apply clippy lints to beacon node

* Add support for YamlConfig in Eth2TestnetDir

* Upgrade tesnet deposit contract version

* Remove unnecessary logging and correct formatting

* Add a hardcoded eth2 testnet config

* Ensure http server flag works. Overwrite configs with flags.

* Ensure boot nodes are loaded from testnet dir

* Fix account manager CLI bugs

* Fix bugs with beacon node cli

* Allow testnet dir without boot nodes

* Write genesis state as SSZ

* Remove ---/n from the start of testnet_dir files

* Set default libp2p address

* Tidy account manager CLI, add logging

* Add check to see if testnet dir exists

* Apply reviewers suggestions

* Add HeadTracker struct

* Add fork choice persistence

* Shorten slot time for simulator

* Add the /beacon/heads API endpoint

* Update hardcoded testnet

* Add tests for BeaconChain persistence + fix bugs

* Extend BeaconChain persistence testing

* Ensure chain is finalized b4 persistence tests

* Ensure boot_enr.yaml is include in binary

* Refactor beacon_chain_sim

* Move files about in beacon sim

* Update beacon_chain_sim

* Fix bug with deposit inclusion

* Increase log in genesis service, fix todo

* Tidy sim, fix broken rest_api tests

* Fix more broken tests

* Update testnet

* Fix broken rest api test

* Tidy account manager CLI

* Use tempdir for account manager

* Stop hardcoded testnet dir from creating dir

* Rename Eth2TestnetDir to Eth2TestnetConfig

* Change hardcoded -> hard_coded

* Tidy account manager

* Add log to account manager

* Tidy, ensure head tracker is loaded from disk

* Tidy beacon chain builder

* Tidy eth1_chain

* Adds log support for simulator

* Revert "Adds log support for simulator"

This reverts commit ec77c66a05.

* Adds log support for simulator

* Tidy after self-review

* Change default log level

* Address Michael's delicious PR comments

* Fix off-by-one in tests

beacon_node/Cargo.toml

@@ -34,3 +34,6 @@ logging = { path = "../eth2/utils/logging" }
 futures = "0.1.29"
 environment = { path = "../lighthouse/environment" }
 genesis = { path = "genesis" }
+eth2_testnet_config = { path = "../eth2/utils/eth2_testnet_config" }
+eth2-libp2p = { path = "./eth2-libp2p" }
+eth2_ssz = { path = "../eth2/utils/ssz" }

beacon_node/beacon_chain/src/beacon_chain.rs

@@ -3,6 +3,7 @@ use crate::errors::{BeaconChainError as Error, BlockProductionError};
 use crate::eth1_chain::{Eth1Chain, Eth1ChainBackend};
 use crate::events::{EventHandler, EventKind};
 use crate::fork_choice::{Error as ForkChoiceError, ForkChoice};
+use crate::head_tracker::HeadTracker;
 use crate::metrics;
 use crate::persisted_beacon_chain::{PersistedBeaconChain, BEACON_CHAIN_DB_KEY};
 use lmd_ghost::LmdGhost;
@@ -126,6 +127,8 @@ pub struct BeaconChain<T: BeaconChainTypes> {
     pub fork_choice: ForkChoice<T>,
     /// A handler for events generated by the beacon chain.
     pub event_handler: T::EventHandler,
+    /// Used to track the heads of the beacon chain.
+    pub(crate) head_tracker: HeadTracker,
     /// Logging to CLI, etc.
     pub(crate) log: Logger,
 }
@@ -137,10 +140,35 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
     pub fn persist(&self) -> Result<(), Error> {
         let timer = metrics::start_timer(&metrics::PERSIST_CHAIN);
 
+        let canonical_head = self.head();
+
+        let finalized_checkpoint = {
+            let beacon_block_root = canonical_head.beacon_state.finalized_checkpoint.root;
+            let beacon_block = self
+                .store
+                .get::<BeaconBlock<_>>(&beacon_block_root)?
+                .ok_or_else(|| Error::MissingBeaconBlock(beacon_block_root))?;
+            let beacon_state_root = beacon_block.state_root;
+            let beacon_state = self
+                .store
+                .get_state(&beacon_state_root, Some(beacon_block.slot))?
+                .ok_or_else(|| Error::MissingBeaconState(beacon_state_root))?;
+
+            CheckPoint {
+                beacon_block_root,
+                beacon_block,
+                beacon_state_root,
+                beacon_state,
+            }
+        };
+
         let p: PersistedBeaconChain<T> = PersistedBeaconChain {
-            canonical_head: self.canonical_head.read().clone(),
+            canonical_head,
+            finalized_checkpoint,
             op_pool: PersistedOperationPool::from_operation_pool(&self.op_pool),
             genesis_block_root: self.genesis_block_root,
+            ssz_head_tracker: self.head_tracker.to_ssz_container(),
+            fork_choice: self.fork_choice.as_ssz_container(),
         };
 
         let key = Hash256::from_slice(&BEACON_CHAIN_DB_KEY.as_bytes());
@@ -331,10 +359,17 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
         self.canonical_head.read().clone()
     }
 
+    /// Returns the current heads of the `BeaconChain`. For the canonical head, see `Self::head`.
+    ///
+    /// Returns `(block_root, block_slot)`.
+    pub fn heads(&self) -> Vec<(Hash256, Slot)> {
+        self.head_tracker.heads()
+    }
+
     /// Returns the `BeaconState` at the given slot.
     ///
-    ///  Returns `None` when the state is not found in the database or there is an error skipping
-    ///  to a future state.
+    /// Returns `None` when the state is not found in the database or there is an error skipping
+    /// to a future state.
     pub fn state_at_slot(&self, slot: Slot) -> Result<BeaconState<T::EthSpec>, Error> {
         let head_state = self.head().beacon_state;
 
@@ -1219,6 +1254,8 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
 
         metrics::stop_timer(db_write_timer);
 
+        self.head_tracker.register_block(block_root, &block);
+
         let fork_choice_register_timer =
             metrics::start_timer(&metrics::BLOCK_PROCESSING_FORK_CHOICE_REGISTER);
 
@@ -1324,6 +1361,11 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
         let (proposer_slashings, attester_slashings) =
             self.op_pool.get_slashings(&state, &self.spec);
 
+        let eth1_data = eth1_chain.eth1_data_for_block_production(&state, &self.spec)?;
+        let deposits = eth1_chain
+            .deposits_for_block_inclusion(&state, &eth1_data, &self.spec)?
+            .into();
+
         let mut block = BeaconBlock {
             slot: state.slot,
             parent_root,
@@ -1332,14 +1374,12 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
             signature: Signature::empty_signature(),
             body: BeaconBlockBody {
                 randao_reveal,
-                eth1_data: eth1_chain.eth1_data_for_block_production(&state, &self.spec)?,
+                eth1_data,
                 graffiti,
                 proposer_slashings: proposer_slashings.into(),
                 attester_slashings: attester_slashings.into(),
                 attestations: self.op_pool.get_attestations(&state, &self.spec).into(),
-                deposits: eth1_chain
-                    .deposits_for_block_inclusion(&state, &self.spec)?
-                    .into(),
+                deposits,
                 voluntary_exits: self.op_pool.get_voluntary_exits(&state, &self.spec).into(),
             },
         };
@@ -1596,6 +1636,23 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
     }
 }
 
+impl<T: BeaconChainTypes> Drop for BeaconChain<T> {
+    fn drop(&mut self) {
+        if let Err(e) = self.persist() {
+            error!(
+                self.log,
+                "Failed to persist BeaconChain on drop";
+                "error" => format!("{:?}", e)
+            )
+        } else {
+            info!(
+                self.log,
+                "Saved beacon chain state";
+            )
+        }
+    }
+}
+
 fn write_state<T: EthSpec>(prefix: &str, state: &BeaconState<T>, log: &Logger) {
     if WRITE_BLOCK_PROCESSING_SSZ {
         let root = Hash256::from_slice(&state.tree_hash_root());

beacon_node/beacon_chain/src/builder.rs

@@ -1,5 +1,6 @@
 use crate::eth1_chain::CachingEth1Backend;
 use crate::events::NullEventHandler;
+use crate::head_tracker::HeadTracker;
 use crate::persisted_beacon_chain::{PersistedBeaconChain, BEACON_CHAIN_DB_KEY};
 use crate::{
     BeaconChain, BeaconChainTypes, CheckPoint, Eth1Chain, Eth1ChainBackend, EventHandler,
@@ -88,6 +89,8 @@ pub struct BeaconChainBuilder<T: BeaconChainTypes> {
     eth1_chain: Option<Eth1Chain<T::Eth1Chain, T::EthSpec>>,
     event_handler: Option<T::EventHandler>,
     slot_clock: Option<T::SlotClock>,
+    persisted_beacon_chain: Option<PersistedBeaconChain<T>>,
+    head_tracker: Option<HeadTracker>,
     spec: ChainSpec,
     log: Option<Logger>,
 }
@@ -128,6 +131,8 @@ where
             eth1_chain: None,
             event_handler: None,
             slot_clock: None,
+            persisted_beacon_chain: None,
+            head_tracker: None,
             spec: TEthSpec::default_spec(),
             log: None,
         }
@@ -208,11 +213,17 @@ where
 
         self.op_pool = Some(
             p.op_pool
+                .clone()
                 .into_operation_pool(&p.canonical_head.beacon_state, &self.spec),
         );
 
-        self.finalized_checkpoint = Some(p.canonical_head);
+        self.finalized_checkpoint = Some(p.finalized_checkpoint.clone());
         self.genesis_block_root = Some(p.genesis_block_root);
+        self.head_tracker = Some(
+            HeadTracker::from_ssz_container(&p.ssz_head_tracker)
+                .map_err(|e| format!("Failed to decode head tracker for database: {:?}", e))?,
+        );
+        self.persisted_beacon_chain = Some(p);
 
         Ok(self)
     }
@@ -264,28 +275,6 @@ where
         Ok(self.empty_op_pool())
     }
 
-    /// Sets the `BeaconChain` fork choice backend.
-    ///
-    /// Requires the store and state to have been specified earlier in the build chain.
-    pub fn fork_choice_backend(mut self, backend: TLmdGhost) -> Result<Self, String> {
-        let store = self
-            .store
-            .clone()
-            .ok_or_else(|| "reduced_tree_fork_choice requires a store")?;
-        let genesis_block_root = self
-            .genesis_block_root
-            .ok_or_else(|| "fork_choice_backend requires a genesis_block_root")?;
-
-        self.fork_choice = Some(ForkChoice::new(
-            store,
-            backend,
-            genesis_block_root,
-            self.spec.genesis_slot,
-        ));
-
-        Ok(self)
-    }
-
     /// Sets the `BeaconChain` eth1 backend.
     pub fn eth1_backend(mut self, backend: Option<TEth1Backend>) -> Self {
         self.eth1_chain = backend.map(Eth1Chain::new);
@@ -337,19 +326,24 @@ where
         >,
         String,
     > {
-        let mut canonical_head = self
-            .finalized_checkpoint
-            .ok_or_else(|| "Cannot build without a state".to_string())?;
+        let log = self
+            .log
+            .ok_or_else(|| "Cannot build without a logger".to_string())?;
+
+        // If this beacon chain is being loaded from disk, use the stored head. Otherwise, just use
+        // the finalized checkpoint (which is probably genesis).
+        let mut canonical_head = if let Some(persisted_beacon_chain) = self.persisted_beacon_chain {
+            persisted_beacon_chain.canonical_head
+        } else {
+            self.finalized_checkpoint
+                .ok_or_else(|| "Cannot build without a state".to_string())?
+        };
 
         canonical_head
             .beacon_state
             .build_all_caches(&self.spec)
             .map_err(|e| format!("Failed to build state caches: {:?}", e))?;
 
-        let log = self
-            .log
-            .ok_or_else(|| "Cannot build without a logger".to_string())?;
-
         if canonical_head.beacon_block.state_root != canonical_head.beacon_state_root {
             return Err("beacon_block.state_root != beacon_state".to_string());
         }
@@ -379,6 +373,7 @@ where
             event_handler: self
                 .event_handler
                 .ok_or_else(|| "Cannot build without an event handler".to_string())?,
+            head_tracker: self.head_tracker.unwrap_or_default(),
             log: log.clone(),
         };
 
@@ -414,27 +409,43 @@ where
     TEthSpec: EthSpec + 'static,
     TEventHandler: EventHandler<TEthSpec> + 'static,
 {
-    /// Initializes a new, empty (no recorded votes or blocks) fork choice, using the
-    /// `ThreadSafeReducedTree` backend.
+    /// Initializes a fork choice with the `ThreadSafeReducedTree` backend.
     ///
-    /// Requires the store and state to be initialized.
-    pub fn empty_reduced_tree_fork_choice(self) -> Result<Self, String> {
+    /// If this builder is being "resumed" from disk, then rebuild the last fork choice stored to
+    /// the database. Otherwise, create a new, empty fork choice.
+    pub fn reduced_tree_fork_choice(mut self) -> Result<Self, String> {
         let store = self
             .store
             .clone()
             .ok_or_else(|| "reduced_tree_fork_choice requires a store")?;
-        let finalized_checkpoint = &self
-            .finalized_checkpoint
-            .as_ref()
-            .expect("should have finalized checkpoint");
 
-        let backend = ThreadSafeReducedTree::new(
-            store.clone(),
-            &finalized_checkpoint.beacon_block,
-            finalized_checkpoint.beacon_block_root,
-        );
+        let fork_choice = if let Some(persisted_beacon_chain) = &self.persisted_beacon_chain {
+            ForkChoice::from_ssz_container(
+                persisted_beacon_chain.fork_choice.clone(),
+                store.clone(),
+            )
+            .map_err(|e| format!("Unable to decode fork choice from db: {:?}", e))?
+        } else {
+            let finalized_checkpoint = &self
+                .finalized_checkpoint
+                .as_ref()
+                .ok_or_else(|| "fork_choice_backend requires a finalized_checkpoint")?;
+            let genesis_block_root = self
+                .genesis_block_root
+                .ok_or_else(|| "fork_choice_backend requires a genesis_block_root")?;
 
-        self.fork_choice_backend(backend)
+            let backend = ThreadSafeReducedTree::new(
+                store.clone(),
+                &finalized_checkpoint.beacon_block,
+                finalized_checkpoint.beacon_block_root,
+            );
+
+            ForkChoice::new(store, backend, genesis_block_root, self.spec.genesis_slot)
+        };
+
+        self.fork_choice = Some(fork_choice);
+
+        Ok(self)
     }
 }
 
@@ -611,7 +622,7 @@ mod test {
             .null_event_handler()
             .testing_slot_clock(Duration::from_secs(1))
             .expect("should configure testing slot clock")
-            .empty_reduced_tree_fork_choice()
+            .reduced_tree_fork_choice()
             .expect("should add fork choice to builder")
             .build()
             .expect("should build");

beacon_node/beacon_chain/src/eth1_chain.rs

@@ -4,7 +4,8 @@ use exit_future::Exit;
 use futures::Future;
 use integer_sqrt::IntegerSquareRoot;
 use rand::prelude::*;
-use slog::{crit, Logger};
+use slog::{crit, debug, error, trace, Logger};
+use state_processing::per_block_processing::get_new_eth1_data;
 use std::collections::HashMap;
 use std::iter::DoubleEndedIterator;
 use std::iter::FromIterator;
@@ -33,7 +34,7 @@ pub enum Error {
     /// voting period.
     UnableToGetPreviousStateRoot(BeaconStateError),
     /// The state required to find the previous eth1 block was not found in the store.
-    PreviousStateNotInDB,
+    PreviousStateNotInDB(Hash256),
     /// There was an error accessing an object in the database.
     StoreError(StoreError),
     /// The eth1 head block at the start of the eth1 voting period is unknown.
@@ -89,16 +90,21 @@ where
     /// Returns a list of `Deposits` that may be included in a block.
     ///
     /// Including all of the returned `Deposits` in a block should _not_ cause it to become
-    /// invalid.
+    /// invalid (i.e., this function should respect the maximum).
+    ///
+    /// `eth1_data_vote` is the `Eth1Data` that the block producer would include in their
+    /// block. This vote may change the `state.eth1_data` value, which would change the deposit
+    /// count and therefore change the output of this function.
     pub fn deposits_for_block_inclusion(
         &self,
         state: &BeaconState<E>,
+        eth1_data_vote: &Eth1Data,
         spec: &ChainSpec,
     ) -> Result<Vec<Deposit>, Error> {
         if self.use_dummy_backend {
-            DummyEth1ChainBackend::default().queued_deposits(state, spec)
+            DummyEth1ChainBackend::default().queued_deposits(state, eth1_data_vote, spec)
         } else {
-            self.backend.queued_deposits(state, spec)
+            self.backend.queued_deposits(state, eth1_data_vote, spec)
         }
     }
 }
@@ -119,6 +125,7 @@ pub trait Eth1ChainBackend<T: EthSpec>: Sized + Send + Sync {
     fn queued_deposits(
         &self,
         beacon_state: &BeaconState<T>,
+        eth1_data_vote: &Eth1Data,
         spec: &ChainSpec,
     ) -> Result<Vec<Deposit>, Error>;
 }
@@ -131,6 +138,7 @@ pub trait Eth1ChainBackend<T: EthSpec>: Sized + Send + Sync {
 pub struct DummyEth1ChainBackend<T: EthSpec>(PhantomData<T>);
 
 impl<T: EthSpec> Eth1ChainBackend<T> for DummyEth1ChainBackend<T> {
+    /// Produce some deterministic junk based upon the current epoch.
     fn eth1_data(&self, state: &BeaconState<T>, _spec: &ChainSpec) -> Result<Eth1Data, Error> {
         let current_epoch = state.current_epoch();
         let slots_per_voting_period = T::slots_per_eth1_voting_period() as u64;
@@ -146,7 +154,13 @@ impl<T: EthSpec> Eth1ChainBackend<T> for DummyEth1ChainBackend<T> {
         })
     }
 
-    fn queued_deposits(&self, _: &BeaconState<T>, _: &ChainSpec) -> Result<Vec<Deposit>, Error> {
+    /// The dummy back-end never produces deposits.
+    fn queued_deposits(
+        &self,
+        _: &BeaconState<T>,
+        _: &Eth1Data,
+        _: &ChainSpec,
+    ) -> Result<Vec<Deposit>, Error> {
         Ok(vec![])
     }
 }
@@ -201,24 +215,102 @@ impl<T: EthSpec, S: Store> CachingEth1Backend<T, S> {
 
 impl<T: EthSpec, S: Store> Eth1ChainBackend<T> for CachingEth1Backend<T, S> {
     fn eth1_data(&self, state: &BeaconState<T>, spec: &ChainSpec) -> Result<Eth1Data, Error> {
+        // Note: we do not return random junk if this function call fails as it would be caused by
+        // an internal error.
         let prev_eth1_hash = eth1_block_hash_at_start_of_voting_period(self.store.clone(), state)?;
 
+        let period = T::SlotsPerEth1VotingPeriod::to_u64();
+        let eth1_follow_distance = spec.eth1_follow_distance;
+        let voting_period_start_slot = (state.slot / period) * period;
+        let voting_period_start_seconds = slot_start_seconds::<T>(
+            state.genesis_time,
+            spec.milliseconds_per_slot,
+            voting_period_start_slot,
+        );
+
         let blocks = self.core.blocks().read();
 
-        let eth1_data = eth1_data_sets(blocks.iter(), state, prev_eth1_hash, spec)
-            .map(|(new_eth1_data, all_eth1_data)| {
-                collect_valid_votes(state, new_eth1_data, all_eth1_data)
-            })
-            .and_then(find_winning_vote)
-            .unwrap_or_else(|| {
-                crit!(
-                    self.log,
-                    "Unable to cast valid vote for Eth1Data";
-                    "hint" => "check connection to eth1 node",
-                    "reason" => "no votes",
-                );
-                random_eth1_data()
-            });
+        let (new_eth1_data, all_eth1_data) = if let Some(sets) = eth1_data_sets(
+            blocks.iter(),
+            prev_eth1_hash,
+            voting_period_start_seconds,
+            spec,
+            &self.log,
+        ) {
+            sets
+        } else {
+            // The algorithm was unable to find the `new_eth1_data` and `all_eth1_data` sets.
+            //
+            // This is likely because the caches are empty or the previous eth1 block hash is not
+            // in the cache.
+            //
+            // This situation can also be caused when a testnet does not have an adequate delay
+            // between the eth1 genesis block and the eth2 genesis block. This delay needs to be at
+            // least `2 * ETH1_FOLLOW_DISTANCE`.
+            crit!(
+                self.log,
+                "Unable to find eth1 data sets";
+                "lowest_block_number" => self.core.lowest_block_number(),
+                "earliest_block_timestamp" => self.core.earliest_block_timestamp(),
+                "genesis_time" => state.genesis_time,
+                "outcome" => "casting random eth1 vote"
+            );
+
+            return Ok(random_eth1_data());
+        };
+
+        trace!(
+            self.log,
+            "Found eth1 data sets";
+            "all_eth1_data" => all_eth1_data.len(),
+            "new_eth1_data" => new_eth1_data.len(),
+        );
+
+        let valid_votes = collect_valid_votes(state, new_eth1_data, all_eth1_data);
+
+        let eth1_data = if let Some(eth1_data) = find_winning_vote(valid_votes) {
+            eth1_data
+        } else {
+            // In this case, there are no other viable votes (perhaps there are no votes yet or all
+            // the existing votes are junk).
+            //
+            // Here we choose the latest block in our voting window.
+            blocks
+                .iter()
+                .rev()
+                .skip_while(|eth1_block| eth1_block.timestamp > voting_period_start_seconds)
+                .skip(eth1_follow_distance as usize)
+                .next()
+                .map(|block| {
+                    trace!(
+                        self.log,
+                        "Choosing default eth1_data";
+                        "eth1_block_number" =>  block.number,
+                        "eth1_block_hash" => format!("{:?}", block.hash),
+                    );
+
+                    block
+                })
+                .and_then(|block| block.clone().eth1_data())
+                .unwrap_or_else(|| {
+                    crit!(
+                        self.log,
+                        "Unable to find a winning eth1 vote";
+                        "outcome" => "casting random eth1 vote"
+                    );
+
+                    random_eth1_data()
+                })
+        };
+
+        debug!(
+            self.log,
+            "Produced vote for eth1 chain";
+            "is_period_tail" => is_period_tail(state),
+            "deposit_root" => format!("{:?}", eth1_data.deposit_root),
+            "deposit_count" => eth1_data.deposit_count,
+            "block_hash" => format!("{:?}", eth1_data.block_hash),
+        );
 
         Ok(eth1_data)
     }
@@ -226,10 +318,15 @@ impl<T: EthSpec, S: Store> Eth1ChainBackend<T> for CachingEth1Backend<T, S> {
     fn queued_deposits(
         &self,
         state: &BeaconState<T>,
+        eth1_data_vote: &Eth1Data,
         _spec: &ChainSpec,
     ) -> Result<Vec<Deposit>, Error> {
-        let deposit_count = state.eth1_data.deposit_count;
         let deposit_index = state.eth1_deposit_index;
+        let deposit_count = if let Some(new_eth1_data) = get_new_eth1_data(state, eth1_data_vote) {
+            new_eth1_data.deposit_count
+        } else {
+            state.eth1_data.deposit_count
+        };
 
         if deposit_index > deposit_count {
             Err(Error::DepositIndexTooHigh)
@@ -281,12 +378,14 @@ fn eth1_block_hash_at_start_of_voting_period<T: EthSpec, S: Store>(
 ) -> Result<Hash256, Error> {
     let period = T::SlotsPerEth1VotingPeriod::to_u64();
 
-    // Find `state.eth1_data.block_hash` for the state at the start of the voting period.
-    if state.slot % period < period / 2 {
-        // When the state is less than half way through the period we can safely assume that
-        // the eth1_data has not changed since the start of the period.
+    if !eth1_data_change_is_possible(state) {
+        // If there are less than 50% of the votes in the current state, it's impossible that the
+        // `eth1_data.block_hash` has changed from the value at `state.eth1_data.block_hash`.
         Ok(state.eth1_data.block_hash)
     } else {
+        // If there have been more than 50% of votes in this period it's possible (but not
+        // necessary) that the `state.eth1_data.block_hash` has been changed since the start of the
+        // voting period.
         let slot = (state.slot / period) * period;
         let prev_state_root = state
             .get_state_root(slot)
@@ -296,33 +395,32 @@ fn eth1_block_hash_at_start_of_voting_period<T: EthSpec, S: Store>(
             .get_state::<T>(&prev_state_root, Some(slot))
             .map_err(Error::StoreError)?
             .map(|state| state.eth1_data.block_hash)
-            .ok_or_else(|| Error::PreviousStateNotInDB)
+            .ok_or_else(|| Error::PreviousStateNotInDB(*prev_state_root))
     }
 }
 
+/// Returns true if there are enough eth1 votes in the given `state` to have updated
+/// `state.eth1_data`.
+fn eth1_data_change_is_possible<E: EthSpec>(state: &BeaconState<E>) -> bool {
+    2 * state.eth1_data_votes.len() > E::SlotsPerEth1VotingPeriod::to_usize()
+}
+
 /// Calculates and returns `(new_eth1_data, all_eth1_data)` for the given `state`, based upon the
 /// blocks in the `block` iterator.
 ///
 /// `prev_eth1_hash` is the `eth1_data.block_hash` at the start of the voting period defined by
 /// `state.slot`.
-fn eth1_data_sets<'a, T: EthSpec, I>(
+fn eth1_data_sets<'a, I>(
     blocks: I,
-    state: &BeaconState<T>,
     prev_eth1_hash: Hash256,
+    voting_period_start_seconds: u64,
     spec: &ChainSpec,
+    log: &Logger,
 ) -> Option<(Eth1DataBlockNumber, Eth1DataBlockNumber)>
 where
-    T: EthSpec,
     I: DoubleEndedIterator<Item = &'a Eth1Block> + Clone,
 {
-    let period = T::SlotsPerEth1VotingPeriod::to_u64();
     let eth1_follow_distance = spec.eth1_follow_distance;
-    let voting_period_start_slot = (state.slot / period) * period;
-    let voting_period_start_seconds = slot_start_seconds::<T>(
-        state.genesis_time,
-        spec.milliseconds_per_slot,
-        voting_period_start_slot,
-    );
 
     let in_scope_eth1_data = blocks
         .rev()
@@ -345,6 +443,12 @@ where
             HashMap::from_iter(all_eth1_data),
         ))
     } else {
+        error!(
+            log,
+            "The previous eth1 hash is not in cache";
+            "previous_hash" => format!("{:?}", prev_eth1_hash)
+        );
+
         None
     }
 }
@@ -356,8 +460,6 @@ fn collect_valid_votes<T: EthSpec>(
     new_eth1_data: Eth1DataBlockNumber,
     all_eth1_data: Eth1DataBlockNumber,
 ) -> Eth1DataVoteCount {
-    let slots_per_eth1_voting_period = T::SlotsPerEth1VotingPeriod::to_u64();
-
     let mut valid_votes = HashMap::new();
 
     state
@@ -368,10 +470,7 @@ fn collect_valid_votes<T: EthSpec>(
                 .get(vote)
                 .map(|block_number| (vote.clone(), *block_number))
                 .or_else(|| {
-                    let slot = state.slot % slots_per_eth1_voting_period;
-                    let period_tail = slot >= slots_per_eth1_voting_period.integer_sqrt();
-
-                    if period_tail {
+                    if is_period_tail(state) {
                         all_eth1_data
                             .get(vote)
                             .map(|block_number| (vote.clone(), *block_number))
@@ -390,6 +489,15 @@ fn collect_valid_votes<T: EthSpec>(
     valid_votes
 }
 
+/// Indicates if the given `state` is in the tail of it's eth1 voting period (i.e., in the later
+/// slots).
+fn is_period_tail<E: EthSpec>(state: &BeaconState<E>) -> bool {
+    let slots_per_eth1_voting_period = E::SlotsPerEth1VotingPeriod::to_u64();
+    let slot = state.slot % slots_per_eth1_voting_period;
+
+    slot >= slots_per_eth1_voting_period.integer_sqrt()
+}
+
 /// Selects the winning vote from `valid_votes`.
 fn find_winning_vote(valid_votes: Eth1DataVoteCount) -> Option<Eth1Data> {
     valid_votes
@@ -417,6 +525,7 @@ fn slot_start_seconds<T: EthSpec>(
 #[cfg(test)]
 mod test {
     use super::*;
+    use environment::null_logger;
     use types::{test_utils::DepositTestTask, MinimalEthSpec};
 
     type E = MinimalEthSpec;
@@ -468,7 +577,6 @@ mod test {
 
     mod eth1_chain_json_backend {
         use super::*;
-        use environment::null_logger;
         use eth1::DepositLog;
         use store::MemoryStore;
         use types::test_utils::{generate_deterministic_keypair, TestingDepositBuilder};
@@ -514,7 +622,7 @@ mod test {
 
             assert!(
                 eth1_chain
-                    .deposits_for_block_inclusion(&state, spec)
+                    .deposits_for_block_inclusion(&state, &random_eth1_data(), spec)
                     .is_ok(),
                 "should succeed if cache is empty but no deposits are required"
             );
@@ -523,7 +631,7 @@ mod test {
 
             assert!(
                 eth1_chain
-                    .deposits_for_block_inclusion(&state, spec)
+                    .deposits_for_block_inclusion(&state, &random_eth1_data(), spec)
                     .is_err(),
                 "should fail to get deposits if required, but cache is empty"
             );
@@ -567,7 +675,7 @@ mod test {
 
             assert!(
                 eth1_chain
-                    .deposits_for_block_inclusion(&state, spec)
+                    .deposits_for_block_inclusion(&state, &random_eth1_data(), spec)
                     .is_ok(),
                 "should succeed if no deposits are required"
             );
@@ -579,7 +687,7 @@ mod test {
                     state.eth1_data.deposit_count = i as u64;
 
                     let deposits_for_inclusion = eth1_chain
-                        .deposits_for_block_inclusion(&state, spec)
+                        .deposits_for_block_inclusion(&state, &random_eth1_data(), spec)
                         .expect(&format!("should find deposit for {}", i));
 
                     let expected_len =
@@ -657,6 +765,14 @@ mod test {
             prev_state.slot = Slot::new(period * 1_000);
             state.slot = Slot::new(period * 1_000 + period / 2);
 
+            // Add 50% of the votes so a lookup is required.
+            for _ in 0..period / 2 + 1 {
+                state
+                    .eth1_data_votes
+                    .push(random_eth1_data())
+                    .expect("should push eth1 vote");
+            }
+
             (0..2048).for_each(|i| {
                 eth1_chain
                     .backend
@@ -728,6 +844,14 @@ mod test {
 
             state.slot = Slot::new(period / 2);
 
+            // Add 50% of the votes so a lookup is required.
+            for _ in 0..period / 2 + 1 {
+                state
+                    .eth1_data_votes
+                    .push(random_eth1_data())
+                    .expect("should push eth1 vote");
+            }
+
             let expected_root = Hash256::from_low_u64_be(42);
 
             prev_state.eth1_data.block_hash = expected_root;
@@ -757,8 +881,20 @@ mod test {
     mod eth1_data_sets {
         use super::*;
 
+        fn get_voting_period_start_seconds(state: &BeaconState<E>, spec: &ChainSpec) -> u64 {
+            let period = <E as EthSpec>::SlotsPerEth1VotingPeriod::to_u64();
+            let voting_period_start_slot = (state.slot / period) * period;
+            slot_start_seconds::<E>(
+                state.genesis_time,
+                spec.milliseconds_per_slot,
+                voting_period_start_slot,
+            )
+        }
+
         #[test]
         fn empty_cache() {
+            let log = null_logger().unwrap();
+
             let spec = &E::default_spec();
             let state: BeaconState<E> = BeaconState::new(0, get_eth1_data(0), spec);
             let prev_eth1_hash = Hash256::zero();
@@ -766,13 +902,21 @@ mod test {
             let blocks = vec![];
 
             assert_eq!(
-                eth1_data_sets(blocks.iter(), &state, prev_eth1_hash, &spec),
+                eth1_data_sets(
+                    blocks.iter(),
+                    prev_eth1_hash,
+                    get_voting_period_start_seconds(&state, spec),
+                    &spec,
+                    &log
+                ),
                 None
             );
         }
 
         #[test]
         fn no_known_block_hash() {
+            let log = null_logger().unwrap();
+
             let mut spec = E::default_spec();
             spec.milliseconds_per_slot = 1_000;
 
@@ -782,13 +926,21 @@ mod test {
             let blocks = vec![get_eth1_block(0, 0)];
 
             assert_eq!(
-                eth1_data_sets(blocks.iter(), &state, prev_eth1_hash, &spec),
+                eth1_data_sets(
+                    blocks.iter(),
+                    prev_eth1_hash,
+                    get_voting_period_start_seconds(&state, &spec),
+                    &spec,
+                    &log
+                ),
                 None
             );
         }
 
         #[test]
         fn ideal_scenario() {
+            let log = null_logger().unwrap();
+
             let mut spec = E::default_spec();
             spec.milliseconds_per_slot = 1_000;
 
@@ -805,9 +957,14 @@ mod test {
                 .map(|i| get_eth1_block(i, i))
                 .collect::<Vec<_>>();
 
-            let (new_eth1_data, all_eth1_data) =
-                eth1_data_sets(blocks.iter(), &state, prev_eth1_hash, &spec)
-                    .expect("should find data");
+            let (new_eth1_data, all_eth1_data) = eth1_data_sets(
+                blocks.iter(),
+                prev_eth1_hash,
+                get_voting_period_start_seconds(&state, &spec),
+                &spec,
+                &log,
+            )
+            .expect("should find data");
 
             assert_eq!(
                 all_eth1_data.len(),

beacon_node/beacon_chain/src/fork_choice.rs

@@ -1,6 +1,7 @@
 use crate::{errors::BeaconChainError, metrics, BeaconChain, BeaconChainTypes};
 use lmd_ghost::LmdGhost;
 use parking_lot::RwLock;
+use ssz_derive::{Decode, Encode};
 use state_processing::{common::get_attesting_indices, per_slot_processing};
 use std::sync::Arc;
 use store::{Error as StoreError, Store};
@@ -34,6 +35,16 @@ pub struct ForkChoice<T: BeaconChainTypes> {
     best_justified_checkpoint: RwLock<Checkpoint>,
 }
 
+impl<T: BeaconChainTypes> PartialEq for ForkChoice<T> {
+    /// This implementation ignores the `store`.
+    fn eq(&self, other: &Self) -> bool {
+        self.backend == other.backend
+            && self.genesis_block_root == other.genesis_block_root
+            && *self.justified_checkpoint.read() == *other.justified_checkpoint.read()
+            && *self.best_justified_checkpoint.read() == *other.best_justified_checkpoint.read()
+    }
+}
+
 impl<T: BeaconChainTypes> ForkChoice<T> {
     /// Instantiate a new fork chooser.
     ///
@@ -291,6 +302,42 @@ impl<T: BeaconChainTypes> ForkChoice<T> {
             .update_finalized_root(finalized_block, finalized_block_root)
             .map_err(Into::into)
     }
+
+    /// Returns a `SszForkChoice` which contains the current state of `Self`.
+    pub fn as_ssz_container(&self) -> SszForkChoice {
+        SszForkChoice {
+            genesis_block_root: self.genesis_block_root.clone(),
+            justified_checkpoint: self.justified_checkpoint.read().clone(),
+            best_justified_checkpoint: self.best_justified_checkpoint.read().clone(),
+            backend_bytes: self.backend.as_bytes(),
+        }
+    }
+
+    /// Instantiates `Self` from a prior `SszForkChoice`.
+    ///
+    /// The created `Self` will have the same state as the `Self` that created the `SszForkChoice`.
+    pub fn from_ssz_container(ssz_container: SszForkChoice, store: Arc<T::Store>) -> Result<Self> {
+        let backend = LmdGhost::from_bytes(&ssz_container.backend_bytes, store.clone())?;
+
+        Ok(Self {
+            store,
+            backend,
+            genesis_block_root: ssz_container.genesis_block_root,
+            justified_checkpoint: RwLock::new(ssz_container.justified_checkpoint),
+            best_justified_checkpoint: RwLock::new(ssz_container.best_justified_checkpoint),
+        })
+    }
+}
+
+/// Helper struct that is used to encode/decode the state of the `ForkChoice` as SSZ bytes.
+///
+/// This is used when persisting the state of the `BeaconChain` to disk.
+#[derive(Encode, Decode, Clone)]
+pub struct SszForkChoice {
+    genesis_block_root: Hash256,
+    justified_checkpoint: Checkpoint,
+    best_justified_checkpoint: Checkpoint,
+    backend_bytes: Vec<u8>,
 }
 
 impl From<BeaconStateError> for Error {

beacon_node/beacon_chain/src/head_tracker.rs

@@ -0,0 +1,205 @@
+use parking_lot::RwLock;
+use ssz_derive::{Decode, Encode};
+use std::collections::HashMap;
+use std::iter::FromIterator;
+use types::{BeaconBlock, EthSpec, Hash256, Slot};
+
+#[derive(Debug, PartialEq)]
+pub enum Error {
+    MismatchingLengths { roots_len: usize, slots_len: usize },
+}
+
+/// Maintains a list of `BeaconChain` head block roots and slots.
+///
+/// Each time a new block is imported, it should be applied to the `Self::register_block` function.
+/// In order for this struct to be effective, every single block that is imported must be
+/// registered here.
+#[derive(Default, Debug)]
+pub struct HeadTracker(RwLock<HashMap<Hash256, Slot>>);
+
+impl HeadTracker {
+    /// Register a block with `Self`, so it may or may not be included in a `Self::heads` call.
+    ///
+    /// This function assumes that no block is imported without its parent having already been
+    /// imported. It cannot detect an error if this is not the case, it is the responsibility of
+    /// the upstream user.
+    pub fn register_block<E: EthSpec>(&self, block_root: Hash256, block: &BeaconBlock<E>) {
+        let mut map = self.0.write();
+
+        map.remove(&block.parent_root);
+        map.insert(block_root, block.slot);
+    }
+
+    /// Returns the list of heads in the chain.
+    pub fn heads(&self) -> Vec<(Hash256, Slot)> {
+        self.0
+            .read()
+            .iter()
+            .map(|(root, slot)| (*root, *slot))
+            .collect()
+    }
+
+    /// Returns a `SszHeadTracker`, which contains all necessary information to restore the state
+    /// of `Self` at some later point.
+    pub fn to_ssz_container(&self) -> SszHeadTracker {
+        let (roots, slots) = self
+            .0
+            .read()
+            .iter()
+            .map(|(hash, slot)| (*hash, *slot))
+            .unzip();
+
+        SszHeadTracker { roots, slots }
+    }
+
+    /// Creates a new `Self` from the given `SszHeadTracker`, restoring `Self` to the same state of
+    /// the `Self` that created the `SszHeadTracker`.
+    pub fn from_ssz_container(ssz_container: &SszHeadTracker) -> Result<Self, Error> {
+        let roots_len = ssz_container.roots.len();
+        let slots_len = ssz_container.slots.len();
+
+        if roots_len != slots_len {
+            return Err(Error::MismatchingLengths {
+                roots_len,
+                slots_len,
+            });
+        } else {
+            let map = HashMap::from_iter(
+                ssz_container
+                    .roots
+                    .iter()
+                    .zip(ssz_container.slots.iter())
+                    .map(|(root, slot)| (*root, *slot)),
+            );
+
+            Ok(Self(RwLock::new(map)))
+        }
+    }
+}
+
+impl PartialEq<HeadTracker> for HeadTracker {
+    fn eq(&self, other: &HeadTracker) -> bool {
+        *self.0.read() == *other.0.read()
+    }
+}
+
+/// Helper struct that is used to encode/decode the state of the `HeadTracker` as SSZ bytes.
+///
+/// This is used when persisting the state of the `BeaconChain` to disk.
+#[derive(Encode, Decode, Clone)]
+pub struct SszHeadTracker {
+    roots: Vec<Hash256>,
+    slots: Vec<Slot>,
+}
+
+#[cfg(test)]
+mod test {
+    use super::*;
+    use ssz::{Decode, Encode};
+    use types::MainnetEthSpec;
+
+    type E = MainnetEthSpec;
+
+    #[test]
+    fn block_add() {
+        let spec = &E::default_spec();
+
+        let head_tracker = HeadTracker::default();
+
+        for i in 0..16 {
+            let mut block = BeaconBlock::empty(spec);
+            let block_root = Hash256::from_low_u64_be(i);
+
+            block.slot = Slot::new(i);
+            block.parent_root = if i == 0 {
+                Hash256::random()
+            } else {
+                Hash256::from_low_u64_be(i - 1)
+            };
+
+            head_tracker.register_block::<E>(block_root, &block);
+        }
+
+        assert_eq!(
+            head_tracker.heads(),
+            vec![(Hash256::from_low_u64_be(15), Slot::new(15))],
+            "should only have one head"
+        );
+
+        let mut block = BeaconBlock::empty(spec);
+        let block_root = Hash256::from_low_u64_be(42);
+        block.slot = Slot::new(15);
+        block.parent_root = Hash256::from_low_u64_be(14);
+        head_tracker.register_block::<E>(block_root, &block);
+
+        let heads = head_tracker.heads();
+
+        assert_eq!(heads.len(), 2, "should only have two heads");
+        assert!(
+            heads
+                .iter()
+                .any(|(root, slot)| *root == Hash256::from_low_u64_be(15) && *slot == Slot::new(15)),
+            "should contain first head"
+        );
+        assert!(
+            heads
+                .iter()
+                .any(|(root, slot)| *root == Hash256::from_low_u64_be(42) && *slot == Slot::new(15)),
+            "should contain second head"
+        );
+    }
+
+    #[test]
+    fn empty_round_trip() {
+        let non_empty = HeadTracker::default();
+        for i in 0..16 {
+            non_empty.0.write().insert(Hash256::random(), Slot::new(i));
+        }
+        let bytes = non_empty.to_ssz_container().as_ssz_bytes();
+
+        assert_eq!(
+            HeadTracker::from_ssz_container(
+                &SszHeadTracker::from_ssz_bytes(&bytes).expect("should decode")
+            ),
+            Ok(non_empty),
+            "non_empty should pass round trip"
+        );
+    }
+
+    #[test]
+    fn non_empty_round_trip() {
+        let non_empty = HeadTracker::default();
+        for i in 0..16 {
+            non_empty.0.write().insert(Hash256::random(), Slot::new(i));
+        }
+        let bytes = non_empty.to_ssz_container().as_ssz_bytes();
+
+        assert_eq!(
+            HeadTracker::from_ssz_container(
+                &SszHeadTracker::from_ssz_bytes(&bytes).expect("should decode")
+            ),
+            Ok(non_empty),
+            "non_empty should pass round trip"
+        );
+    }
+
+    #[test]
+    fn bad_length() {
+        let container = SszHeadTracker {
+            roots: vec![Hash256::random()],
+            slots: vec![],
+        };
+        let bytes = container.as_ssz_bytes();
+
+        assert_eq!(
+            HeadTracker::from_ssz_container(
+                &SszHeadTracker::from_ssz_bytes(&bytes).expect("should decode")
+            ),
+            Err(Error::MismatchingLengths {
+                roots_len: 1,
+                slots_len: 0
+            }),
+            "should fail decoding with bad lengths"
+        );
+    }
+}

beacon_node/beacon_chain/src/lib.rs

@@ -9,6 +9,7 @@ mod errors;
 pub mod eth1_chain;
 pub mod events;
 mod fork_choice;
+mod head_tracker;
 mod metrics;
 mod persisted_beacon_chain;
 pub mod test_utils;

beacon_node/beacon_chain/src/persisted_beacon_chain.rs

@@ -1,3 +1,5 @@
+use crate::fork_choice::SszForkChoice;
+use crate::head_tracker::SszHeadTracker;
 use crate::{BeaconChainTypes, CheckPoint};
 use operation_pool::PersistedOperationPool;
 use ssz::{Decode, Encode};
@@ -8,11 +10,14 @@ use types::Hash256;
 /// 32-byte key for accessing the `PersistedBeaconChain`.
 pub const BEACON_CHAIN_DB_KEY: &str = "PERSISTEDBEACONCHAINPERSISTEDBEA";
 
-#[derive(Encode, Decode)]
+#[derive(Clone, Encode, Decode)]
 pub struct PersistedBeaconChain<T: BeaconChainTypes> {
     pub canonical_head: CheckPoint<T::EthSpec>,
+    pub finalized_checkpoint: CheckPoint<T::EthSpec>,
     pub op_pool: PersistedOperationPool<T::EthSpec>,
     pub genesis_block_root: Hash256,
+    pub ssz_head_tracker: SszHeadTracker,
+    pub fork_choice: SszForkChoice,
 }
 
 impl<T: BeaconChainTypes> SimpleStoreItem for PersistedBeaconChain<T> {

beacon_node/beacon_chain/src/test_utils.rs

@@ -25,7 +25,10 @@ use types::{
 pub use crate::persisted_beacon_chain::{PersistedBeaconChain, BEACON_CHAIN_DB_KEY};
 pub use types::test_utils::generate_deterministic_keypairs;
 
-pub const HARNESS_GENESIS_TIME: u64 = 1_567_552_690; // 4th September 2019
+// 4th September 2019
+pub const HARNESS_GENESIS_TIME: u64 = 1_567_552_690;
+// This parameter is required by a builder but not used because we use the `TestingSlotClock`.
+pub const HARNESS_SLOT_TIME: Duration = Duration::from_secs(1);
 
 pub type BaseHarnessType<TStore, TStoreMigrator, TEthSpec> = Witness<
     TStore,
@@ -98,9 +101,9 @@ impl<E: EthSpec> BeaconChainHarness<HarnessType<E>> {
             .dummy_eth1_backend()
             .expect("should build dummy backend")
             .null_event_handler()
-            .testing_slot_clock(Duration::from_secs(1))
+            .testing_slot_clock(HARNESS_SLOT_TIME)
             .expect("should configure testing slot clock")
-            .empty_reduced_tree_fork_choice()
+            .reduced_tree_fork_choice()
             .expect("should add fork choice to builder")
             .build()
             .expect("should build");
@@ -115,7 +118,7 @@ impl<E: EthSpec> BeaconChainHarness<HarnessType<E>> {
 
 impl<E: EthSpec> BeaconChainHarness<DiskHarnessType<E>> {
     /// Instantiate a new harness with `validator_count` initial validators.
-    pub fn with_disk_store(
+    pub fn new_with_disk_store(
         eth_spec_instance: E,
         store: Arc<DiskStore>,
         keypairs: Vec<Keypair>,
@@ -140,9 +143,46 @@ impl<E: EthSpec> BeaconChainHarness<DiskHarnessType<E>> {
             .dummy_eth1_backend()
             .expect("should build dummy backend")
             .null_event_handler()
+            .testing_slot_clock(HARNESS_SLOT_TIME)
+            .expect("should configure testing slot clock")
+            .reduced_tree_fork_choice()
+            .expect("should add fork choice to builder")
+            .build()
+            .expect("should build");
+
+        Self {
+            spec: chain.spec.clone(),
+            chain,
+            keypairs,
+        }
+    }
+
+    /// Instantiate a new harness with `validator_count` initial validators.
+    pub fn resume_from_disk_store(
+        eth_spec_instance: E,
+        store: Arc<DiskStore>,
+        keypairs: Vec<Keypair>,
+    ) -> Self {
+        let spec = E::default_spec();
+
+        let log = TerminalLoggerBuilder::new()
+            .level(Severity::Warning)
+            .build()
+            .expect("logger should build");
+
+        let chain = BeaconChainBuilder::new(eth_spec_instance)
+            .logger(log.clone())
+            .custom_spec(spec.clone())
+            .store(store.clone())
+            .store_migrator(<BlockingMigrator<_> as Migrate<_, E>>::new(store))
+            .resume_from_db()
+            .expect("should resume beacon chain from db")
+            .dummy_eth1_backend()
+            .expect("should build dummy backend")
+            .null_event_handler()
             .testing_slot_clock(Duration::from_secs(1))
             .expect("should configure testing slot clock")
-            .empty_reduced_tree_fork_choice()
+            .reduced_tree_fork_choice()
             .expect("should add fork choice to builder")
             .build()
             .expect("should build");

beacon_node/beacon_chain/tests/persistence_tests.rs

@@ -0,0 +1,134 @@
+#![cfg(not(debug_assertions))]
+
+#[macro_use]
+extern crate lazy_static;
+
+use beacon_chain::{
+    test_utils::{AttestationStrategy, BeaconChainHarness, BlockStrategy},
+    BeaconChain, BeaconChainTypes,
+};
+use sloggers::{null::NullLoggerBuilder, Build};
+use std::sync::Arc;
+use store::DiskStore;
+use tempfile::{tempdir, TempDir};
+use types::{EthSpec, Keypair, MinimalEthSpec};
+
+type E = MinimalEthSpec;
+
+// Should ideally be divisible by 3.
+pub const VALIDATOR_COUNT: usize = 24;
+
+lazy_static! {
+    /// A cached set of keys.
+    static ref KEYPAIRS: Vec<Keypair> = types::test_utils::generate_deterministic_keypairs(VALIDATOR_COUNT);
+}
+
+fn get_store(db_path: &TempDir) -> Arc<DiskStore> {
+    let spec = E::default_spec();
+    let hot_path = db_path.path().join("hot_db");
+    let cold_path = db_path.path().join("cold_db");
+    let log = NullLoggerBuilder.build().expect("logger should build");
+    Arc::new(
+        DiskStore::open(&hot_path, &cold_path, spec, log).expect("disk store should initialize"),
+    )
+}
+
+#[test]
+fn finalizes_after_resuming_from_db() {
+    let validator_count = 16;
+    let num_blocks_produced = MinimalEthSpec::slots_per_epoch() * 8;
+    let first_half = num_blocks_produced / 2;
+
+    let db_path = tempdir().unwrap();
+    let store = get_store(&db_path);
+
+    let harness = BeaconChainHarness::new_with_disk_store(
+        MinimalEthSpec,
+        store.clone(),
+        KEYPAIRS[0..validator_count].to_vec(),
+    );
+
+    harness.advance_slot();
+
+    harness.extend_chain(
+        first_half as usize,
+        BlockStrategy::OnCanonicalHead,
+        AttestationStrategy::AllValidators,
+    );
+
+    assert!(
+        harness.chain.head().beacon_state.finalized_checkpoint.epoch > 0,
+        "the chain should have already finalized"
+    );
+
+    let latest_slot = harness.chain.slot().expect("should have a slot");
+
+    harness.chain.persist().expect("should persist the chain");
+
+    let resumed_harness = BeaconChainHarness::resume_from_disk_store(
+        MinimalEthSpec,
+        store,
+        KEYPAIRS[0..validator_count].to_vec(),
+    );
+
+    assert_chains_pretty_much_the_same(&harness.chain, &resumed_harness.chain);
+
+    // Ensures we don't accidentally use it again.
+    //
+    // Note: this will persist the chain again, but that shouldn't matter since nothing has
+    // changed.
+    drop(harness);
+
+    // Set the slot clock of the resumed harness to be in the slot following the previous harness.
+    //
+    // This allows us to produce the block at the next slot.
+    resumed_harness
+        .chain
+        .slot_clock
+        .set_slot(latest_slot.as_u64() + 1);
+
+    resumed_harness.extend_chain(
+        (num_blocks_produced - first_half) as usize,
+        BlockStrategy::OnCanonicalHead,
+        AttestationStrategy::AllValidators,
+    );
+
+    let state = &resumed_harness.chain.head().beacon_state;
+    assert_eq!(
+        state.slot, num_blocks_produced,
+        "head should be at the current slot"
+    );
+    assert_eq!(
+        state.current_epoch(),
+        num_blocks_produced / MinimalEthSpec::slots_per_epoch(),
+        "head should be at the expected epoch"
+    );
+    assert_eq!(
+        state.current_justified_checkpoint.epoch,
+        state.current_epoch() - 1,
+        "the head should be justified one behind the current epoch"
+    );
+    assert_eq!(
+        state.finalized_checkpoint.epoch,
+        state.current_epoch() - 2,
+        "the head should be finalized two behind the current epoch"
+    );
+}
+
+/// Checks that two chains are the same, for the purpose of this tests.
+///
+/// Several fields that are hard/impossible to check are ignored (e.g., the store).
+fn assert_chains_pretty_much_the_same<T: BeaconChainTypes>(a: &BeaconChain<T>, b: &BeaconChain<T>) {
+    assert_eq!(a.spec, b.spec, "spec should be equal");
+    assert_eq!(a.op_pool, b.op_pool, "op_pool should be equal");
+    assert_eq!(a.head(), b.head(), "head() should be equal");
+    assert_eq!(a.heads(), b.heads(), "heads() should be equal");
+    assert_eq!(
+        a.genesis_block_root, b.genesis_block_root,
+        "genesis_block_root should be equal"
+    );
+    assert!(
+        a.fork_choice == b.fork_choice,
+        "fork_choice should be equal"
+    );
+}

beacon_node/beacon_chain/tests/store_tests.rs

@@ -37,7 +37,7 @@ fn get_store(db_path: &TempDir) -> Arc<DiskStore> {
 }
 
 fn get_harness(store: Arc<DiskStore>, validator_count: usize) -> TestHarness {
-    let harness = BeaconChainHarness::with_disk_store(
+    let harness = BeaconChainHarness::new_with_disk_store(
         MinimalEthSpec,
         store,
         KEYPAIRS[0..validator_count].to_vec(),

beacon_node/client/Cargo.toml

@@ -28,7 +28,6 @@ serde_yaml = "0.8.11"
 slog = { version = "2.5.2", features = ["max_level_trace"] }
 slog-async = "2.3.0"
 tokio = "0.1.22"
-clap = "2.33.0"
 dirs = "2.0.2"
 exit-future = "0.1.4"
 futures = "0.1.29"
@@ -39,3 +38,4 @@ eth1 = { path = "../eth1" }
 genesis = { path = "../genesis" }
 environment = { path = "../../lighthouse/environment" }
 lighthouse_bootstrap = { path = "../../eth2/utils/lighthouse_bootstrap" }
+eth2_ssz = { path = "../../eth2/utils/ssz" }

beacon_node/client/src/builder.rs

@@ -23,13 +23,14 @@ use lighthouse_bootstrap::Bootstrapper;
 use lmd_ghost::LmdGhost;
 use network::{NetworkConfig, NetworkMessage, Service as NetworkService};
 use slog::{debug, error, info, warn};
+use ssz::Decode;
 use std::net::SocketAddr;
 use std::path::Path;
 use std::sync::Arc;
 use std::time::{Duration, Instant};
 use tokio::sync::mpsc::UnboundedSender;
 use tokio::timer::Interval;
-use types::{ChainSpec, EthSpec};
+use types::{BeaconState, ChainSpec, EthSpec};
 use websocket_server::{Config as WebSocketConfig, WebSocketSender};
 
 /// The interval between notifier events.
@@ -37,7 +38,7 @@ pub const NOTIFIER_INTERVAL_SECONDS: u64 = 15;
 /// Create a warning log whenever the peer count is at or below this value.
 pub const WARN_PEER_COUNT: usize = 1;
 /// Interval between polling the eth1 node for genesis information.
-pub const ETH1_GENESIS_UPDATE_INTERVAL_MILLIS: u64 = 500;
+pub const ETH1_GENESIS_UPDATE_INTERVAL_MILLIS: u64 = 7_000;
 
 /// Builds a `Client` instance.
 ///
@@ -148,7 +149,7 @@ where
                 })?;
                 let context = runtime_context
                     .ok_or_else(|| "beacon_chain_start_method requires a log".to_string())?
-                    .service_context("beacon");
+                    .service_context("beacon".into());
                 let spec = chain_spec
                     .ok_or_else(|| "beacon_chain_start_method requires a chain spec".to_string())?;
 
@@ -187,40 +188,34 @@ where
 
                             Box::new(future)
                         }
-                        ClientGenesis::DepositContract => {
-                            let genesis_service = Eth1GenesisService::new(
-                                // Some of the configuration options for `Eth1Config` are
-                                // hard-coded when listening for genesis from the deposit contract.
-                                //
-                                // The idea is that the `Eth1Config` supplied to this function
-                                // (`config`) is intended for block production duties (i.e.,
-                                // listening for deposit events and voting on eth1 data) and that
-                                // we can make listening for genesis more efficient if we modify
-                                // some params.
-                                Eth1Config {
-                                    // Truncating the block cache makes searching for genesis more
-                                    // complicated.
-                                    block_cache_truncation: None,
-                                    // Scan large ranges of blocks when awaiting genesis.
-                                    blocks_per_log_query: 1_000,
-                                    // Only perform a single log request each time the eth1 node is
-                                    // polled.
-                                    //
-                                    // For small testnets this makes finding genesis much faster,
-                                    // as it usually happens within 1,000 blocks.
-                                    max_log_requests_per_update: Some(1),
-                                    // Only perform a single block request each time the eth1 node
-                                    // is polled.
-                                    //
-                                    // For small testnets, this is much faster as they do not have
-                                    // a `MIN_GENESIS_SECONDS`, so after `MIN_GENESIS_VALIDATOR_COUNT`
-                                    // has been reached only a single block needs to be read.
-                                    max_blocks_per_update: Some(1),
-                                    ..config
-                                },
-                                context.log.clone(),
+                        ClientGenesis::SszBytes {
+                            genesis_state_bytes,
+                        } => {
+                            info!(
+                                context.log,
+                                "Starting from known genesis state";
                             );
 
+                            let result = BeaconState::from_ssz_bytes(&genesis_state_bytes)
+                                .map_err(|e| format!("Unable to parse genesis state SSZ: {:?}", e));
+
+                            let future = result
+                                .and_then(move |genesis_state| builder.genesis_state(genesis_state))
+                                .into_future()
+                                .map(|v| (v, None));
+
+                            Box::new(future)
+                        }
+                        ClientGenesis::DepositContract => {
+                            info!(
+                                context.log,
+                                "Waiting for eth2 genesis from eth1";
+                                "eth1_node" => &config.endpoint
+                            );
+
+                            let genesis_service =
+                                Eth1GenesisService::new(config, context.log.clone());
+
                             let future = genesis_service
                                 .wait_for_genesis_state(
                                     Duration::from_millis(ETH1_GENESIS_UPDATE_INTERVAL_MILLIS),
@@ -275,7 +270,7 @@ where
             .runtime_context
             .as_ref()
             .ok_or_else(|| "libp2p_network requires a runtime_context")?
-            .service_context("network");
+            .service_context("network".into());
 
         let (network, network_send) =
             NetworkService::new(beacon_chain, config, &context.executor, context.log)
@@ -301,7 +296,7 @@ where
             .runtime_context
             .as_ref()
             .ok_or_else(|| "http_server requires a runtime_context")?
-            .service_context("http");
+            .service_context("http".into());
         let network = self
             .libp2p_network
             .clone()
@@ -341,7 +336,7 @@ where
             .runtime_context
             .as_ref()
             .ok_or_else(|| "peer_count_notifier requires a runtime_context")?
-            .service_context("peer_notifier");
+            .service_context("peer_notifier".into());
         let log = context.log.clone();
         let log_2 = context.log.clone();
         let network = self
@@ -384,7 +379,7 @@ where
             .runtime_context
             .as_ref()
             .ok_or_else(|| "slot_notifier requires a runtime_context")?
-            .service_context("slot_notifier");
+            .service_context("slot_notifier".into());
         let log = context.log.clone();
         let log_2 = log.clone();
         let beacon_chain = self
@@ -498,7 +493,7 @@ where
                     .clone()
                     .ok_or_else(|| "beacon_chain requires a slot clock")?,
             )
-            .empty_reduced_tree_fork_choice()
+            .reduced_tree_fork_choice()
             .map_err(|e| format!("Failed to init fork choice: {}", e))?
             .build()
             .map_err(|e| format!("Failed to build beacon chain: {}", e))?;
@@ -537,7 +532,7 @@ where
             .runtime_context
             .as_ref()
             .ok_or_else(|| "websocket_event_handler requires a runtime_context")?
-            .service_context("ws");
+            .service_context("ws".into());
 
         let (sender, exit_signal, listening_addr): (
             WebSocketSender<TEthSpec>,
@@ -587,7 +582,7 @@ where
             .runtime_context
             .as_ref()
             .ok_or_else(|| "disk_store requires a log".to_string())?
-            .service_context("freezer_db");
+            .service_context("freezer_db".into());
         let spec = self
             .chain_spec
             .clone()
@@ -715,7 +710,7 @@ where
             .runtime_context
             .as_ref()
             .ok_or_else(|| "caching_eth1_backend requires a runtime_context")?
-            .service_context("eth1_rpc");
+            .service_context("eth1_rpc".into());
         let beacon_chain_builder = self
             .beacon_chain_builder
             .ok_or_else(|| "caching_eth1_backend requires a beacon_chain_builder")?;
@@ -726,6 +721,17 @@ where
 
         let backend = if let Some(eth1_service_from_genesis) = self.eth1_service {
             eth1_service_from_genesis.update_config(config.clone())?;
+
+            // This cache is not useful because it's first (earliest) block likely the block that
+            // triggered genesis.
+            //
+            // In order to vote we need to be able to go back at least 2 * `ETH1_FOLLOW_DISTANCE`
+            // from the genesis-triggering block.  Presently the block cache does not support
+            // importing blocks with decreasing block numbers, it only accepts them in increasing
+            // order. If this turns out to be a bottleneck we can update the block cache to allow
+            // adding earlier blocks too.
+            eth1_service_from_genesis.drop_block_cache();
+
             CachingEth1Backend::from_service(eth1_service_from_genesis, store)
         } else {
             CachingEth1Backend::new(config, context.log, store)

beacon_node/client/src/config.rs

@@ -1,4 +1,3 @@
-use clap::ArgMatches;
 use network::NetworkConfig;
 use serde_derive::{Deserialize, Serialize};
 use std::fs;
@@ -25,6 +24,11 @@ pub enum ClientGenesis {
     DepositContract,
     /// Loads the genesis state from a SSZ-encoded `BeaconState` file.
     SszFile { path: PathBuf },
+    /// Loads the genesis state from SSZ-encoded `BeaconState` bytes.
+    ///
+    /// We include the bytes instead of the `BeaconState<E>` because the `EthSpec` type
+    /// parameter would be very annoying.
+    SszBytes { genesis_state_bytes: Vec<u8> },
     /// Connects to another Lighthouse instance and reads the genesis state and other data via the
     /// HTTP API.
     RemoteNode { server: String, port: Option<u16> },
@@ -40,9 +44,10 @@ impl Default for ClientGenesis {
 #[derive(Debug, Clone, Serialize, Deserialize)]
 pub struct Config {
     pub data_dir: PathBuf,
+    pub testnet_dir: Option<PathBuf>,
     pub db_type: String,
-    db_name: String,
-    freezer_db_path: Option<PathBuf>,
+    pub db_name: String,
+    pub freezer_db_path: Option<PathBuf>,
     pub log_file: PathBuf,
     pub spec_constants: String,
     /// If true, the node will use co-ordinated junk for eth1 values.
@@ -64,12 +69,13 @@ impl Default for Config {
     fn default() -> Self {
         Self {
             data_dir: PathBuf::from(".lighthouse"),
+            testnet_dir: None,
             log_file: PathBuf::from(""),
             db_type: "disk".to_string(),
             db_name: "chain_db".to_string(),
             freezer_db_path: None,
             genesis: <_>::default(),
-            network: NetworkConfig::new(),
+            network: NetworkConfig::default(),
             rest_api: <_>::default(),
             websocket_server: <_>::default(),
             spec_constants: TESTNET_SPEC_CONSTANTS.into(),
@@ -135,26 +141,6 @@ impl Config {
             .ok_or_else(|| "Unable to locate user home directory".to_string())?;
         ensure_dir_exists(path)
     }
-
-    /// Apply the following arguments to `self`, replacing values if they are specified in `args`.
-    ///
-    /// Returns an error if arguments are obviously invalid. May succeed even if some values are
-    /// invalid.
-    pub fn apply_cli_args(&mut self, args: &ArgMatches, _log: &slog::Logger) -> Result<(), String> {
-        if let Some(dir) = args.value_of("datadir") {
-            self.data_dir = PathBuf::from(dir);
-        };
-
-        if let Some(freezer_dir) = args.value_of("freezer-dir") {
-            self.freezer_db_path = Some(PathBuf::from(freezer_dir));
-        }
-
-        self.network.apply_cli_args(args)?;
-        self.rest_api.apply_cli_args(args)?;
-        self.websocket_server.apply_cli_args(args)?;
-
-        Ok(())
-    }
 }
 
 /// Ensure that the directory at `path` exists, by creating it and all parents if necessary.

beacon_node/eth1/src/block_cache.rs

@@ -54,6 +54,21 @@ impl BlockCache {
         self.blocks.is_empty()
     }
 
+    /// Returns the timestamp of the earliest block in the cache (if any).
+    pub fn earliest_block_timestamp(&self) -> Option<u64> {
+        self.blocks.first().map(|block| block.timestamp)
+    }
+
+    /// Returns the timestamp of the latest block in the cache (if any).
+    pub fn latest_block_timestamp(&self) -> Option<u64> {
+        self.blocks.last().map(|block| block.timestamp)
+    }
+
+    /// Returns the lowest block number stored.
+    pub fn lowest_block_number(&self) -> Option<u64> {
+        self.blocks.first().map(|block| block.number)
+    }
+
     /// Returns the highest block number stored.
     pub fn highest_block_number(&self) -> Option<u64> {
         self.blocks.last().map(|block| block.number)

beacon_node/eth1/src/http.rs

@@ -23,7 +23,7 @@ use types::Hash256;
 pub const DEPOSIT_EVENT_TOPIC: &str =
     "0x649bbc62d0e31342afea4e5cd82d4049e7e1ee912fc0889aa790803be39038c5";
 /// `keccak("get_deposit_root()")[0..4]`
-pub const DEPOSIT_ROOT_FN_SIGNATURE: &str = "0x863a311b";
+pub const DEPOSIT_ROOT_FN_SIGNATURE: &str = "0xc5f2892f";
 /// `keccak("get_deposit_count()")[0..4]`
 pub const DEPOSIT_COUNT_FN_SIGNATURE: &str = "0x621fd130";
 
@@ -137,19 +137,21 @@ pub fn get_deposit_count(
         block_number,
         timeout,
     )
-    .and_then(|result| result.ok_or_else(|| "No response to deposit count".to_string()))
-    .and_then(|bytes| {
-        if bytes.is_empty() {
-            Ok(None)
-        } else if bytes.len() == DEPOSIT_COUNT_RESPONSE_BYTES {
-            let mut array = [0; 8];
-            array.copy_from_slice(&bytes[32 + 32..32 + 32 + 8]);
-            Ok(Some(u64::from_le_bytes(array)))
-        } else {
-            Err(format!(
-                "Deposit count response was not {} bytes: {:?}",
-                DEPOSIT_COUNT_RESPONSE_BYTES, bytes
-            ))
+    .and_then(|result| match result {
+        None => Err(format!("Deposit root response was none")),
+        Some(bytes) => {
+            if bytes.is_empty() {
+                Ok(None)
+            } else if bytes.len() == DEPOSIT_COUNT_RESPONSE_BYTES {
+                let mut array = [0; 8];
+                array.copy_from_slice(&bytes[32 + 32..32 + 32 + 8]);
+                Ok(Some(u64::from_le_bytes(array)))
+            } else {
+                Err(format!(
+                    "Deposit count response was not {} bytes: {:?}",
+                    DEPOSIT_COUNT_RESPONSE_BYTES, bytes
+                ))
+            }
         }
     })
 }
@@ -172,17 +174,19 @@ pub fn get_deposit_root(
         block_number,
         timeout,
     )
-    .and_then(|result| result.ok_or_else(|| "No response to deposit root".to_string()))
-    .and_then(|bytes| {
-        if bytes.is_empty() {
-            Ok(None)
-        } else if bytes.len() == DEPOSIT_ROOT_BYTES {
-            Ok(Some(Hash256::from_slice(&bytes)))
-        } else {
-            Err(format!(
-                "Deposit root response was not {} bytes: {:?}",
-                DEPOSIT_ROOT_BYTES, bytes
-            ))
+    .and_then(|result| match result {
+        None => Err(format!("Deposit root response was none")),
+        Some(bytes) => {
+            if bytes.is_empty() {
+                Ok(None)
+            } else if bytes.len() == DEPOSIT_ROOT_BYTES {
+                Ok(Some(Hash256::from_slice(&bytes)))
+            } else {
+                Err(format!(
+                    "Deposit root response was not {} bytes: {:?}",
+                    DEPOSIT_ROOT_BYTES, bytes
+                ))
+            }
         }
     })
 }
@@ -369,12 +373,18 @@ pub fn send_rpc_request(
 
 /// Accepts an entire HTTP body (as a string) and returns the `result` field, as a serde `Value`.
 fn response_result(response: &str) -> Result<Option<Value>, String> {
-    Ok(serde_json::from_str::<Value>(&response)
-        .map_err(|e| format!("Failed to parse response: {:?}", e))?
-        .get("result")
-        .cloned()
-        .map(Some)
-        .unwrap_or_else(|| None))
+    let json = serde_json::from_str::<Value>(&response)
+        .map_err(|e| format!("Failed to parse response: {:?}", e))?;
+
+    if let Some(error) = json.get("error") {
+        Err(format!("Eth1 node returned error: {}", error))
+    } else {
+        Ok(json
+            .get("result")
+            .cloned()
+            .map(Some)
+            .unwrap_or_else(|| None))
+    }
 }
 
 /// Parses a `0x`-prefixed, **big-endian** hex string as a u64.

beacon_node/eth1/src/service.rs

@@ -121,7 +121,7 @@ impl Default for Config {
             lowest_cached_block_number: 0,
             follow_distance: 128,
             block_cache_truncation: Some(4_096),
-            auto_update_interval_millis: 500,
+            auto_update_interval_millis: 7_000,
             blocks_per_log_query: 1_000,
             max_log_requests_per_update: None,
             max_blocks_per_update: None,
@@ -163,6 +163,26 @@ impl Service {
         &self.inner.deposit_cache
     }
 
+    /// Drop the block cache, replacing it with an empty one.
+    pub fn drop_block_cache(&self) {
+        *(self.inner.block_cache.write()) = BlockCache::default();
+    }
+
+    /// Returns the timestamp of the earliest block in the cache (if any).
+    pub fn earliest_block_timestamp(&self) -> Option<u64> {
+        self.inner.block_cache.read().earliest_block_timestamp()
+    }
+
+    /// Returns the timestamp of the latest block in the cache (if any).
+    pub fn latest_block_timestamp(&self) -> Option<u64> {
+        self.inner.block_cache.read().latest_block_timestamp()
+    }
+
+    /// Returns the lowest block number stored.
+    pub fn lowest_block_number(&self) -> Option<u64> {
+        self.inner.block_cache.read().lowest_block_number()
+    }
+
     /// Returns the number of currently cached blocks.
     pub fn block_cache_len(&self) -> usize {
         self.blocks().read().len()
@@ -220,6 +240,8 @@ impl Service {
     {
         let log_a = self.log.clone();
         let log_b = self.log.clone();
+        let inner_1 = self.inner.clone();
+        let inner_2 = self.inner.clone();
 
         let deposit_future = self
             .update_deposit_cache()
@@ -229,6 +251,7 @@ impl Service {
                     Ok(DepositCacheUpdateOutcome::Success { logs_imported }) => trace!(
                         log_a,
                         "Updated eth1 deposit cache";
+                        "cached_deposits" => inner_1.deposit_cache.read().cache.len(),
                         "logs_imported" => logs_imported,
                     ),
                     Err(e) => error!(
@@ -252,6 +275,7 @@ impl Service {
                     }) => trace!(
                         log_b,
                         "Updated eth1 block cache";
+                        "cached_blocks" => inner_2.block_cache.read().len(),
                         "blocks_imported" => blocks_imported,
                         "head_block" => head_block_number,
                     ),

beacon_node/eth2-libp2p/Cargo.toml

@@ -5,7 +5,6 @@ authors = ["Age Manning <Age@AgeManning.com>"]
 edition = "2018"
 
 [dependencies]
-clap = "2.33.0"
 hex = "0.3"
 # rust-libp2p is presently being sourced from a Sigma Prime fork of the
 # `libp2p/rust-libp2p` repository.

beacon_node/eth2-libp2p/src/config.rs

@@ -1,4 +1,3 @@
-use clap::ArgMatches;
 use enr::Enr;
 use libp2p::gossipsub::{GossipsubConfig, GossipsubConfigBuilder};
 use libp2p::Multiaddr;
@@ -97,84 +96,3 @@ impl Default for Config {
         }
     }
 }
-
-/// Generates a default Config.
-impl Config {
-    pub fn new() -> Self {
-        Config::default()
-    }
-
-    pub fn apply_cli_args(&mut self, args: &ArgMatches) -> Result<(), String> {
-        // If a `datadir` has been specified, set the network dir to be inside it.
-        if let Some(dir) = args.value_of("datadir") {
-            self.network_dir = PathBuf::from(dir).join("network");
-        };
-
-        // If a network dir has been specified, override the `datadir` definition.
-        if let Some(dir) = args.value_of("network-dir") {
-            self.network_dir = PathBuf::from(dir);
-        };
-
-        if let Some(listen_address_str) = args.value_of("listen-address") {
-            let listen_address = listen_address_str
-                .parse()
-                .map_err(|_| format!("Invalid listen address: {:?}", listen_address_str))?;
-            self.listen_address = listen_address;
-            self.discovery_address = listen_address;
-        }
-
-        if let Some(max_peers_str) = args.value_of("maxpeers") {
-            self.max_peers = max_peers_str
-                .parse::<usize>()
-                .map_err(|_| format!("Invalid number of max peers: {}", max_peers_str))?;
-        }
-
-        if let Some(port_str) = args.value_of("port") {
-            let port = port_str
-                .parse::<u16>()
-                .map_err(|_| format!("Invalid port: {}", port_str))?;
-            self.libp2p_port = port;
-            self.discovery_port = port;
-        }
-
-        if let Some(boot_enr_str) = args.value_of("boot-nodes") {
-            self.boot_nodes = boot_enr_str
-                .split(',')
-                .map(|enr| enr.parse().map_err(|_| format!("Invalid ENR: {}", enr)))
-                .collect::<Result<Vec<Enr>, _>>()?;
-        }
-
-        if let Some(libp2p_addresses_str) = args.value_of("libp2p-addresses") {
-            self.libp2p_nodes = libp2p_addresses_str
-                .split(',')
-                .map(|multiaddr| {
-                    multiaddr
-                        .parse()
-                        .map_err(|_| format!("Invalid Multiaddr: {}", multiaddr))
-                })
-                .collect::<Result<Vec<Multiaddr>, _>>()?;
-        }
-
-        if let Some(topics_str) = args.value_of("topics") {
-            self.topics = topics_str.split(',').map(|s| s.into()).collect();
-        }
-
-        if let Some(discovery_address_str) = args.value_of("discovery-address") {
-            self.discovery_address = discovery_address_str
-                .parse()
-                .map_err(|_| format!("Invalid discovery address: {:?}", discovery_address_str))?
-        }
-
-        if let Some(disc_port_str) = args.value_of("disc-port") {
-            self.discovery_port = disc_port_str
-                .parse::<u16>()
-                .map_err(|_| format!("Invalid discovery port: {}", disc_port_str))?;
-        }
-
-        if let Some(p2p_priv_key) = args.value_of("p2p-priv-key") {
-            self.secret_key_hex = Some(p2p_priv_key.to_string());
-        }
-
-        Ok(())
-    }
-}

beacon_node/genesis/src/eth1_genesis_service.rs

@@ -8,7 +8,7 @@ use futures::{
     Future,
 };
 use parking_lot::Mutex;
-use slog::{debug, error, info, Logger};
+use slog::{debug, error, info, trace, Logger};
 use state_processing::{
     initialize_beacon_state_from_eth1, is_valid_genesis_state,
     per_block_processing::process_deposit, process_activations,
@@ -37,7 +37,29 @@ pub struct Eth1GenesisService {
 
 impl Eth1GenesisService {
     /// Creates a new service. Does not attempt to connect to the Eth1 node.
+    ///
+    /// Modifies the given `config` to make it more suitable to the task of listening to genesis.
     pub fn new(config: Eth1Config, log: Logger) -> Self {
+        let config = Eth1Config {
+            // Truncating the block cache makes searching for genesis more
+            // complicated.
+            block_cache_truncation: None,
+            // Scan large ranges of blocks when awaiting genesis.
+            blocks_per_log_query: 1_000,
+            // Only perform a few log requests each time the eth1 node is polled.
+            //
+            // For small testnets this makes finding genesis much faster,
+            // as it usually happens within 1,000 blocks.
+            max_log_requests_per_update: Some(5),
+            // Only perform a few logs requests each time the eth1 node is polled.
+            //
+            // For small testnets, this is much faster as they do not have
+            // a `MIN_GENESIS_SECONDS`, so after `MIN_GENESIS_VALIDATOR_COUNT`
+            // has been reached only a single block needs to be read.
+            max_blocks_per_update: Some(5),
+            ..config
+        };
+
         Self {
             core: Service::new(config, log),
             highest_processed_block: Arc::new(Mutex::new(None)),
@@ -81,6 +103,7 @@ impl Eth1GenesisService {
                 let service_4 = service.clone();
                 let log = service.core.log.clone();
                 let min_genesis_active_validator_count = spec.min_genesis_active_validator_count;
+                let min_genesis_time = spec.min_genesis_time;
 
                 Delay::new(Instant::now() + update_interval)
                     .map_err(|e| format!("Delay between genesis deposit checks failed: {:?}", e))
@@ -161,6 +184,9 @@ impl Eth1GenesisService {
                             debug!(
                                 service_4.core.log,
                                 "No eth1 genesis block found";
+                                "latest_block_timestamp" => service_4.core.latest_block_timestamp(),
+                                "min_genesis_time" => min_genesis_time,
+                                "min_validator_count" => min_genesis_active_validator_count,
                                 "cached_blocks" => service_4.core.block_cache_len(),
                                 "cached_deposits" => service_4.core.deposit_cache_len(),
                                 "cache_head" => service_4.highest_known_block(),
@@ -205,23 +231,35 @@ impl Eth1GenesisService {
             .filter(|block| {
                 self.highest_known_block()
                     .map(|n| block.number <= n)
-                    .unwrap_or_else(|| false)
+                    .unwrap_or_else(|| true)
             })
             .find(|block| {
                 let mut highest_processed_block = self.highest_processed_block.lock();
+                let block_number = block.number;
 
                 let next_new_block_number =
                     highest_processed_block.map(|n| n + 1).unwrap_or_else(|| 0);
 
-                if block.number < next_new_block_number {
+                if block_number < next_new_block_number {
                     return false;
                 }
 
-                self.is_valid_genesis_eth1_block::<E>(block, &spec)
+                self.is_valid_genesis_eth1_block::<E>(block, &spec, &self.core.log)
                     .and_then(|val| {
                         *highest_processed_block = Some(block.number);
                         Ok(val)
                     })
+                    .map(|is_valid| {
+                        if !is_valid {
+                            info!(
+                                self.core.log,
+                                "Inspected new eth1 block";
+                                "msg" => "did not trigger genesis",
+                                "block_number" => block_number
+                            );
+                        };
+                        is_valid
+                    })
                     .unwrap_or_else(|_| {
                         error!(
                             self.core.log,
@@ -301,6 +339,7 @@ impl Eth1GenesisService {
         &self,
         target_block: &Eth1Block,
         spec: &ChainSpec,
+        log: &Logger,
     ) -> Result<bool, String> {
         if target_block.timestamp < spec.min_genesis_time {
             Ok(false)
@@ -345,8 +384,16 @@ impl Eth1GenesisService {
                 })?;
 
             process_activations(&mut local_state, spec);
+            let is_valid = is_valid_genesis_state(&local_state, spec);
 
-            Ok(is_valid_genesis_state(&local_state, spec))
+            trace!(
+                log,
+                "Eth1 block inspected for genesis";
+                "active_validators" => local_state.get_active_validator_indices(local_state.current_epoch()).len(),
+                "validators" => local_state.validators.len()
+            );
+
+            Ok(is_valid)
         }
     }
 

beacon_node/rest_api/Cargo.toml

@@ -22,7 +22,6 @@ eth2_ssz = { path = "../../eth2/utils/ssz" }
 eth2_ssz_derive = { path = "../../eth2/utils/ssz_derive" }
 state_processing = { path = "../../eth2/state_processing" }
 types = { path = "../../eth2/types" }
-clap = "2.33"
 http = "0.1"
 hyper = "0.12"
 exit-future = "0.1.4"

beacon_node/rest_api/src/beacon.rs

@@ -56,6 +56,29 @@ pub fn get_head<T: BeaconChainTypes>(
     ResponseBuilder::new(&req)?.body(&head)
 }
 
+#[derive(Serialize, Deserialize, Encode)]
+pub struct HeadBeaconBlock {
+    beacon_block_root: Hash256,
+    beacon_block_slot: Slot,
+}
+
+/// HTTP handler to return a list of head BeaconBlocks.
+pub fn get_heads<T: BeaconChainTypes>(
+    req: Request<Body>,
+    beacon_chain: Arc<BeaconChain<T>>,
+) -> ApiResult {
+    let heads = beacon_chain
+        .heads()
+        .into_iter()
+        .map(|(beacon_block_root, beacon_block_slot)| HeadBeaconBlock {
+            beacon_block_root,
+            beacon_block_slot,
+        })
+        .collect::<Vec<_>>();
+
+    ResponseBuilder::new(&req)?.body(&heads)
+}
+
 #[derive(Serialize, Encode)]
 #[serde(bound = "T: EthSpec")]
 pub struct BlockResponse<T: EthSpec> {

beacon_node/rest_api/src/config.rs

@@ -1,4 +1,3 @@
-use clap::ArgMatches;
 use serde::{Deserialize, Serialize};
 use std::net::Ipv4Addr;
 
@@ -50,25 +49,3 @@ impl Default for Config {
         }
     }
 }
-
-impl Config {
-    pub fn apply_cli_args(&mut self, args: &ArgMatches) -> Result<(), &'static str> {
-        if args.is_present("no-api") {
-            self.enabled = false;
-        }
-
-        if let Some(rpc_address) = args.value_of("api-address") {
-            self.listen_address = rpc_address
-                .parse::<Ipv4Addr>()
-                .map_err(|_| "api-address is not a valid IPv4 address.")?;
-        }
-
-        if let Some(rpc_port) = args.value_of("api-port") {
-            self.port = rpc_port
-                .parse::<u16>()
-                .map_err(|_| "api-port is not a valid u16.")?;
-        }
-
-        Ok(())
-    }
-}

beacon_node/rest_api/src/router.rs

@@ -64,6 +64,9 @@ pub fn route<T: BeaconChainTypes>(
 
             // Methods for Beacon Node
             (&Method::GET, "/beacon/head") => into_boxfut(beacon::get_head::<T>(req, beacon_chain)),
+            (&Method::GET, "/beacon/heads") => {
+                into_boxfut(beacon::get_heads::<T>(req, beacon_chain))
+            }
             (&Method::GET, "/beacon/block") => {
                 into_boxfut(beacon::get_block::<T>(req, beacon_chain))
             }
@@ -154,7 +157,7 @@ pub fn route<T: BeaconChainTypes>(
     // (e.g., a response with a 404 or 500 status).
     request_result.then(move |result| match result {
         Ok(response) => {
-            debug!(local_log, "Request successful: {:?}", path);
+            debug!(local_log, "HTTP API request successful"; "path" => path);
             metrics::inc_counter(&metrics::SUCCESS_COUNT);
             metrics::stop_timer(timer);
 
@@ -163,7 +166,7 @@ pub fn route<T: BeaconChainTypes>(
         Err(e) => {
             let error_response = e.into();
 
-            debug!(local_log, "Request failure: {:?}", path);
+            debug!(local_log, "HTTP API request failure"; "path" => path);
             metrics::stop_timer(timer);
 
             Ok(error_response)

beacon_node/rest_api/tests/test.rs

@@ -3,7 +3,7 @@
 use beacon_chain::{BeaconChain, BeaconChainTypes};
 use node_test_rig::{
     environment::{Environment, EnvironmentBuilder},
-    testing_client_config, ClientGenesis, LocalBeaconNode,
+    testing_client_config, ClientConfig, ClientGenesis, LocalBeaconNode,
 };
 use remote_beacon_node::{PublishStatus, ValidatorDuty};
 use std::sync::Arc;
@@ -26,6 +26,13 @@ fn build_env() -> Environment<E> {
         .expect("environment should build")
 }
 
+fn build_node<E: EthSpec>(env: &mut Environment<E>, config: ClientConfig) -> LocalBeaconNode<E> {
+    let context = env.core_context();
+    env.runtime()
+        .block_on(LocalBeaconNode::production(context, config))
+        .expect("should block until node created")
+}
+
 /// Returns the randao reveal for the given slot (assuming the given `beacon_chain` uses
 /// deterministic keypairs).
 fn get_randao_reveal<T: BeaconChainTypes>(
@@ -64,7 +71,7 @@ fn validator_produce_attestation() {
 
     let spec = &E::default_spec();
 
-    let node = LocalBeaconNode::production(env.core_context(), testing_client_config());
+    let node = build_node(&mut env, testing_client_config());
     let remote_node = node.remote_node().expect("should produce remote node");
 
     let beacon_chain = node
@@ -160,7 +167,7 @@ fn validator_duties_bulk() {
 
     let spec = &E::default_spec();
 
-    let node = LocalBeaconNode::production(env.core_context(), testing_client_config());
+    let node = build_node(&mut env, testing_client_config());
     let remote_node = node.remote_node().expect("should produce remote node");
 
     let beacon_chain = node
@@ -197,7 +204,7 @@ fn validator_duties() {
 
     let spec = &E::default_spec();
 
-    let node = LocalBeaconNode::production(env.core_context(), testing_client_config());
+    let node = build_node(&mut env, testing_client_config());
     let remote_node = node.remote_node().expect("should produce remote node");
 
     let beacon_chain = node
@@ -321,7 +328,7 @@ fn validator_block_post() {
         genesis_time: 13_371_337,
     };
 
-    let node = LocalBeaconNode::production(env.core_context(), config);
+    let node = build_node(&mut env, config);
     let remote_node = node.remote_node().expect("should produce remote node");
 
     let beacon_chain = node
@@ -387,7 +394,7 @@ fn validator_block_get() {
 
     let spec = &E::default_spec();
 
-    let node = LocalBeaconNode::production(env.core_context(), testing_client_config());
+    let node = build_node(&mut env, testing_client_config());
     let remote_node = node.remote_node().expect("should produce remote node");
 
     let beacon_chain = node
@@ -425,7 +432,7 @@ fn validator_block_get() {
 fn beacon_state() {
     let mut env = build_env();
 
-    let node = LocalBeaconNode::production(env.core_context(), testing_client_config());
+    let node = build_node(&mut env, testing_client_config());
     let remote_node = node.remote_node().expect("should produce remote node");
 
     let (state_by_slot, root) = env
@@ -469,7 +476,7 @@ fn beacon_state() {
 fn beacon_block() {
     let mut env = build_env();
 
-    let node = LocalBeaconNode::production(env.core_context(), testing_client_config());
+    let node = build_node(&mut env, testing_client_config());
     let remote_node = node.remote_node().expect("should produce remote node");
 
     let (block_by_slot, root) = env
@@ -513,7 +520,7 @@ fn beacon_block() {
 fn genesis_time() {
     let mut env = build_env();
 
-    let node = LocalBeaconNode::production(env.core_context(), testing_client_config());
+    let node = build_node(&mut env, testing_client_config());
     let remote_node = node.remote_node().expect("should produce remote node");
 
     let genesis_time = env
@@ -537,7 +544,7 @@ fn genesis_time() {
 fn fork() {
     let mut env = build_env();
 
-    let node = LocalBeaconNode::production(env.core_context(), testing_client_config());
+    let node = build_node(&mut env, testing_client_config());
     let remote_node = node.remote_node().expect("should produce remote node");
 
     let fork = env
@@ -561,7 +568,7 @@ fn fork() {
 fn eth2_config() {
     let mut env = build_env();
 
-    let node = LocalBeaconNode::production(env.core_context(), testing_client_config());
+    let node = build_node(&mut env, testing_client_config());
     let remote_node = node.remote_node().expect("should produce remote node");
 
     let eth2_config = env
@@ -585,7 +592,7 @@ fn eth2_config() {
 fn get_version() {
     let mut env = build_env();
 
-    let node = LocalBeaconNode::production(env.core_context(), testing_client_config());
+    let node = build_node(&mut env, testing_client_config());
     let remote_node = node.remote_node().expect("should produce remote node");
 
     let version = env

beacon_node/src/cli.rs

@@ -13,9 +13,9 @@ pub fn cli_app<'a, 'b>() -> App<'a, 'b> {
             Arg::with_name("network-dir")
                 .long("network-dir")
                 .value_name("DIR")
-                .help("Data directory for network keys.")
+                .help("Data directory for network keys. Defaults to network/ inside the beacon node \
+                       dir.")
                 .takes_value(true)
-                .global(true)
         )
         .arg(
             Arg::with_name("freezer-dir")
@@ -23,26 +23,33 @@ pub fn cli_app<'a, 'b>() -> App<'a, 'b> {
                 .value_name("DIR")
                 .help("Data directory for the freezer database.")
                 .takes_value(true)
-                .global(true)
+        )
+        .arg(
+            Arg::with_name("testnet-dir")
+                .long("testnet-dir")
+                .value_name("DIR")
+                .help("Path to directory containing eth2_testnet specs. Defaults to \
+                      a hard-coded Lighthouse testnet. Only effective if there is no \
+                      existing database.")
+                .takes_value(true)
         )
         /*
          * Network parameters.
          */
         .arg(
-            Arg::with_name("port-bump")
-                .long("port-bump")
-                .short("b")
-                .value_name("INCREMENT")
-                .help("Sets all listening TCP/UDP ports to default values, but with each port increased by \
-                      INCREMENT. Useful when starting multiple nodes on a single machine. Using increments \
-                      in multiples of 10 is recommended.")
-                .takes_value(true),
+            Arg::with_name("zero-ports")
+                .long("zero-ports")
+                .short("z")
+                .help("Sets all listening TCP/UDP ports to 0, allowing the OS to choose some \
+                       arbitrary free ports.")
+                .takes_value(false),
         )
         .arg(
             Arg::with_name("listen-address")
                 .long("listen-address")
                 .value_name("ADDRESS")
-                .help("The address lighthouse will listen for UDP and TCP connections. (default 127.0.0.1).")
+                .help("The address lighthouse will listen for UDP and TCP connections.")
+                .default_value("0.0.0.0")
                 .takes_value(true)
         )
         .arg(
@@ -50,13 +57,14 @@ pub fn cli_app<'a, 'b>() -> App<'a, 'b> {
                 .long("port")
                 .value_name("PORT")
                 .help("The TCP/UDP port to listen on. The UDP port can be modified by the --discovery-port flag.")
-                .conflicts_with("port-bump")
+                .default_value("9000")
                 .takes_value(true),
         )
         .arg(
             Arg::with_name("maxpeers")
                 .long("maxpeers")
-                .help("The maximum number of peers (default 10).")
+                .help("The maximum number of peers.")
+                .default_value("10")
                 .takes_value(true),
         )
         .arg(
@@ -72,64 +80,70 @@ pub fn cli_app<'a, 'b>() -> App<'a, 'b> {
                 .long("disc-port")
                 .value_name("PORT")
                 .help("The discovery UDP port.")
-                .conflicts_with("port-bump")
+                .default_value("9000")
                 .takes_value(true),
         )
         .arg(
             Arg::with_name("discovery-address")
                 .long("discovery-address")
                 .value_name("ADDRESS")
-                .help("The IP address to broadcast to other peers on how to reach this node.")
+                .help("The IP address to broadcast to other peers on how to reach this node. \
+                       Default is determined automatically.")
                 .takes_value(true),
         )
         .arg(
             Arg::with_name("topics")
                 .long("topics")
                 .value_name("STRING")
-                .help("One or more comma-delimited gossipsub topic strings to subscribe to.")
+                .help("One or more comma-delimited gossipsub topic strings to subscribe to. Default \
+                       is determined automatically.")
                 .takes_value(true),
         )
         .arg(
             Arg::with_name("libp2p-addresses")
                 .long("libp2p-addresses")
                 .value_name("MULTIADDR")
-                .help("One or more comma-delimited multiaddrs to manually connect to a libp2p peer without an ENR.")
+                .help("One or more comma-delimited multiaddrs to manually connect to a libp2p peer \
+                       without an ENR.")
                 .takes_value(true),
         )
         .arg(
             Arg::with_name("p2p-priv-key")
                 .long("p2p-priv-key")
                 .value_name("HEX")
-                .help("A secp256k1 secret key, represented as ASCII-encoded hex bytes (with or without 0x prefix).")
+                .help("A secp256k1 secret key, represented as ASCII-encoded hex bytes (with or \
+                       without 0x prefix). Default is either loaded from disk or generated \
+                       automatically.")
                 .takes_value(true),
         )
         /* REST API related arguments */
         .arg(
-            Arg::with_name("no-api")
-                .long("no-api")
-                .help("Disable RESTful HTTP API server.")
+            Arg::with_name("http")
+                .long("http")
+                .help("Enable RESTful HTTP API server. Disabled by default.")
                 .takes_value(false),
         )
         .arg(
-            Arg::with_name("api-address")
-                .long("api-address")
+            Arg::with_name("http-address")
+                .long("http-address")
                 .value_name("ADDRESS")
                 .help("Set the listen address for the RESTful HTTP API server.")
+                .default_value("127.0.0.1")
                 .takes_value(true),
         )
         .arg(
-            Arg::with_name("api-port")
-                .long("api-port")
+            Arg::with_name("http-port")
+                .long("http-port")
                 .value_name("PORT")
                 .help("Set the listen TCP port for the RESTful HTTP API server.")
-                .conflicts_with("port-bump")
+                .default_value("5052")
                 .takes_value(true),
         )
         /* Websocket related arguments */
         .arg(
-            Arg::with_name("no-ws")
-                .long("no-ws")
-                .help("Disable websocket server.")
+            Arg::with_name("ws")
+                .long("ws")
+                .help("Enable the websocket server. Disabled by default.")
                 .takes_value(false),
         )
         .arg(
@@ -137,7 +151,7 @@ pub fn cli_app<'a, 'b>() -> App<'a, 'b> {
                 .long("ws-address")
                 .value_name("ADDRESS")
                 .help("Set the listen address for the websocket server.")
-                .conflicts_with_all(&["no-ws"])
+                .default_value("127.0.0.1")
                 .takes_value(true),
         )
         .arg(
@@ -145,16 +159,24 @@ pub fn cli_app<'a, 'b>() -> App<'a, 'b> {
                 .long("ws-port")
                 .value_name("PORT")
                 .help("Set the listen TCP port for the websocket server.")
-                .conflicts_with_all(&["no-ws", "port-bump"])
+                .default_value("5053")
                 .takes_value(true),
         )
 
         /*
          * Eth1 Integration
          */
+        .arg(
+            Arg::with_name("eth1")
+                .long("eth1")
+                .help("If present the node will connect to an eth1 node. This is required for \
+                       block production, you must use this flag if you wish to serve a validator.")
+                .takes_value(false),
+        )
         .arg(
             Arg::with_name("dummy-eth1")
                 .long("dummy-eth1")
+                .conflicts_with("eth1")
                 .help("If present, uses an eth1 backend that generates static dummy data.\
                       Identical to the method used at the 2019 Canada interop.")
         )
@@ -166,32 +188,6 @@ pub fn cli_app<'a, 'b>() -> App<'a, 'b> {
                 .takes_value(true)
                 .default_value("http://localhost:8545")
         )
-        .arg(
-            Arg::with_name("eth1-follow")
-                .long("eth1-follow")
-                .value_name("BLOCK_COUNT")
-                .help("Specifies how many blocks we should cache behind the eth1 head. A larger number means a smaller cache.")
-                .takes_value(true)
-                // TODO: set this higher once we're not using testnets all the time.
-                .default_value("0")
-        )
-        .arg(
-            Arg::with_name("deposit-contract")
-                .long("deposit-contract")
-                .short("e")
-                .value_name("DEPOSIT-CONTRACT")
-                .help("Specifies the deposit contract address on the Eth1 chain.")
-                .takes_value(true)
-        )
-        .arg(
-            Arg::with_name("deposit-contract-deploy")
-                .long("deposit-contract-deploy")
-                .value_name("BLOCK_NUMBER")
-                .help("Specifies the block number that the deposit contract was deployed at.")
-                .takes_value(true)
-                // TODO: set this higher once we're not using testnets all the time.
-                .default_value("0")
-        )
         /*
          * The "testnet" sub-command.
          *
@@ -199,21 +195,6 @@ pub fn cli_app<'a, 'b>() -> App<'a, 'b> {
          */
         .subcommand(SubCommand::with_name("testnet")
             .about("Create a new Lighthouse datadir using a testnet strategy.")
-            .arg(
-                Arg::with_name("eth2-config")
-                    .long("eth2-config")
-                    .value_name("TOML_FILE")
-                    .help("A existing eth2_spec TOML file (e.g., eth2_spec.toml).")
-                    .takes_value(true)
-                    .conflicts_with("spec")
-            )
-            .arg(
-                Arg::with_name("client-config")
-                    .long("client-config")
-                    .value_name("TOML_FILE")
-                    .help("An existing beacon_node TOML file (e.g., beacon_node.toml).")
-                    .takes_value(true)
-            )
             .arg(
                 Arg::with_name("random-datadir")
                     .long("random-datadir")
@@ -221,13 +202,6 @@ pub fn cli_app<'a, 'b>() -> App<'a, 'b> {
                     .help("If present, append a random string to the datadir path. Useful for fast development \
                           iteration.")
             )
-            .arg(
-                Arg::with_name("random-propagation")
-                    .long("random-propagation")
-                    .value_name("INTEGER")
-                    .takes_value(true)
-                    .help("Specifies (as a percentage) the likelihood of propagating blocks and attestations. This should only be used for testing networking elements. The value must like in the range 1-100.")
-            )
             .arg(
                 Arg::with_name("force")
                     .long("force")
@@ -236,33 +210,22 @@ pub fn cli_app<'a, 'b>() -> App<'a, 'b> {
                            backup directory.")
                     .conflicts_with("random-datadir")
             )
+            .arg(
+                Arg::with_name("random-propagation")
+                    .long("random-propagation")
+                    .value_name("INTEGER")
+                    .takes_value(true)
+                    .help("Specifies (as a percentage) the likelihood of propagating blocks and \
+                           attestations. This should only be used for testing networking elements. The \
+                           value must like in the range 1-100. Default is 100.")
+            )
             .arg(
                 Arg::with_name("slot-time")
                     .long("slot-time")
                     .short("t")
                     .value_name("MILLISECONDS")
-                    .help("Defines the slot time when creating a new testnet.")
-            )
-            /*
-             * `boostrap`
-             *
-             * Start a new node by downloading genesis and network info from another node via the
-             * HTTP API.
-             */
-            .subcommand(SubCommand::with_name("bootstrap")
-                .about("Connects to the given HTTP server, downloads a genesis state and attempts to peer with it.")
-                .arg(Arg::with_name("server")
-                    .value_name("HTTP_SERVER")
-                    .required(true)
-                    .default_value("http://localhost:5052")
-                    .help("A HTTP server, with a http:// prefix"))
-                .arg(Arg::with_name("libp2p-port")
-                    .short("p")
-                    .long("port")
-                    .value_name("TCP_PORT")
-                    .help("A libp2p listen port used to peer with the bootstrap server. This flag is useful \
-                           when port-fowarding is used: you may connect using a different port than \
-                           the one the server is immediately listening on."))
+                    .help("Defines the slot time when creating a new testnet. The default is \
+                           specified by the spec.")
             )
             /*
              * `recent`
@@ -326,7 +289,8 @@ pub fn cli_app<'a, 'b>() -> App<'a, 'b> {
              * Connect to the Prysmatic Labs testnet.
              */
             .subcommand(SubCommand::with_name("prysm")
-                .about("Connect to the Prysmatic Labs testnet on Goerli.")
+                .about("Connect to the Prysmatic Labs testnet on Goerli. Not guaranteed to be \
+                    up-to-date or functioning.")
             )
         )
 }

beacon_node/src/config.rs

@@ -1,18 +1,21 @@
 use clap::ArgMatches;
 use client::{ClientConfig, ClientGenesis, Eth2Config};
 use eth2_config::{read_from_file, write_to_file};
+use eth2_libp2p::{Enr, Multiaddr};
+use eth2_testnet_config::Eth2TestnetConfig;
 use genesis::recent_genesis_time;
-use lighthouse_bootstrap::Bootstrapper;
 use rand::{distributions::Alphanumeric, Rng};
-use slog::{crit, info, warn, Logger};
+use slog::{crit, info, Logger};
+use ssz::Encode;
 use std::fs;
 use std::net::Ipv4Addr;
-use std::path::{Path, PathBuf};
-use types::{Address, Epoch, Fork};
+use std::path::PathBuf;
+use types::{Epoch, EthSpec, Fork};
 
-pub const DEFAULT_DATA_DIR: &str = ".lighthouse";
 pub const CLIENT_CONFIG_FILENAME: &str = "beacon-node.toml";
 pub const ETH2_CONFIG_FILENAME: &str = "eth2-spec.toml";
+pub const BEACON_NODE_DIR: &str = "beacon";
+pub const NETWORK_DIR: &str = "network";
 
 type Result<T> = std::result::Result<T, String>;
 type Config = (ClientConfig, Eth2Config, Logger);
@@ -24,89 +27,391 @@ type Config = (ClientConfig, Eth2Config, Logger);
 /// The output of this function depends primarily upon the given `cli_args`, however it's behaviour
 /// may be influenced by other external services like the contents of the file system or the
 /// response of some remote server.
-pub fn get_configs(cli_args: &ArgMatches, core_log: Logger) -> Result<Config> {
+pub fn get_configs<E: EthSpec>(
+    cli_args: &ArgMatches,
+    mut eth2_config: Eth2Config,
+    core_log: Logger,
+) -> Result<Config> {
     let log = core_log.clone();
 
-    let mut builder = ConfigBuilder::new(cli_args, core_log)?;
+    let mut client_config = ClientConfig::default();
 
+    // Read the `--datadir` flag.
+    //
+    // If it's not present, try and find the home directory (`~`) and push the default data
+    // directory onto it.
+    client_config.data_dir = cli_args
+        .value_of("datadir")
+        .map(PathBuf::from)
+        .or_else(|| dirs::home_dir().map(|home| home.join(".lighthouse").join(BEACON_NODE_DIR)))
+        .unwrap_or_else(|| PathBuf::from("."));
+
+    // Load the client config, if it exists .
+    let path = client_config.data_dir.join(CLIENT_CONFIG_FILENAME);
+    if path.exists() {
+        client_config = read_from_file(path.clone())
+            .map_err(|e| format!("Unable to parse {:?} file: {:?}", path, e))?
+            .ok_or_else(|| format!("{:?} file does not exist", path))?;
+    }
+
+    // Load the eth2 config, if it exists .
+    let path = client_config.data_dir.join(ETH2_CONFIG_FILENAME);
+    if path.exists() {
+        eth2_config = read_from_file(path.clone())
+            .map_err(|e| format!("Unable to parse {:?} file: {:?}", path, e))?
+            .ok_or_else(|| format!("{:?} file does not exist", path))?;
+    }
+
+    // Read the `--testnet-dir` flag.
+    if let Some(val) = cli_args.value_of("testnet-dir") {
+        client_config.testnet_dir = Some(PathBuf::from(val));
+    }
+
+    /*
+     * Networking
+     */
+    // If a network dir has been specified, override the `datadir` definition.
+    if let Some(dir) = cli_args.value_of("network-dir") {
+        client_config.network.network_dir = PathBuf::from(dir);
+    } else {
+        client_config.network.network_dir = client_config.data_dir.join(NETWORK_DIR);
+    };
+
+    if let Some(listen_address_str) = cli_args.value_of("listen-address") {
+        let listen_address = listen_address_str
+            .parse()
+            .map_err(|_| format!("Invalid listen address: {:?}", listen_address_str))?;
+        client_config.network.listen_address = listen_address;
+        client_config.network.discovery_address = listen_address;
+    }
+
+    if let Some(max_peers_str) = cli_args.value_of("maxpeers") {
+        client_config.network.max_peers = max_peers_str
+            .parse::<usize>()
+            .map_err(|_| format!("Invalid number of max peers: {}", max_peers_str))?;
+    }
+
+    if let Some(port_str) = cli_args.value_of("port") {
+        let port = port_str
+            .parse::<u16>()
+            .map_err(|_| format!("Invalid port: {}", port_str))?;
+        client_config.network.libp2p_port = port;
+        client_config.network.discovery_port = port;
+    }
+
+    if let Some(boot_enr_str) = cli_args.value_of("boot-nodes") {
+        client_config.network.boot_nodes = boot_enr_str
+            .split(',')
+            .map(|enr| enr.parse().map_err(|_| format!("Invalid ENR: {}", enr)))
+            .collect::<Result<Vec<Enr>>>()?;
+    }
+
+    if let Some(libp2p_addresses_str) = cli_args.value_of("libp2p-addresses") {
+        client_config.network.libp2p_nodes = libp2p_addresses_str
+            .split(',')
+            .map(|multiaddr| {
+                multiaddr
+                    .parse()
+                    .map_err(|_| format!("Invalid Multiaddr: {}", multiaddr))
+            })
+            .collect::<Result<Vec<Multiaddr>>>()?;
+    }
+
+    if let Some(topics_str) = cli_args.value_of("topics") {
+        client_config.network.topics = topics_str.split(',').map(|s| s.into()).collect();
+    }
+
+    if let Some(discovery_address_str) = cli_args.value_of("discovery-address") {
+        client_config.network.discovery_address = discovery_address_str
+            .parse()
+            .map_err(|_| format!("Invalid discovery address: {:?}", discovery_address_str))?
+    }
+
+    if let Some(disc_port_str) = cli_args.value_of("disc-port") {
+        client_config.network.discovery_port = disc_port_str
+            .parse::<u16>()
+            .map_err(|_| format!("Invalid discovery port: {}", disc_port_str))?;
+    }
+
+    if let Some(p2p_priv_key) = cli_args.value_of("p2p-priv-key") {
+        client_config.network.secret_key_hex = Some(p2p_priv_key.to_string());
+    }
+
+    /*
+     * Http server
+     */
+
+    if cli_args.is_present("http") {
+        client_config.rest_api.enabled = true;
+    }
+
+    if let Some(address) = cli_args.value_of("http-address") {
+        client_config.rest_api.listen_address = address
+            .parse::<Ipv4Addr>()
+            .map_err(|_| "http-address is not a valid IPv4 address.")?;
+    }
+
+    if let Some(port) = cli_args.value_of("http-port") {
+        client_config.rest_api.port = port
+            .parse::<u16>()
+            .map_err(|_| "http-port is not a valid u16.")?;
+    }
+
+    /*
+     * Websocket server
+     */
+
+    if cli_args.is_present("ws") {
+        client_config.websocket_server.enabled = true;
+    }
+
+    if let Some(address) = cli_args.value_of("ws-address") {
+        client_config.websocket_server.listen_address = address
+            .parse::<Ipv4Addr>()
+            .map_err(|_| "ws-address is not a valid IPv4 address.")?;
+    }
+
+    if let Some(port) = cli_args.value_of("ws-port") {
+        client_config.websocket_server.port = port
+            .parse::<u16>()
+            .map_err(|_| "ws-port is not a valid u16.")?;
+    }
+
+    /*
+     * Eth1
+     */
+
+    // When present, use an eth1 backend that generates deterministic junk.
+    //
+    // Useful for running testnets without the overhead of a deposit contract.
     if cli_args.is_present("dummy-eth1") {
-        builder.client_config.dummy_eth1_backend = true;
+        client_config.dummy_eth1_backend = true;
     }
 
+    // When present, attempt to sync to an eth1 node.
+    //
+    // Required for block production.
+    if cli_args.is_present("eth1") {
+        client_config.sync_eth1_chain = true;
+    }
+
+    // Defines the URL to reach the eth1 node.
     if let Some(val) = cli_args.value_of("eth1-endpoint") {
-        builder.set_eth1_endpoint(val)
-    }
-
-    if let Some(val) = cli_args.value_of("deposit-contract") {
-        builder.set_deposit_contract(
-            val.parse::<Address>()
-                .map_err(|e| format!("Unable to parse deposit-contract address: {:?}", e))?,
-        )
-    }
-
-    if let Some(val) = cli_args.value_of("deposit-contract-deploy") {
-        builder.set_deposit_contract_deploy_block(
-            val.parse::<u64>()
-                .map_err(|e| format!("Unable to parse deposit-contract-deploy: {:?}", e))?,
-        )
-    }
-
-    if let Some(val) = cli_args.value_of("eth1-follow") {
-        builder.set_eth1_follow(
-            val.parse::<u64>()
-                .map_err(|e| format!("Unable to parse follow distance: {:?}", e))?,
-        )
+        client_config.eth1.endpoint = val.to_string();
     }
 
     match cli_args.subcommand() {
         ("testnet", Some(sub_cmd_args)) => {
-            process_testnet_subcommand(&mut builder, sub_cmd_args, &log)?
+            process_testnet_subcommand(&mut client_config, &mut eth2_config, sub_cmd_args)?
         }
         // No sub-command assumes a resume operation.
         _ => {
-            info!(
-                log,
-                "Resuming from existing datadir";
-                "path" => format!("{:?}", builder.client_config.data_dir)
-            );
-
             // If no primary subcommand was given, start the beacon chain from an existing
             // database.
-            builder.set_genesis(ClientGenesis::Resume);
+            client_config.genesis = ClientGenesis::Resume;
 
             // Whilst there is no large testnet or mainnet force the user to specify how they want
             // to start a new chain (e.g., from a genesis YAML file, another node, etc).
-            if !builder.client_config.data_dir.exists() {
-                return Err(
-                    "No datadir found. To start a new beacon chain, see `testnet --help`. \
-                     Use `--datadir` to specify a different directory"
-                        .into(),
+            if !client_config.data_dir.exists() {
+                info!(
+                    log,
+                    "Starting from an empty database";
+                    "data_dir" => format!("{:?}", client_config.data_dir)
                 );
+                init_new_client::<E>(&mut client_config, &mut eth2_config)?
+            } else {
+                info!(
+                    log,
+                    "Resuming from existing datadir";
+                    "data_dir" => format!("{:?}", client_config.data_dir)
+                );
+                // If the `testnet` command was not provided, attempt to load an existing datadir and
+                // continue with an existing chain.
+                load_from_datadir(&mut client_config)?
             }
-
-            // If the `testnet` command was not provided, attempt to load an existing datadir and
-            // continue with an existing chain.
-            builder.load_from_datadir()?;
         }
     };
 
-    builder.build(cli_args)
+    if let Some(freezer_dir) = cli_args.value_of("freezer-dir") {
+        client_config.freezer_db_path = Some(PathBuf::from(freezer_dir));
+    }
+
+    if eth2_config.spec_constants != client_config.spec_constants {
+        crit!(log, "Specification constants do not match.";
+              "client_config" => client_config.spec_constants.to_string(),
+              "eth2_config" => eth2_config.spec_constants.to_string()
+        );
+        return Err("Specification constant mismatch".into());
+    }
+
+    /*
+     * Zero-ports
+     *
+     * Replaces previously set flags.
+     */
+    if cli_args.is_present("zero-ports") {
+        client_config.network.libp2p_port = 0;
+        client_config.network.discovery_port = 0;
+        client_config.rest_api.port = 0;
+        client_config.websocket_server.port = 0;
+    }
+
+    Ok((client_config, eth2_config, log))
+}
+
+/// Load from an existing database.
+fn load_from_datadir(client_config: &mut ClientConfig) -> Result<()> {
+    // Check to ensure the datadir exists.
+    //
+    // For now we return an error. In the future we may decide to boot a default (e.g.,
+    // public testnet or mainnet).
+    if !client_config.get_data_dir().map_or(false, |d| d.exists()) {
+        return Err(
+            "No datadir found. Either create a new testnet or specify a different `--datadir`."
+                .into(),
+        );
+    }
+
+    // If there is a path to a database in the config, ensure it exists.
+    if !client_config
+        .get_db_path()
+        .map_or(false, |path| path.exists())
+    {
+        return Err(
+            "No database found in datadir. Use 'testnet -f' to overwrite the existing \
+             datadir, or specify a different `--datadir`."
+                .into(),
+        );
+    }
+
+    client_config.genesis = ClientGenesis::Resume;
+
+    Ok(())
+}
+
+/// Create a new client with the default configuration.
+fn init_new_client<E: EthSpec>(
+    client_config: &mut ClientConfig,
+    eth2_config: &mut Eth2Config,
+) -> Result<()> {
+    let eth2_testnet_config: Eth2TestnetConfig<E> =
+        if let Some(testnet_dir) = &client_config.testnet_dir {
+            Eth2TestnetConfig::load(testnet_dir.clone())
+                .map_err(|e| format!("Unable to open testnet dir at {:?}: {}", testnet_dir, e))?
+        } else {
+            Eth2TestnetConfig::hard_coded()
+                .map_err(|e| format!("Unable to load hard-coded testnet dir: {}", e))?
+        };
+
+    eth2_config.spec = eth2_testnet_config
+        .yaml_config
+        .as_ref()
+        .ok_or_else(|| "The testnet directory must contain a spec config".to_string())?
+        .apply_to_chain_spec::<E>(&eth2_config.spec)
+        .ok_or_else(|| {
+            format!(
+                "The loaded config is not compatible with the {} spec",
+                &eth2_config.spec_constants
+            )
+        })?;
+
+    let spec = &mut eth2_config.spec;
+
+    client_config.eth1.deposit_contract_address =
+        format!("{:?}", eth2_testnet_config.deposit_contract_address()?);
+    client_config.eth1.deposit_contract_deploy_block =
+        eth2_testnet_config.deposit_contract_deploy_block;
+
+    client_config.eth1.follow_distance = spec.eth1_follow_distance / 2;
+    client_config.dummy_eth1_backend = false;
+    client_config.eth1.lowest_cached_block_number = client_config
+        .eth1
+        .deposit_contract_deploy_block
+        .saturating_sub(client_config.eth1.follow_distance * 2);
+
+    if let Some(boot_nodes) = eth2_testnet_config.boot_enr {
+        client_config
+            .network
+            .boot_nodes
+            .append(&mut boot_nodes.clone())
+    }
+
+    if let Some(genesis_state) = eth2_testnet_config.genesis_state {
+        // Note: re-serializing the genesis state is not so efficient, however it avoids adding
+        // trait bounds to the `ClientGenesis` enum. This would have significant flow-on
+        // effects.
+        client_config.genesis = ClientGenesis::SszBytes {
+            genesis_state_bytes: genesis_state.as_ssz_bytes(),
+        };
+    } else {
+        client_config.genesis = ClientGenesis::DepositContract;
+    }
+
+    create_new_datadir(&client_config, &eth2_config)?;
+
+    Ok(())
+}
+
+/// Writes the configs in `self` to `self.data_dir`.
+///
+/// Returns an error if `self.data_dir` already exists.
+pub fn create_new_datadir(client_config: &ClientConfig, eth2_config: &Eth2Config) -> Result<()> {
+    if client_config.data_dir.exists() {
+        return Err(format!(
+            "Data dir already exists at {:?}",
+            client_config.data_dir
+        ))?;
+    }
+
+    // Create `datadir` and any non-existing parent directories.
+    fs::create_dir_all(&client_config.data_dir)
+        .map_err(|e| format!("Failed to create data dir: {}", e))?;
+
+    macro_rules! write_to_file {
+        ($file: ident, $variable: ident) => {
+            let file = client_config.data_dir.join($file);
+            if file.exists() {
+                return Err(format!("Datadir is not clean, {} exists.", $file));
+            } else {
+                // Write the onfig to a TOML file in the datadir.
+                write_to_file(client_config.data_dir.join($file), $variable)
+                    .map_err(|e| format!("Unable to write {} file: {:?}", $file, e))?;
+            }
+        };
+    }
+
+    write_to_file!(CLIENT_CONFIG_FILENAME, client_config);
+    write_to_file!(ETH2_CONFIG_FILENAME, eth2_config);
+
+    Ok(())
 }
 
 /// Process the `testnet` CLI subcommand arguments, updating the `builder`.
 fn process_testnet_subcommand(
-    builder: &mut ConfigBuilder,
+    client_config: &mut ClientConfig,
+    eth2_config: &mut Eth2Config,
     cli_args: &ArgMatches,
-    log: &Logger,
 ) -> Result<()> {
+    // Specifies that a random datadir should be used.
     if cli_args.is_present("random-datadir") {
-        builder.set_random_datadir()?;
+        client_config
+            .data_dir
+            .push(format!("random_{}", random_string(6)));
+        client_config.network.network_dir = client_config.data_dir.join("network");
     }
 
+    // Deletes the existing datadir.
     if cli_args.is_present("force") {
-        builder.clean_datadir()?;
+        if client_config.data_dir.exists() {
+            fs::remove_dir_all(&client_config.data_dir)
+                .map_err(|e| format!("Unable to delete existing datadir: {:?}", e))?;
+        }
     }
 
+    // Define a percentage of messages that should be propogated, useful for simulating bad network
+    // conditions.
+    //
+    // WARNING: setting this to anything less than 100 will cause bad behaviour.
     if let Some(propagation_percentage_string) = cli_args.value_of("random-propagation") {
         let percentage = propagation_percentage_string
             .parse::<u8>()
@@ -114,72 +419,20 @@ fn process_testnet_subcommand(
         if percentage > 100 {
             return Err("Propagation percentage greater than 100".to_string());
         }
-        builder.client_config.network.propagation_percentage = Some(percentage);
-    }
-
-    let is_bootstrap = cli_args.subcommand_name() == Some("bootstrap");
-
-    if let Some(path_string) = cli_args.value_of("eth2-config") {
-        if is_bootstrap {
-            return Err("Cannot supply --eth2-config when using bootstrap".to_string());
-        }
-
-        let path = path_string
-            .parse::<PathBuf>()
-            .map_err(|e| format!("Unable to parse eth2-config path: {:?}", e))?;
-        builder.load_eth2_config(path)?;
-    } else {
-        builder.update_spec_from_subcommand(&cli_args)?;
+        client_config.network.propagation_percentage = Some(percentage);
     }
 
+    // Modify the `SECONDS_PER_SLOT` "constant".
     if let Some(slot_time) = cli_args.value_of("slot-time") {
-        if is_bootstrap {
-            return Err("Cannot supply --slot-time flag whilst using bootstrap.".into());
-        }
-
         let slot_time = slot_time
             .parse::<u64>()
             .map_err(|e| format!("Unable to parse slot-time: {:?}", e))?;
 
-        builder.set_slot_time(slot_time);
+        eth2_config.spec.milliseconds_per_slot = slot_time;
     }
 
-    if let Some(path_string) = cli_args.value_of("client-config") {
-        let path = path_string
-            .parse::<PathBuf>()
-            .map_err(|e| format!("Unable to parse client config path: {:?}", e))?;
-        builder.load_client_config(path)?;
-    }
-
-    info!(
-        log,
-        "Creating new datadir";
-        "path" => format!("{:?}", builder.client_config.data_dir)
-    );
-
-    // When using the testnet command we listen on all addresses.
-    builder.set_listen_addresses("0.0.0.0".into())?;
-    warn!(log, "All services listening on 0.0.0.0");
-
     // Start matching on the second subcommand (e.g., `testnet bootstrap ...`).
     match cli_args.subcommand() {
-        ("bootstrap", Some(cli_args)) => {
-            let server = cli_args
-                .value_of("server")
-                .ok_or_else(|| "No bootstrap server specified")?;
-            let port: Option<u16> = cli_args
-                .value_of("libp2p-port")
-                .and_then(|s| s.parse::<u16>().ok());
-
-            builder.import_bootstrap_libp2p_address(server, port)?;
-            builder.import_bootstrap_enr_address(server)?;
-            builder.import_bootstrap_eth2_config(server)?;
-
-            builder.set_genesis(ClientGenesis::RemoteNode {
-                server: server.to_string(),
-                port,
-            })
-        }
         ("recent", Some(cli_args)) => {
             let validator_count = cli_args
                 .value_of("validator_count")
@@ -193,12 +446,12 @@ fn process_testnet_subcommand(
                 .parse::<u64>()
                 .map_err(|e| format!("Unable to parse minutes: {:?}", e))?;
 
-            builder.client_config.dummy_eth1_backend = true;
+            client_config.dummy_eth1_backend = true;
 
-            builder.set_genesis(ClientGenesis::Interop {
+            client_config.genesis = ClientGenesis::Interop {
                 validator_count,
                 genesis_time: recent_genesis_time(minutes),
-            })
+            };
         }
         ("quick", Some(cli_args)) => {
             let validator_count = cli_args
@@ -213,12 +466,12 @@ fn process_testnet_subcommand(
                 .parse::<u64>()
                 .map_err(|e| format!("Unable to parse genesis time: {:?}", e))?;
 
-            builder.client_config.dummy_eth1_backend = true;
+            client_config.dummy_eth1_backend = true;
 
-            builder.set_genesis(ClientGenesis::Interop {
+            client_config.genesis = ClientGenesis::Interop {
                 validator_count,
                 genesis_time,
-            })
+            };
         }
         ("file", Some(cli_args)) => {
             let path = cli_args
@@ -236,11 +489,10 @@ fn process_testnet_subcommand(
                 other => return Err(format!("Unknown genesis file format: {}", other)),
             };
 
-            builder.set_genesis(start_method)
+            client_config.genesis = start_method;
         }
         ("prysm", Some(_)) => {
-            let mut spec = &mut builder.eth2_config.spec;
-            let mut client_config = &mut builder.client_config;
+            let mut spec = &mut eth2_config.spec;
 
             spec.min_deposit_amount = 100;
             spec.max_effective_balance = 3_200_000_000;
@@ -259,7 +511,7 @@ fn process_testnet_subcommand(
             client_config.eth1.follow_distance = 16;
             client_config.dummy_eth1_backend = false;
 
-            builder.set_genesis(ClientGenesis::DepositContract)
+            client_config.genesis = ClientGenesis::DepositContract;
         }
         (cmd, Some(_)) => {
             return Err(format!(
@@ -270,369 +522,11 @@ fn process_testnet_subcommand(
         _ => return Err("No testnet method specified. See 'testnet --help'.".into()),
     };
 
-    builder.write_configs_to_new_datadir()?;
+    create_new_datadir(&client_config, &eth2_config)?;
 
     Ok(())
 }
 
-/// Allows for building a set of configurations based upon `clap` arguments.
-struct ConfigBuilder {
-    log: Logger,
-    pub eth2_config: Eth2Config,
-    pub client_config: ClientConfig,
-}
-
-impl ConfigBuilder {
-    /// Create a new builder with default settings.
-    pub fn new(cli_args: &ArgMatches, log: Logger) -> Result<Self> {
-        // Read the `--datadir` flag.
-        //
-        // If it's not present, try and find the home directory (`~`) and push the default data
-        // directory onto it.
-        let data_dir: PathBuf = cli_args
-            .value_of("datadir")
-            .map(PathBuf::from)
-            .or_else(|| {
-                dirs::home_dir().map(|mut home| {
-                    home.push(DEFAULT_DATA_DIR);
-                    home
-                })
-            })
-            .ok_or_else(|| "Unable to find a home directory for the datadir".to_string())?;
-
-        let mut client_config = ClientConfig::default();
-        client_config.data_dir = data_dir;
-
-        Ok(Self {
-            log,
-            eth2_config: Eth2Config::minimal(),
-            client_config,
-        })
-    }
-
-    /// Clears any configuration files that would interfere with writing new configs.
-    ///
-    /// Moves the following files in `data_dir` into a backup directory:
-    ///
-    /// - Client config
-    /// - Eth2 config
-    /// - All database directories
-    pub fn clean_datadir(&mut self) -> Result<()> {
-        let backup_dir = {
-            let mut s = String::from("backup_");
-            s.push_str(&random_string(6));
-            self.client_config.data_dir.join(s)
-        };
-
-        fs::create_dir_all(&backup_dir)
-            .map_err(|e| format!("Unable to create config backup dir: {:?}", e))?;
-
-        let move_to_backup_dir = |path: &Path| -> Result<()> {
-            let file_name = path
-                .file_name()
-                .ok_or_else(|| "Invalid path found during datadir clean (no filename).")?;
-
-            let mut new = path.to_path_buf();
-            new.pop();
-            new.push(backup_dir.clone());
-            new.push(file_name);
-
-            let _ = fs::rename(path, new);
-
-            Ok(())
-        };
-
-        move_to_backup_dir(&self.client_config.data_dir.join(CLIENT_CONFIG_FILENAME))?;
-        move_to_backup_dir(&self.client_config.data_dir.join(ETH2_CONFIG_FILENAME))?;
-        move_to_backup_dir(&self.client_config.create_db_path()?)?;
-        move_to_backup_dir(&self.client_config.create_freezer_db_path()?)?;
-
-        Ok(())
-    }
-
-    pub fn set_eth1_endpoint(&mut self, endpoint: &str) {
-        self.client_config.eth1.endpoint = endpoint.to_string();
-    }
-
-    pub fn set_deposit_contract(&mut self, deposit_contract: Address) {
-        self.client_config.eth1.deposit_contract_address = format!("{:?}", deposit_contract);
-    }
-
-    pub fn set_deposit_contract_deploy_block(&mut self, eth1_block_number: u64) {
-        self.client_config.eth1.deposit_contract_deploy_block = eth1_block_number;
-    }
-
-    pub fn set_eth1_follow(&mut self, distance: u64) {
-        self.client_config.eth1.follow_distance = distance;
-    }
-
-    pub fn set_genesis(&mut self, method: ClientGenesis) {
-        self.client_config.genesis = method;
-    }
-
-    /// Import the libp2p address for `server` into the list of libp2p nodes to connect with.
-    ///
-    /// If `port` is `Some`, it is used as the port for the `Multiaddr`. If `port` is `None`,
-    /// attempts to connect to the `server` via HTTP and retrieve it's libp2p listen port.
-    pub fn import_bootstrap_libp2p_address(
-        &mut self,
-        server: &str,
-        port: Option<u16>,
-    ) -> Result<()> {
-        let bootstrapper = Bootstrapper::connect(server.to_string(), &self.log)?;
-
-        if let Some(server_multiaddr) = bootstrapper.best_effort_multiaddr(port) {
-            info!(
-                self.log,
-                "Estimated bootstrapper libp2p address";
-                "multiaddr" => format!("{:?}", server_multiaddr)
-            );
-
-            self.client_config
-                .network
-                .libp2p_nodes
-                .push(server_multiaddr);
-        } else {
-            warn!(
-                self.log,
-                "Unable to estimate a bootstrapper libp2p address, this node may not find any peers."
-            );
-        };
-
-        Ok(())
-    }
-
-    /// Import the enr address for `server` into the list of initial enrs (boot nodes).
-    pub fn import_bootstrap_enr_address(&mut self, server: &str) -> Result<()> {
-        let bootstrapper = Bootstrapper::connect(server.to_string(), &self.log)?;
-
-        if let Ok(enr) = bootstrapper.enr() {
-            info!(
-                self.log,
-                "Loaded bootstrapper libp2p address";
-                "enr" => format!("{:?}", enr)
-            );
-
-            self.client_config.network.boot_nodes.push(enr);
-        } else {
-            warn!(
-                self.log,
-                "Unable to estimate a bootstrapper enr address, this node may not find any peers."
-            );
-        };
-
-        Ok(())
-    }
-
-    /// Set the config data_dir to be an random directory.
-    ///
-    /// Useful for easily spinning up ephemeral testnets.
-    pub fn set_random_datadir(&mut self) -> Result<()> {
-        self.client_config
-            .data_dir
-            .push(format!("random_{}", random_string(6)));
-        self.client_config.network.network_dir = self.client_config.data_dir.join("network");
-
-        Ok(())
-    }
-
-    /// Imports an `Eth2Config` from `server`, returning an error if this fails.
-    pub fn import_bootstrap_eth2_config(&mut self, server: &str) -> Result<()> {
-        let bootstrapper = Bootstrapper::connect(server.to_string(), &self.log)?;
-
-        self.update_eth2_config(bootstrapper.eth2_config()?);
-
-        Ok(())
-    }
-
-    fn update_eth2_config(&mut self, eth2_config: Eth2Config) {
-        self.eth2_config = eth2_config;
-    }
-
-    fn set_slot_time(&mut self, milliseconds_per_slot: u64) {
-        self.eth2_config.spec.milliseconds_per_slot = milliseconds_per_slot;
-    }
-
-    /// Reads the subcommand and tries to update `self.eth2_config` based up on the `--spec` flag.
-    ///
-    /// Returns an error if the `--spec` flag is not present in the given `cli_args`.
-    pub fn update_spec_from_subcommand(&mut self, cli_args: &ArgMatches) -> Result<()> {
-        // Re-initialise the `Eth2Config`.
-        //
-        // If a CLI parameter is set, overwrite any config file present.
-        // If a parameter is not set, use either the config file present or default to minimal.
-        let eth2_config = match cli_args.value_of("spec") {
-            Some("mainnet") => Eth2Config::mainnet(),
-            Some("minimal") => Eth2Config::minimal(),
-            Some("interop") => Eth2Config::interop(),
-            _ => return Err("Unable to determine specification type.".into()),
-        };
-
-        self.client_config.spec_constants = cli_args
-            .value_of("spec")
-            .expect("Guarded by prior match statement")
-            .to_string();
-        self.eth2_config = eth2_config;
-
-        Ok(())
-    }
-
-    /// Writes the configs in `self` to `self.data_dir`.
-    ///
-    /// Returns an error if `self.data_dir` already exists.
-    pub fn write_configs_to_new_datadir(&mut self) -> Result<()> {
-        let db_exists = self
-            .client_config
-            .get_db_path()
-            .map(|d| d.exists())
-            .unwrap_or_else(|| false);
-
-        // Do not permit creating a new config when the datadir exists.
-        if db_exists {
-            return Err("Database already exists. See `-f` or `-r` in `testnet --help`".into());
-        }
-
-        // Create `datadir` and any non-existing parent directories.
-        fs::create_dir_all(&self.client_config.data_dir).map_err(|e| {
-            crit!(self.log, "Failed to initialize data dir"; "error" => format!("{}", e));
-            format!("{}", e)
-        })?;
-
-        let client_config_file = self.client_config.data_dir.join(CLIENT_CONFIG_FILENAME);
-        if client_config_file.exists() {
-            return Err(format!(
-                "Datadir is not clean, {} exists. See `-f` in `testnet --help`.",
-                CLIENT_CONFIG_FILENAME
-            ));
-        } else {
-            // Write the onfig to a TOML file in the datadir.
-            write_to_file(
-                self.client_config.data_dir.join(CLIENT_CONFIG_FILENAME),
-                &self.client_config,
-            )
-            .map_err(|e| format!("Unable to write {} file: {:?}", CLIENT_CONFIG_FILENAME, e))?;
-        }
-
-        let eth2_config_file = self.client_config.data_dir.join(ETH2_CONFIG_FILENAME);
-        if eth2_config_file.exists() {
-            return Err(format!(
-                "Datadir is not clean, {} exists. See `-f` in `testnet --help`.",
-                ETH2_CONFIG_FILENAME
-            ));
-        } else {
-            // Write the config to a TOML file in the datadir.
-            write_to_file(
-                self.client_config.data_dir.join(ETH2_CONFIG_FILENAME),
-                &self.eth2_config,
-            )
-            .map_err(|e| format!("Unable to write {} file: {:?}", ETH2_CONFIG_FILENAME, e))?;
-        }
-
-        Ok(())
-    }
-
-    /// Attempts to load the client and eth2 configs from `self.data_dir`.
-    ///
-    /// Returns an error if any files are not found or are invalid.
-    pub fn load_from_datadir(&mut self) -> Result<()> {
-        // Check to ensure the datadir exists.
-        //
-        // For now we return an error. In the future we may decide to boot a default (e.g.,
-        // public testnet or mainnet).
-        if !self
-            .client_config
-            .get_data_dir()
-            .map_or(false, |d| d.exists())
-        {
-            return Err(
-                "No datadir found. Either create a new testnet or specify a different `--datadir`."
-                    .into(),
-            );
-        }
-
-        // If there is a path to a database in the config, ensure it exists.
-        if !self
-            .client_config
-            .get_db_path()
-            .map_or(false, |path| path.exists())
-        {
-            return Err(
-                "No database found in datadir. Use 'testnet -f' to overwrite the existing \
-                 datadir, or specify a different `--datadir`."
-                    .into(),
-            );
-        }
-
-        self.load_eth2_config(self.client_config.data_dir.join(ETH2_CONFIG_FILENAME))?;
-        self.load_client_config(self.client_config.data_dir.join(CLIENT_CONFIG_FILENAME))?;
-
-        Ok(())
-    }
-
-    /// Attempts to load the client config from `path`.
-    ///
-    /// Returns an error if any files are not found or are invalid.
-    pub fn load_client_config(&mut self, path: PathBuf) -> Result<()> {
-        self.client_config = read_from_file::<ClientConfig>(path.clone())
-            .map_err(|e| format!("Unable to parse {:?} file: {:?}", path, e))?
-            .ok_or_else(|| format!("{:?} file does not exist", path))?;
-
-        Ok(())
-    }
-
-    /// Attempts to load the eth2 config from `path`.
-    ///
-    /// Returns an error if any files are not found or are invalid.
-    pub fn load_eth2_config(&mut self, path: PathBuf) -> Result<()> {
-        self.eth2_config = read_from_file::<Eth2Config>(path.clone())
-            .map_err(|e| format!("Unable to parse {:?} file: {:?}", path, e))?
-            .ok_or_else(|| format!("{:?} file does not exist", path))?;
-
-        Ok(())
-    }
-
-    /// Sets all listening addresses to the given `addr`.
-    pub fn set_listen_addresses(&mut self, addr: String) -> Result<()> {
-        let addr = addr
-            .parse::<Ipv4Addr>()
-            .map_err(|e| format!("Unable to parse default listen address: {:?}", e))?;
-
-        self.client_config.network.listen_address = addr.into();
-        self.client_config.rest_api.listen_address = addr;
-
-        Ok(())
-    }
-
-    /// Consumes self, returning the configs.
-    ///
-    /// The supplied `cli_args` should be the base-level `clap` cli_args (i.e., not a subcommand
-    /// cli_args).
-    pub fn build(mut self, cli_args: &ArgMatches) -> Result<Config> {
-        self.client_config.apply_cli_args(cli_args, &self.log)?;
-
-        if let Some(bump) = cli_args.value_of("port-bump") {
-            let bump = bump
-                .parse::<u16>()
-                .map_err(|e| format!("Unable to parse port bump: {}", e))?;
-
-            self.client_config.network.libp2p_port += bump;
-            self.client_config.network.discovery_port += bump;
-            self.client_config.rest_api.port += bump;
-            self.client_config.websocket_server.port += bump;
-        }
-
-        if self.eth2_config.spec_constants != self.client_config.spec_constants {
-            crit!(self.log, "Specification constants do not match.";
-                  "client_config" => self.client_config.spec_constants.to_string(),
-                  "eth2_config" => self.eth2_config.spec_constants.to_string()
-            );
-            return Err("Specification constant mismatch".into());
-        }
-
-        Ok((self.client_config, self.eth2_config, self.log))
-    }
-}
-
 fn random_string(len: usize) -> String {
     rand::thread_rng()
         .sample_iter(&Alphanumeric)

beacon_node/src/lib.rs

@@ -57,15 +57,15 @@ impl<E: EthSpec> ProductionBeaconNode<E> {
     ) -> impl Future<Item = Self, Error = String> + 'a {
         let log = context.log.clone();
 
-        // TODO: the eth2 config in the env is being completely ignored.
+        // TODO: the eth2 config in the env is being modified.
         //
         // See https://github.com/sigp/lighthouse/issues/602
-        get_configs(&matches, log).into_future().and_then(
-            move |(client_config, eth2_config, _log)| {
+        get_configs::<E>(&matches, context.eth2_config.clone(), log)
+            .into_future()
+            .and_then(move |(client_config, eth2_config, _log)| {
                 context.eth2_config = eth2_config;
                 Self::new(context, client_config)
-            },
-        )
+            })
     }
 
     /// Starts a new beacon node `Client` in the given `environment`.
@@ -126,10 +126,15 @@ impl<E: EthSpec> ProductionBeaconNode<E> {
                     .system_time_slot_clock()?
                     .websocket_event_handler(client_config.websocket_server.clone())?
                     .build_beacon_chain()?
-                    .libp2p_network(&client_config.network)?
-                    .http_server(&client_config, &http_eth2_config)?
-                    .peer_count_notifier()?
-                    .slot_notifier()?;
+                    .libp2p_network(&client_config.network)?;
+
+                let builder = if client_config.rest_api.enabled {
+                    builder.http_server(&client_config, &http_eth2_config)?
+                } else {
+                    builder
+                };
+
+                let builder = builder.peer_count_notifier()?.slot_notifier()?;
 
                 Ok(Self(builder.build()))
             })

beacon_node/tests/test.rs

@@ -1,12 +1,25 @@
 #![cfg(test)]
 
-use node_test_rig::{environment::EnvironmentBuilder, testing_client_config, LocalBeaconNode};
-use types::{MinimalEthSpec, Slot};
+use node_test_rig::{
+    environment::{Environment, EnvironmentBuilder},
+    testing_client_config, LocalBeaconNode,
+};
+use types::{EthSpec, MinimalEthSpec, Slot};
 
 fn env_builder() -> EnvironmentBuilder<MinimalEthSpec> {
     EnvironmentBuilder::minimal()
 }
 
+fn build_node<E: EthSpec>(env: &mut Environment<E>) -> LocalBeaconNode<E> {
+    let context = env.core_context();
+    env.runtime()
+        .block_on(LocalBeaconNode::production(
+            context,
+            testing_client_config(),
+        ))
+        .expect("should block until node created")
+}
+
 #[test]
 fn http_server_genesis_state() {
     let mut env = env_builder()
@@ -17,7 +30,7 @@ fn http_server_genesis_state() {
         .build()
         .expect("environment should build");
 
-    let node = LocalBeaconNode::production(env.core_context(), testing_client_config());
+    let node = build_node(&mut env);
     let remote_node = node.remote_node().expect("should produce remote node");
 
     let (api_state, _root) = env

beacon_node/websocket_server/Cargo.toml

@@ -7,7 +7,6 @@ edition = "2018"
 # See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
 
 [dependencies]
-clap = "2.33.0"
 exit-future = "0.1.4"
 futures = "0.1.29"
 serde = "1.0.102"

beacon_node/websocket_server/src/config.rs

@@ -1,5 +1,4 @@
-use clap::ArgMatches;
-use serde::{Deserialize, Serialize};
+use serde_derive::{Deserialize, Serialize};
 use std::net::Ipv4Addr;
 
 /// The core configuration of a Lighthouse beacon node.
@@ -21,25 +20,3 @@ impl Default for Config {
         }
     }
 }
-
-impl Config {
-    pub fn apply_cli_args(&mut self, args: &ArgMatches) -> Result<(), &'static str> {
-        if args.is_present("no-ws") {
-            self.enabled = false;
-        }
-
-        if let Some(rpc_address) = args.value_of("ws-address") {
-            self.listen_address = rpc_address
-                .parse::<Ipv4Addr>()
-                .map_err(|_| "ws-address is not a valid IPv4 address.")?;
-        }
-
-        if let Some(rpc_port) = args.value_of("ws-port") {
-            self.port = rpc_port
-                .parse::<u16>()
-                .map_err(|_| "ws-port is not a valid u16.")?;
-        }
-
-        Ok(())
-    }
-}