Prepare for public testnet (#628)

* Update to spec v0.9.0

* Update to v0.9.1

* Bump spec tags for v0.9.1

* Formatting, fix CI failures

* Resolve accidental KeyPair merge conflict

* Document new BeaconState functions

* Add `validator` changes from `validator-to-rest`

* Add initial (failing) REST api tests

* Fix signature parsing

* Add more tests

* Refactor http router

* Add working tests for publish beacon block

* Add validator duties tests

* Move account_manager under `lighthouse` binary

* Unify logfile handling in `environment` crate.

* Fix incorrect cache drops in `advance_caches`

* Update fork choice for v0.9.1

* Add `deposit_contract` crate

* Add progress on validator onboarding

* Add unfinished attesation code

* Update account manager CLI

* Write eth1 data file as hex string

* Integrate ValidatorDirectory with validator_client

* Move ValidatorDirectory into validator_client

* Clean up some FIXMEs

* Add beacon_chain_sim

* Fix a few docs/logs

* Expand `beacon_chain_sim`

* Fix spec for `beacon_chain_sim

* More testing for api

* Start work on attestation endpoint

* Reject empty attestations

* Allow attestations to genesis block

* Add working tests for `rest_api` validator endpoint

* Remove grpc from beacon_node

* Start heavy refactor of validator client

- Block production is working

* Prune old validator client files

* Start works on attestation service

* Add attestation service to validator client

* Use full pubkey for validator directories

* Add validator duties post endpoint

* Use par_iter for keypair generation

* Use bulk duties request in validator client

* Add version http endpoint tests

* Add interop keys and startup wait

* Ensure a prompt exit

* Add duties pruning

* Fix compile error in beacon node tests

* Add github workflow

* Modify rust.yaml

* Modify gitlab actions

* Add to CI file

* Add sudo to CI npm install

* Move cargo fmt to own job in tests

* Fix cargo fmt in CI

* Add rustup update before cargo fmt

* Change name of CI job

* Make other CI jobs require cargo fmt

* Add CI badge

* Remove gitlab and travis files

* Add different http timeout for debug

* Update docker file, use makefile in CI

* Use make in the dockerfile, skip the test

* Use the makefile for debug GI test

* Update book

* Tidy grpc and misc things

* Apply discv5 fixes

* Address other minor issues

* Fix warnings

* Attempt fix for addr parsing

* Tidy validator config, CLIs

* Tidy comments

* Tidy signing, reduce ForkService duplication

* Fail if skipping too many slots

* Set default recent genesis time to 0

* Add custom http timeout to validator

* Fix compile bug in node_test_rig

* Remove old bootstrap flag from val CLI

* Update docs

* Tidy val client

* Change val client log levels

* Add comments, more validity checks

* Fix compile error, add comments

* Undo changes to eth2-libp2p/src

* Reduce duplication of keypair generation

* Add more logging for validator duties

* Fix beacon_chain_sim, nitpicks

* Fix compile error, minor nits

* Update to use v0.9.2 version of deposit contract

* Add efforts to automate eth1 testnet deployment

* Fix lcli testnet deployer

* Modify bn CLI to parse eth2_testnet_dir

* Progress with account_manager deposit tools

* Make account manager submit deposits

* Add password option for submitting deposits

* Allow custom deposit amount

* Add long names to lcli clap

* Add password option to lcli deploy command

* Fix minor bugs whilst testing

* Address Michael's comments

* Add refund-deposit-contract to lcli

* Use time instead of skip count for denying long skips

* Improve logging for eth1

* Fix bug with validator services exiting on error

* Drop the block cache after genesis

* Modify eth1 testnet config

* Improve eth1 logging

* Make validator wait until genesis time

* Fix bug in eth1 voting

* Add more logging to eth1 voting

* Handle errors in eth1 http module

* Set SECONDS_PER_DAY to sensible minimum

* Shorten delay before testnet start

* Ensure eth1 block is produced without any votes

* Improve eth1 logging

* Fix broken tests in eth1

* Tidy code in rest_api

* Fix failing test in deposit_contract

* Make CLI args more consistent

* Change validator/duties endpoint

* Add time-based skip slot limiting

* Add new error type missed in previous commit

* Add log when waiting for genesis

* Refactor beacon node CLI

* Remove unused dep

* Add lcli eth1-genesis command

* Fix bug in master merge

* Apply clippy lints to beacon node

* Add support for YamlConfig in Eth2TestnetDir

* Upgrade tesnet deposit contract version

* Remove unnecessary logging and correct formatting

* Add a hardcoded eth2 testnet config

* Ensure http server flag works. Overwrite configs with flags.

* Ensure boot nodes are loaded from testnet dir

* Fix account manager CLI bugs

* Fix bugs with beacon node cli

* Allow testnet dir without boot nodes

* Write genesis state as SSZ

* Remove ---/n from the start of testnet_dir files

* Set default libp2p address

* Tidy account manager CLI, add logging

* Add check to see if testnet dir exists

* Apply reviewers suggestions

* Add HeadTracker struct

* Add fork choice persistence

* Shorten slot time for simulator

* Add the /beacon/heads API endpoint

* Update hardcoded testnet

* Add tests for BeaconChain persistence + fix bugs

* Extend BeaconChain persistence testing

* Ensure chain is finalized b4 persistence tests

* Ensure boot_enr.yaml is include in binary

* Refactor beacon_chain_sim

* Move files about in beacon sim

* Update beacon_chain_sim

* Fix bug with deposit inclusion

* Increase log in genesis service, fix todo

* Tidy sim, fix broken rest_api tests

* Fix more broken tests

* Update testnet

* Fix broken rest api test

* Tidy account manager CLI

* Use tempdir for account manager

* Stop hardcoded testnet dir from creating dir

* Rename Eth2TestnetDir to Eth2TestnetConfig

* Change hardcoded -> hard_coded

* Tidy account manager

* Add log to account manager

* Tidy, ensure head tracker is loaded from disk

* Tidy beacon chain builder

* Tidy eth1_chain

* Adds log support for simulator

* Revert "Adds log support for simulator"

This reverts commit ec77c66a05.

* Adds log support for simulator

* Tidy after self-review

* Change default log level

* Address Michael's delicious PR comments

* Fix off-by-one in tests

beacon_node/beacon_chain/src/beacon_chain.rs

@@ -3,6 +3,7 @@ use crate::errors::{BeaconChainError as Error, BlockProductionError};
 use crate::eth1_chain::{Eth1Chain, Eth1ChainBackend};
 use crate::events::{EventHandler, EventKind};
 use crate::fork_choice::{Error as ForkChoiceError, ForkChoice};
+use crate::head_tracker::HeadTracker;
 use crate::metrics;
 use crate::persisted_beacon_chain::{PersistedBeaconChain, BEACON_CHAIN_DB_KEY};
 use lmd_ghost::LmdGhost;
@@ -126,6 +127,8 @@ pub struct BeaconChain<T: BeaconChainTypes> {
     pub fork_choice: ForkChoice<T>,
     /// A handler for events generated by the beacon chain.
     pub event_handler: T::EventHandler,
+    /// Used to track the heads of the beacon chain.
+    pub(crate) head_tracker: HeadTracker,
     /// Logging to CLI, etc.
     pub(crate) log: Logger,
 }
@@ -137,10 +140,35 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
     pub fn persist(&self) -> Result<(), Error> {
         let timer = metrics::start_timer(&metrics::PERSIST_CHAIN);
 
+        let canonical_head = self.head();
+
+        let finalized_checkpoint = {
+            let beacon_block_root = canonical_head.beacon_state.finalized_checkpoint.root;
+            let beacon_block = self
+                .store
+                .get::<BeaconBlock<_>>(&beacon_block_root)?
+                .ok_or_else(|| Error::MissingBeaconBlock(beacon_block_root))?;
+            let beacon_state_root = beacon_block.state_root;
+            let beacon_state = self
+                .store
+                .get_state(&beacon_state_root, Some(beacon_block.slot))?
+                .ok_or_else(|| Error::MissingBeaconState(beacon_state_root))?;
+
+            CheckPoint {
+                beacon_block_root,
+                beacon_block,
+                beacon_state_root,
+                beacon_state,
+            }
+        };
+
         let p: PersistedBeaconChain<T> = PersistedBeaconChain {
-            canonical_head: self.canonical_head.read().clone(),
+            canonical_head,
+            finalized_checkpoint,
             op_pool: PersistedOperationPool::from_operation_pool(&self.op_pool),
             genesis_block_root: self.genesis_block_root,
+            ssz_head_tracker: self.head_tracker.to_ssz_container(),
+            fork_choice: self.fork_choice.as_ssz_container(),
         };
 
         let key = Hash256::from_slice(&BEACON_CHAIN_DB_KEY.as_bytes());
@@ -331,10 +359,17 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
         self.canonical_head.read().clone()
     }
 
+    /// Returns the current heads of the `BeaconChain`. For the canonical head, see `Self::head`.
+    ///
+    /// Returns `(block_root, block_slot)`.
+    pub fn heads(&self) -> Vec<(Hash256, Slot)> {
+        self.head_tracker.heads()
+    }
+
     /// Returns the `BeaconState` at the given slot.
     ///
-    ///  Returns `None` when the state is not found in the database or there is an error skipping
-    ///  to a future state.
+    /// Returns `None` when the state is not found in the database or there is an error skipping
+    /// to a future state.
     pub fn state_at_slot(&self, slot: Slot) -> Result<BeaconState<T::EthSpec>, Error> {
         let head_state = self.head().beacon_state;
 
@@ -1219,6 +1254,8 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
 
         metrics::stop_timer(db_write_timer);
 
+        self.head_tracker.register_block(block_root, &block);
+
         let fork_choice_register_timer =
             metrics::start_timer(&metrics::BLOCK_PROCESSING_FORK_CHOICE_REGISTER);
 
@@ -1324,6 +1361,11 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
         let (proposer_slashings, attester_slashings) =
             self.op_pool.get_slashings(&state, &self.spec);
 
+        let eth1_data = eth1_chain.eth1_data_for_block_production(&state, &self.spec)?;
+        let deposits = eth1_chain
+            .deposits_for_block_inclusion(&state, &eth1_data, &self.spec)?
+            .into();
+
         let mut block = BeaconBlock {
             slot: state.slot,
             parent_root,
@@ -1332,14 +1374,12 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
             signature: Signature::empty_signature(),
             body: BeaconBlockBody {
                 randao_reveal,
-                eth1_data: eth1_chain.eth1_data_for_block_production(&state, &self.spec)?,
+                eth1_data,
                 graffiti,
                 proposer_slashings: proposer_slashings.into(),
                 attester_slashings: attester_slashings.into(),
                 attestations: self.op_pool.get_attestations(&state, &self.spec).into(),
-                deposits: eth1_chain
-                    .deposits_for_block_inclusion(&state, &self.spec)?
-                    .into(),
+                deposits,
                 voluntary_exits: self.op_pool.get_voluntary_exits(&state, &self.spec).into(),
             },
         };
@@ -1596,6 +1636,23 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
     }
 }
 
+impl<T: BeaconChainTypes> Drop for BeaconChain<T> {
+    fn drop(&mut self) {
+        if let Err(e) = self.persist() {
+            error!(
+                self.log,
+                "Failed to persist BeaconChain on drop";
+                "error" => format!("{:?}", e)
+            )
+        } else {
+            info!(
+                self.log,
+                "Saved beacon chain state";
+            )
+        }
+    }
+}
+
 fn write_state<T: EthSpec>(prefix: &str, state: &BeaconState<T>, log: &Logger) {
     if WRITE_BLOCK_PROCESSING_SSZ {
         let root = Hash256::from_slice(&state.tree_hash_root());