Prepare for public testnet (#628)

* Update to spec v0.9.0

* Update to v0.9.1

* Bump spec tags for v0.9.1

* Formatting, fix CI failures

* Resolve accidental KeyPair merge conflict

* Document new BeaconState functions

* Add `validator` changes from `validator-to-rest`

* Add initial (failing) REST api tests

* Fix signature parsing

* Add more tests

* Refactor http router

* Add working tests for publish beacon block

* Add validator duties tests

* Move account_manager under `lighthouse` binary

* Unify logfile handling in `environment` crate.

* Fix incorrect cache drops in `advance_caches`

* Update fork choice for v0.9.1

* Add `deposit_contract` crate

* Add progress on validator onboarding

* Add unfinished attesation code

* Update account manager CLI

* Write eth1 data file as hex string

* Integrate ValidatorDirectory with validator_client

* Move ValidatorDirectory into validator_client

* Clean up some FIXMEs

* Add beacon_chain_sim

* Fix a few docs/logs

* Expand `beacon_chain_sim`

* Fix spec for `beacon_chain_sim

* More testing for api

* Start work on attestation endpoint

* Reject empty attestations

* Allow attestations to genesis block

* Add working tests for `rest_api` validator endpoint

* Remove grpc from beacon_node

* Start heavy refactor of validator client

- Block production is working

* Prune old validator client files

* Start works on attestation service

* Add attestation service to validator client

* Use full pubkey for validator directories

* Add validator duties post endpoint

* Use par_iter for keypair generation

* Use bulk duties request in validator client

* Add version http endpoint tests

* Add interop keys and startup wait

* Ensure a prompt exit

* Add duties pruning

* Fix compile error in beacon node tests

* Add github workflow

* Modify rust.yaml

* Modify gitlab actions

* Add to CI file

* Add sudo to CI npm install

* Move cargo fmt to own job in tests

* Fix cargo fmt in CI

* Add rustup update before cargo fmt

* Change name of CI job

* Make other CI jobs require cargo fmt

* Add CI badge

* Remove gitlab and travis files

* Add different http timeout for debug

* Update docker file, use makefile in CI

* Use make in the dockerfile, skip the test

* Use the makefile for debug GI test

* Update book

* Tidy grpc and misc things

* Apply discv5 fixes

* Address other minor issues

* Fix warnings

* Attempt fix for addr parsing

* Tidy validator config, CLIs

* Tidy comments

* Tidy signing, reduce ForkService duplication

* Fail if skipping too many slots

* Set default recent genesis time to 0

* Add custom http timeout to validator

* Fix compile bug in node_test_rig

* Remove old bootstrap flag from val CLI

* Update docs

* Tidy val client

* Change val client log levels

* Add comments, more validity checks

* Fix compile error, add comments

* Undo changes to eth2-libp2p/src

* Reduce duplication of keypair generation

* Add more logging for validator duties

* Fix beacon_chain_sim, nitpicks

* Fix compile error, minor nits

* Update to use v0.9.2 version of deposit contract

* Add efforts to automate eth1 testnet deployment

* Fix lcli testnet deployer

* Modify bn CLI to parse eth2_testnet_dir

* Progress with account_manager deposit tools

* Make account manager submit deposits

* Add password option for submitting deposits

* Allow custom deposit amount

* Add long names to lcli clap

* Add password option to lcli deploy command

* Fix minor bugs whilst testing

* Address Michael's comments

* Add refund-deposit-contract to lcli

* Use time instead of skip count for denying long skips

* Improve logging for eth1

* Fix bug with validator services exiting on error

* Drop the block cache after genesis

* Modify eth1 testnet config

* Improve eth1 logging

* Make validator wait until genesis time

* Fix bug in eth1 voting

* Add more logging to eth1 voting

* Handle errors in eth1 http module

* Set SECONDS_PER_DAY to sensible minimum

* Shorten delay before testnet start

* Ensure eth1 block is produced without any votes

* Improve eth1 logging

* Fix broken tests in eth1

* Tidy code in rest_api

* Fix failing test in deposit_contract

* Make CLI args more consistent

* Change validator/duties endpoint

* Add time-based skip slot limiting

* Add new error type missed in previous commit

* Add log when waiting for genesis

* Refactor beacon node CLI

* Remove unused dep

* Add lcli eth1-genesis command

* Fix bug in master merge

* Apply clippy lints to beacon node

* Add support for YamlConfig in Eth2TestnetDir

* Upgrade tesnet deposit contract version

* Remove unnecessary logging and correct formatting

* Add a hardcoded eth2 testnet config

* Ensure http server flag works. Overwrite configs with flags.

* Ensure boot nodes are loaded from testnet dir

* Fix account manager CLI bugs

* Fix bugs with beacon node cli

* Allow testnet dir without boot nodes

* Write genesis state as SSZ

* Remove ---/n from the start of testnet_dir files

* Set default libp2p address

* Tidy account manager CLI, add logging

* Add check to see if testnet dir exists

* Apply reviewers suggestions

* Add HeadTracker struct

* Add fork choice persistence

* Shorten slot time for simulator

* Add the /beacon/heads API endpoint

* Update hardcoded testnet

* Add tests for BeaconChain persistence + fix bugs

* Extend BeaconChain persistence testing

* Ensure chain is finalized b4 persistence tests

* Ensure boot_enr.yaml is include in binary

* Refactor beacon_chain_sim

* Move files about in beacon sim

* Update beacon_chain_sim

* Fix bug with deposit inclusion

* Increase log in genesis service, fix todo

* Tidy sim, fix broken rest_api tests

* Fix more broken tests

* Update testnet

* Fix broken rest api test

* Tidy account manager CLI

* Use tempdir for account manager

* Stop hardcoded testnet dir from creating dir

* Rename Eth2TestnetDir to Eth2TestnetConfig

* Change hardcoded -> hard_coded

* Tidy account manager

* Add log to account manager

* Tidy, ensure head tracker is loaded from disk

* Tidy beacon chain builder

* Tidy eth1_chain

* Adds log support for simulator

* Revert "Adds log support for simulator"

This reverts commit ec77c66a05.

* Adds log support for simulator

* Tidy after self-review

* Change default log level

* Address Michael's delicious PR comments

* Fix off-by-one in tests

tests/eth1_test_rig/src/lib.rs

@@ -7,17 +7,17 @@
 //! some initial issues.
 mod ganache;
 
-use deposit_contract::{eth1_tx_data, ABI, BYTECODE, CONTRACT_DEPLOY_GAS, DEPOSIT_GAS};
-use futures::{stream, Future, IntoFuture, Stream};
+use deposit_contract::{eth1_tx_data, testnet, ABI, BYTECODE, CONTRACT_DEPLOY_GAS, DEPOSIT_GAS};
+use futures::{future, stream, Future, IntoFuture, Stream};
 use ganache::GanacheInstance;
 use std::time::{Duration, Instant};
 use tokio::{runtime::Runtime, timer::Delay};
 use types::DepositData;
-use types::{EthSpec, Hash256, Keypair, Signature};
+use types::{test_utils::generate_deterministic_keypair, EthSpec, Hash256, Keypair, Signature};
 use web3::contract::{Contract, Options};
 use web3::transports::Http;
 use web3::types::{Address, TransactionRequest, U256};
-use web3::{Transport, Web3};
+use web3::Web3;
 
 pub const DEPLOYER_ACCOUNTS_INDEX: usize = 0;
 pub const DEPOSIT_ACCOUNTS_INDEX: usize = 0;
@@ -31,7 +31,7 @@ pub struct GanacheEth1Instance {
 impl GanacheEth1Instance {
     pub fn new() -> impl Future<Item = Self, Error = String> {
         GanacheInstance::new().into_future().and_then(|ganache| {
-            DepositContract::deploy(ganache.web3.clone(), 0).map(|deposit_contract| Self {
+            DepositContract::deploy(ganache.web3.clone(), 0, None).map(|deposit_contract| Self {
                 ganache,
                 deposit_contract,
             })
@@ -58,21 +58,52 @@ impl DepositContract {
     pub fn deploy(
         web3: Web3<Http>,
         confirmations: usize,
+        password: Option<String>,
+    ) -> impl Future<Item = Self, Error = String> {
+        Self::deploy_bytecode(web3, confirmations, BYTECODE, ABI, password)
+    }
+
+    pub fn deploy_testnet(
+        web3: Web3<Http>,
+        confirmations: usize,
+        password: Option<String>,
+    ) -> impl Future<Item = Self, Error = String> {
+        Self::deploy_bytecode(
+            web3,
+            confirmations,
+            testnet::BYTECODE,
+            testnet::ABI,
+            password,
+        )
+    }
+
+    fn deploy_bytecode(
+        web3: Web3<Http>,
+        confirmations: usize,
+        bytecode: &[u8],
+        abi: &[u8],
+        password: Option<String>,
     ) -> impl Future<Item = Self, Error = String> {
         let web3_1 = web3.clone();
 
-        deploy_deposit_contract(web3.clone(), confirmations)
-            .map_err(|e| {
-                format!(
-                    "Failed to deploy contract: {}. Is scripts/ganache_tests_node.sh running?.",
-                    e
-                )
-            })
-            .and_then(move |address| {
-                Contract::from_json(web3_1.eth(), address, ABI)
-                    .map_err(|e| format!("Failed to init contract: {:?}", e))
-            })
-            .map(|contract| Self { contract, web3 })
+        deploy_deposit_contract(
+            web3.clone(),
+            confirmations,
+            bytecode.to_vec(),
+            abi.to_vec(),
+            password,
+        )
+        .map_err(|e| {
+            format!(
+                "Failed to deploy contract: {}. Is scripts/ganache_tests_node.sh running?.",
+                e
+            )
+        })
+        .and_then(move |address| {
+            Contract::from_json(web3_1.eth(), address, ABI)
+                .map_err(|e| format!("Failed to init contract: {:?}", e))
+        })
+        .map(|contract| Self { contract, web3 })
     }
 
     /// The deposit contract's address in `0x00ab...` format.
@@ -125,6 +156,25 @@ impl DepositContract {
             .map_err(|e| format!("Deposit failed: {:?}", e))
     }
 
+    pub fn deposit_deterministic_async<E: EthSpec>(
+        &self,
+        keypair_index: usize,
+        amount: u64,
+    ) -> impl Future<Item = (), Error = String> {
+        let keypair = generate_deterministic_keypair(keypair_index);
+
+        let mut deposit = DepositData {
+            pubkey: keypair.pk.into(),
+            withdrawal_credentials: Hash256::zero(),
+            amount,
+            signature: Signature::empty_signature().into(),
+        };
+
+        deposit.signature = deposit.create_signature(&keypair.sk, &E::default_spec());
+
+        self.deposit_async(deposit)
+    }
+
     /// Performs a non-blocking deposit.
     pub fn deposit_async(
         &self,
@@ -208,11 +258,15 @@ fn from_gwei(gwei: u64) -> U256 {
 
 /// Deploys the deposit contract to the given web3 instance using the account with index
 /// `DEPLOYER_ACCOUNTS_INDEX`.
-fn deploy_deposit_contract<T: Transport>(
-    web3: Web3<T>,
+fn deploy_deposit_contract(
+    web3: Web3<Http>,
     confirmations: usize,
+    bytecode: Vec<u8>,
+    abi: Vec<u8>,
+    password_opt: Option<String>,
 ) -> impl Future<Item = Address, Error = String> {
-    let bytecode = String::from_utf8_lossy(&BYTECODE);
+    let bytecode = String::from_utf8(bytecode).expect("bytecode must be valid utf8");
+    let web3_1 = web3.clone();
 
     web3.eth()
         .accounts()
@@ -223,8 +277,30 @@ fn deploy_deposit_contract<T: Transport>(
                 .cloned()
                 .ok_or_else(|| "Insufficient accounts for deployer".to_string())
         })
+        .and_then(move |from_address| {
+            let future: Box<dyn Future<Item = Address, Error = String> + Send> =
+                if let Some(password) = password_opt {
+                    // Unlock for only a single transaction.
+                    let duration = None;
+
+                    let future = web3_1
+                        .personal()
+                        .unlock_account(from_address, &password, duration)
+                        .then(move |result| match result {
+                            Ok(true) => Ok(from_address),
+                            Ok(false) => Err("Eth1 node refused to unlock account".to_string()),
+                            Err(e) => Err(format!("Eth1 unlock request failed: {:?}", e)),
+                        });
+
+                    Box::new(future)
+                } else {
+                    Box::new(future::ok(from_address))
+                };
+
+            future
+        })
         .and_then(move |deploy_address| {
-            Contract::deploy(web3.eth(), &ABI)
+            Contract::deploy(web3.eth(), &abi)
                 .map_err(|e| format!("Unable to build contract deployer: {:?}", e))?
                 .confirmations(confirmations)
                 .options(Options {