Prepare for public testnet (#628)

* Update to spec v0.9.0

* Update to v0.9.1

* Bump spec tags for v0.9.1

* Formatting, fix CI failures

* Resolve accidental KeyPair merge conflict

* Document new BeaconState functions

* Add `validator` changes from `validator-to-rest`

* Add initial (failing) REST api tests

* Fix signature parsing

* Add more tests

* Refactor http router

* Add working tests for publish beacon block

* Add validator duties tests

* Move account_manager under `lighthouse` binary

* Unify logfile handling in `environment` crate.

* Fix incorrect cache drops in `advance_caches`

* Update fork choice for v0.9.1

* Add `deposit_contract` crate

* Add progress on validator onboarding

* Add unfinished attesation code

* Update account manager CLI

* Write eth1 data file as hex string

* Integrate ValidatorDirectory with validator_client

* Move ValidatorDirectory into validator_client

* Clean up some FIXMEs

* Add beacon_chain_sim

* Fix a few docs/logs

* Expand `beacon_chain_sim`

* Fix spec for `beacon_chain_sim

* More testing for api

* Start work on attestation endpoint

* Reject empty attestations

* Allow attestations to genesis block

* Add working tests for `rest_api` validator endpoint

* Remove grpc from beacon_node

* Start heavy refactor of validator client

- Block production is working

* Prune old validator client files

* Start works on attestation service

* Add attestation service to validator client

* Use full pubkey for validator directories

* Add validator duties post endpoint

* Use par_iter for keypair generation

* Use bulk duties request in validator client

* Add version http endpoint tests

* Add interop keys and startup wait

* Ensure a prompt exit

* Add duties pruning

* Fix compile error in beacon node tests

* Add github workflow

* Modify rust.yaml

* Modify gitlab actions

* Add to CI file

* Add sudo to CI npm install

* Move cargo fmt to own job in tests

* Fix cargo fmt in CI

* Add rustup update before cargo fmt

* Change name of CI job

* Make other CI jobs require cargo fmt

* Add CI badge

* Remove gitlab and travis files

* Add different http timeout for debug

* Update docker file, use makefile in CI

* Use make in the dockerfile, skip the test

* Use the makefile for debug GI test

* Update book

* Tidy grpc and misc things

* Apply discv5 fixes

* Address other minor issues

* Fix warnings

* Attempt fix for addr parsing

* Tidy validator config, CLIs

* Tidy comments

* Tidy signing, reduce ForkService duplication

* Fail if skipping too many slots

* Set default recent genesis time to 0

* Add custom http timeout to validator

* Fix compile bug in node_test_rig

* Remove old bootstrap flag from val CLI

* Update docs

* Tidy val client

* Change val client log levels

* Add comments, more validity checks

* Fix compile error, add comments

* Undo changes to eth2-libp2p/src

* Reduce duplication of keypair generation

* Add more logging for validator duties

* Fix beacon_chain_sim, nitpicks

* Fix compile error, minor nits

* Update to use v0.9.2 version of deposit contract

* Add efforts to automate eth1 testnet deployment

* Fix lcli testnet deployer

* Modify bn CLI to parse eth2_testnet_dir

* Progress with account_manager deposit tools

* Make account manager submit deposits

* Add password option for submitting deposits

* Allow custom deposit amount

* Add long names to lcli clap

* Add password option to lcli deploy command

* Fix minor bugs whilst testing

* Address Michael's comments

* Add refund-deposit-contract to lcli

* Use time instead of skip count for denying long skips

* Improve logging for eth1

* Fix bug with validator services exiting on error

* Drop the block cache after genesis

* Modify eth1 testnet config

* Improve eth1 logging

* Make validator wait until genesis time

* Fix bug in eth1 voting

* Add more logging to eth1 voting

* Handle errors in eth1 http module

* Set SECONDS_PER_DAY to sensible minimum

* Shorten delay before testnet start

* Ensure eth1 block is produced without any votes

* Improve eth1 logging

* Fix broken tests in eth1

* Tidy code in rest_api

* Fix failing test in deposit_contract

* Make CLI args more consistent

* Change validator/duties endpoint

* Add time-based skip slot limiting

* Add new error type missed in previous commit

* Add log when waiting for genesis

* Refactor beacon node CLI

* Remove unused dep

* Add lcli eth1-genesis command

* Fix bug in master merge

* Apply clippy lints to beacon node

* Add support for YamlConfig in Eth2TestnetDir

* Upgrade tesnet deposit contract version

* Remove unnecessary logging and correct formatting

* Add a hardcoded eth2 testnet config

* Ensure http server flag works. Overwrite configs with flags.

* Ensure boot nodes are loaded from testnet dir

* Fix account manager CLI bugs

* Fix bugs with beacon node cli

* Allow testnet dir without boot nodes

* Write genesis state as SSZ

* Remove ---/n from the start of testnet_dir files

* Set default libp2p address

* Tidy account manager CLI, add logging

* Add check to see if testnet dir exists

* Apply reviewers suggestions

* Add HeadTracker struct

* Add fork choice persistence

* Shorten slot time for simulator

* Add the /beacon/heads API endpoint

* Update hardcoded testnet

* Add tests for BeaconChain persistence + fix bugs

* Extend BeaconChain persistence testing

* Ensure chain is finalized b4 persistence tests

* Ensure boot_enr.yaml is include in binary

* Refactor beacon_chain_sim

* Move files about in beacon sim

* Update beacon_chain_sim

* Fix bug with deposit inclusion

* Increase log in genesis service, fix todo

* Tidy sim, fix broken rest_api tests

* Fix more broken tests

* Update testnet

* Fix broken rest api test

* Tidy account manager CLI

* Use tempdir for account manager

* Stop hardcoded testnet dir from creating dir

* Rename Eth2TestnetDir to Eth2TestnetConfig

* Change hardcoded -> hard_coded

* Tidy account manager

* Add log to account manager

* Tidy, ensure head tracker is loaded from disk

* Tidy beacon chain builder

* Tidy eth1_chain

* Adds log support for simulator

* Revert "Adds log support for simulator"

This reverts commit ec77c66a05.

* Adds log support for simulator

* Tidy after self-review

* Change default log level

* Address Michael's delicious PR comments

* Fix off-by-one in tests

beacon_node/eth1/src/block_cache.rs

@@ -54,6 +54,21 @@ impl BlockCache {
         self.blocks.is_empty()
     }
 
+    /// Returns the timestamp of the earliest block in the cache (if any).
+    pub fn earliest_block_timestamp(&self) -> Option<u64> {
+        self.blocks.first().map(|block| block.timestamp)
+    }
+
+    /// Returns the timestamp of the latest block in the cache (if any).
+    pub fn latest_block_timestamp(&self) -> Option<u64> {
+        self.blocks.last().map(|block| block.timestamp)
+    }
+
+    /// Returns the lowest block number stored.
+    pub fn lowest_block_number(&self) -> Option<u64> {
+        self.blocks.first().map(|block| block.number)
+    }
+
     /// Returns the highest block number stored.
     pub fn highest_block_number(&self) -> Option<u64> {
         self.blocks.last().map(|block| block.number)

beacon_node/eth1/src/http.rs

@@ -23,7 +23,7 @@ use types::Hash256;
 pub const DEPOSIT_EVENT_TOPIC: &str =
     "0x649bbc62d0e31342afea4e5cd82d4049e7e1ee912fc0889aa790803be39038c5";
 /// `keccak("get_deposit_root()")[0..4]`
-pub const DEPOSIT_ROOT_FN_SIGNATURE: &str = "0x863a311b";
+pub const DEPOSIT_ROOT_FN_SIGNATURE: &str = "0xc5f2892f";
 /// `keccak("get_deposit_count()")[0..4]`
 pub const DEPOSIT_COUNT_FN_SIGNATURE: &str = "0x621fd130";
 
@@ -137,19 +137,21 @@ pub fn get_deposit_count(
         block_number,
         timeout,
     )
-    .and_then(|result| result.ok_or_else(|| "No response to deposit count".to_string()))
-    .and_then(|bytes| {
-        if bytes.is_empty() {
-            Ok(None)
-        } else if bytes.len() == DEPOSIT_COUNT_RESPONSE_BYTES {
-            let mut array = [0; 8];
-            array.copy_from_slice(&bytes[32 + 32..32 + 32 + 8]);
-            Ok(Some(u64::from_le_bytes(array)))
-        } else {
-            Err(format!(
-                "Deposit count response was not {} bytes: {:?}",
-                DEPOSIT_COUNT_RESPONSE_BYTES, bytes
-            ))
+    .and_then(|result| match result {
+        None => Err(format!("Deposit root response was none")),
+        Some(bytes) => {
+            if bytes.is_empty() {
+                Ok(None)
+            } else if bytes.len() == DEPOSIT_COUNT_RESPONSE_BYTES {
+                let mut array = [0; 8];
+                array.copy_from_slice(&bytes[32 + 32..32 + 32 + 8]);
+                Ok(Some(u64::from_le_bytes(array)))
+            } else {
+                Err(format!(
+                    "Deposit count response was not {} bytes: {:?}",
+                    DEPOSIT_COUNT_RESPONSE_BYTES, bytes
+                ))
+            }
         }
     })
 }
@@ -172,17 +174,19 @@ pub fn get_deposit_root(
         block_number,
         timeout,
     )
-    .and_then(|result| result.ok_or_else(|| "No response to deposit root".to_string()))
-    .and_then(|bytes| {
-        if bytes.is_empty() {
-            Ok(None)
-        } else if bytes.len() == DEPOSIT_ROOT_BYTES {
-            Ok(Some(Hash256::from_slice(&bytes)))
-        } else {
-            Err(format!(
-                "Deposit root response was not {} bytes: {:?}",
-                DEPOSIT_ROOT_BYTES, bytes
-            ))
+    .and_then(|result| match result {
+        None => Err(format!("Deposit root response was none")),
+        Some(bytes) => {
+            if bytes.is_empty() {
+                Ok(None)
+            } else if bytes.len() == DEPOSIT_ROOT_BYTES {
+                Ok(Some(Hash256::from_slice(&bytes)))
+            } else {
+                Err(format!(
+                    "Deposit root response was not {} bytes: {:?}",
+                    DEPOSIT_ROOT_BYTES, bytes
+                ))
+            }
         }
     })
 }
@@ -369,12 +373,18 @@ pub fn send_rpc_request(
 
 /// Accepts an entire HTTP body (as a string) and returns the `result` field, as a serde `Value`.
 fn response_result(response: &str) -> Result<Option<Value>, String> {
-    Ok(serde_json::from_str::<Value>(&response)
-        .map_err(|e| format!("Failed to parse response: {:?}", e))?
-        .get("result")
-        .cloned()
-        .map(Some)
-        .unwrap_or_else(|| None))
+    let json = serde_json::from_str::<Value>(&response)
+        .map_err(|e| format!("Failed to parse response: {:?}", e))?;
+
+    if let Some(error) = json.get("error") {
+        Err(format!("Eth1 node returned error: {}", error))
+    } else {
+        Ok(json
+            .get("result")
+            .cloned()
+            .map(Some)
+            .unwrap_or_else(|| None))
+    }
 }
 
 /// Parses a `0x`-prefixed, **big-endian** hex string as a u64.

beacon_node/eth1/src/service.rs

@@ -121,7 +121,7 @@ impl Default for Config {
             lowest_cached_block_number: 0,
             follow_distance: 128,
             block_cache_truncation: Some(4_096),
-            auto_update_interval_millis: 500,
+            auto_update_interval_millis: 7_000,
             blocks_per_log_query: 1_000,
             max_log_requests_per_update: None,
             max_blocks_per_update: None,
@@ -163,6 +163,26 @@ impl Service {
         &self.inner.deposit_cache
     }
 
+    /// Drop the block cache, replacing it with an empty one.
+    pub fn drop_block_cache(&self) {
+        *(self.inner.block_cache.write()) = BlockCache::default();
+    }
+
+    /// Returns the timestamp of the earliest block in the cache (if any).
+    pub fn earliest_block_timestamp(&self) -> Option<u64> {
+        self.inner.block_cache.read().earliest_block_timestamp()
+    }
+
+    /// Returns the timestamp of the latest block in the cache (if any).
+    pub fn latest_block_timestamp(&self) -> Option<u64> {
+        self.inner.block_cache.read().latest_block_timestamp()
+    }
+
+    /// Returns the lowest block number stored.
+    pub fn lowest_block_number(&self) -> Option<u64> {
+        self.inner.block_cache.read().lowest_block_number()
+    }
+
     /// Returns the number of currently cached blocks.
     pub fn block_cache_len(&self) -> usize {
         self.blocks().read().len()
@@ -220,6 +240,8 @@ impl Service {
     {
         let log_a = self.log.clone();
         let log_b = self.log.clone();
+        let inner_1 = self.inner.clone();
+        let inner_2 = self.inner.clone();
 
         let deposit_future = self
             .update_deposit_cache()
@@ -229,6 +251,7 @@ impl Service {
                     Ok(DepositCacheUpdateOutcome::Success { logs_imported }) => trace!(
                         log_a,
                         "Updated eth1 deposit cache";
+                        "cached_deposits" => inner_1.deposit_cache.read().cache.len(),
                         "logs_imported" => logs_imported,
                     ),
                     Err(e) => error!(
@@ -252,6 +275,7 @@ impl Service {
                     }) => trace!(
                         log_b,
                         "Updated eth1 block cache";
+                        "cached_blocks" => inner_2.block_cache.read().len(),
                         "blocks_imported" => blocks_imported,
                         "head_block" => head_block_number,
                     ),