Eth1 Integration (#542)

* Refactor to cache Eth1Data * Fix merge conflicts and minor refactorings * Rename Eth1Cache to Eth1DataCache * Refactor events subscription * Add deposits module to interface with BeaconChain deposits * Remove utils * Rename to types.rs and add trait constraints to Eth1DataFetcher * Confirm to trait constraints. Make Web3DataFetcher cloneable * Make fetcher object member of deposit and eth1_data cache and other fixes * Fix update_cache function * Move fetch_eth1_data to impl block * Fix deposit tests * Create Eth1 object for interfacing with Beacon chain * Add `run` function for running update_cache and subscribe_deposit_logs tasks * Add logging * Run `cargo fmt` and make tests pass * Convert sync functions to async * Add timeouts to web3 functions * Return futures from cache functions * Add failed chaining of futures * Working cache updation * Clean up tests and `update_cache` function * Refactor `get_eth1_data` functions to work with future returning functions * Refactor eth1 `run` function to work with modified `update_cache` api * Minor changes * Add distance parameter to `update_cache` * Fix tests and other minor fixes * Working integration with cache and deposits * Add merkle_tree construction, proof generation and verification code * Add function to construct and fetch Deposits for BeaconNode * Add error handling * Import ssz * Add error handling to eth1 cache and fix minor errors * Run rustfmt * Fix minor bug * Rename Eth1Error and change to Result<T> * Change deposit fetching mechanism from notification based to poll based * Add deposits from eth1 chain in a given range every `x` blocks * Modify `run` function to accommodate changes * Minor fixes * Fix formatting * Initial commit. web3 api working. * Tidied up lib. Add function for fetching logs. * Refactor with `Eth1DataFetcher` trait * Add parsing for deposit contract logs and get_eth1_data function * Add `get_eth1_votes` function * Refactor to cache Eth1Data * Fix merge conflicts and minor refactorings * Rename Eth1Cache to Eth1DataCache * Refactor events subscription * Add deposits module to interface with BeaconChain deposits * Remove utils * Rename to types.rs and add trait constraints to Eth1DataFetcher * Confirm to trait constraints. Make Web3DataFetcher cloneable * Make fetcher object member of deposit and eth1_data cache and other fixes * Fix update_cache function * Move fetch_eth1_data to impl block * Fix deposit tests * Create Eth1 object for interfacing with Beacon chain * Add `run` function for running update_cache and subscribe_deposit_logs tasks * Add logging * Run `cargo fmt` and make tests pass * Convert sync functions to async * Add timeouts to web3 functions * Return futures from cache functions * Add failed chaining of futures * Working cache updation * Clean up tests and `update_cache` function * Refactor `get_eth1_data` functions to work with future returning functions * Refactor eth1 `run` function to work with modified `update_cache` api * Minor changes * Add distance parameter to `update_cache` * Fix tests and other minor fixes * Working integration with cache and deposits * Add merkle_tree construction, proof generation and verification code * Add function to construct and fetch Deposits for BeaconNode * Add error handling * Import ssz * Add error handling to eth1 cache and fix minor errors * Run rustfmt * Fix minor bug * Rename Eth1Error and change to Result<T> * Change deposit fetching mechanism from notification based to poll based * Add deposits from eth1 chain in a given range every `x` blocks * Modify `run` function to accommodate changes * Minor fixes * Fix formatting * Fix merge issue * Refactor with `Config` struct. Remote `ContractConfig` * Rename eth1_chain crate to eth1 * Rename files and read abi file using `fs::read` * Move eth1 to lib * Remove unnecessary mutability constraint * Add `Web3Backend` for returning actual eth1 data * Refactor `get_eth1_votes` to return a Result * Delete `eth1_chain` crate * Return `Result` from `get_deposits` * Fix range of deposits to return to beacon chain * Add `get_block_height_by_hash` trait function * Add naive method for getting `previous_eth1_distance` * Add eth1 config params to main config * Add instructions for setting up eth1 testing environment * Add build script to fetch deposit contract abi * Contract ABI is part of compiled binary * Fix minor bugs * Move docs to lib * Add timeout to config * Remove print statements * Change warn to error * Fix typos * Removed prints in test and get timeout value from config * Fixed error types * Added logging to web3_fetcher * Refactor for modified web3 api * Fix minor stuff * Add build script * Tidy, hide eth1 integration tests behind flag * Add http crate * Add first stages of eth1_test_rig * Fix deposits on test rig * Fix bug with deposit count method * Add block hash getter to http eth1 * Clean eth1 http crate and tests * Add script to start ganache * Adds deposit tree to eth1-http * Extend deposit tree tests * Tidy tests in eth1-http * Add more detail to get block request * Add block cache to eth1-http * Rename deposit tree to deposit cache * Add inital updating to eth1-http * Tidy updater * Fix compile bugs in tests * Adds an Eth1DataCache builder * Reorg eth1-http files * Add (failing) tests for eth1 updater * Rename files, fix bug in eth1-http * Ensure that ganache timestamps are increasing * Fix bugs with getting eth1data ancestors * Improve eth1 testing, fix bugs * Add truncate method to block cache * Add pruning to block cache update process * Add tests for block pruning * Allow for dropping an expired cache. * Add more comments * Add first compiling version of deposit updater * Add common fn for getting range of required blocks * Add passing deposit update test * Improve tests * Fix block pruning bug * Add tests for running two updates at once * Add updater services to eth1 * Add deposit collection to beacon chain * Add incomplete builder experiments * Add first working version of beacon chain builder * Update test harness to new beacon chain type * Rename builder file, tidy * Add first working client builder * Progress further on client builder * Update becaon node binary to use client builder * Ensure release tests compile * Remove old eth1 crate * Add first pass of new lighthouse binary * Fix websocket server startup * Remove old binary code from beacon_node crate * Add first working beacon node tests * Add genesis crate, new eth1 cache_2 * Add Serivce to Eth1Cache * Refactor with general eth1 improvements * Add passing genesis test * Tidy, add comments * Add more comments to eth1 service * Add further eth1 progress * Fix some bugs with genesis * Fix eth1 bugs, make eth1 linking more efficient * Shift logic in genesis service * Add more comments to genesis service * Add gzip, max request values, timeouts to http * Update testnet parameters to suit goerli testnet * Add ability to vary Fork, fix custom spec * Be more explicit about deposit fork version * Start adding beacon chain eth1 option * Add more flexibility to prod client * Further runtime refactoring * Allow for starting from store * Add bootstrapping to client config * Add remote_beacon_node crate * Update eth1 service for more configurability * Update eth1 tests to use less runtimes * Patch issues with tests using too many files * Move dummy eth1 backend flag * Ensure all tests pass * Add ganache-cli to Dockerfile * Use a special docker hub image for testing * Appease clippy * Move validator client into lighthouse binary * Allow starting with dummy eth1 backend * Improve logging * Fix dummy eth1 backend from cli * Add extra testnet command * Ensure consistent spec in beacon node * Update eth1 rig to work on goerli * Tidy lcli, start adding support for yaml config * Add incomplete YamlConfig struct * Remove efforts at YamlConfig * Add incomplete eth1 voting. Blocked on spec issues * Add (untested) first pass at eth1 vote algo * Add tests for winning vote * Add more tests for eth1 chain * Add more eth1 voting tests * Added more eth1 voting testing * Change test name * Add more tests to eth1 chain * Tidy eth1 generics, add more tests * Improve comments * Tidy beacon_node tests * Tidy, rename JsonRpc.. to Caching.. * Tidy voting logic * Tidy builder docs * Add comments, tidy eth1 * Add more comments to eth1 * Fix bug with winning_vote * Add doc comments to the `ClientBuilder` * Remove commented-out code * Improve `ClientBuilder` docs * Add comments to client config * Add decoding test for `ClientConfig` * Remove unused `DepositSet` struct * Tidy `block_cache` * Remove commented out lines * Remove unused code in `eth1` crate * Remove old validator binary `main.rs` * Tidy, fix tests compile error * Add initial tests for get_deposits * Remove dead code in eth1_test_rig * Update TestingDepositBuilder * Add testing for getting eth1 deposits * Fix duplicate rand dep * Remove dead code * Remove accidentally-added files * Fix comment in eth1_genesis_service * Add .gitignore for eth1_test_rig * Fix bug in eth1_genesis_service * Remove dead code from eth2_config * Fix tabs/spaces in root Cargo.toml * Tidy eth1 crate * Allow for re-use of eth1 service after genesis * Update docs for new CLI * Change README gif * Tidy eth1 http module * Tidy eth1 service * Tidy environment crate * Remove unused file * Tidy, add comments * Remove commented-out code * Address majority of Michael's comments * Address other PR comments * Add link to issue alongside TODO
2026-05-30 12:47:05 +00:00 · 2019-11-15 14:47:51 +11:00
parent 97729f8654
commit f229bbba1c
99 changed files with 8263 additions and 1631 deletions
--- a/lighthouse/environment/Cargo.toml
+++ b/lighthouse/environment/Cargo.toml
@@ -0,0 +1,19 @@
+[package]
+name = "environment"
+version = "0.1.0"
+authors = ["Paul Hauner <paul@paulhauner.com>"]
+edition = "2018"
+
+[dependencies]
+tokio = "0.1.15"
+slog = { version = "^2.2.3" , features = ["max_level_trace"] }
+sloggers = "0.3.4"
+types = { "path" = "../../eth2/types" }
+eth2_config = { "path" = "../../eth2/utils/eth2_config" }
+env_logger = "0.6.1"
+logging = { path = "../../eth2/utils/logging" }
+slog-term = "^2.4.0"
+slog-async = "^2.3.0"
+ctrlc = { version = "3.1.1", features = ["termination"] }
+futures = "0.1.25"
+parking_lot = "0.7"
--- a/lighthouse/environment/src/lib.rs
+++ b/lighthouse/environment/src/lib.rs
@@ -0,0 +1,241 @@
+//! This crate aims to provide a common set of tools that can be used to create a "environment" to
+//! run Lighthouse services like the `beacon_node` or `validator_client`. This allows for the
+//! unification of creating tokio runtimes, loggers and eth2 specifications in production and in
+//! testing.
+//!
+//! The idea is that the main thread creates an `Environment`, which is then used to spawn a
+//! `Context` which can be handed to any service that wishes to start async tasks or perform
+//! logging.
+
+use eth2_config::Eth2Config;
+use futures::{sync::oneshot, Future};
+use slog::{o, Drain, Level, Logger};
+use sloggers::{null::NullLoggerBuilder, Build};
+use std::cell::RefCell;
+use tokio::runtime::{Builder as RuntimeBuilder, Runtime, TaskExecutor};
+use types::{EthSpec, InteropEthSpec, MainnetEthSpec, MinimalEthSpec};
+
+/// Builds an `Environment`.
+pub struct EnvironmentBuilder<E: EthSpec> {
+    runtime: Option<Runtime>,
+    log: Option<Logger>,
+    eth_spec_instance: E,
+    eth2_config: Eth2Config,
+}
+
+impl EnvironmentBuilder<MinimalEthSpec> {
+    /// Creates a new builder using the `minimal` eth2 specification.
+    pub fn minimal() -> Self {
+        Self {
+            runtime: None,
+            log: None,
+            eth_spec_instance: MinimalEthSpec,
+            eth2_config: Eth2Config::minimal(),
+        }
+    }
+}
+
+impl EnvironmentBuilder<MainnetEthSpec> {
+    /// Creates a new builder using the `mainnet` eth2 specification.
+    pub fn mainnet() -> Self {
+        Self {
+            runtime: None,
+            log: None,
+            eth_spec_instance: MainnetEthSpec,
+            eth2_config: Eth2Config::mainnet(),
+        }
+    }
+}
+
+impl EnvironmentBuilder<InteropEthSpec> {
+    /// Creates a new builder using the `interop` eth2 specification.
+    pub fn interop() -> Self {
+        Self {
+            runtime: None,
+            log: None,
+            eth_spec_instance: InteropEthSpec,
+            eth2_config: Eth2Config::interop(),
+        }
+    }
+}
+
+impl<E: EthSpec> EnvironmentBuilder<E> {
+    /// Specifies that a multi-threaded tokio runtime should be used. Ideal for production uses.
+    ///
+    /// The `Runtime` used is just the standard tokio runtime.
+    pub fn multi_threaded_tokio_runtime(mut self) -> Result<Self, String> {
+        self.runtime =
+            Some(Runtime::new().map_err(|e| format!("Failed to start runtime: {:?}", e))?);
+        Ok(self)
+    }
+
+    /// Specifies that a single-threaded tokio runtime should be used. Ideal for testing purposes
+    /// where tests are already multi-threaded.
+    ///
+    /// This can solve problems if "too many open files" errors are thrown during tests.
+    pub fn single_thread_tokio_runtime(mut self) -> Result<Self, String> {
+        self.runtime = Some(
+            RuntimeBuilder::new()
+                .core_threads(1)
+                .build()
+                .map_err(|e| format!("Failed to start runtime: {:?}", e))?,
+        );
+        Ok(self)
+    }
+
+    /// Specifies that all logs should be sent to `null` (i.e., ignored).
+    pub fn null_logger(mut self) -> Result<Self, String> {
+        self.log = Some(null_logger()?);
+        Ok(self)
+    }
+
+    /// Specifies that the `slog` asynchronous logger should be used. Ideal for production.
+    ///
+    /// The logger is "async" because it has a dedicated thread that accepts logs and then
+    /// asynchronously flushes them to stdout/files/etc. This means the thread that raised the log
+    /// does not have to wait for the logs to be flushed.
+    pub fn async_logger(mut self, debug_level: &str) -> Result<Self, String> {
+        // Build the initial logger.
+        let decorator = slog_term::TermDecorator::new().build();
+        let decorator = logging::AlignedTermDecorator::new(decorator, logging::MAX_MESSAGE_WIDTH);
+        let drain = slog_term::FullFormat::new(decorator).build().fuse();
+        let drain = slog_async::Async::new(drain).build();
+
+        let drain = match debug_level {
+            "info" => drain.filter_level(Level::Info),
+            "debug" => drain.filter_level(Level::Debug),
+            "trace" => drain.filter_level(Level::Trace),
+            "warn" => drain.filter_level(Level::Warning),
+            "error" => drain.filter_level(Level::Error),
+            "crit" => drain.filter_level(Level::Critical),
+            unknown => return Err(format!("Unknown debug-level: {}", unknown)),
+        };
+
+        self.log = Some(Logger::root(drain.fuse(), o!()));
+        Ok(self)
+    }
+
+    /// Consumes the builder, returning an `Environment`.
+    pub fn build(self) -> Result<Environment<E>, String> {
+        Ok(Environment {
+            runtime: self
+                .runtime
+                .ok_or_else(|| "Cannot build environment without runtime".to_string())?,
+            log: self
+                .log
+                .ok_or_else(|| "Cannot build environment without log".to_string())?,
+            eth_spec_instance: self.eth_spec_instance,
+            eth2_config: self.eth2_config,
+        })
+    }
+}
+
+/// An execution context that can be used by a service.
+///
+/// Distinct from an `Environment` because a `Context` is not able to give a mutable reference to a
+/// `Runtime`, instead it only has access to a `TaskExecutor`.
+#[derive(Clone)]
+pub struct RuntimeContext<E: EthSpec> {
+    pub executor: TaskExecutor,
+    pub log: Logger,
+    pub eth_spec_instance: E,
+    pub eth2_config: Eth2Config,
+}
+
+impl<E: EthSpec> RuntimeContext<E> {
+    /// Returns a sub-context of this context.
+    ///
+    /// The generated service will have the `service_name` in all it's logs.
+    pub fn service_context(&self, service_name: &'static str) -> Self {
+        Self {
+            executor: self.executor.clone(),
+            log: self.log.new(o!("service" => service_name)),
+            eth_spec_instance: self.eth_spec_instance.clone(),
+            eth2_config: self.eth2_config.clone(),
+        }
+    }
+
+    /// Returns the `eth2_config` for this service.
+    pub fn eth2_config(&self) -> &Eth2Config {
+        &self.eth2_config
+    }
+}
+
+/// An environment where Lighthouse services can run. Used to start a production beacon node or
+/// validator client, or to run tests that involve logging and async task execution.
+pub struct Environment<E: EthSpec> {
+    runtime: Runtime,
+    log: Logger,
+    eth_spec_instance: E,
+    eth2_config: Eth2Config,
+}
+
+impl<E: EthSpec> Environment<E> {
+    /// Returns a mutable reference to the `tokio` runtime.
+    ///
+    /// Useful in the rare scenarios where it's necessary to block the current thread until a task
+    /// is finished (e.g., during testing).
+    pub fn runtime(&mut self) -> &mut Runtime {
+        &mut self.runtime
+    }
+
+    /// Returns a `Context` where no "service" has been added to the logger output.
+    pub fn core_context(&mut self) -> RuntimeContext<E> {
+        RuntimeContext {
+            executor: self.runtime.executor(),
+            log: self.log.clone(),
+            eth_spec_instance: self.eth_spec_instance.clone(),
+            eth2_config: self.eth2_config.clone(),
+        }
+    }
+
+    /// Returns a `Context` where the `service_name` is added to the logger output.
+    pub fn service_context(&mut self, service_name: &'static str) -> RuntimeContext<E> {
+        RuntimeContext {
+            executor: self.runtime.executor(),
+            log: self.log.new(o!("service" => service_name)),
+            eth_spec_instance: self.eth_spec_instance.clone(),
+            eth2_config: self.eth2_config.clone(),
+        }
+    }
+
+    /// Block the current thread until Ctrl+C is received.
+    pub fn block_until_ctrl_c(&mut self) -> Result<(), String> {
+        let (ctrlc_send, ctrlc_oneshot) = oneshot::channel();
+        let ctrlc_send_c = RefCell::new(Some(ctrlc_send));
+        ctrlc::set_handler(move || {
+            if let Some(ctrlc_send) = ctrlc_send_c.try_borrow_mut().unwrap().take() {
+                ctrlc_send.send(()).expect("Error sending ctrl-c message");
+            }
+        })
+        .map_err(|e| format!("Could not set ctrlc handler: {:?}", e))?;
+
+        // Block this thread until Crtl+C is pressed.
+        self.runtime()
+            .block_on(ctrlc_oneshot)
+            .map_err(|e| format!("Ctrlc oneshot failed: {:?}", e))
+    }
+
+    /// Shutdown the `tokio` runtime when all tasks are idle.
+    pub fn shutdown_on_idle(self) -> Result<(), String> {
+        self.runtime
+            .shutdown_on_idle()
+            .wait()
+            .map_err(|e| format!("Tokio runtime shutdown returned an error: {:?}", e))
+    }
+
+    pub fn eth_spec_instance(&self) -> &E {
+        &self.eth_spec_instance
+    }
+
+    pub fn eth2_config(&self) -> &Eth2Config {
+        &self.eth2_config
+    }
+}
+
+pub fn null_logger() -> Result<Logger, String> {
+    let log_builder = NullLoggerBuilder;
+    log_builder
+        .build()
+        .map_err(|e| format!("Failed to start null logger: {:?}", e))
+}