Eth1 Integration (#542)

* Refactor to cache Eth1Data * Fix merge conflicts and minor refactorings * Rename Eth1Cache to Eth1DataCache * Refactor events subscription * Add deposits module to interface with BeaconChain deposits * Remove utils * Rename to types.rs and add trait constraints to Eth1DataFetcher * Confirm to trait constraints. Make Web3DataFetcher cloneable * Make fetcher object member of deposit and eth1_data cache and other fixes * Fix update_cache function * Move fetch_eth1_data to impl block * Fix deposit tests * Create Eth1 object for interfacing with Beacon chain * Add `run` function for running update_cache and subscribe_deposit_logs tasks * Add logging * Run `cargo fmt` and make tests pass * Convert sync functions to async * Add timeouts to web3 functions * Return futures from cache functions * Add failed chaining of futures * Working cache updation * Clean up tests and `update_cache` function * Refactor `get_eth1_data` functions to work with future returning functions * Refactor eth1 `run` function to work with modified `update_cache` api * Minor changes * Add distance parameter to `update_cache` * Fix tests and other minor fixes * Working integration with cache and deposits * Add merkle_tree construction, proof generation and verification code * Add function to construct and fetch Deposits for BeaconNode * Add error handling * Import ssz * Add error handling to eth1 cache and fix minor errors * Run rustfmt * Fix minor bug * Rename Eth1Error and change to Result<T> * Change deposit fetching mechanism from notification based to poll based * Add deposits from eth1 chain in a given range every `x` blocks * Modify `run` function to accommodate changes * Minor fixes * Fix formatting * Initial commit. web3 api working. * Tidied up lib. Add function for fetching logs. * Refactor with `Eth1DataFetcher` trait * Add parsing for deposit contract logs and get_eth1_data function * Add `get_eth1_votes` function * Refactor to cache Eth1Data * Fix merge conflicts and minor refactorings * Rename Eth1Cache to Eth1DataCache * Refactor events subscription * Add deposits module to interface with BeaconChain deposits * Remove utils * Rename to types.rs and add trait constraints to Eth1DataFetcher * Confirm to trait constraints. Make Web3DataFetcher cloneable * Make fetcher object member of deposit and eth1_data cache and other fixes * Fix update_cache function * Move fetch_eth1_data to impl block * Fix deposit tests * Create Eth1 object for interfacing with Beacon chain * Add `run` function for running update_cache and subscribe_deposit_logs tasks * Add logging * Run `cargo fmt` and make tests pass * Convert sync functions to async * Add timeouts to web3 functions * Return futures from cache functions * Add failed chaining of futures * Working cache updation * Clean up tests and `update_cache` function * Refactor `get_eth1_data` functions to work with future returning functions * Refactor eth1 `run` function to work with modified `update_cache` api * Minor changes * Add distance parameter to `update_cache` * Fix tests and other minor fixes * Working integration with cache and deposits * Add merkle_tree construction, proof generation and verification code * Add function to construct and fetch Deposits for BeaconNode * Add error handling * Import ssz * Add error handling to eth1 cache and fix minor errors * Run rustfmt * Fix minor bug * Rename Eth1Error and change to Result<T> * Change deposit fetching mechanism from notification based to poll based * Add deposits from eth1 chain in a given range every `x` blocks * Modify `run` function to accommodate changes * Minor fixes * Fix formatting * Fix merge issue * Refactor with `Config` struct. Remote `ContractConfig` * Rename eth1_chain crate to eth1 * Rename files and read abi file using `fs::read` * Move eth1 to lib * Remove unnecessary mutability constraint * Add `Web3Backend` for returning actual eth1 data * Refactor `get_eth1_votes` to return a Result * Delete `eth1_chain` crate * Return `Result` from `get_deposits` * Fix range of deposits to return to beacon chain * Add `get_block_height_by_hash` trait function * Add naive method for getting `previous_eth1_distance` * Add eth1 config params to main config * Add instructions for setting up eth1 testing environment * Add build script to fetch deposit contract abi * Contract ABI is part of compiled binary * Fix minor bugs * Move docs to lib * Add timeout to config * Remove print statements * Change warn to error * Fix typos * Removed prints in test and get timeout value from config * Fixed error types * Added logging to web3_fetcher * Refactor for modified web3 api * Fix minor stuff * Add build script * Tidy, hide eth1 integration tests behind flag * Add http crate * Add first stages of eth1_test_rig * Fix deposits on test rig * Fix bug with deposit count method * Add block hash getter to http eth1 * Clean eth1 http crate and tests * Add script to start ganache * Adds deposit tree to eth1-http * Extend deposit tree tests * Tidy tests in eth1-http * Add more detail to get block request * Add block cache to eth1-http * Rename deposit tree to deposit cache * Add inital updating to eth1-http * Tidy updater * Fix compile bugs in tests * Adds an Eth1DataCache builder * Reorg eth1-http files * Add (failing) tests for eth1 updater * Rename files, fix bug in eth1-http * Ensure that ganache timestamps are increasing * Fix bugs with getting eth1data ancestors * Improve eth1 testing, fix bugs * Add truncate method to block cache * Add pruning to block cache update process * Add tests for block pruning * Allow for dropping an expired cache. * Add more comments * Add first compiling version of deposit updater * Add common fn for getting range of required blocks * Add passing deposit update test * Improve tests * Fix block pruning bug * Add tests for running two updates at once * Add updater services to eth1 * Add deposit collection to beacon chain * Add incomplete builder experiments * Add first working version of beacon chain builder * Update test harness to new beacon chain type * Rename builder file, tidy * Add first working client builder * Progress further on client builder * Update becaon node binary to use client builder * Ensure release tests compile * Remove old eth1 crate * Add first pass of new lighthouse binary * Fix websocket server startup * Remove old binary code from beacon_node crate * Add first working beacon node tests * Add genesis crate, new eth1 cache_2 * Add Serivce to Eth1Cache * Refactor with general eth1 improvements * Add passing genesis test * Tidy, add comments * Add more comments to eth1 service * Add further eth1 progress * Fix some bugs with genesis * Fix eth1 bugs, make eth1 linking more efficient * Shift logic in genesis service * Add more comments to genesis service * Add gzip, max request values, timeouts to http * Update testnet parameters to suit goerli testnet * Add ability to vary Fork, fix custom spec * Be more explicit about deposit fork version * Start adding beacon chain eth1 option * Add more flexibility to prod client * Further runtime refactoring * Allow for starting from store * Add bootstrapping to client config * Add remote_beacon_node crate * Update eth1 service for more configurability * Update eth1 tests to use less runtimes * Patch issues with tests using too many files * Move dummy eth1 backend flag * Ensure all tests pass * Add ganache-cli to Dockerfile * Use a special docker hub image for testing * Appease clippy * Move validator client into lighthouse binary * Allow starting with dummy eth1 backend * Improve logging * Fix dummy eth1 backend from cli * Add extra testnet command * Ensure consistent spec in beacon node * Update eth1 rig to work on goerli * Tidy lcli, start adding support for yaml config * Add incomplete YamlConfig struct * Remove efforts at YamlConfig * Add incomplete eth1 voting. Blocked on spec issues * Add (untested) first pass at eth1 vote algo * Add tests for winning vote * Add more tests for eth1 chain * Add more eth1 voting tests * Added more eth1 voting testing * Change test name * Add more tests to eth1 chain * Tidy eth1 generics, add more tests * Improve comments * Tidy beacon_node tests * Tidy, rename JsonRpc.. to Caching.. * Tidy voting logic * Tidy builder docs * Add comments, tidy eth1 * Add more comments to eth1 * Fix bug with winning_vote * Add doc comments to the `ClientBuilder` * Remove commented-out code * Improve `ClientBuilder` docs * Add comments to client config * Add decoding test for `ClientConfig` * Remove unused `DepositSet` struct * Tidy `block_cache` * Remove commented out lines * Remove unused code in `eth1` crate * Remove old validator binary `main.rs` * Tidy, fix tests compile error * Add initial tests for get_deposits * Remove dead code in eth1_test_rig * Update TestingDepositBuilder * Add testing for getting eth1 deposits * Fix duplicate rand dep * Remove dead code * Remove accidentally-added files * Fix comment in eth1_genesis_service * Add .gitignore for eth1_test_rig * Fix bug in eth1_genesis_service * Remove dead code from eth2_config * Fix tabs/spaces in root Cargo.toml * Tidy eth1 crate * Allow for re-use of eth1 service after genesis * Update docs for new CLI * Change README gif * Tidy eth1 http module * Tidy eth1 service * Tidy environment crate * Remove unused file * Tidy, add comments * Remove commented-out code * Address majority of Michael's comments * Address other PR comments * Add link to issue alongside TODO
2026-03-14 02:12:33 +00:00 · 2019-11-15 14:47:51 +11:00
parent 97729f8654
commit f229bbba1c
99 changed files with 8263 additions and 1631 deletions
--- a/beacon_node/eth1/src/block_cache.rs
+++ b/beacon_node/eth1/src/block_cache.rs
@@ -0,0 +1,271 @@
+use std::ops::RangeInclusive;
+use types::{Eth1Data, Hash256};
+
+#[derive(Debug, PartialEq, Clone)]
+pub enum Error {
+    /// The timestamp of each block equal to or later than the block prior to it.
+    InconsistentTimestamp { parent: u64, child: u64 },
+    /// Some `Eth1Block` was provided with the same block number but different data. The source
+    /// of eth1 data is inconsistent.
+    Conflicting(u64),
+    /// The given block was not one block number higher than the higest known block number.
+    NonConsecutive { given: u64, expected: u64 },
+    /// Some invariant was violated, there is a likely bug in the code.
+    Internal(String),
+}
+
+/// A block of the eth1 chain.
+///
+/// Contains all information required to add a `BlockCache` entry.
+#[derive(Debug, PartialEq, Clone, Eq, Hash)]
+pub struct Eth1Block {
+    pub hash: Hash256,
+    pub timestamp: u64,
+    pub number: u64,
+    pub deposit_root: Option<Hash256>,
+    pub deposit_count: Option<u64>,
+}
+
+impl Eth1Block {
+    pub fn eth1_data(self) -> Option<Eth1Data> {
+        Some(Eth1Data {
+            deposit_root: self.deposit_root?,
+            deposit_count: self.deposit_count?,
+            block_hash: self.hash,
+        })
+    }
+}
+
+/// Stores block and deposit contract information and provides queries based upon the block
+/// timestamp.
+#[derive(Debug, PartialEq, Clone, Default)]
+pub struct BlockCache {
+    blocks: Vec<Eth1Block>,
+}
+
+impl BlockCache {
+    /// Returns the number of blocks stored in `self`.
+    pub fn len(&self) -> usize {
+        self.blocks.len()
+    }
+
+    /// True if the cache does not store any blocks.
+    pub fn is_empty(&self) -> bool {
+        self.blocks.is_empty()
+    }
+
+    /// Returns the highest block number stored.
+    pub fn highest_block_number(&self) -> Option<u64> {
+        self.blocks.last().map(|block| block.number)
+    }
+
+    /// Returns an iterator over all blocks.
+    ///
+    /// Blocks a guaranteed to be returned with;
+    ///
+    /// - Monotonically increasing block numbers.
+    /// - Non-uniformly increasing block timestamps.
+    pub fn iter(&self) -> impl DoubleEndedIterator<Item = &Eth1Block> + Clone {
+        self.blocks.iter()
+    }
+
+    /// Shortens the cache, keeping the latest (by block number) `len` blocks while dropping the
+    /// rest.
+    ///
+    /// If `len` is greater than the vector's current length, this has no effect.
+    pub fn truncate(&mut self, len: usize) {
+        if len < self.blocks.len() {
+            self.blocks = self.blocks.split_off(self.blocks.len() - len);
+        }
+    }
+
+    /// Returns the range of block numbers stored in the block cache. All blocks in this range can
+    /// be accessed.
+    fn available_block_numbers(&self) -> Option<RangeInclusive<u64>> {
+        Some(self.blocks.first()?.number..=self.blocks.last()?.number)
+    }
+
+    /// Returns a block with the corresponding number, if any.
+    pub fn block_by_number(&self, block_number: u64) -> Option<&Eth1Block> {
+        self.blocks.get(
+            self.blocks
+                .as_slice()
+                .binary_search_by(|block| block.number.cmp(&block_number))
+                .ok()?,
+        )
+    }
+
+    /// Insert an `Eth1Snapshot` into `self`, allowing future queries.
+    ///
+    /// Allows inserting either:
+    ///
+    /// - The root block (i.e., any block if there are no existing blocks), or,
+    /// - An immediate child of the most recent (highest block number) block.
+    ///
+    /// ## Errors
+    ///
+    /// - If the cache is not empty and `item.block.block_number - 1` is not already in `self`.
+    /// - If `item.block.block_number` is in `self`, but is not identical to the supplied
+    /// `Eth1Snapshot`.
+    /// - If `item.block.timestamp` is prior to the parent.
+    pub fn insert_root_or_child(&mut self, block: Eth1Block) -> Result<(), Error> {
+        let expected_block_number = self
+            .highest_block_number()
+            .map(|n| n + 1)
+            .unwrap_or_else(|| block.number);
+
+        // If there are already some cached blocks, check to see if the new block number is one of
+        // them.
+        //
+        // If the block is already known, check to see the given block is identical to it. If not,
+        // raise an inconsistency error. This is mostly likely caused by some fork on the eth1
+        // chain.
+        if let Some(local) = self.available_block_numbers() {
+            if local.contains(&block.number) {
+                let known_block = self.block_by_number(block.number).ok_or_else(|| {
+                    Error::Internal("An expected block was not present".to_string())
+                })?;
+
+                if known_block == &block {
+                    return Ok(());
+                } else {
+                    return Err(Error::Conflicting(block.number));
+                };
+            }
+        }
+
+        // Only permit blocks when it's either:
+        //
+        // - The first block inserted.
+        // - Exactly one block number higher than the highest known block number.
+        if block.number != expected_block_number {
+            return Err(Error::NonConsecutive {
+                given: block.number,
+                expected: expected_block_number,
+            });
+        }
+
+        // If the block is not the first block inserted, ensure that its timestamp is not higher
+        // than its parents.
+        if let Some(previous_block) = self.blocks.last() {
+            if previous_block.timestamp > block.timestamp {
+                return Err(Error::InconsistentTimestamp {
+                    parent: previous_block.timestamp,
+                    child: block.timestamp,
+                });
+            }
+        }
+
+        self.blocks.push(block);
+
+        Ok(())
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    fn get_block(i: u64, interval_secs: u64) -> Eth1Block {
+        Eth1Block {
+            hash: Hash256::from_low_u64_be(i),
+            timestamp: i * interval_secs,
+            number: i,
+            deposit_root: Some(Hash256::from_low_u64_be(i << 32)),
+            deposit_count: Some(i),
+        }
+    }
+
+    fn get_blocks(n: usize, interval_secs: u64) -> Vec<Eth1Block> {
+        (0..n as u64)
+            .into_iter()
+            .map(|i| get_block(i, interval_secs))
+            .collect()
+    }
+
+    fn insert(cache: &mut BlockCache, s: Eth1Block) -> Result<(), Error> {
+        cache.insert_root_or_child(s)
+    }
+
+    #[test]
+    fn truncate() {
+        let n = 16;
+        let blocks = get_blocks(n, 10);
+
+        let mut cache = BlockCache::default();
+
+        for block in blocks {
+            insert(&mut cache, block.clone()).expect("should add consecutive blocks");
+        }
+
+        for len in vec![0, 1, 2, 3, 4, 8, 15, 16] {
+            let mut cache = cache.clone();
+
+            cache.truncate(len);
+
+            assert_eq!(
+                cache.blocks.len(),
+                len,
+                "should truncate to length: {}",
+                len
+            );
+        }
+
+        let mut cache_2 = cache.clone();
+        cache_2.truncate(17);
+        assert_eq!(
+            cache_2.blocks.len(),
+            n,
+            "truncate to larger than n should be a no-op"
+        );
+    }
+
+    #[test]
+    fn inserts() {
+        let n = 16;
+        let blocks = get_blocks(n, 10);
+
+        let mut cache = BlockCache::default();
+
+        for block in blocks {
+            insert(&mut cache, block.clone()).expect("should add consecutive blocks");
+        }
+
+        // No error for re-adding a block identical to one that exists.
+        assert!(insert(&mut cache, get_block(n as u64 - 1, 10)).is_ok());
+
+        // Error for re-adding a block that is different to the one that exists.
+        assert!(insert(&mut cache, get_block(n as u64 - 1, 11)).is_err());
+
+        // Error for adding non-consecutive blocks.
+        assert!(insert(&mut cache, get_block(n as u64 + 1, 10)).is_err());
+        assert!(insert(&mut cache, get_block(n as u64 + 2, 10)).is_err());
+
+        // Error for adding timestamp prior to previous.
+        assert!(insert(&mut cache, get_block(n as u64, 1)).is_err());
+        // Double check to make sure previous test was only affected by timestamp.
+        assert!(insert(&mut cache, get_block(n as u64, 10)).is_ok());
+    }
+
+    #[test]
+    fn duplicate_timestamp() {
+        let mut blocks = get_blocks(7, 10);
+
+        blocks[0].timestamp = 0;
+        blocks[1].timestamp = 10;
+        blocks[2].timestamp = 10;
+        blocks[3].timestamp = 20;
+        blocks[4].timestamp = 30;
+        blocks[5].timestamp = 40;
+        blocks[6].timestamp = 40;
+
+        let mut cache = BlockCache::default();
+
+        for block in &blocks {
+            insert(&mut cache, block.clone())
+                .expect("should add consecutive blocks with duplicate timestamps");
+        }
+
+        assert_eq!(cache.blocks, blocks, "should have added all blocks");
+    }
+}