Prepare for public testnet (#628)

* Update to spec v0.9.0 * Update to v0.9.1 * Bump spec tags for v0.9.1 * Formatting, fix CI failures * Resolve accidental KeyPair merge conflict * Document new BeaconState functions * Add `validator` changes from `validator-to-rest` * Add initial (failing) REST api tests * Fix signature parsing * Add more tests * Refactor http router * Add working tests for publish beacon block * Add validator duties tests * Move account_manager under `lighthouse` binary * Unify logfile handling in `environment` crate. * Fix incorrect cache drops in `advance_caches` * Update fork choice for v0.9.1 * Add `deposit_contract` crate * Add progress on validator onboarding * Add unfinished attesation code * Update account manager CLI * Write eth1 data file as hex string * Integrate ValidatorDirectory with validator_client * Move ValidatorDirectory into validator_client * Clean up some FIXMEs * Add beacon_chain_sim * Fix a few docs/logs * Expand `beacon_chain_sim` * Fix spec for `beacon_chain_sim * More testing for api * Start work on attestation endpoint * Reject empty attestations * Allow attestations to genesis block * Add working tests for `rest_api` validator endpoint * Remove grpc from beacon_node * Start heavy refactor of validator client - Block production is working * Prune old validator client files * Start works on attestation service * Add attestation service to validator client * Use full pubkey for validator directories * Add validator duties post endpoint * Use par_iter for keypair generation * Use bulk duties request in validator client * Add version http endpoint tests * Add interop keys and startup wait * Ensure a prompt exit * Add duties pruning * Fix compile error in beacon node tests * Add github workflow * Modify rust.yaml * Modify gitlab actions * Add to CI file * Add sudo to CI npm install * Move cargo fmt to own job in tests * Fix cargo fmt in CI * Add rustup update before cargo fmt * Change name of CI job * Make other CI jobs require cargo fmt * Add CI badge * Remove gitlab and travis files * Add different http timeout for debug * Update docker file, use makefile in CI * Use make in the dockerfile, skip the test * Use the makefile for debug GI test * Update book * Tidy grpc and misc things * Apply discv5 fixes * Address other minor issues * Fix warnings * Attempt fix for addr parsing * Tidy validator config, CLIs * Tidy comments * Tidy signing, reduce ForkService duplication * Fail if skipping too many slots * Set default recent genesis time to 0 * Add custom http timeout to validator * Fix compile bug in node_test_rig * Remove old bootstrap flag from val CLI * Update docs * Tidy val client * Change val client log levels * Add comments, more validity checks * Fix compile error, add comments * Undo changes to eth2-libp2p/src * Reduce duplication of keypair generation * Add more logging for validator duties * Fix beacon_chain_sim, nitpicks * Fix compile error, minor nits * Update to use v0.9.2 version of deposit contract * Add efforts to automate eth1 testnet deployment * Fix lcli testnet deployer * Modify bn CLI to parse eth2_testnet_dir * Progress with account_manager deposit tools * Make account manager submit deposits * Add password option for submitting deposits * Allow custom deposit amount * Add long names to lcli clap * Add password option to lcli deploy command * Fix minor bugs whilst testing * Address Michael's comments * Add refund-deposit-contract to lcli * Use time instead of skip count for denying long skips * Improve logging for eth1 * Fix bug with validator services exiting on error * Drop the block cache after genesis * Modify eth1 testnet config * Improve eth1 logging * Make validator wait until genesis time * Fix bug in eth1 voting * Add more logging to eth1 voting * Handle errors in eth1 http module * Set SECONDS_PER_DAY to sensible minimum * Shorten delay before testnet start * Ensure eth1 block is produced without any votes * Improve eth1 logging * Fix broken tests in eth1 * Tidy code in rest_api * Fix failing test in deposit_contract * Make CLI args more consistent * Change validator/duties endpoint * Add time-based skip slot limiting * Add new error type missed in previous commit * Add log when waiting for genesis * Refactor beacon node CLI * Remove unused dep * Add lcli eth1-genesis command * Fix bug in master merge * Apply clippy lints to beacon node * Add support for YamlConfig in Eth2TestnetDir * Upgrade tesnet deposit contract version * Remove unnecessary logging and correct formatting * Add a hardcoded eth2 testnet config * Ensure http server flag works. Overwrite configs with flags. * Ensure boot nodes are loaded from testnet dir * Fix account manager CLI bugs * Fix bugs with beacon node cli * Allow testnet dir without boot nodes * Write genesis state as SSZ * Remove ---/n from the start of testnet_dir files * Set default libp2p address * Tidy account manager CLI, add logging * Add check to see if testnet dir exists * Apply reviewers suggestions * Add HeadTracker struct * Add fork choice persistence * Shorten slot time for simulator * Add the /beacon/heads API endpoint * Update hardcoded testnet * Add tests for BeaconChain persistence + fix bugs * Extend BeaconChain persistence testing * Ensure chain is finalized b4 persistence tests * Ensure boot_enr.yaml is include in binary * Refactor beacon_chain_sim * Move files about in beacon sim * Update beacon_chain_sim * Fix bug with deposit inclusion * Increase log in genesis service, fix todo * Tidy sim, fix broken rest_api tests * Fix more broken tests * Update testnet * Fix broken rest api test * Tidy account manager CLI * Use tempdir for account manager * Stop hardcoded testnet dir from creating dir * Rename Eth2TestnetDir to Eth2TestnetConfig * Change hardcoded -> hard_coded * Tidy account manager * Add log to account manager * Tidy, ensure head tracker is loaded from disk * Tidy beacon chain builder * Tidy eth1_chain * Adds log support for simulator * Revert "Adds log support for simulator" This reverts commit ec77c66a05. * Adds log support for simulator * Tidy after self-review * Change default log level * Address Michael's delicious PR comments * Fix off-by-one in tests
2026-03-10 12:11:59 +00:00 · 2019-12-03 15:28:57 +11:00
parent 4b4bc6247d
commit a0549e3842
74 changed files with 3421 additions and 1298 deletions
--- a/eth2/utils/eth2_testnet_config/src/lib.rs
+++ b/eth2/utils/eth2_testnet_config/src/lib.rs
@@ -0,0 +1,260 @@
+//! This crate should eventually represent the structure at this repo:
+//!
+//! https://github.com/eth2-clients/eth2-testnets/tree/master/nimbus/testnet1
+//!
+//! It is not accurate at the moment, we include extra files and we also don't support a few
+//! others. We are unable to confirm to the repo until we have the following PR merged:
+//!
+//! https://github.com/sigp/lighthouse/pull/605
+
+use eth2_libp2p::Enr;
+use ssz::{Decode, Encode};
+use std::fs::{create_dir_all, File};
+use std::io::{Read, Write};
+use std::path::PathBuf;
+use types::{Address, BeaconState, EthSpec, YamlConfig};
+
+pub const ADDRESS_FILE: &str = "deposit_contract.txt";
+pub const DEPLOY_BLOCK_FILE: &str = "deploy_block.txt";
+pub const BOOT_ENR_FILE: &str = "boot_enr.yaml";
+pub const GENESIS_STATE_FILE: &str = "genesis.ssz";
+pub const YAML_CONFIG_FILE: &str = "config.yaml";
+
+pub const HARDCODED_YAML_CONFIG: &[u8] = include_bytes!("../testnet/config.yaml");
+pub const HARDCODED_DEPLOY_BLOCK: &[u8] = include_bytes!("../testnet/deploy_block.txt");
+pub const HARDCODED_DEPOSIT_CONTRACT: &[u8] = include_bytes!("../testnet/deposit_contract.txt");
+pub const HARDCODED_GENESIS_STATE: &[u8] = include_bytes!("../testnet/genesis.ssz");
+pub const HARDCODED_BOOT_ENR: &[u8] = include_bytes!("../testnet/boot_enr.yaml");
+
+/// Specifies an Eth2 testnet.
+///
+/// See the crate-level documentation for more details.
+#[derive(Clone, PartialEq, Debug)]
+pub struct Eth2TestnetConfig<E: EthSpec> {
+    pub deposit_contract_address: String,
+    pub deposit_contract_deploy_block: u64,
+    pub boot_enr: Option<Vec<Enr>>,
+    pub genesis_state: Option<BeaconState<E>>,
+    pub yaml_config: Option<YamlConfig>,
+}
+
+impl<E: EthSpec> Eth2TestnetConfig<E> {
+    // Creates the `Eth2TestnetConfig` that was included in the binary at compile time. This can be
+    // considered the default Lighthouse testnet.
+    //
+    // Returns an error if those included bytes are invalid (this is unlikely).
+    pub fn hard_coded() -> Result<Self, String> {
+        Ok(Self {
+            deposit_contract_address: serde_yaml::from_reader(HARDCODED_DEPOSIT_CONTRACT)
+                .map_err(|e| format!("Unable to parse contract address: {:?}", e))?,
+            deposit_contract_deploy_block: serde_yaml::from_reader(HARDCODED_DEPLOY_BLOCK)
+                .map_err(|e| format!("Unable to parse deploy block: {:?}", e))?,
+            boot_enr: Some(
+                serde_yaml::from_reader(HARDCODED_BOOT_ENR)
+                    .map_err(|e| format!("Unable to parse boot enr: {:?}", e))?,
+            ),
+            genesis_state: Some(
+                BeaconState::from_ssz_bytes(HARDCODED_GENESIS_STATE)
+                    .map_err(|e| format!("Unable to parse genesis state: {:?}", e))?,
+            ),
+            yaml_config: Some(
+                serde_yaml::from_reader(HARDCODED_YAML_CONFIG)
+                    .map_err(|e| format!("Unable to parse genesis state: {:?}", e))?,
+            ),
+        })
+    }
+
+    // Write the files to the directory, only if the directory doesn't already exist.
+    pub fn write_to_file(&self, base_dir: PathBuf) -> Result<(), String> {
+        if base_dir.exists() {
+            return Err("Testnet directory already exists".to_string());
+        }
+
+        self.force_write_to_file(base_dir)
+    }
+
+    // Write the files to the directory, even if the directory already exists.
+    pub fn force_write_to_file(&self, base_dir: PathBuf) -> Result<(), String> {
+        create_dir_all(&base_dir)
+            .map_err(|e| format!("Unable to create testnet directory: {:?}", e))?;
+
+        macro_rules! write_to_yaml_file {
+            ($file: ident, $variable: expr) => {
+                File::create(base_dir.join($file))
+                    .map_err(|e| format!("Unable to create {}: {:?}", $file, e))
+                    .and_then(|mut file| {
+                        let yaml = serde_yaml::to_string(&$variable)
+                            .map_err(|e| format!("Unable to YAML encode {}: {:?}", $file, e))?;
+
+                        // Remove the doc header from the YAML file.
+                        //
+                        // This allows us to play nice with other clients that are expecting
+                        // plain-text, not YAML.
+                        let no_doc_header = if yaml.starts_with("---\n") {
+                            &yaml[4..]
+                        } else {
+                            &yaml
+                        };
+
+                        file.write_all(no_doc_header.as_bytes())
+                            .map_err(|e| format!("Unable to write {}: {:?}", $file, e))
+                    })?;
+            };
+        }
+
+        write_to_yaml_file!(ADDRESS_FILE, self.deposit_contract_address);
+        write_to_yaml_file!(DEPLOY_BLOCK_FILE, self.deposit_contract_deploy_block);
+
+        if let Some(boot_enr) = &self.boot_enr {
+            write_to_yaml_file!(BOOT_ENR_FILE, boot_enr);
+        }
+
+        if let Some(yaml_config) = &self.yaml_config {
+            write_to_yaml_file!(YAML_CONFIG_FILE, yaml_config);
+        }
+
+        // The genesis state is a special case because it uses SSZ, not YAML.
+        if let Some(genesis_state) = &self.genesis_state {
+            let file = base_dir.join(GENESIS_STATE_FILE);
+
+            File::create(&file)
+                .map_err(|e| format!("Unable to create {:?}: {:?}", file, e))
+                .and_then(|mut file| {
+                    file.write_all(&genesis_state.as_ssz_bytes())
+                        .map_err(|e| format!("Unable to write {:?}: {:?}", file, e))
+                })?;
+        }
+
+        Ok(())
+    }
+
+    pub fn load(base_dir: PathBuf) -> Result<Self, String> {
+        macro_rules! load_from_file {
+            ($file: ident) => {
+                File::open(base_dir.join($file))
+                    .map_err(|e| format!("Unable to open {}: {:?}", $file, e))
+                    .and_then(|file| {
+                        serde_yaml::from_reader(file)
+                            .map_err(|e| format!("Unable to parse {}: {:?}", $file, e))
+                    })?;
+            };
+        }
+
+        macro_rules! optional_load_from_file {
+            ($file: ident) => {
+                if base_dir.join($file).exists() {
+                    Some(load_from_file!($file))
+                } else {
+                    None
+                }
+            };
+        }
+
+        let deposit_contract_address = load_from_file!(ADDRESS_FILE);
+        let deposit_contract_deploy_block = load_from_file!(DEPLOY_BLOCK_FILE);
+        let boot_enr = optional_load_from_file!(BOOT_ENR_FILE);
+        let yaml_config = optional_load_from_file!(YAML_CONFIG_FILE);
+
+        // The genesis state is a special case because it uses SSZ, not YAML.
+        let file = base_dir.join(GENESIS_STATE_FILE);
+        let genesis_state = if base_dir.join(&file).exists() {
+            Some(
+                File::open(base_dir.join(&file))
+                    .map_err(|e| format!("Unable to open {:?}: {:?}", file, e))
+                    .and_then(|mut file| {
+                        let mut bytes = vec![];
+                        file.read_to_end(&mut bytes)
+                            .map_err(|e| format!("Unable to read {:?}: {:?}", file, e))?;
+
+                        BeaconState::from_ssz_bytes(&bytes)
+                            .map_err(|e| format!("Unable to SSZ decode {:?}: {:?}", file, e))
+                    })?,
+            )
+        } else {
+            None
+        };
+
+        Ok(Self {
+            deposit_contract_address,
+            deposit_contract_deploy_block,
+            boot_enr,
+            genesis_state,
+            yaml_config,
+        })
+    }
+
+    pub fn deposit_contract_address(&self) -> Result<Address, String> {
+        if self.deposit_contract_address.starts_with("0x") {
+            self.deposit_contract_address[2..]
+                .parse()
+                .map_err(|e| format!("Corrupted address, unable to parse: {:?}", e))
+        } else {
+            Err("Corrupted address, must start with 0x".to_string())
+        }
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use tempdir::TempDir;
+    use types::{Eth1Data, Hash256, MinimalEthSpec, YamlConfig};
+
+    type E = MinimalEthSpec;
+
+    #[test]
+    fn hard_coded_works() {
+        let dir: Eth2TestnetConfig<E> =
+            Eth2TestnetConfig::hard_coded().expect("should decode hard_coded params");
+
+        assert!(dir.boot_enr.is_some());
+        assert!(dir.genesis_state.is_some());
+        assert!(dir.yaml_config.is_some());
+    }
+
+    #[test]
+    fn round_trip() {
+        let spec = &E::default_spec();
+
+        let eth1_data = Eth1Data {
+            deposit_root: Hash256::zero(),
+            deposit_count: 0,
+            block_hash: Hash256::zero(),
+        };
+
+        // TODO: figure out how to generate ENR and add some here.
+        let boot_enr = None;
+        let genesis_state = Some(BeaconState::new(42, eth1_data, spec));
+        let yaml_config = Some(YamlConfig::from_spec::<E>(spec));
+
+        do_test::<E>(boot_enr, genesis_state.clone(), yaml_config.clone());
+        do_test::<E>(None, None, None);
+    }
+
+    fn do_test<E: EthSpec>(
+        boot_enr: Option<Vec<Enr>>,
+        genesis_state: Option<BeaconState<E>>,
+        yaml_config: Option<YamlConfig>,
+    ) {
+        let temp_dir = TempDir::new("eth2_testnet_test").expect("should create temp dir");
+        let base_dir = PathBuf::from(temp_dir.path().join("my_testnet"));
+        let deposit_contract_address = "0xBB9bc244D798123fDe783fCc1C72d3Bb8C189413".to_string();
+        let deposit_contract_deploy_block = 42;
+
+        let testnet: Eth2TestnetConfig<E> = Eth2TestnetConfig {
+            deposit_contract_address: deposit_contract_address.clone(),
+            deposit_contract_deploy_block: deposit_contract_deploy_block,
+            boot_enr,
+            genesis_state,
+            yaml_config,
+        };
+
+        testnet
+            .write_to_file(base_dir.clone())
+            .expect("should write to file");
+
+        let decoded = Eth2TestnetConfig::load(base_dir).expect("should load struct");
+
+        assert_eq!(testnet, decoded, "should decode as encoded");
+    }
+}