gnosis/lighthouse
1
0
Fork 0
mirror of https://github.com/sigp/lighthouse.git synced 2026-03-03 00:31:50 +00:00
Code Issues Packages Projects Releases Wiki Activity

Prepare for public testnet (#628)

Browse Source 
* Update to spec v0.9.0

* Update to v0.9.1

* Bump spec tags for v0.9.1

* Formatting, fix CI failures

* Resolve accidental KeyPair merge conflict

* Document new BeaconState functions

* Add `validator` changes from `validator-to-rest`

* Add initial (failing) REST api tests

* Fix signature parsing

* Add more tests

* Refactor http router

* Add working tests for publish beacon block

* Add validator duties tests

* Move account_manager under `lighthouse` binary

* Unify logfile handling in `environment` crate.

* Fix incorrect cache drops in `advance_caches`

* Update fork choice for v0.9.1

* Add `deposit_contract` crate

* Add progress on validator onboarding

* Add unfinished attesation code

* Update account manager CLI

* Write eth1 data file as hex string

* Integrate ValidatorDirectory with validator_client

* Move ValidatorDirectory into validator_client

* Clean up some FIXMEs

* Add beacon_chain_sim

* Fix a few docs/logs

* Expand `beacon_chain_sim`

* Fix spec for `beacon_chain_sim

* More testing for api

* Start work on attestation endpoint

* Reject empty attestations

* Allow attestations to genesis block

* Add working tests for `rest_api` validator endpoint

* Remove grpc from beacon_node

* Start heavy refactor of validator client

- Block production is working

* Prune old validator client files

* Start works on attestation service

* Add attestation service to validator client

* Use full pubkey for validator directories

* Add validator duties post endpoint

* Use par_iter for keypair generation

* Use bulk duties request in validator client

* Add version http endpoint tests

* Add interop keys and startup wait

* Ensure a prompt exit

* Add duties pruning

* Fix compile error in beacon node tests

* Add github workflow

* Modify rust.yaml

* Modify gitlab actions

* Add to CI file

* Add sudo to CI npm install

* Move cargo fmt to own job in tests

* Fix cargo fmt in CI

* Add rustup update before cargo fmt

* Change name of CI job

* Make other CI jobs require cargo fmt

* Add CI badge

* Remove gitlab and travis files

* Add different http timeout for debug

* Update docker file, use makefile in CI

* Use make in the dockerfile, skip the test

* Use the makefile for debug GI test

* Update book

* Tidy grpc and misc things

* Apply discv5 fixes

* Address other minor issues

* Fix warnings

* Attempt fix for addr parsing

* Tidy validator config, CLIs

* Tidy comments

* Tidy signing, reduce ForkService duplication

* Fail if skipping too many slots

* Set default recent genesis time to 0

* Add custom http timeout to validator

* Fix compile bug in node_test_rig

* Remove old bootstrap flag from val CLI

* Update docs

* Tidy val client

* Change val client log levels

* Add comments, more validity checks

* Fix compile error, add comments

* Undo changes to eth2-libp2p/src

* Reduce duplication of keypair generation

* Add more logging for validator duties

* Fix beacon_chain_sim, nitpicks

* Fix compile error, minor nits

* Update to use v0.9.2 version of deposit contract

* Add efforts to automate eth1 testnet deployment

* Fix lcli testnet deployer

* Modify bn CLI to parse eth2_testnet_dir

* Progress with account_manager deposit tools

* Make account manager submit deposits

* Add password option for submitting deposits

* Allow custom deposit amount

* Add long names to lcli clap

* Add password option to lcli deploy command

* Fix minor bugs whilst testing

* Address Michael's comments

* Add refund-deposit-contract to lcli

* Use time instead of skip count for denying long skips

* Improve logging for eth1

* Fix bug with validator services exiting on error

* Drop the block cache after genesis

* Modify eth1 testnet config

* Improve eth1 logging

* Make validator wait until genesis time

* Fix bug in eth1 voting

* Add more logging to eth1 voting

* Handle errors in eth1 http module

* Set SECONDS_PER_DAY to sensible minimum

* Shorten delay before testnet start

* Ensure eth1 block is produced without any votes

* Improve eth1 logging

* Fix broken tests in eth1

* Tidy code in rest_api

* Fix failing test in deposit_contract

* Make CLI args more consistent

* Change validator/duties endpoint

* Add time-based skip slot limiting

* Add new error type missed in previous commit

* Add log when waiting for genesis

* Refactor beacon node CLI

* Remove unused dep

* Add lcli eth1-genesis command

* Fix bug in master merge

* Apply clippy lints to beacon node

* Add support for YamlConfig in Eth2TestnetDir

* Upgrade tesnet deposit contract version

* Remove unnecessary logging and correct formatting

* Add a hardcoded eth2 testnet config

* Ensure http server flag works. Overwrite configs with flags.

* Ensure boot nodes are loaded from testnet dir

* Fix account manager CLI bugs

* Fix bugs with beacon node cli

* Allow testnet dir without boot nodes

* Write genesis state as SSZ

* Remove ---/n from the start of testnet_dir files

* Set default libp2p address

* Tidy account manager CLI, add logging

* Add check to see if testnet dir exists

* Apply reviewers suggestions

* Add HeadTracker struct

* Add fork choice persistence

* Shorten slot time for simulator

* Add the /beacon/heads API endpoint

* Update hardcoded testnet

* Add tests for BeaconChain persistence + fix bugs

* Extend BeaconChain persistence testing

* Ensure chain is finalized b4 persistence tests

* Ensure boot_enr.yaml is include in binary

* Refactor beacon_chain_sim

* Move files about in beacon sim

* Update beacon_chain_sim

* Fix bug with deposit inclusion

* Increase log in genesis service, fix todo

* Tidy sim, fix broken rest_api tests

* Fix more broken tests

* Update testnet

* Fix broken rest api test

* Tidy account manager CLI

* Use tempdir for account manager

* Stop hardcoded testnet dir from creating dir

* Rename Eth2TestnetDir to Eth2TestnetConfig

* Change hardcoded -> hard_coded

* Tidy account manager

* Add log to account manager

* Tidy, ensure head tracker is loaded from disk

* Tidy beacon chain builder

* Tidy eth1_chain

* Adds log support for simulator

* Revert "Adds log support for simulator"

This reverts commit ec77c66a05.

* Adds log support for simulator

* Tidy after self-review

* Change default log level

* Address Michael's delicious PR comments

* Fix off-by-one in tests
This commit is contained in:
Paul Hauner
2019-12-03 15:28:57 +11:00
committed by GitHub
parent 4b4bc6247d
commit a0549e3842
74 changed files with 3421 additions and 1298 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
account_manager/Cargo.toml
View File

@@ -23,3 +23,6 @@ eth2_ssz_derive = { path = "../eth2/utils/ssz_derive" }
hex = "0.4"
validator_client = { path = "../validator_client" }
rayon = "1.2.0"
eth2_testnet_config = { path = "../eth2/utils/eth2_testnet_config" }
web3 = "0.8.0"
futures = "0.1.25"

52
account_manager/src/cli.rs
View File

@@ -10,6 +10,56 @@ pub fn cli_app<'a, 'b>() -> App<'a, 'b> {
.subcommand(
SubCommand::with_name("new")
.about("Create a new Ethereum 2.0 validator.")
.arg(
Arg::with_name("deposit-value")
.short("v")
.long("deposit-value")
.value_name("GWEI")
.takes_value(true)
.default_value("32000000000")
.help("The deposit amount in Gwei (not Wei). Default is 32 ETH."),
)
.arg(
Arg::with_name("send-deposits")
.long("send-deposits")
.help("If present, submit validator deposits to an eth1 endpoint /
defined by the --eth1-endpoint. Requires either the /
--deposit-contract or --testnet-dir flag.")
)
.arg(
Arg::with_name("eth1-endpoint")
.short("e")
.long("eth1-endpoint")
.value_name("HTTP_SERVER")
.takes_value(true)
.default_value("http://localhost:8545")
.help("The URL to the Eth1 JSON-RPC HTTP API (e.g., Geth/Parity-Ethereum)."),
)
.arg(
Arg::with_name("account-index")
.short("i")
.long("account-index")
.value_name("INDEX")
.takes_value(true)
.default_value("0")
.help("The eth1 accounts[] index which will send the transaction"),
)
.arg(
Arg::with_name("password")
.short("p")
.long("password")
.value_name("FILE")
.takes_value(true)
.help("The password file to unlock the eth1 account (see --index)"),
)
.arg(
Arg::with_name("testnet-dir")
.long("testnet-dir")
.value_name("DIRECTORY")
.takes_value(true)
.help("The directory from which to read the deposit contract /
address. Defaults to the current Lighthouse testnet."),
)
.subcommand(
SubCommand::with_name("insecure")
.about("Produce insecure, ephemeral validators. DO NOT USE TO STORE VALUE.")
@@ -25,7 +75,7 @@ pub fn cli_app<'a, 'b>() -> App<'a, 'b> {
Arg::with_name("last")
.index(2)
.value_name("INDEX")
.help("Index of the first validator")
.help("Index of the last validator")
.takes_value(true)
.required(true),
),

333
account_manager/src/lib.rs
View File

@@ -1,20 +1,30 @@
mod cli;
use clap::ArgMatches;
use environment::RuntimeContext;
use deposit_contract::DEPOSIT_GAS;
use environment::{Environment, RuntimeContext};
use eth2_testnet_config::Eth2TestnetConfig;
use futures::{future, Future, IntoFuture, Stream};
use rayon::prelude::*;
use slog::{crit, info};
use slog::{crit, error, info, Logger};
use std::fs;
use std::fs::File;
use std::io::Read;
use std::path::PathBuf;
use types::{ChainSpec, EthSpec};
use validator_client::validator_directory::{ValidatorDirectory, ValidatorDirectoryBuilder};
use web3::{
transports::Http,
types::{Address, TransactionRequest, U256},
Web3,
};
pub use cli::cli_app;
/// Run the account manager, logging an error if the operation did not succeed.
pub fn run<T: EthSpec>(matches: &ArgMatches, context: RuntimeContext<T>) {
let log = context.log.clone();
match run_account_manager(matches, context) {
pub fn run<T: EthSpec>(matches: &ArgMatches, mut env: Environment<T>) {
let log = env.core_context().log.clone();
match run_account_manager(matches, env) {
Ok(()) => (),
Err(e) => crit!(log, "Account manager failed"; "error" => e),
}
@@ -23,26 +33,24 @@ pub fn run<T: EthSpec>(matches: &ArgMatches, context: RuntimeContext<T>) {
/// Run the account manager, returning an error if the operation did not succeed.
fn run_account_manager<T: EthSpec>(
matches: &ArgMatches,
context: RuntimeContext<T>,
mut env: Environment<T>,
) -> Result<(), String> {
let context = env.core_context();
let log = context.log.clone();
// If the `datadir` was not provided, default to the home directory. If the home directory is
// not known, use the current directory.
let datadir = matches
.value_of("datadir")
.map(PathBuf::from)
.unwrap_or_else(|| {
let mut default_dir = match dirs::home_dir() {
Some(v) => v,
None => {
panic!("Failed to find a home directory");
}
};
default_dir.push(".lighthouse");
default_dir.push("validators");
default_dir
dirs::home_dir()
.unwrap_or_else(|| PathBuf::from("."))
.join(".lighthouse")
.join("validators")
});
fs::create_dir_all(&datadir).map_err(|e| format!("Failed to initialize datadir: {}", e))?;
fs::create_dir_all(&datadir).map_err(|e| format!("Failed to create datadir: {}", e))?;
info!(
log,
@@ -52,7 +60,7 @@ fn run_account_manager<T: EthSpec>(
match matches.subcommand() {
("validator", Some(matches)) => match matches.subcommand() {
("new", Some(matches)) => run_new_validator_subcommand(matches, datadir, context)?,
("new", Some(matches)) => run_new_validator_subcommand(matches, datadir, env)?,
_ => {
return Err("Invalid 'validator new' command. See --help.".to_string());
}
@@ -77,8 +85,9 @@ enum KeygenMethod {
fn run_new_validator_subcommand<T: EthSpec>(
matches: &ArgMatches,
datadir: PathBuf,
context: RuntimeContext<T>,
mut env: Environment<T>,
) -> Result<(), String> {
let context = env.core_context();
let log = context.log.clone();
let methods: Vec<KeygenMethod> = match matches.subcommand() {
@@ -110,7 +119,119 @@ fn run_new_validator_subcommand<T: EthSpec>(
}
};
let validators = make_validators(datadir.clone(), &methods, context.eth2_config.spec)?;
let deposit_value = matches
.value_of("deposit-value")
.ok_or_else(|| "No deposit-value".to_string())?
.parse::<u64>()
.map_err(|e| format!("Unable to parse deposit-value: {}", e))?;
let validators = make_validators(
datadir.clone(),
&methods,
deposit_value,
&context.eth2_config.spec,
)?;
if matches.is_present("send-deposits") {
let eth1_endpoint = matches
.value_of("eth1-endpoint")
.ok_or_else(|| "No eth1-endpoint".to_string())?;
let account_index = matches
.value_of("account-index")
.ok_or_else(|| "No account-index".to_string())?
.parse::<usize>()
.map_err(|e| format!("Unable to parse account-index: {}", e))?;
// If supplied, load the eth1 account password from file.
let password = if let Some(password_path) = matches.value_of("password") {
Some(
File::open(password_path)
.map_err(|e| format!("Unable to open password file: {:?}", e))
.and_then(|mut file| {
let mut password = String::new();
file.read_to_string(&mut password)
.map_err(|e| format!("Unable to read password file to string: {:?}", e))
.map(|_| password)
})
.map(|password| {
// Trim the line feed from the end of the password file, if present.
if password.ends_with("\n") {
password[0..password.len() - 1].to_string()
} else {
password
}
})?,
)
} else {
None
};
info!(
log,
"Submitting validator deposits";
"eth1_node_http_endpoint" => eth1_endpoint
);
// Load the testnet configuration from disk, or use the default testnet.
let eth2_testnet_config: Eth2TestnetConfig<T> = if let Some(testnet_dir_str) =
matches.value_of("testnet-dir")
{
let testnet_dir = testnet_dir_str
.parse::<PathBuf>()
.map_err(|e| format!("Unable to parse testnet-dir: {}", e))?;
if !testnet_dir.exists() {
return Err(format!(
"Testnet directory at {:?} does not exist",
testnet_dir
));
}
info!(
log,
"Loading deposit contract address";
"testnet_dir" => format!("{:?}", &testnet_dir)
);
Eth2TestnetConfig::load(testnet_dir.clone())
.map_err(|e| format!("Failed to load testnet dir at {:?}: {}", testnet_dir, e))?
} else {
info!(
log,
"Using Lighthouse testnet deposit contract";
);
Eth2TestnetConfig::hard_coded()
.map_err(|e| format!("Failed to load hard_coded testnet dir: {}", e))?
};
// Convert from `types::Address` to `web3::types::Address`.
let deposit_contract = Address::from_slice(
eth2_testnet_config
.deposit_contract_address()?
.as_fixed_bytes(),
);
if let Err(()) = env.runtime().block_on(deposit_validators(
context.clone(),
eth1_endpoint.to_string(),
deposit_contract,
validators.clone(),
account_index,
deposit_value,
password,
)) {
error!(
log,
"Created validators but could not submit deposits";
)
} else {
info!(
log,
"Validator deposits complete";
);
}
}
info!(
log,
@@ -126,14 +247,15 @@ fn run_new_validator_subcommand<T: EthSpec>(
fn make_validators(
datadir: PathBuf,
methods: &[KeygenMethod],
spec: ChainSpec,
deposit_value: u64,
spec: &ChainSpec,
) -> Result<Vec<ValidatorDirectory>, String> {
methods
.par_iter()
.map(|method| {
let mut builder = ValidatorDirectoryBuilder::default()
.spec(spec.clone())
.full_deposit_amount()?;
.custom_deposit_amount(deposit_value);
builder = match method {
KeygenMethod::Insecure(index) => builder.insecure_keypairs(*index),
@@ -148,3 +270,172 @@ fn make_validators(
})
.collect()
}
/// For each `ValidatorDirectory`, submit a deposit transaction to the `eth1_endpoint`.
///
/// Returns success as soon as the eth1 endpoint accepts the transaction (i.e., does not wait for
/// transaction success/revert).
fn deposit_validators<E: EthSpec>(
context: RuntimeContext<E>,
eth1_endpoint: String,
deposit_contract: Address,
validators: Vec<ValidatorDirectory>,
account_index: usize,
deposit_value: u64,
password: Option<String>,
) -> impl Future<Item = (), Error = ()> {
let log_1 = context.log.clone();
let log_2 = context.log.clone();
Http::new(&eth1_endpoint)
.map_err(move |e| {
error!(
log_1,
"Failed to start web3 HTTP transport";
"error" => format!("{:?}", e)
)
})
.into_future()
/*
* Loop through the validator directories and submit the deposits.
*/
.and_then(move |(event_loop, transport)| {
let web3 = Web3::new(transport);
futures::stream::iter_ok(validators)
.for_each(move |validator| {
let web3 = web3.clone();
let log = log_2.clone();
let password = password.clone();
deposit_validator(
web3,
deposit_contract,
&validator,
deposit_value,
account_index,
password,
log,
)
})
.map(|_| event_loop)
})
// Web3 gives errors if the event loop is dropped whilst performing requests.
.map(|event_loop| drop(event_loop))
}
/// For the given `ValidatorDirectory`, submit a deposit transaction to the `web3` node.
///
/// Returns success as soon as the eth1 endpoint accepts the transaction (i.e., does not wait for
/// transaction success/revert).
fn deposit_validator(
web3: Web3<Http>,
deposit_contract: Address,
validator: &ValidatorDirectory,
deposit_amount: u64,
account_index: usize,
password_opt: Option<String>,
log: Logger,
) -> impl Future<Item = (), Error = ()> {
validator
.voting_keypair
.clone()
.ok_or_else(|| error!(log, "Validator does not have voting keypair"))
.and_then(|voting_keypair| {
validator
.deposit_data
.clone()
.ok_or_else(|| error!(log, "Validator does not have deposit data"))
.map(|deposit_data| (voting_keypair, deposit_data))
})
.into_future()
.and_then(move |(voting_keypair, deposit_data)| {
let pubkey_1 = voting_keypair.pk.clone();
let pubkey_2 = voting_keypair.pk.clone();
let web3_1 = web3.clone();
let web3_2 = web3.clone();
let log_1 = log.clone();
let log_2 = log.clone();
web3.eth()
.accounts()
.map_err(|e| format!("Failed to get accounts: {:?}", e))
.and_then(move |accounts| {
accounts
.get(account_index)
.cloned()
.ok_or_else(|| "Insufficient accounts for deposit".to_string())
})
/*
* If a password was supplied, unlock the account.
*/
.and_then(move |from_address| {
let future: Box<dyn Future<Item = Address, Error = String> + Send> =
if let Some(password) = password_opt {
// Unlock for only a single transaction.
let duration = None;
let future = web3_1
.personal()
.unlock_account(from_address, &password, duration)
.then(move |result| match result {
Ok(true) => Ok(from_address),
Ok(false) => {
Err("Eth1 node refused to unlock account. Check password."
.to_string())
}
Err(e) => Err(format!("Eth1 unlock request failed: {:?}", e)),
});
Box::new(future)
} else {
Box::new(future::ok(from_address))
};
future
})
/*
* Submit the deposit transaction.
*/
.and_then(move |from| {
let tx_request = TransactionRequest {
from,
to: Some(deposit_contract),
gas: Some(U256::from(DEPOSIT_GAS)),
gas_price: None,
value: Some(U256::from(from_gwei(deposit_amount))),
data: Some(deposit_data.into()),
nonce: None,
condition: None,
};
web3_2
.eth()
.send_transaction(tx_request)
.map_err(|e| format!("Failed to call deposit fn: {:?}", e))
})
.map(move |tx| {
info!(
log_1,
"Validator deposit successful";
"eth1_tx_hash" => format!("{:?}", tx),
"validator_voting_pubkey" => format!("{:?}", pubkey_1)
)
})
.map_err(move |e| {
error!(
log_2,
"Validator deposit_failed";
"error" => e,
"validator_voting_pubkey" => format!("{:?}", pubkey_2)
)
})
})
}
/// Converts gwei to wei.
fn from_gwei(gwei: u64) -> U256 {
U256::from(gwei) * U256::exp10(9)
}