gnosis/lighthouse
1
0
Fork 0
mirror of https://github.com/sigp/lighthouse.git synced 2026-03-14 18:32:42 +00:00
Code Issues Packages Projects Releases Wiki Activity

Prepare for public testnet (#628)

Browse Source 
* Update to spec v0.9.0

* Update to v0.9.1

* Bump spec tags for v0.9.1

* Formatting, fix CI failures

* Resolve accidental KeyPair merge conflict

* Document new BeaconState functions

* Add `validator` changes from `validator-to-rest`

* Add initial (failing) REST api tests

* Fix signature parsing

* Add more tests

* Refactor http router

* Add working tests for publish beacon block

* Add validator duties tests

* Move account_manager under `lighthouse` binary

* Unify logfile handling in `environment` crate.

* Fix incorrect cache drops in `advance_caches`

* Update fork choice for v0.9.1

* Add `deposit_contract` crate

* Add progress on validator onboarding

* Add unfinished attesation code

* Update account manager CLI

* Write eth1 data file as hex string

* Integrate ValidatorDirectory with validator_client

* Move ValidatorDirectory into validator_client

* Clean up some FIXMEs

* Add beacon_chain_sim

* Fix a few docs/logs

* Expand `beacon_chain_sim`

* Fix spec for `beacon_chain_sim

* More testing for api

* Start work on attestation endpoint

* Reject empty attestations

* Allow attestations to genesis block

* Add working tests for `rest_api` validator endpoint

* Remove grpc from beacon_node

* Start heavy refactor of validator client

- Block production is working

* Prune old validator client files

* Start works on attestation service

* Add attestation service to validator client

* Use full pubkey for validator directories

* Add validator duties post endpoint

* Use par_iter for keypair generation

* Use bulk duties request in validator client

* Add version http endpoint tests

* Add interop keys and startup wait

* Ensure a prompt exit

* Add duties pruning

* Fix compile error in beacon node tests

* Add github workflow

* Modify rust.yaml

* Modify gitlab actions

* Add to CI file

* Add sudo to CI npm install

* Move cargo fmt to own job in tests

* Fix cargo fmt in CI

* Add rustup update before cargo fmt

* Change name of CI job

* Make other CI jobs require cargo fmt

* Add CI badge

* Remove gitlab and travis files

* Add different http timeout for debug

* Update docker file, use makefile in CI

* Use make in the dockerfile, skip the test

* Use the makefile for debug GI test

* Update book

* Tidy grpc and misc things

* Apply discv5 fixes

* Address other minor issues

* Fix warnings

* Attempt fix for addr parsing

* Tidy validator config, CLIs

* Tidy comments

* Tidy signing, reduce ForkService duplication

* Fail if skipping too many slots

* Set default recent genesis time to 0

* Add custom http timeout to validator

* Fix compile bug in node_test_rig

* Remove old bootstrap flag from val CLI

* Update docs

* Tidy val client

* Change val client log levels

* Add comments, more validity checks

* Fix compile error, add comments

* Undo changes to eth2-libp2p/src

* Reduce duplication of keypair generation

* Add more logging for validator duties

* Fix beacon_chain_sim, nitpicks

* Fix compile error, minor nits

* Update to use v0.9.2 version of deposit contract

* Add efforts to automate eth1 testnet deployment

* Fix lcli testnet deployer

* Modify bn CLI to parse eth2_testnet_dir

* Progress with account_manager deposit tools

* Make account manager submit deposits

* Add password option for submitting deposits

* Allow custom deposit amount

* Add long names to lcli clap

* Add password option to lcli deploy command

* Fix minor bugs whilst testing

* Address Michael's comments

* Add refund-deposit-contract to lcli

* Use time instead of skip count for denying long skips

* Improve logging for eth1

* Fix bug with validator services exiting on error

* Drop the block cache after genesis

* Modify eth1 testnet config

* Improve eth1 logging

* Make validator wait until genesis time

* Fix bug in eth1 voting

* Add more logging to eth1 voting

* Handle errors in eth1 http module

* Set SECONDS_PER_DAY to sensible minimum

* Shorten delay before testnet start

* Ensure eth1 block is produced without any votes

* Improve eth1 logging

* Fix broken tests in eth1

* Tidy code in rest_api

* Fix failing test in deposit_contract

* Make CLI args more consistent

* Change validator/duties endpoint

* Add time-based skip slot limiting

* Add new error type missed in previous commit

* Add log when waiting for genesis

* Refactor beacon node CLI

* Remove unused dep

* Add lcli eth1-genesis command

* Fix bug in master merge

* Apply clippy lints to beacon node

* Add support for YamlConfig in Eth2TestnetDir

* Upgrade tesnet deposit contract version

* Remove unnecessary logging and correct formatting

* Add a hardcoded eth2 testnet config

* Ensure http server flag works. Overwrite configs with flags.

* Ensure boot nodes are loaded from testnet dir

* Fix account manager CLI bugs

* Fix bugs with beacon node cli

* Allow testnet dir without boot nodes

* Write genesis state as SSZ

* Remove ---/n from the start of testnet_dir files

* Set default libp2p address

* Tidy account manager CLI, add logging

* Add check to see if testnet dir exists

* Apply reviewers suggestions

* Add HeadTracker struct

* Add fork choice persistence

* Shorten slot time for simulator

* Add the /beacon/heads API endpoint

* Update hardcoded testnet

* Add tests for BeaconChain persistence + fix bugs

* Extend BeaconChain persistence testing

* Ensure chain is finalized b4 persistence tests

* Ensure boot_enr.yaml is include in binary

* Refactor beacon_chain_sim

* Move files about in beacon sim

* Update beacon_chain_sim

* Fix bug with deposit inclusion

* Increase log in genesis service, fix todo

* Tidy sim, fix broken rest_api tests

* Fix more broken tests

* Update testnet

* Fix broken rest api test

* Tidy account manager CLI

* Use tempdir for account manager

* Stop hardcoded testnet dir from creating dir

* Rename Eth2TestnetDir to Eth2TestnetConfig

* Change hardcoded -> hard_coded

* Tidy account manager

* Add log to account manager

* Tidy, ensure head tracker is loaded from disk

* Tidy beacon chain builder

* Tidy eth1_chain

* Adds log support for simulator

* Revert "Adds log support for simulator"

This reverts commit ec77c66a05.

* Adds log support for simulator

* Tidy after self-review

* Change default log level

* Address Michael's delicious PR comments

* Fix off-by-one in tests
This commit is contained in:
Paul Hauner
2019-12-03 15:28:57 +11:00
committed by GitHub
parent 4b4bc6247d
commit a0549e3842
74 changed files with 3421 additions and 1298 deletions
Download Patch File Download Diff File Expand all files Collapse all files

71
beacon_node/beacon_chain/src/beacon_chain.rs
View File

@@ -3,6 +3,7 @@ use crate::errors::{BeaconChainError as Error, BlockProductionError};
use crate::eth1_chain::{Eth1Chain, Eth1ChainBackend};
use crate::events::{EventHandler, EventKind};
use crate::fork_choice::{Error as ForkChoiceError, ForkChoice};
use crate::head_tracker::HeadTracker;
use crate::metrics;
use crate::persisted_beacon_chain::{PersistedBeaconChain, BEACON_CHAIN_DB_KEY};
use lmd_ghost::LmdGhost;
@@ -126,6 +127,8 @@ pub struct BeaconChain<T: BeaconChainTypes> {
pub fork_choice: ForkChoice<T>,
/// A handler for events generated by the beacon chain.
pub event_handler: T::EventHandler,
/// Used to track the heads of the beacon chain.
pub(crate) head_tracker: HeadTracker,
/// Logging to CLI, etc.
pub(crate) log: Logger,
}
@@ -137,10 +140,35 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
pub fn persist(&self) -> Result<(), Error> {
let timer = metrics::start_timer(&metrics::PERSIST_CHAIN);
let canonical_head = self.head();
let finalized_checkpoint = {
let beacon_block_root = canonical_head.beacon_state.finalized_checkpoint.root;
let beacon_block = self
.store
.get::<BeaconBlock<_>>(&beacon_block_root)?
.ok_or_else(|| Error::MissingBeaconBlock(beacon_block_root))?;
let beacon_state_root = beacon_block.state_root;
let beacon_state = self
.store
.get_state(&beacon_state_root, Some(beacon_block.slot))?
.ok_or_else(|| Error::MissingBeaconState(beacon_state_root))?;
CheckPoint {
beacon_block_root,
beacon_block,
beacon_state_root,
beacon_state,
}
};
let p: PersistedBeaconChain<T> = PersistedBeaconChain {
canonical_head: self.canonical_head.read().clone(),
canonical_head,
finalized_checkpoint,
op_pool: PersistedOperationPool::from_operation_pool(&self.op_pool),
genesis_block_root: self.genesis_block_root,
ssz_head_tracker: self.head_tracker.to_ssz_container(),
fork_choice: self.fork_choice.as_ssz_container(),
};
let key = Hash256::from_slice(&BEACON_CHAIN_DB_KEY.as_bytes());
@@ -331,10 +359,17 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
self.canonical_head.read().clone()
}
/// Returns the current heads of the `BeaconChain`. For the canonical head, see `Self::head`.
///
/// Returns `(block_root, block_slot)`.
pub fn heads(&self) -> Vec<(Hash256, Slot)> {
self.head_tracker.heads()
}
/// Returns the `BeaconState` at the given slot.
///
/// Returns `None` when the state is not found in the database or there is an error skipping
/// to a future state.
/// Returns `None` when the state is not found in the database or there is an error skipping
/// to a future state.
pub fn state_at_slot(&self, slot: Slot) -> Result<BeaconState<T::EthSpec>, Error> {
let head_state = self.head().beacon_state;
@@ -1219,6 +1254,8 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
metrics::stop_timer(db_write_timer);
self.head_tracker.register_block(block_root, &block);
let fork_choice_register_timer =
metrics::start_timer(&metrics::BLOCK_PROCESSING_FORK_CHOICE_REGISTER);
@@ -1324,6 +1361,11 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
let (proposer_slashings, attester_slashings) =
self.op_pool.get_slashings(&state, &self.spec);
let eth1_data = eth1_chain.eth1_data_for_block_production(&state, &self.spec)?;
let deposits = eth1_chain
.deposits_for_block_inclusion(&state, &eth1_data, &self.spec)?
.into();
let mut block = BeaconBlock {
slot: state.slot,
parent_root,
@@ -1332,14 +1374,12 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
signature: Signature::empty_signature(),
body: BeaconBlockBody {
randao_reveal,
eth1_data: eth1_chain.eth1_data_for_block_production(&state, &self.spec)?,
eth1_data,
graffiti,
proposer_slashings: proposer_slashings.into(),
attester_slashings: attester_slashings.into(),
attestations: self.op_pool.get_attestations(&state, &self.spec).into(),
deposits: eth1_chain
.deposits_for_block_inclusion(&state, &self.spec)?
.into(),
deposits,
voluntary_exits: self.op_pool.get_voluntary_exits(&state, &self.spec).into(),
},
};
@@ -1596,6 +1636,23 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
}
}
impl<T: BeaconChainTypes> Drop for BeaconChain<T> {
fn drop(&mut self) {
if let Err(e) = self.persist() {
error!(
self.log,
"Failed to persist BeaconChain on drop";
"error" => format!("{:?}", e)
)
} else {
info!(
self.log,
"Saved beacon chain state";
)
}
}
}
fn write_state<T: EthSpec>(prefix: &str, state: &BeaconState<T>, log: &Logger) {
if WRITE_BLOCK_PROCESSING_SSZ {
let root = Hash256::from_slice(&state.tree_hash_root());

101
beacon_node/beacon_chain/src/builder.rs
View File

@@ -1,5 +1,6 @@
use crate::eth1_chain::CachingEth1Backend;
use crate::events::NullEventHandler;
use crate::head_tracker::HeadTracker;
use crate::persisted_beacon_chain::{PersistedBeaconChain, BEACON_CHAIN_DB_KEY};
use crate::{
BeaconChain, BeaconChainTypes, CheckPoint, Eth1Chain, Eth1ChainBackend, EventHandler,
@@ -88,6 +89,8 @@ pub struct BeaconChainBuilder<T: BeaconChainTypes> {
eth1_chain: Option<Eth1Chain<T::Eth1Chain, T::EthSpec>>,
event_handler: Option<T::EventHandler>,
slot_clock: Option<T::SlotClock>,
persisted_beacon_chain: Option<PersistedBeaconChain<T>>,
head_tracker: Option<HeadTracker>,
spec: ChainSpec,
log: Option<Logger>,
}
@@ -128,6 +131,8 @@ where
eth1_chain: None,
event_handler: None,
slot_clock: None,
persisted_beacon_chain: None,
head_tracker: None,
spec: TEthSpec::default_spec(),
log: None,
}
@@ -208,11 +213,17 @@ where
self.op_pool = Some(
p.op_pool
.clone()
.into_operation_pool(&p.canonical_head.beacon_state, &self.spec),
);
self.finalized_checkpoint = Some(p.canonical_head);
self.finalized_checkpoint = Some(p.finalized_checkpoint.clone());
self.genesis_block_root = Some(p.genesis_block_root);
self.head_tracker = Some(
HeadTracker::from_ssz_container(&p.ssz_head_tracker)
.map_err(|e| format!("Failed to decode head tracker for database: {:?}", e))?,
);
self.persisted_beacon_chain = Some(p);
Ok(self)
}
@@ -264,28 +275,6 @@ where
Ok(self.empty_op_pool())
}
/// Sets the `BeaconChain` fork choice backend.
///
/// Requires the store and state to have been specified earlier in the build chain.
pub fn fork_choice_backend(mut self, backend: TLmdGhost) -> Result<Self, String> {
let store = self
.store
.clone()
.ok_or_else(|| "reduced_tree_fork_choice requires a store")?;
let genesis_block_root = self
.genesis_block_root
.ok_or_else(|| "fork_choice_backend requires a genesis_block_root")?;
self.fork_choice = Some(ForkChoice::new(
store,
backend,
genesis_block_root,
self.spec.genesis_slot,
));
Ok(self)
}
/// Sets the `BeaconChain` eth1 backend.
pub fn eth1_backend(mut self, backend: Option<TEth1Backend>) -> Self {
self.eth1_chain = backend.map(Eth1Chain::new);
@@ -337,19 +326,24 @@ where
>,
String,
> {
let mut canonical_head = self
.finalized_checkpoint
.ok_or_else(|| "Cannot build without a state".to_string())?;
let log = self
.log
.ok_or_else(|| "Cannot build without a logger".to_string())?;
// If this beacon chain is being loaded from disk, use the stored head. Otherwise, just use
// the finalized checkpoint (which is probably genesis).
let mut canonical_head = if let Some(persisted_beacon_chain) = self.persisted_beacon_chain {
persisted_beacon_chain.canonical_head
} else {
self.finalized_checkpoint
.ok_or_else(|| "Cannot build without a state".to_string())?
};
canonical_head
.beacon_state
.build_all_caches(&self.spec)
.map_err(|e| format!("Failed to build state caches: {:?}", e))?;
let log = self
.log
.ok_or_else(|| "Cannot build without a logger".to_string())?;
if canonical_head.beacon_block.state_root != canonical_head.beacon_state_root {
return Err("beacon_block.state_root != beacon_state".to_string());
}
@@ -379,6 +373,7 @@ where
event_handler: self
.event_handler
.ok_or_else(|| "Cannot build without an event handler".to_string())?,
head_tracker: self.head_tracker.unwrap_or_default(),
log: log.clone(),
};
@@ -414,27 +409,43 @@ where
TEthSpec: EthSpec + 'static,
TEventHandler: EventHandler<TEthSpec> + 'static,
{
/// Initializes a new, empty (no recorded votes or blocks) fork choice, using the
/// `ThreadSafeReducedTree` backend.
/// Initializes a fork choice with the `ThreadSafeReducedTree` backend.
///
/// Requires the store and state to be initialized.
pub fn empty_reduced_tree_fork_choice(self) -> Result<Self, String> {
/// If this builder is being "resumed" from disk, then rebuild the last fork choice stored to
/// the database. Otherwise, create a new, empty fork choice.
pub fn reduced_tree_fork_choice(mut self) -> Result<Self, String> {
let store = self
.store
.clone()
.ok_or_else(|| "reduced_tree_fork_choice requires a store")?;
let finalized_checkpoint = &self
.finalized_checkpoint
.as_ref()
.expect("should have finalized checkpoint");
let backend = ThreadSafeReducedTree::new(
store.clone(),
&finalized_checkpoint.beacon_block,
finalized_checkpoint.beacon_block_root,
);
let fork_choice = if let Some(persisted_beacon_chain) = &self.persisted_beacon_chain {
ForkChoice::from_ssz_container(
persisted_beacon_chain.fork_choice.clone(),
store.clone(),
)
.map_err(|e| format!("Unable to decode fork choice from db: {:?}", e))?
} else {
let finalized_checkpoint = &self
.finalized_checkpoint
.as_ref()
.ok_or_else(|| "fork_choice_backend requires a finalized_checkpoint")?;
let genesis_block_root = self
.genesis_block_root
.ok_or_else(|| "fork_choice_backend requires a genesis_block_root")?;
self.fork_choice_backend(backend)
let backend = ThreadSafeReducedTree::new(
store.clone(),
&finalized_checkpoint.beacon_block,
finalized_checkpoint.beacon_block_root,
);
ForkChoice::new(store, backend, genesis_block_root, self.spec.genesis_slot)
};
self.fork_choice = Some(fork_choice);
Ok(self)
}
}
@@ -611,7 +622,7 @@ mod test {
.null_event_handler()
.testing_slot_clock(Duration::from_secs(1))
.expect("should configure testing slot clock")
.empty_reduced_tree_fork_choice()
.reduced_tree_fork_choice()
.expect("should add fork choice to builder")
.build()
.expect("should build");

261
beacon_node/beacon_chain/src/eth1_chain.rs
View File

@@ -4,7 +4,8 @@ use exit_future::Exit;
use futures::Future;
use integer_sqrt::IntegerSquareRoot;
use rand::prelude::*;
use slog::{crit, Logger};
use slog::{crit, debug, error, trace, Logger};
use state_processing::per_block_processing::get_new_eth1_data;
use std::collections::HashMap;
use std::iter::DoubleEndedIterator;
use std::iter::FromIterator;
@@ -33,7 +34,7 @@ pub enum Error {
/// voting period.
UnableToGetPreviousStateRoot(BeaconStateError),
/// The state required to find the previous eth1 block was not found in the store.
PreviousStateNotInDB,
PreviousStateNotInDB(Hash256),
/// There was an error accessing an object in the database.
StoreError(StoreError),
/// The eth1 head block at the start of the eth1 voting period is unknown.
@@ -89,16 +90,21 @@ where
/// Returns a list of `Deposits` that may be included in a block.
///
/// Including all of the returned `Deposits` in a block should _not_ cause it to become
/// invalid.
/// invalid (i.e., this function should respect the maximum).
///
/// `eth1_data_vote` is the `Eth1Data` that the block producer would include in their
/// block. This vote may change the `state.eth1_data` value, which would change the deposit
/// count and therefore change the output of this function.
pub fn deposits_for_block_inclusion(
&self,
state: &BeaconState<E>,
eth1_data_vote: &Eth1Data,
spec: &ChainSpec,
) -> Result<Vec<Deposit>, Error> {
if self.use_dummy_backend {
DummyEth1ChainBackend::default().queued_deposits(state, spec)
DummyEth1ChainBackend::default().queued_deposits(state, eth1_data_vote, spec)
} else {
self.backend.queued_deposits(state, spec)
self.backend.queued_deposits(state, eth1_data_vote, spec)
}
}
}
@@ -119,6 +125,7 @@ pub trait Eth1ChainBackend<T: EthSpec>: Sized + Send + Sync {
fn queued_deposits(
&self,
beacon_state: &BeaconState<T>,
eth1_data_vote: &Eth1Data,
spec: &ChainSpec,
) -> Result<Vec<Deposit>, Error>;
}
@@ -131,6 +138,7 @@ pub trait Eth1ChainBackend<T: EthSpec>: Sized + Send + Sync {
pub struct DummyEth1ChainBackend<T: EthSpec>(PhantomData<T>);
impl<T: EthSpec> Eth1ChainBackend<T> for DummyEth1ChainBackend<T> {
/// Produce some deterministic junk based upon the current epoch.
fn eth1_data(&self, state: &BeaconState<T>, _spec: &ChainSpec) -> Result<Eth1Data, Error> {
let current_epoch = state.current_epoch();
let slots_per_voting_period = T::slots_per_eth1_voting_period() as u64;
@@ -146,7 +154,13 @@ impl<T: EthSpec> Eth1ChainBackend<T> for DummyEth1ChainBackend<T> {
})
}
fn queued_deposits(&self, _: &BeaconState<T>, _: &ChainSpec) -> Result<Vec<Deposit>, Error> {
/// The dummy back-end never produces deposits.
fn queued_deposits(
&self,
_: &BeaconState<T>,
_: &Eth1Data,
_: &ChainSpec,
) -> Result<Vec<Deposit>, Error> {
Ok(vec![])
}
}
@@ -201,24 +215,102 @@ impl<T: EthSpec, S: Store> CachingEth1Backend<T, S> {
impl<T: EthSpec, S: Store> Eth1ChainBackend<T> for CachingEth1Backend<T, S> {
fn eth1_data(&self, state: &BeaconState<T>, spec: &ChainSpec) -> Result<Eth1Data, Error> {
// Note: we do not return random junk if this function call fails as it would be caused by
// an internal error.
let prev_eth1_hash = eth1_block_hash_at_start_of_voting_period(self.store.clone(), state)?;
let period = T::SlotsPerEth1VotingPeriod::to_u64();
let eth1_follow_distance = spec.eth1_follow_distance;
let voting_period_start_slot = (state.slot / period) * period;
let voting_period_start_seconds = slot_start_seconds::<T>(
state.genesis_time,
spec.milliseconds_per_slot,
voting_period_start_slot,
);
let blocks = self.core.blocks().read();
let eth1_data = eth1_data_sets(blocks.iter(), state, prev_eth1_hash, spec)
.map(|(new_eth1_data, all_eth1_data)| {
collect_valid_votes(state, new_eth1_data, all_eth1_data)
})
.and_then(find_winning_vote)
.unwrap_or_else(|| {
crit!(
self.log,
"Unable to cast valid vote for Eth1Data";
"hint" => "check connection to eth1 node",
"reason" => "no votes",
);
random_eth1_data()
});
let (new_eth1_data, all_eth1_data) = if let Some(sets) = eth1_data_sets(
blocks.iter(),
prev_eth1_hash,
voting_period_start_seconds,
spec,
&self.log,
) {
sets
} else {
// The algorithm was unable to find the `new_eth1_data` and `all_eth1_data` sets.
//
// This is likely because the caches are empty or the previous eth1 block hash is not
// in the cache.
//
// This situation can also be caused when a testnet does not have an adequate delay
// between the eth1 genesis block and the eth2 genesis block. This delay needs to be at
// least `2 * ETH1_FOLLOW_DISTANCE`.
crit!(
self.log,
"Unable to find eth1 data sets";
"lowest_block_number" => self.core.lowest_block_number(),
"earliest_block_timestamp" => self.core.earliest_block_timestamp(),
"genesis_time" => state.genesis_time,
"outcome" => "casting random eth1 vote"
);
return Ok(random_eth1_data());
};
trace!(
self.log,
"Found eth1 data sets";
"all_eth1_data" => all_eth1_data.len(),
"new_eth1_data" => new_eth1_data.len(),
);
let valid_votes = collect_valid_votes(state, new_eth1_data, all_eth1_data);
let eth1_data = if let Some(eth1_data) = find_winning_vote(valid_votes) {
eth1_data
} else {
// In this case, there are no other viable votes (perhaps there are no votes yet or all
// the existing votes are junk).
//
// Here we choose the latest block in our voting window.
blocks
.iter()
.rev()
.skip_while(|eth1_block| eth1_block.timestamp > voting_period_start_seconds)
.skip(eth1_follow_distance as usize)
.next()
.map(|block| {
trace!(
self.log,
"Choosing default eth1_data";
"eth1_block_number" => block.number,
"eth1_block_hash" => format!("{:?}", block.hash),
);
block
})
.and_then(|block| block.clone().eth1_data())
.unwrap_or_else(|| {
crit!(
self.log,
"Unable to find a winning eth1 vote";
"outcome" => "casting random eth1 vote"
);
random_eth1_data()
})
};
debug!(
self.log,
"Produced vote for eth1 chain";
"is_period_tail" => is_period_tail(state),
"deposit_root" => format!("{:?}", eth1_data.deposit_root),
"deposit_count" => eth1_data.deposit_count,
"block_hash" => format!("{:?}", eth1_data.block_hash),
);
Ok(eth1_data)
}
@@ -226,10 +318,15 @@ impl<T: EthSpec, S: Store> Eth1ChainBackend<T> for CachingEth1Backend<T, S> {
fn queued_deposits(
&self,
state: &BeaconState<T>,
eth1_data_vote: &Eth1Data,
_spec: &ChainSpec,
) -> Result<Vec<Deposit>, Error> {
let deposit_count = state.eth1_data.deposit_count;
let deposit_index = state.eth1_deposit_index;
let deposit_count = if let Some(new_eth1_data) = get_new_eth1_data(state, eth1_data_vote) {
new_eth1_data.deposit_count
} else {
state.eth1_data.deposit_count
};
if deposit_index > deposit_count {
Err(Error::DepositIndexTooHigh)
@@ -281,12 +378,14 @@ fn eth1_block_hash_at_start_of_voting_period<T: EthSpec, S: Store>(
) -> Result<Hash256, Error> {
let period = T::SlotsPerEth1VotingPeriod::to_u64();
// Find `state.eth1_data.block_hash` for the state at the start of the voting period.
if state.slot % period < period / 2 {
// When the state is less than half way through the period we can safely assume that
// the eth1_data has not changed since the start of the period.
if !eth1_data_change_is_possible(state) {
// If there are less than 50% of the votes in the current state, it's impossible that the
// `eth1_data.block_hash` has changed from the value at `state.eth1_data.block_hash`.
Ok(state.eth1_data.block_hash)
} else {
// If there have been more than 50% of votes in this period it's possible (but not
// necessary) that the `state.eth1_data.block_hash` has been changed since the start of the
// voting period.
let slot = (state.slot / period) * period;
let prev_state_root = state
.get_state_root(slot)
@@ -296,33 +395,32 @@ fn eth1_block_hash_at_start_of_voting_period<T: EthSpec, S: Store>(
.get_state::<T>(&prev_state_root, Some(slot))
.map_err(Error::StoreError)?
.map(|state| state.eth1_data.block_hash)
.ok_or_else(|| Error::PreviousStateNotInDB)
.ok_or_else(|| Error::PreviousStateNotInDB(*prev_state_root))
}
}
/// Returns true if there are enough eth1 votes in the given `state` to have updated
/// `state.eth1_data`.
fn eth1_data_change_is_possible<E: EthSpec>(state: &BeaconState<E>) -> bool {
2 * state.eth1_data_votes.len() > E::SlotsPerEth1VotingPeriod::to_usize()
}
/// Calculates and returns `(new_eth1_data, all_eth1_data)` for the given `state`, based upon the
/// blocks in the `block` iterator.
///
/// `prev_eth1_hash` is the `eth1_data.block_hash` at the start of the voting period defined by
/// `state.slot`.
fn eth1_data_sets<'a, T: EthSpec, I>(
fn eth1_data_sets<'a, I>(
blocks: I,
state: &BeaconState<T>,
prev_eth1_hash: Hash256,
voting_period_start_seconds: u64,
spec: &ChainSpec,
log: &Logger,
) -> Option<(Eth1DataBlockNumber, Eth1DataBlockNumber)>
where
T: EthSpec,
I: DoubleEndedIterator<Item = &'a Eth1Block> + Clone,
{
let period = T::SlotsPerEth1VotingPeriod::to_u64();
let eth1_follow_distance = spec.eth1_follow_distance;
let voting_period_start_slot = (state.slot / period) * period;
let voting_period_start_seconds = slot_start_seconds::<T>(
state.genesis_time,
spec.milliseconds_per_slot,
voting_period_start_slot,
);
let in_scope_eth1_data = blocks
.rev()
@@ -345,6 +443,12 @@ where
HashMap::from_iter(all_eth1_data),
))
} else {
error!(
log,
"The previous eth1 hash is not in cache";
"previous_hash" => format!("{:?}", prev_eth1_hash)
);
None
}
}
@@ -356,8 +460,6 @@ fn collect_valid_votes<T: EthSpec>(
new_eth1_data: Eth1DataBlockNumber,
all_eth1_data: Eth1DataBlockNumber,
) -> Eth1DataVoteCount {
let slots_per_eth1_voting_period = T::SlotsPerEth1VotingPeriod::to_u64();
let mut valid_votes = HashMap::new();
state
@@ -368,10 +470,7 @@ fn collect_valid_votes<T: EthSpec>(
.get(vote)
.map(|block_number| (vote.clone(), *block_number))
.or_else(|| {
let slot = state.slot % slots_per_eth1_voting_period;
let period_tail = slot >= slots_per_eth1_voting_period.integer_sqrt();
if period_tail {
if is_period_tail(state) {
all_eth1_data
.get(vote)
.map(|block_number| (vote.clone(), *block_number))
@@ -390,6 +489,15 @@ fn collect_valid_votes<T: EthSpec>(
valid_votes
}
/// Indicates if the given `state` is in the tail of it's eth1 voting period (i.e., in the later
/// slots).
fn is_period_tail<E: EthSpec>(state: &BeaconState<E>) -> bool {
let slots_per_eth1_voting_period = E::SlotsPerEth1VotingPeriod::to_u64();
let slot = state.slot % slots_per_eth1_voting_period;
slot >= slots_per_eth1_voting_period.integer_sqrt()
}
/// Selects the winning vote from `valid_votes`.
fn find_winning_vote(valid_votes: Eth1DataVoteCount) -> Option<Eth1Data> {
valid_votes
@@ -417,6 +525,7 @@ fn slot_start_seconds<T: EthSpec>(
#[cfg(test)]
mod test {
use super::*;
use environment::null_logger;
use types::{test_utils::DepositTestTask, MinimalEthSpec};
type E = MinimalEthSpec;
@@ -468,7 +577,6 @@ mod test {
mod eth1_chain_json_backend {
use super::*;
use environment::null_logger;
use eth1::DepositLog;
use store::MemoryStore;
use types::test_utils::{generate_deterministic_keypair, TestingDepositBuilder};
@@ -514,7 +622,7 @@ mod test {
assert!(
eth1_chain
.deposits_for_block_inclusion(&state, spec)
.deposits_for_block_inclusion(&state, &random_eth1_data(), spec)
.is_ok(),
"should succeed if cache is empty but no deposits are required"
);
@@ -523,7 +631,7 @@ mod test {
assert!(
eth1_chain
.deposits_for_block_inclusion(&state, spec)
.deposits_for_block_inclusion(&state, &random_eth1_data(), spec)
.is_err(),
"should fail to get deposits if required, but cache is empty"
);
@@ -567,7 +675,7 @@ mod test {
assert!(
eth1_chain
.deposits_for_block_inclusion(&state, spec)
.deposits_for_block_inclusion(&state, &random_eth1_data(), spec)
.is_ok(),
"should succeed if no deposits are required"
);
@@ -579,7 +687,7 @@ mod test {
state.eth1_data.deposit_count = i as u64;
let deposits_for_inclusion = eth1_chain
.deposits_for_block_inclusion(&state, spec)
.deposits_for_block_inclusion(&state, &random_eth1_data(), spec)
.expect(&format!("should find deposit for {}", i));
let expected_len =
@@ -657,6 +765,14 @@ mod test {
prev_state.slot = Slot::new(period * 1_000);
state.slot = Slot::new(period * 1_000 + period / 2);
// Add 50% of the votes so a lookup is required.
for _ in 0..period / 2 + 1 {
state
.eth1_data_votes
.push(random_eth1_data())
.expect("should push eth1 vote");
}
(0..2048).for_each(|i| {
eth1_chain
.backend
@@ -728,6 +844,14 @@ mod test {
state.slot = Slot::new(period / 2);
// Add 50% of the votes so a lookup is required.
for _ in 0..period / 2 + 1 {
state
.eth1_data_votes
.push(random_eth1_data())
.expect("should push eth1 vote");
}
let expected_root = Hash256::from_low_u64_be(42);
prev_state.eth1_data.block_hash = expected_root;
@@ -757,8 +881,20 @@ mod test {
mod eth1_data_sets {
use super::*;
fn get_voting_period_start_seconds(state: &BeaconState<E>, spec: &ChainSpec) -> u64 {
let period = <E as EthSpec>::SlotsPerEth1VotingPeriod::to_u64();
let voting_period_start_slot = (state.slot / period) * period;
slot_start_seconds::<E>(
state.genesis_time,
spec.milliseconds_per_slot,
voting_period_start_slot,
)
}
#[test]
fn empty_cache() {
let log = null_logger().unwrap();
let spec = &E::default_spec();
let state: BeaconState<E> = BeaconState::new(0, get_eth1_data(0), spec);
let prev_eth1_hash = Hash256::zero();
@@ -766,13 +902,21 @@ mod test {
let blocks = vec![];
assert_eq!(
eth1_data_sets(blocks.iter(), &state, prev_eth1_hash, &spec),
eth1_data_sets(
blocks.iter(),
prev_eth1_hash,
get_voting_period_start_seconds(&state, spec),
&spec,
&log
),
None
);
}
#[test]
fn no_known_block_hash() {
let log = null_logger().unwrap();
let mut spec = E::default_spec();
spec.milliseconds_per_slot = 1_000;
@@ -782,13 +926,21 @@ mod test {
let blocks = vec![get_eth1_block(0, 0)];
assert_eq!(
eth1_data_sets(blocks.iter(), &state, prev_eth1_hash, &spec),
eth1_data_sets(
blocks.iter(),
prev_eth1_hash,
get_voting_period_start_seconds(&state, &spec),
&spec,
&log
),
None
);
}
#[test]
fn ideal_scenario() {
let log = null_logger().unwrap();
let mut spec = E::default_spec();
spec.milliseconds_per_slot = 1_000;
@@ -805,9 +957,14 @@ mod test {
.map(|i| get_eth1_block(i, i))
.collect::<Vec<_>>();
let (new_eth1_data, all_eth1_data) =
eth1_data_sets(blocks.iter(), &state, prev_eth1_hash, &spec)
.expect("should find data");
let (new_eth1_data, all_eth1_data) = eth1_data_sets(
blocks.iter(),
prev_eth1_hash,
get_voting_period_start_seconds(&state, &spec),
&spec,
&log,
)
.expect("should find data");
assert_eq!(
all_eth1_data.len(),

47
beacon_node/beacon_chain/src/fork_choice.rs
View File

@@ -1,6 +1,7 @@
use crate::{errors::BeaconChainError, metrics, BeaconChain, BeaconChainTypes};
use lmd_ghost::LmdGhost;
use parking_lot::RwLock;
use ssz_derive::{Decode, Encode};
use state_processing::{common::get_attesting_indices, per_slot_processing};
use std::sync::Arc;
use store::{Error as StoreError, Store};
@@ -34,6 +35,16 @@ pub struct ForkChoice<T: BeaconChainTypes> {
best_justified_checkpoint: RwLock<Checkpoint>,
}
impl<T: BeaconChainTypes> PartialEq for ForkChoice<T> {
/// This implementation ignores the `store`.
fn eq(&self, other: &Self) -> bool {
self.backend == other.backend
&& self.genesis_block_root == other.genesis_block_root
&& *self.justified_checkpoint.read() == *other.justified_checkpoint.read()
&& *self.best_justified_checkpoint.read() == *other.best_justified_checkpoint.read()
}
}
impl<T: BeaconChainTypes> ForkChoice<T> {
/// Instantiate a new fork chooser.
///
@@ -291,6 +302,42 @@ impl<T: BeaconChainTypes> ForkChoice<T> {
.update_finalized_root(finalized_block, finalized_block_root)
.map_err(Into::into)
}
/// Returns a `SszForkChoice` which contains the current state of `Self`.
pub fn as_ssz_container(&self) -> SszForkChoice {
SszForkChoice {
genesis_block_root: self.genesis_block_root.clone(),
justified_checkpoint: self.justified_checkpoint.read().clone(),
best_justified_checkpoint: self.best_justified_checkpoint.read().clone(),
backend_bytes: self.backend.as_bytes(),
}
}
/// Instantiates `Self` from a prior `SszForkChoice`.
///
/// The created `Self` will have the same state as the `Self` that created the `SszForkChoice`.
pub fn from_ssz_container(ssz_container: SszForkChoice, store: Arc<T::Store>) -> Result<Self> {
let backend = LmdGhost::from_bytes(&ssz_container.backend_bytes, store.clone())?;
Ok(Self {
store,
backend,
genesis_block_root: ssz_container.genesis_block_root,
justified_checkpoint: RwLock::new(ssz_container.justified_checkpoint),
best_justified_checkpoint: RwLock::new(ssz_container.best_justified_checkpoint),
})
}
}
/// Helper struct that is used to encode/decode the state of the `ForkChoice` as SSZ bytes.
///
/// This is used when persisting the state of the `BeaconChain` to disk.
#[derive(Encode, Decode, Clone)]
pub struct SszForkChoice {
genesis_block_root: Hash256,
justified_checkpoint: Checkpoint,
best_justified_checkpoint: Checkpoint,
backend_bytes: Vec<u8>,
}
impl From<BeaconStateError> for Error {

205
beacon_node/beacon_chain/src/head_tracker.rs Normal file
View File

@@ -0,0 +1,205 @@
use parking_lot::RwLock;
use ssz_derive::{Decode, Encode};
use std::collections::HashMap;
use std::iter::FromIterator;
use types::{BeaconBlock, EthSpec, Hash256, Slot};
#[derive(Debug, PartialEq)]
pub enum Error {
MismatchingLengths { roots_len: usize, slots_len: usize },
}
/// Maintains a list of `BeaconChain` head block roots and slots.
///
/// Each time a new block is imported, it should be applied to the `Self::register_block` function.
/// In order for this struct to be effective, every single block that is imported must be
/// registered here.
#[derive(Default, Debug)]
pub struct HeadTracker(RwLock<HashMap<Hash256, Slot>>);
impl HeadTracker {
/// Register a block with `Self`, so it may or may not be included in a `Self::heads` call.
///
/// This function assumes that no block is imported without its parent having already been
/// imported. It cannot detect an error if this is not the case, it is the responsibility of
/// the upstream user.
pub fn register_block<E: EthSpec>(&self, block_root: Hash256, block: &BeaconBlock<E>) {
let mut map = self.0.write();
map.remove(&block.parent_root);
map.insert(block_root, block.slot);
}
/// Returns the list of heads in the chain.
pub fn heads(&self) -> Vec<(Hash256, Slot)> {
self.0
.read()
.iter()
.map(|(root, slot)| (*root, *slot))
.collect()
}
/// Returns a `SszHeadTracker`, which contains all necessary information to restore the state
/// of `Self` at some later point.
pub fn to_ssz_container(&self) -> SszHeadTracker {
let (roots, slots) = self
.0
.read()
.iter()
.map(|(hash, slot)| (*hash, *slot))
.unzip();
SszHeadTracker { roots, slots }
}
/// Creates a new `Self` from the given `SszHeadTracker`, restoring `Self` to the same state of
/// the `Self` that created the `SszHeadTracker`.
pub fn from_ssz_container(ssz_container: &SszHeadTracker) -> Result<Self, Error> {
let roots_len = ssz_container.roots.len();
let slots_len = ssz_container.slots.len();
if roots_len != slots_len {
return Err(Error::MismatchingLengths {
roots_len,
slots_len,
});
} else {
let map = HashMap::from_iter(
ssz_container
.roots
.iter()
.zip(ssz_container.slots.iter())
.map(|(root, slot)| (*root, *slot)),
);
Ok(Self(RwLock::new(map)))
}
}
}
impl PartialEq<HeadTracker> for HeadTracker {
fn eq(&self, other: &HeadTracker) -> bool {
*self.0.read() == *other.0.read()
}
}
/// Helper struct that is used to encode/decode the state of the `HeadTracker` as SSZ bytes.
///
/// This is used when persisting the state of the `BeaconChain` to disk.
#[derive(Encode, Decode, Clone)]
pub struct SszHeadTracker {
roots: Vec<Hash256>,
slots: Vec<Slot>,
}
#[cfg(test)]
mod test {
use super::*;
use ssz::{Decode, Encode};
use types::MainnetEthSpec;
type E = MainnetEthSpec;
#[test]
fn block_add() {
let spec = &E::default_spec();
let head_tracker = HeadTracker::default();
for i in 0..16 {
let mut block = BeaconBlock::empty(spec);
let block_root = Hash256::from_low_u64_be(i);
block.slot = Slot::new(i);
block.parent_root = if i == 0 {
Hash256::random()
} else {
Hash256::from_low_u64_be(i - 1)
};
head_tracker.register_block::<E>(block_root, &block);
}
assert_eq!(
head_tracker.heads(),
vec![(Hash256::from_low_u64_be(15), Slot::new(15))],
"should only have one head"
);
let mut block = BeaconBlock::empty(spec);
let block_root = Hash256::from_low_u64_be(42);
block.slot = Slot::new(15);
block.parent_root = Hash256::from_low_u64_be(14);
head_tracker.register_block::<E>(block_root, &block);
let heads = head_tracker.heads();
assert_eq!(heads.len(), 2, "should only have two heads");
assert!(
heads
.iter()
.any(|(root, slot)| *root == Hash256::from_low_u64_be(15) && *slot == Slot::new(15)),
"should contain first head"
);
assert!(
heads
.iter()
.any(|(root, slot)| *root == Hash256::from_low_u64_be(42) && *slot == Slot::new(15)),
"should contain second head"
);
}
#[test]
fn empty_round_trip() {
let non_empty = HeadTracker::default();
for i in 0..16 {
non_empty.0.write().insert(Hash256::random(), Slot::new(i));
}
let bytes = non_empty.to_ssz_container().as_ssz_bytes();
assert_eq!(
HeadTracker::from_ssz_container(
&SszHeadTracker::from_ssz_bytes(&bytes).expect("should decode")
),
Ok(non_empty),
"non_empty should pass round trip"
);
}
#[test]
fn non_empty_round_trip() {
let non_empty = HeadTracker::default();
for i in 0..16 {
non_empty.0.write().insert(Hash256::random(), Slot::new(i));
}
let bytes = non_empty.to_ssz_container().as_ssz_bytes();
assert_eq!(
HeadTracker::from_ssz_container(
&SszHeadTracker::from_ssz_bytes(&bytes).expect("should decode")
),
Ok(non_empty),
"non_empty should pass round trip"
);
}
#[test]
fn bad_length() {
let container = SszHeadTracker {
roots: vec![Hash256::random()],
slots: vec![],
};
let bytes = container.as_ssz_bytes();
assert_eq!(
HeadTracker::from_ssz_container(
&SszHeadTracker::from_ssz_bytes(&bytes).expect("should decode")
),
Err(Error::MismatchingLengths {
roots_len: 1,
slots_len: 0
}),
"should fail decoding with bad lengths"
);
}
}

1
beacon_node/beacon_chain/src/lib.rs
View File

@@ -9,6 +9,7 @@ mod errors;
pub mod eth1_chain;
pub mod events;
mod fork_choice;
mod head_tracker;
mod metrics;
mod persisted_beacon_chain;
pub mod test_utils;

7
beacon_node/beacon_chain/src/persisted_beacon_chain.rs
View File

@@ -1,3 +1,5 @@
use crate::fork_choice::SszForkChoice;
use crate::head_tracker::SszHeadTracker;
use crate::{BeaconChainTypes, CheckPoint};
use operation_pool::PersistedOperationPool;
use ssz::{Decode, Encode};
@@ -8,11 +10,14 @@ use types::Hash256;
/// 32-byte key for accessing the `PersistedBeaconChain`.
pub const BEACON_CHAIN_DB_KEY: &str = "PERSISTEDBEACONCHAINPERSISTEDBEA";
#[derive(Encode, Decode)]
#[derive(Clone, Encode, Decode)]
pub struct PersistedBeaconChain<T: BeaconChainTypes> {
pub canonical_head: CheckPoint<T::EthSpec>,
pub finalized_checkpoint: CheckPoint<T::EthSpec>,
pub op_pool: PersistedOperationPool<T::EthSpec>,
pub genesis_block_root: Hash256,
pub ssz_head_tracker: SszHeadTracker,
pub fork_choice: SszForkChoice,
}
impl<T: BeaconChainTypes> SimpleStoreItem for PersistedBeaconChain<T> {

50
beacon_node/beacon_chain/src/test_utils.rs
View File

@@ -25,7 +25,10 @@ use types::{
pub use crate::persisted_beacon_chain::{PersistedBeaconChain, BEACON_CHAIN_DB_KEY};
pub use types::test_utils::generate_deterministic_keypairs;
pub const HARNESS_GENESIS_TIME: u64 = 1_567_552_690; // 4th September 2019
// 4th September 2019
pub const HARNESS_GENESIS_TIME: u64 = 1_567_552_690;
// This parameter is required by a builder but not used because we use the `TestingSlotClock`.
pub const HARNESS_SLOT_TIME: Duration = Duration::from_secs(1);
pub type BaseHarnessType<TStore, TStoreMigrator, TEthSpec> = Witness<
TStore,
@@ -98,9 +101,9 @@ impl<E: EthSpec> BeaconChainHarness<HarnessType<E>> {
.dummy_eth1_backend()
.expect("should build dummy backend")
.null_event_handler()
.testing_slot_clock(Duration::from_secs(1))
.testing_slot_clock(HARNESS_SLOT_TIME)
.expect("should configure testing slot clock")
.empty_reduced_tree_fork_choice()
.reduced_tree_fork_choice()
.expect("should add fork choice to builder")
.build()
.expect("should build");
@@ -115,7 +118,7 @@ impl<E: EthSpec> BeaconChainHarness<HarnessType<E>> {
impl<E: EthSpec> BeaconChainHarness<DiskHarnessType<E>> {
/// Instantiate a new harness with `validator_count` initial validators.
pub fn with_disk_store(
pub fn new_with_disk_store(
eth_spec_instance: E,
store: Arc<DiskStore>,
keypairs: Vec<Keypair>,
@@ -140,9 +143,46 @@ impl<E: EthSpec> BeaconChainHarness<DiskHarnessType<E>> {
.dummy_eth1_backend()
.expect("should build dummy backend")
.null_event_handler()
.testing_slot_clock(HARNESS_SLOT_TIME)
.expect("should configure testing slot clock")
.reduced_tree_fork_choice()
.expect("should add fork choice to builder")
.build()
.expect("should build");
Self {
spec: chain.spec.clone(),
chain,
keypairs,
}
}
/// Instantiate a new harness with `validator_count` initial validators.
pub fn resume_from_disk_store(
eth_spec_instance: E,
store: Arc<DiskStore>,
keypairs: Vec<Keypair>,
) -> Self {
let spec = E::default_spec();
let log = TerminalLoggerBuilder::new()
.level(Severity::Warning)
.build()
.expect("logger should build");
let chain = BeaconChainBuilder::new(eth_spec_instance)
.logger(log.clone())
.custom_spec(spec.clone())
.store(store.clone())
.store_migrator(<BlockingMigrator<_> as Migrate<_, E>>::new(store))
.resume_from_db()
.expect("should resume beacon chain from db")
.dummy_eth1_backend()
.expect("should build dummy backend")
.null_event_handler()
.testing_slot_clock(Duration::from_secs(1))
.expect("should configure testing slot clock")
.empty_reduced_tree_fork_choice()
.reduced_tree_fork_choice()
.expect("should add fork choice to builder")
.build()
.expect("should build");