gnosis/lighthouse
1
0
Fork 0
mirror of https://github.com/sigp/lighthouse.git synced 2026-03-15 19:02:42 +00:00
Code Issues Packages Projects Releases Wiki Activity

Update to frozen spec ❄️ (v0.8.1) (#444)

Browse Source 
* types: first updates for v0.8

* state_processing: epoch processing v0.8.0

* state_processing: block processing v0.8.0

* tree_hash_derive: support generics in SignedRoot

* types v0.8: update to use ssz_types

* state_processing v0.8: use ssz_types

* ssz_types: add bitwise methods and from_elem

* types: fix v0.8 FIXMEs

* ssz_types: add bitfield shift_up

* ssz_types: iterators and DerefMut for VariableList

* types,state_processing: use VariableList

* ssz_types: fix BitVector Decode impl

Fixed a typo in the implementation of ssz::Decode for BitVector, which caused it
to be considered variable length!

* types: fix test modules for v0.8 update

* types: remove slow type-level arithmetic

* state_processing: fix tests for v0.8

* op_pool: update for v0.8

* ssz_types: Bitfield difference length-independent

Allow computing the difference of two bitfields of different lengths.

* Implement compact committee support

* epoch_processing: committee & active index roots

* state_processing: genesis state builder v0.8

* state_processing: implement v0.8.1

* Further improve tree_hash

* Strip examples, tests from cached_tree_hash

* Update TreeHash, un-impl CachedTreeHash

* Update bitfield TreeHash, un-impl CachedTreeHash

* Update FixedLenVec TreeHash, unimpl CachedTreeHash

* Update update tree_hash_derive for new TreeHash

* Fix TreeHash, un-impl CachedTreeHash for ssz_types

* Remove fixed_len_vec, ssz benches

SSZ benches relied upon fixed_len_vec -- it is easier to just delete
them and rebuild them later (when necessary)

* Remove boolean_bitfield crate

* Fix fake_crypto BLS compile errors

* Update ef_tests for new v.8 type params

* Update ef_tests submodule to v0.8.1 tag

* Make fixes to support parsing ssz ef_tests

* `compact_committee...` to `compact_committees...`

* Derive more traits for `CompactCommittee`

* Flip bitfield byte-endianness

* Fix tree_hash for bitfields

* Modify CLI output for ef_tests

* Bump ssz crate version

* Update ssz_types doc comment

* Del cached tree hash tests from ssz_static tests

* Tidy SSZ dependencies

* Rename ssz_types crate to eth2_ssz_types

* validator_client: update for v0.8

* ssz_types: update union/difference for bit order swap

* beacon_node: update for v0.8, EthSpec

* types: disable cached tree hash, update min spec

* state_processing: fix slot bug in committee update

* tests: temporarily disable fork choice harness test

See #447

* committee cache: prevent out-of-bounds access

In the case where we tried to access the committee of a shard that didn't have a committee in the
current epoch, we were accessing elements beyond the end of the shuffling vector and panicking! This
commit adds a check to make the failure safe and explicit.

* fix bug in get_indexed_attestation and simplify

There was a bug in our implementation of get_indexed_attestation whereby
incorrect "committee indices" were used to index into the custody bitfield. The
bug was only observable in the case where some bits of the custody bitfield were
set to 1. The implementation has been simplified to remove the bug, and a test
added.

* state_proc: workaround for compact committees bug

https://github.com/ethereum/eth2.0-specs/issues/1315

* v0.8: updates to make the EF tests pass

* Remove redundant max operation checks.
* Always supply both messages when checking attestation signatures -- allowing
  verification of an attestation with no signatures.
* Swap the order of the fork and domain constant in `get_domain`, to match
  the spec.

* rustfmt

* ef_tests: add new epoch processing tests

* Integrate v0.8 into master (compiles)

* Remove unused crates, fix clippy lints

* Replace v0.6.3 tags w/ v0.8.1

* Remove old comment

* Ensure lmd ghost tests only run in release

* Update readme
This commit is contained in:
Michael Sproul
2019-07-30 12:44:51 +10:00
committed by Paul Hauner
parent 177df12149
commit a236003a7b
184 changed files with 3332 additions and 4542 deletions
Download Patch File Download Diff File Expand all files Collapse all files

33
eth2/state_processing/src/common/convert_to_indexed.rs
View File

@@ -1,33 +0,0 @@
use super::{get_attesting_indices, get_attesting_indices_unsorted};
use itertools::{Either, Itertools};
use types::*;
/// Convert `attestation` to (almost) indexed-verifiable form.
///
/// Spec v0.6.3
pub fn convert_to_indexed<T: EthSpec>(
state: &BeaconState<T>,
attestation: &Attestation,
) -> Result<IndexedAttestation, BeaconStateError> {
let attesting_indices =
get_attesting_indices(state, &attestation.data, &attestation.aggregation_bitfield)?;
// We verify the custody bitfield by calling `get_attesting_indices_unsorted` and throwing
// away the result. This avoids double-sorting - the partition below takes care of the ordering.
get_attesting_indices_unsorted(state, &attestation.data, &attestation.custody_bitfield)?;
let (custody_bit_0_indices, custody_bit_1_indices) =
attesting_indices.into_iter().enumerate().partition_map(
|(committee_idx, validator_idx)| match attestation.custody_bitfield.get(committee_idx) {
Ok(true) => Either::Right(validator_idx as u64),
_ => Either::Left(validator_idx as u64),
},
);
Ok(IndexedAttestation {
custody_bit_0_indices,
custody_bit_1_indices,
data: attestation.data.clone(),
signature: attestation.signature.clone(),
})
}

37
eth2/state_processing/src/common/get_attesting_indices.rs
View File

@@ -1,44 +1,33 @@
use crate::common::verify_bitfield_length;
use std::collections::BTreeSet;
use types::*;
/// Returns validator indices which participated in the attestation, sorted by increasing index.
///
/// Spec v0.6.3
/// Spec v0.8.1
pub fn get_attesting_indices<T: EthSpec>(
state: &BeaconState<T>,
attestation_data: &AttestationData,
bitfield: &Bitfield,
) -> Result<Vec<usize>, BeaconStateError> {
get_attesting_indices_unsorted(state, attestation_data, bitfield).map(|mut indices| {
// Fast unstable sort is safe because validator indices are unique
indices.sort_unstable();
indices
})
}
/// Returns validator indices which participated in the attestation, unsorted.
///
/// Spec v0.6.3
pub fn get_attesting_indices_unsorted<T: EthSpec>(
state: &BeaconState<T>,
attestation_data: &AttestationData,
bitfield: &Bitfield,
) -> Result<Vec<usize>, BeaconStateError> {
bitlist: &BitList<T::MaxValidatorsPerCommittee>,
) -> Result<BTreeSet<usize>, BeaconStateError> {
let target_relative_epoch =
RelativeEpoch::from_epoch(state.current_epoch(), attestation_data.target_epoch)?;
RelativeEpoch::from_epoch(state.current_epoch(), attestation_data.target.epoch)?;
let committee =
state.get_crosslink_committee_for_shard(attestation_data.shard, target_relative_epoch)?;
let committee = state.get_crosslink_committee_for_shard(
attestation_data.crosslink.shard,
target_relative_epoch,
)?;
if !verify_bitfield_length(&bitfield, committee.committee.len()) {
/* TODO(freeze): re-enable this?
if bitlist.len() > committee.committee.len() {
return Err(BeaconStateError::InvalidBitfield);
}
*/
Ok(committee
.committee
.iter()
.enumerate()
.filter_map(|(i, validator_index)| match bitfield.get(i) {
.filter_map(|(i, validator_index)| match bitlist.get(i) {
Ok(true) => Some(*validator_index),
_ => None,
})

49
eth2/state_processing/src/common/get_compact_committees_root.rs Normal file
View File

@@ -0,0 +1,49 @@
use tree_hash::TreeHash;
use types::*;
/// Return the compact committee root at `relative_epoch`.
///
/// Spec v0.8.0
pub fn get_compact_committees_root<T: EthSpec>(
state: &BeaconState<T>,
relative_epoch: RelativeEpoch,
spec: &ChainSpec,
) -> Result<Hash256, BeaconStateError> {
let mut committees =
FixedVector::<_, T::ShardCount>::from_elem(CompactCommittee::<T>::default());
// FIXME: this is a spec bug, whereby the start shard for the epoch after the next epoch
// is mistakenly used. The start shard from the cache SHOULD work.
// Waiting on a release to fix https://github.com/ethereum/eth2.0-specs/issues/1315
// let start_shard = state.get_epoch_start_shard(relative_epoch)?;
let start_shard = state.next_epoch_start_shard(spec)?;
for committee_number in 0..state.get_committee_count(relative_epoch)? {
let shard = (start_shard + committee_number) % T::ShardCount::to_u64();
// FIXME: this is a partial workaround for the above, but it only works in the case
// where there's a committee for every shard in every epoch. It works for the minimal
// tests but not the mainnet ones.
let fake_shard = (shard + 1) % T::ShardCount::to_u64();
for &index in state
.get_crosslink_committee_for_shard(fake_shard, relative_epoch)?
.committee
{
let validator = state
.validators
.get(index)
.ok_or(BeaconStateError::UnknownValidator)?;
committees[shard as usize]
.pubkeys
.push(validator.pubkey.clone())?;
let compact_balance = validator.effective_balance / spec.effective_balance_increment;
// `index` (top 6 bytes) + `slashed` (16th bit) + `compact_balance` (bottom 15 bits)
let compact_validator: u64 =
((index as u64) << 16) + (u64::from(validator.slashed) << 15) + compact_balance;
committees[shard as usize]
.compact_validators
.push(compact_validator)?;
}
}
Ok(Hash256::from_slice(&committees.tree_hash_root()))
}

122
eth2/state_processing/src/common/get_indexed_attestation.rs Normal file
View File

@@ -0,0 +1,122 @@
use super::get_attesting_indices;
use crate::per_block_processing::errors::{
AttestationInvalid as Invalid, AttestationValidationError as Error,
};
use types::*;
/// Convert `attestation` to (almost) indexed-verifiable form.
///
/// Spec v0.8.0
pub fn get_indexed_attestation<T: EthSpec>(
state: &BeaconState<T>,
attestation: &Attestation<T>,
) -> Result<IndexedAttestation<T>, Error> {
let attesting_indices =
get_attesting_indices(state, &attestation.data, &attestation.aggregation_bits)?;
let custody_bit_1_indices =
get_attesting_indices(state, &attestation.data, &attestation.custody_bits)?;
verify!(
custody_bit_1_indices.is_subset(&attesting_indices),
Invalid::CustodyBitfieldNotSubset
);
let custody_bit_0_indices = &attesting_indices - &custody_bit_1_indices;
Ok(IndexedAttestation {
custody_bit_0_indices: VariableList::new(
custody_bit_0_indices
.into_iter()
.map(|x| x as u64)
.collect(),
)?,
custody_bit_1_indices: VariableList::new(
custody_bit_1_indices
.into_iter()
.map(|x| x as u64)
.collect(),
)?,
data: attestation.data.clone(),
signature: attestation.signature.clone(),
})
}
#[cfg(test)]
mod test {
use super::*;
use itertools::{Either, Itertools};
use types::test_utils::*;
#[test]
fn custody_bitfield_indexing() {
let validator_count = 128;
let spec = MinimalEthSpec::default_spec();
let state_builder =
TestingBeaconStateBuilder::<MinimalEthSpec>::from_default_keypairs_file_if_exists(
validator_count,
&spec,
);
let (mut state, keypairs) = state_builder.build();
state.build_all_caches(&spec).unwrap();
state.slot += 1;
let shard = 0;
let cc = state
.get_crosslink_committee_for_shard(shard, RelativeEpoch::Current)
.unwrap();
// Make a third of the validators sign with custody bit 0, a third with custody bit 1
// and a third not sign at all.
assert!(
cc.committee.len() >= 4,
"need at least 4 validators per committee for this test to work"
);
let (mut bit_0_indices, mut bit_1_indices): (Vec<_>, Vec<_>) = cc
.committee
.iter()
.enumerate()
.filter(|(i, _)| i % 3 != 0)
.partition_map(|(i, index)| {
if i % 3 == 1 {
Either::Left(*index)
} else {
Either::Right(*index)
}
});
assert!(!bit_0_indices.is_empty());
assert!(!bit_1_indices.is_empty());
let bit_0_keys = bit_0_indices
.iter()
.map(|validator_index| &keypairs[*validator_index].sk)
.collect::<Vec<_>>();
let bit_1_keys = bit_1_indices
.iter()
.map(|validator_index| &keypairs[*validator_index].sk)
.collect::<Vec<_>>();
let mut attestation_builder =
TestingAttestationBuilder::new(&state, &cc.committee, cc.slot, shard, &spec);
attestation_builder
.sign(&bit_0_indices, &bit_0_keys, &state.fork, &spec, false)
.sign(&bit_1_indices, &bit_1_keys, &state.fork, &spec, true);
let attestation = attestation_builder.build();
let indexed_attestation = get_indexed_attestation(&state, &attestation).unwrap();
bit_0_indices.sort();
bit_1_indices.sort();
assert!(indexed_attestation
.custody_bit_0_indices
.iter()
.copied()
.eq(bit_0_indices.iter().map(|idx| *idx as u64)));
assert!(indexed_attestation
.custody_bit_1_indices
.iter()
.copied()
.eq(bit_1_indices.iter().map(|idx| *idx as u64)));
}
}

12
eth2/state_processing/src/common/initiate_validator_exit.rs
View File

@@ -3,23 +3,23 @@ use types::{BeaconStateError as Error, *};
/// Initiate the exit of the validator of the given `index`.
///
/// Spec v0.6.3
/// Spec v0.8.1
pub fn initiate_validator_exit<T: EthSpec>(
state: &mut BeaconState<T>,
index: usize,
spec: &ChainSpec,
) -> Result<(), Error> {
if index >= state.validator_registry.len() {
if index >= state.validators.len() {
return Err(Error::UnknownValidator);
}
// Return if the validator already initiated exit
if state.validator_registry[index].exit_epoch != spec.far_future_epoch {
if state.validators[index].exit_epoch != spec.far_future_epoch {
return Ok(());
}
// Compute exit queue epoch
let delayed_epoch = state.get_delayed_activation_exit_epoch(state.current_epoch(), spec);
let delayed_epoch = state.compute_activation_exit_epoch(state.current_epoch(), spec);
let mut exit_queue_epoch = state
.exit_cache
.max_epoch()
@@ -31,8 +31,8 @@ pub fn initiate_validator_exit<T: EthSpec>(
}
state.exit_cache.record_validator_exit(exit_queue_epoch);
state.validator_registry[index].exit_epoch = exit_queue_epoch;
state.validator_registry[index].withdrawable_epoch =
state.validators[index].exit_epoch = exit_queue_epoch;
state.validators[index].withdrawable_epoch =
exit_queue_epoch + spec.min_validator_withdrawability_delay;
Ok(())

10
eth2/state_processing/src/common/mod.rs
View File

@@ -1,11 +1,11 @@
mod convert_to_indexed;
mod get_attesting_indices;
mod get_compact_committees_root;
mod get_indexed_attestation;
mod initiate_validator_exit;
mod slash_validator;
mod verify_bitfield;
pub use convert_to_indexed::convert_to_indexed;
pub use get_attesting_indices::{get_attesting_indices, get_attesting_indices_unsorted};
pub use get_attesting_indices::get_attesting_indices;
pub use get_compact_committees_root::get_compact_committees_root;
pub use get_indexed_attestation::get_indexed_attestation;
pub use initiate_validator_exit::initiate_validator_exit;
pub use slash_validator::slash_validator;
pub use verify_bitfield::verify_bitfield_length;

36
eth2/state_processing/src/common/slash_validator.rs
View File

@@ -1,45 +1,51 @@
use crate::common::initiate_validator_exit;
use std::cmp;
use types::{BeaconStateError as Error, *};
/// Slash the validator with index ``index``.
///
/// Spec v0.6.3
/// Spec v0.8.0
pub fn slash_validator<T: EthSpec>(
state: &mut BeaconState<T>,
slashed_index: usize,
opt_whistleblower_index: Option<usize>,
spec: &ChainSpec,
) -> Result<(), Error> {
if slashed_index >= state.validator_registry.len() || slashed_index >= state.balances.len() {
if slashed_index >= state.validators.len() || slashed_index >= state.balances.len() {
return Err(BeaconStateError::UnknownValidator);
}
let current_epoch = state.current_epoch();
let epoch = state.current_epoch();
initiate_validator_exit(state, slashed_index, spec)?;
state.validator_registry[slashed_index].slashed = true;
state.validator_registry[slashed_index].withdrawable_epoch =
current_epoch + Epoch::from(T::latest_slashed_exit_length());
let slashed_balance = state.get_effective_balance(slashed_index, spec)?;
state.set_slashed_balance(
current_epoch,
state.get_slashed_balance(current_epoch)? + slashed_balance,
state.validators[slashed_index].slashed = true;
state.validators[slashed_index].withdrawable_epoch = cmp::max(
state.validators[slashed_index].withdrawable_epoch,
epoch + Epoch::from(T::EpochsPerSlashingsVector::to_u64()),
);
let validator_effective_balance = state.get_effective_balance(slashed_index, spec)?;
state.set_slashings(
epoch,
state.get_slashings(epoch)? + validator_effective_balance,
)?;
safe_sub_assign!(
state.balances[slashed_index],
validator_effective_balance / spec.min_slashing_penalty_quotient
);
// Apply proposer and whistleblower rewards
let proposer_index =
state.get_beacon_proposer_index(state.slot, RelativeEpoch::Current, spec)?;
let whistleblower_index = opt_whistleblower_index.unwrap_or(proposer_index);
let whistleblowing_reward = slashed_balance / spec.whistleblowing_reward_quotient;
let proposer_reward = whistleblowing_reward / spec.proposer_reward_quotient;
let whistleblower_reward = validator_effective_balance / spec.whistleblower_reward_quotient;
let proposer_reward = whistleblower_reward / spec.proposer_reward_quotient;
safe_add_assign!(state.balances[proposer_index], proposer_reward);
safe_add_assign!(
state.balances[whistleblower_index],
whistleblowing_reward.saturating_sub(proposer_reward)
whistleblower_reward.saturating_sub(proposer_reward)
);
safe_sub_assign!(state.balances[slashed_index], whistleblowing_reward);
Ok(())
}

79
eth2/state_processing/src/common/verify_bitfield.rs
View File

@@ -1,79 +0,0 @@
use types::*;
/// Verify ``bitfield`` against the ``committee_size``.
///
/// Is title `verify_bitfield` in spec.
///
/// Spec v0.6.3
pub fn verify_bitfield_length(bitfield: &Bitfield, committee_size: usize) -> bool {
if bitfield.num_bytes() != ((committee_size + 7) / 8) {
return false;
}
for i in committee_size..(bitfield.num_bytes() * 8) {
if bitfield.get(i).unwrap_or(false) {
return false;
}
}
true
}
#[cfg(test)]
mod test {
use super::*;
#[test]
fn bitfield_length() {
assert_eq!(
verify_bitfield_length(&Bitfield::from_bytes(&[0b0000_0001]), 4),
true
);
assert_eq!(
verify_bitfield_length(&Bitfield::from_bytes(&[0b0001_0001]), 4),
false
);
assert_eq!(
verify_bitfield_length(&Bitfield::from_bytes(&[0b0000_0000]), 4),
true
);
assert_eq!(
verify_bitfield_length(&Bitfield::from_bytes(&[0b1000_0000]), 8),
true
);
assert_eq!(
verify_bitfield_length(&Bitfield::from_bytes(&[0b1000_0000, 0b0000_0000]), 16),
true
);
assert_eq!(
verify_bitfield_length(&Bitfield::from_bytes(&[0b1000_0000, 0b0000_0000]), 15),
false
);
assert_eq!(
verify_bitfield_length(&Bitfield::from_bytes(&[0b0000_0000, 0b0000_0000]), 8),
false
);
assert_eq!(
verify_bitfield_length(
&Bitfield::from_bytes(&[0b0000_0000, 0b0000_0000, 0b0000_0000]),
8
),
false
);
assert_eq!(
verify_bitfield_length(
&Bitfield::from_bytes(&[0b0000_0000, 0b0000_0000, 0b0000_0000]),
24
),
true
);
}
}