Files
lighthouse/testing/ef_tests/src/cases/epoch_processing.rs
Paul Hauner 6e3ca48cb9 Cache participating indices for Altair epoch processing (#2416)
## Issue Addressed

NA

## Proposed Changes

This PR addresses two things:

1. Allows the `ValidatorMonitor` to work with Altair states.
1. Optimizes `altair::process_epoch` (see [code](https://github.com/paulhauner/lighthouse/blob/participation-cache/consensus/state_processing/src/per_epoch_processing/altair/participation_cache.rs) for description)

## Breaking Changes

The breaking changes in this PR revolve around one premise:

*After the Altair fork, it's not longer possible (given only a `BeaconState`) to identify if a validator had *any* attestation included during some epoch. The best we can do is see if that validator made the "timely" source/target/head flags.*

Whilst this seems annoying, it's not actually too bad. Finalization is based upon "timely target" attestations, so that's really the most important thing. Although there's *some* value in knowing if a validator had *any* attestation included, it's far more important to know about "timely target" participation, since this is what affects finality and justification.

For simplicity and consistency, I've also removed the ability to determine if *any* attestation was included from metrics and API endpoints. Now, all Altair and non-Altair states will simply report on the head/target attestations.

The following section details where we've removed fields and provides replacement values.

### Breaking Changes: Prometheus Metrics

Some participation metrics have been removed and replaced. Some were removed since they are no longer relevant to Altair (e.g., total attesting balance) and others replaced with gwei values instead of pre-computed values. This provides more flexibility at display-time (e.g., Grafana).

The following metrics were added as replacements:

- `beacon_participation_prev_epoch_head_attesting_gwei_total`
- `beacon_participation_prev_epoch_target_attesting_gwei_total`
- `beacon_participation_prev_epoch_source_attesting_gwei_total`
- `beacon_participation_prev_epoch_active_gwei_total`

The following metrics were removed:

- `beacon_participation_prev_epoch_attester`
   - instead use `beacon_participation_prev_epoch_source_attesting_gwei_total / beacon_participation_prev_epoch_active_gwei_total`.
- `beacon_participation_prev_epoch_target_attester`
   - instead use `beacon_participation_prev_epoch_target_attesting_gwei_total / beacon_participation_prev_epoch_active_gwei_total`.
- `beacon_participation_prev_epoch_head_attester`
   - instead use `beacon_participation_prev_epoch_head_attesting_gwei_total / beacon_participation_prev_epoch_active_gwei_total`.

The `beacon_participation_prev_epoch_attester` endpoint has been removed. Users should instead use the pre-existing `beacon_participation_prev_epoch_target_attester`. 

### Breaking Changes: HTTP API

The `/lighthouse/validator_inclusion/{epoch}/{validator_id}` endpoint loses the following fields:

- `current_epoch_attesting_gwei` (use `current_epoch_target_attesting_gwei` instead)
- `previous_epoch_attesting_gwei` (use `previous_epoch_target_attesting_gwei` instead)

The `/lighthouse/validator_inclusion/{epoch}/{validator_id}` endpoint lose the following fields:

- `is_current_epoch_attester` (use `is_current_epoch_target_attester` instead)
- `is_previous_epoch_attester` (use `is_previous_epoch_target_attester` instead)
- `is_active_in_current_epoch` becomes `is_active_unslashed_in_current_epoch`.
- `is_active_in_previous_epoch` becomes `is_active_unslashed_in_previous_epoch`.

## Additional Info

NA

## TODO

- [x] Deal with total balances
- [x] Update validator_inclusion API
- [ ] Ensure `beacon_participation_prev_epoch_target_attester` and `beacon_participation_prev_epoch_head_attester` work before Altair

Co-authored-by: realbigsean <seananderson33@gmail.com>
2021-07-27 07:01:01 +00:00

292 lines
9.9 KiB
Rust

use super::*;
use crate::bls_setting::BlsSetting;
use crate::case_result::compare_beacon_state_results_without_caches;
use crate::decode::{ssz_decode_state, yaml_decode_file};
use crate::type_name;
use crate::type_name::TypeName;
use serde_derive::Deserialize;
use state_processing::per_epoch_processing::{
altair, base,
effective_balance_updates::process_effective_balance_updates,
historical_roots_update::process_historical_roots_update,
process_registry_updates, process_slashings,
resets::{process_eth1_data_reset, process_randao_mixes_reset, process_slashings_reset},
};
use state_processing::EpochProcessingError;
use std::marker::PhantomData;
use std::path::{Path, PathBuf};
use types::{BeaconState, ChainSpec, EthSpec, ForkName};
#[derive(Debug, Clone, Default, Deserialize)]
pub struct Metadata {
pub description: Option<String>,
pub bls_setting: Option<BlsSetting>,
}
#[derive(Debug, Clone, Deserialize)]
#[serde(bound = "E: EthSpec")]
pub struct EpochProcessing<E: EthSpec, T: EpochTransition<E>> {
pub path: PathBuf,
pub metadata: Metadata,
pub pre: BeaconState<E>,
pub post: Option<BeaconState<E>>,
#[serde(skip_deserializing)]
_phantom: PhantomData<T>,
}
pub trait EpochTransition<E: EthSpec>: TypeName + Debug + Sync {
fn run(state: &mut BeaconState<E>, spec: &ChainSpec) -> Result<(), EpochProcessingError>;
}
#[derive(Debug)]
pub struct JustificationAndFinalization;
#[derive(Debug)]
pub struct RewardsAndPenalties;
#[derive(Debug)]
pub struct RegistryUpdates;
#[derive(Debug)]
pub struct Slashings;
#[derive(Debug)]
pub struct Eth1DataReset;
#[derive(Debug)]
pub struct EffectiveBalanceUpdates;
#[derive(Debug)]
pub struct SlashingsReset;
#[derive(Debug)]
pub struct RandaoMixesReset;
#[derive(Debug)]
pub struct HistoricalRootsUpdate;
#[derive(Debug)]
pub struct ParticipationRecordUpdates;
#[derive(Debug)]
pub struct SyncCommitteeUpdates;
#[derive(Debug)]
pub struct InactivityUpdates;
#[derive(Debug)]
pub struct ParticipationFlagUpdates;
type_name!(
JustificationAndFinalization,
"justification_and_finalization"
);
type_name!(RewardsAndPenalties, "rewards_and_penalties");
type_name!(RegistryUpdates, "registry_updates");
type_name!(Slashings, "slashings");
type_name!(Eth1DataReset, "eth1_data_reset");
type_name!(EffectiveBalanceUpdates, "effective_balance_updates");
type_name!(SlashingsReset, "slashings_reset");
type_name!(RandaoMixesReset, "randao_mixes_reset");
type_name!(HistoricalRootsUpdate, "historical_roots_update");
type_name!(ParticipationRecordUpdates, "participation_record_updates");
type_name!(SyncCommitteeUpdates, "sync_committee_updates");
type_name!(InactivityUpdates, "inactivity_updates");
type_name!(ParticipationFlagUpdates, "participation_flag_updates");
impl<E: EthSpec> EpochTransition<E> for JustificationAndFinalization {
fn run(state: &mut BeaconState<E>, spec: &ChainSpec) -> Result<(), EpochProcessingError> {
match state {
BeaconState::Base(_) => {
let mut validator_statuses = base::ValidatorStatuses::new(state, spec)?;
validator_statuses.process_attestations(state)?;
base::process_justification_and_finalization(
state,
&validator_statuses.total_balances,
spec,
)
}
BeaconState::Altair(_) => altair::process_justification_and_finalization(
state,
&altair::ParticipationCache::new(state, spec).unwrap(),
),
}
}
}
impl<E: EthSpec> EpochTransition<E> for RewardsAndPenalties {
fn run(state: &mut BeaconState<E>, spec: &ChainSpec) -> Result<(), EpochProcessingError> {
match state {
BeaconState::Base(_) => {
let mut validator_statuses = base::ValidatorStatuses::new(state, spec)?;
validator_statuses.process_attestations(state)?;
base::process_rewards_and_penalties(state, &mut validator_statuses, spec)
}
BeaconState::Altair(_) => altair::process_rewards_and_penalties(
state,
&altair::ParticipationCache::new(state, spec).unwrap(),
spec,
),
}
}
}
impl<E: EthSpec> EpochTransition<E> for RegistryUpdates {
fn run(state: &mut BeaconState<E>, spec: &ChainSpec) -> Result<(), EpochProcessingError> {
process_registry_updates(state, spec)
}
}
impl<E: EthSpec> EpochTransition<E> for Slashings {
fn run(state: &mut BeaconState<E>, spec: &ChainSpec) -> Result<(), EpochProcessingError> {
match state {
BeaconState::Base(_) => {
let mut validator_statuses = base::ValidatorStatuses::new(&state, spec)?;
validator_statuses.process_attestations(&state)?;
process_slashings(
state,
validator_statuses.total_balances.current_epoch(),
spec.proportional_slashing_multiplier,
spec,
)?;
}
BeaconState::Altair(_) => {
process_slashings(
state,
altair::ParticipationCache::new(state, spec)
.unwrap()
.current_epoch_total_active_balance(),
spec.proportional_slashing_multiplier_altair,
spec,
)?;
}
};
Ok(())
}
}
impl<E: EthSpec> EpochTransition<E> for Eth1DataReset {
fn run(state: &mut BeaconState<E>, _spec: &ChainSpec) -> Result<(), EpochProcessingError> {
process_eth1_data_reset(state)
}
}
impl<E: EthSpec> EpochTransition<E> for EffectiveBalanceUpdates {
fn run(state: &mut BeaconState<E>, spec: &ChainSpec) -> Result<(), EpochProcessingError> {
process_effective_balance_updates(state, spec)
}
}
impl<E: EthSpec> EpochTransition<E> for SlashingsReset {
fn run(state: &mut BeaconState<E>, _spec: &ChainSpec) -> Result<(), EpochProcessingError> {
process_slashings_reset(state)
}
}
impl<E: EthSpec> EpochTransition<E> for RandaoMixesReset {
fn run(state: &mut BeaconState<E>, _spec: &ChainSpec) -> Result<(), EpochProcessingError> {
process_randao_mixes_reset(state)
}
}
impl<E: EthSpec> EpochTransition<E> for HistoricalRootsUpdate {
fn run(state: &mut BeaconState<E>, _spec: &ChainSpec) -> Result<(), EpochProcessingError> {
process_historical_roots_update(state)
}
}
impl<E: EthSpec> EpochTransition<E> for ParticipationRecordUpdates {
fn run(state: &mut BeaconState<E>, _spec: &ChainSpec) -> Result<(), EpochProcessingError> {
if let BeaconState::Base(_) = state {
base::process_participation_record_updates(state)
} else {
Ok(())
}
}
}
impl<E: EthSpec> EpochTransition<E> for SyncCommitteeUpdates {
fn run(state: &mut BeaconState<E>, spec: &ChainSpec) -> Result<(), EpochProcessingError> {
match state {
BeaconState::Base(_) => Ok(()),
BeaconState::Altair(_) => altair::process_sync_committee_updates(state, spec),
}
}
}
impl<E: EthSpec> EpochTransition<E> for InactivityUpdates {
fn run(state: &mut BeaconState<E>, spec: &ChainSpec) -> Result<(), EpochProcessingError> {
match state {
BeaconState::Base(_) => Ok(()),
BeaconState::Altair(_) => altair::process_inactivity_updates(
state,
&altair::ParticipationCache::new(state, spec).unwrap(),
spec,
),
}
}
}
impl<E: EthSpec> EpochTransition<E> for ParticipationFlagUpdates {
fn run(state: &mut BeaconState<E>, _: &ChainSpec) -> Result<(), EpochProcessingError> {
match state {
BeaconState::Base(_) => Ok(()),
BeaconState::Altair(_) => altair::process_participation_flag_updates(state),
}
}
}
impl<E: EthSpec, T: EpochTransition<E>> LoadCase for EpochProcessing<E, T> {
fn load_from_dir(path: &Path, fork_name: ForkName) -> Result<Self, Error> {
let spec = &testing_spec::<E>(fork_name);
let metadata_path = path.join("meta.yaml");
let metadata: Metadata = if metadata_path.is_file() {
yaml_decode_file(&metadata_path)?
} else {
Metadata::default()
};
let pre = ssz_decode_state(&path.join("pre.ssz_snappy"), spec)?;
let post_file = path.join("post.ssz_snappy");
let post = if post_file.is_file() {
Some(ssz_decode_state(&post_file, spec)?)
} else {
None
};
Ok(Self {
path: path.into(),
metadata,
pre,
post,
_phantom: PhantomData,
})
}
}
impl<E: EthSpec, T: EpochTransition<E>> Case for EpochProcessing<E, T> {
fn description(&self) -> String {
self.metadata
.description
.clone()
.unwrap_or_else(String::new)
}
fn is_enabled_for_fork(fork_name: ForkName) -> bool {
match fork_name {
// No Altair tests for genesis fork.
ForkName::Base => {
T::name() != "sync_committee_updates"
&& T::name() != "inactivity_updates"
&& T::name() != "participation_flag_updates"
}
ForkName::Altair => true,
}
}
fn result(&self, _case_index: usize, fork_name: ForkName) -> Result<(), Error> {
self.metadata.bls_setting.unwrap_or_default().check()?;
let mut state = self.pre.clone();
let mut expected = self.post.clone();
let spec = &testing_spec::<E>(fork_name);
let mut result = (|| {
// Processing requires the committee caches.
state.build_all_committee_caches(spec)?;
T::run(&mut state, spec).map(|_| state)
})();
compare_beacon_state_results_without_caches(&mut result, &mut expected)
}
}