gnosis/lighthouse
1
0
Fork 0
mirror of https://github.com/sigp/lighthouse.git synced 2026-04-30 19:23:50 +00:00
Code Issues Packages Projects Releases Wiki Activity

Add Holesky (#4653)

Browse Source 
## Issue Addressed

NA

## Proposed Changes

Add the Holesky network config as per 36e4ff2d51/custom_config_data.

Since the genesis state is ~190MB, I've opted to *not* include it in the binary and instead download it at runtime (see #4564 for context). To download this file we have:

- A hard-coded URL for a SigP-hosted S3 bucket with the Holesky genesis state. Assuming this download works correctly, users will be none the wiser that the state wasn't included in the binary (apart from some additional logs)
- If the user provides a `--checkpoint-sync-url` flag, then LH will download the genesis state from that server rather than our S3 bucket.
- If the user provides a `--genesis-state-url` flag, then LH will download the genesis state from that server regardless of the S3 bucket or `--checkpoint-sync-url` flag.
- Whenever a genesis state is downloaded it is checked against a checksum baked into the binary.
- A genesis state will never be downloaded if it's already included in the binary.
- There is a `--genesis-state-url-timeout` flag to tweak the timeout for downloading the genesis state file.

## Log Output

Example of log output when a state is downloaded:

```bash
Aug 23 05:40:13.424 INFO Logging to file                         path: "/Users/paul/.lighthouse/holesky/beacon/logs/beacon.log"
Aug 23 05:40:13.425 INFO Lighthouse started                      version: Lighthouse/v4.3.0-bd9931f+
Aug 23 05:40:13.425 INFO Configured for network                  name: holesky
Aug 23 05:40:13.426 INFO Data directory initialised              datadir: /Users/paul/.lighthouse/holesky
Aug 23 05:40:13.427 INFO Deposit contract                        address: 0x4242424242424242424242424242424242424242, deploy_block: 0
Aug 23 05:40:13.427 INFO Downloading genesis state               info: this may take some time on testnets with large validator counts, timeout: 60s, server: https://sigp-public-genesis-states.s3.ap-southeast-2.amazonaws.com/
Aug 23 05:40:29.895 INFO Starting from known genesis state       service: beacon
```

Example of log output when there are no URLs specified:

```
Aug 23 06:29:51.645 INFO Logging to file                         path: "/Users/paul/.lighthouse/goerli/beacon/logs/beacon.log"
Aug 23 06:29:51.646 INFO Lighthouse started                      version: Lighthouse/v4.3.0-666a39c+
Aug 23 06:29:51.646 INFO Configured for network                  name: goerli
Aug 23 06:29:51.647 INFO Data directory initialised              datadir: /Users/paul/.lighthouse/goerli
Aug 23 06:29:51.647 INFO Deposit contract                        address: 0xff50ed3d0ec03ac01d4c79aad74928bff48a7b2b, deploy_block: 4367322
The genesis state is not present in the binary and there are no known download URLs. Please use --checkpoint-sync-url or --genesis-state-url.
```

## Additional Info

I tested the `--genesis-state-url` flag with all 9 Goerli checkpoint sync servers on https://eth-clients.github.io/checkpoint-sync-endpoints/ and they all worked 🎉 

My IDE eagerly formatted some `Cargo.toml`. I've disabled it but I don't see the value in spending time reverting the changes that are already there.

I also added the `GenesisStateBytes` enum to avoid an unnecessary clone on the genesis state bytes baked into the binary. This is not a huge deal on Mainnet, but will become more relevant when testing with big genesis states.

When we do a fresh checkpoint sync we're downloading the genesis state to check the `genesis_validators_root` against the finalised state we receive. This is not *entirely* pointless, since we verify the checksum when we download the genesis state so we are actually guaranteeing that the finalised state is on the same network. There might be a smarter/less-download-y way to go about this, but I've run out of cycles to figure that out. Perhaps we can grab it in the next release?
This commit is contained in:
Paul Hauner
2023-08-28 05:34:27 +00:00
parent e056c279aa
commit d61f507184
22 changed files with 651 additions and 117 deletions
Download Patch File Download Diff File Expand all files Collapse all files

275
common/eth2_network_config/src/lib.rs
View File

@@ -13,10 +13,20 @@
use discv5::enr::{CombinedKey, Enr};
use eth2_config::{instantiate_hardcoded_nets, HardcodedNet};
use pretty_reqwest_error::PrettyReqwestError;
use reqwest::blocking::Client;
use sensitive_url::SensitiveUrl;
use sha2::{Digest, Sha256};
use slog::{info, warn, Logger};
use std::fs::{create_dir_all, File};
use std::io::{Read, Write};
use std::path::PathBuf;
use types::{BeaconState, ChainSpec, Config, EthSpec, EthSpecId};
use std::str::FromStr;
use std::time::Duration;
use types::{BeaconState, ChainSpec, Config, EthSpec, EthSpecId, Hash256};
use url::Url;
pub use eth2_config::GenesisStateSource;
pub const DEPLOY_BLOCK_FILE: &str = "deploy_block.txt";
pub const BOOT_ENR_FILE: &str = "boot_enr.yaml";
@@ -32,6 +42,35 @@ instantiate_hardcoded_nets!(eth2_config);
pub const DEFAULT_HARDCODED_NETWORK: &str = "mainnet";
/// A simple slice-or-vec enum to avoid cloning the beacon state bytes in the
/// binary whilst also supporting loading them from a file at runtime.
#[derive(Clone, PartialEq, Debug)]
pub enum GenesisStateBytes {
Slice(&'static [u8]),
Vec(Vec<u8>),
}
impl AsRef<[u8]> for GenesisStateBytes {
fn as_ref(&self) -> &[u8] {
match self {
GenesisStateBytes::Slice(slice) => slice,
GenesisStateBytes::Vec(vec) => vec.as_ref(),
}
}
}
impl From<&'static [u8]> for GenesisStateBytes {
fn from(slice: &'static [u8]) -> Self {
GenesisStateBytes::Slice(slice)
}
}
impl From<Vec<u8>> for GenesisStateBytes {
fn from(vec: Vec<u8>) -> Self {
GenesisStateBytes::Vec(vec)
}
}
/// Specifies an Eth2 network.
///
/// See the crate-level documentation for more details.
@@ -41,7 +80,8 @@ pub struct Eth2NetworkConfig {
/// value to be the block number where the first deposit occurs.
pub deposit_contract_deploy_block: u64,
pub boot_enr: Option<Vec<Enr<CombinedKey>>>,
pub genesis_state_bytes: Option<Vec<u8>>,
pub genesis_state_source: GenesisStateSource,
pub genesis_state_bytes: Option<GenesisStateBytes>,
pub config: Config,
}
@@ -65,8 +105,10 @@ impl Eth2NetworkConfig {
serde_yaml::from_reader(net.boot_enr)
.map_err(|e| format!("Unable to parse boot enr: {:?}", e))?,
),
genesis_state_bytes: Some(net.genesis_state_bytes.to_vec())
.filter(|bytes| !bytes.is_empty()),
genesis_state_source: net.genesis_state_source,
genesis_state_bytes: Some(net.genesis_state_bytes)
.filter(|bytes| !bytes.is_empty())
.map(Into::into),
config: serde_yaml::from_reader(net.config)
.map_err(|e| format!("Unable to parse yaml config: {:?}", e))?,
})
@@ -81,8 +123,37 @@ impl Eth2NetworkConfig {
}
/// Returns `true` if this configuration contains a `BeaconState`.
pub fn beacon_state_is_known(&self) -> bool {
self.genesis_state_bytes.is_some()
pub fn genesis_state_is_known(&self) -> bool {
self.genesis_state_source != GenesisStateSource::Unknown
}
/// The `genesis_validators_root` of the genesis state. May download the
/// genesis state if the value is not already available.
pub fn genesis_validators_root<E: EthSpec>(
&self,
genesis_state_url: Option<&str>,
timeout: Duration,
log: &Logger,
) -> Result<Option<Hash256>, String> {
if let GenesisStateSource::Url {
genesis_validators_root,
..
} = self.genesis_state_source
{
Hash256::from_str(genesis_validators_root)
.map(Option::Some)
.map_err(|e| {
format!(
"Unable to parse genesis state genesis_validators_root: {:?}",
e
)
})
} else {
self.genesis_state::<E>(genesis_state_url, timeout, log)?
.map(|state| state.genesis_validators_root())
.map(Result::Ok)
.transpose()
}
}
/// Construct a consolidated `ChainSpec` from the YAML config.
@@ -96,15 +167,65 @@ impl Eth2NetworkConfig {
}
/// Attempts to deserialize `self.beacon_state`, returning an error if it's missing or invalid.
pub fn beacon_state<E: EthSpec>(&self) -> Result<BeaconState<E>, String> {
///
/// If the genesis state is configured to be downloaded from a URL, then the
/// `genesis_state_url` will override the built-in list of download URLs.
pub fn genesis_state<E: EthSpec>(
&self,
genesis_state_url: Option<&str>,
timeout: Duration,
log: &Logger,
) -> Result<Option<BeaconState<E>>, String> {
let spec = self.chain_spec::<E>()?;
let genesis_state_bytes = self
.genesis_state_bytes
.as_ref()
.ok_or("Genesis state is unknown")?;
match &self.genesis_state_source {
GenesisStateSource::Unknown => Ok(None),
GenesisStateSource::IncludedBytes => {
let state = self
.genesis_state_bytes
.as_ref()
.map(|bytes| {
BeaconState::from_ssz_bytes(bytes.as_ref(), &spec).map_err(|e| {
format!("Built-in genesis state SSZ bytes are invalid: {:?}", e)
})
})
.ok_or("Genesis state bytes missing from Eth2NetworkConfig")??;
Ok(Some(state))
}
GenesisStateSource::Url {
urls: built_in_urls,
checksum,
genesis_validators_root,
} => {
let checksum = Hash256::from_str(checksum).map_err(|e| {
format!("Unable to parse genesis state bytes checksum: {:?}", e)
})?;
let bytes = if let Some(specified_url) = genesis_state_url {
download_genesis_state(&[specified_url], timeout, checksum, log)
} else {
download_genesis_state(built_in_urls, timeout, checksum, log)
}?;
let state = BeaconState::from_ssz_bytes(bytes.as_ref(), &spec).map_err(|e| {
format!("Downloaded genesis state SSZ bytes are invalid: {:?}", e)
})?;
BeaconState::from_ssz_bytes(genesis_state_bytes, &spec)
.map_err(|e| format!("Genesis state SSZ bytes are invalid: {:?}", e))
let genesis_validators_root =
Hash256::from_str(genesis_validators_root).map_err(|e| {
format!(
"Unable to parse genesis state genesis_validators_root: {:?}",
e
)
})?;
if state.genesis_validators_root() != genesis_validators_root {
return Err(format!(
"Downloaded genesis validators root {:?} does not match expected {:?}",
state.genesis_validators_root(),
genesis_validators_root
));
}
Ok(Some(state))
}
}
}
/// Write the files to the directory.
@@ -162,7 +283,7 @@ impl Eth2NetworkConfig {
File::create(&file)
.map_err(|e| format!("Unable to create {:?}: {:?}", file, e))
.and_then(|mut file| {
file.write_all(genesis_state_bytes)
file.write_all(genesis_state_bytes.as_ref())
.map_err(|e| format!("Unable to write {:?}: {:?}", file, e))
})?;
}
@@ -198,7 +319,7 @@ impl Eth2NetworkConfig {
// The genesis state is a special case because it uses SSZ, not YAML.
let genesis_file_path = base_dir.join(GENESIS_STATE_FILE);
let genesis_state_bytes = if genesis_file_path.exists() {
let (genesis_state_bytes, genesis_state_source) = if genesis_file_path.exists() {
let mut bytes = vec![];
File::open(&genesis_file_path)
.map_err(|e| format!("Unable to open {:?}: {:?}", genesis_file_path, e))
@@ -207,20 +328,105 @@ impl Eth2NetworkConfig {
.map_err(|e| format!("Unable to read {:?}: {:?}", file, e))
})?;
Some(bytes).filter(|bytes| !bytes.is_empty())
let state = Some(bytes).filter(|bytes| !bytes.is_empty());
let genesis_state_source = if state.is_some() {
GenesisStateSource::IncludedBytes
} else {
GenesisStateSource::Unknown
};
(state, genesis_state_source)
} else {
None
(None, GenesisStateSource::Unknown)
};
Ok(Self {
deposit_contract_deploy_block,
boot_enr,
genesis_state_bytes,
genesis_state_source,
genesis_state_bytes: genesis_state_bytes.map(Into::into),
config,
})
}
}
/// Try to download a genesis state from each of the `urls` in the order they
/// are defined. Return `Ok` if any url returns a response that matches the
/// given `checksum`.
fn download_genesis_state(
urls: &[&str],
timeout: Duration,
checksum: Hash256,
log: &Logger,
) -> Result<Vec<u8>, String> {
if urls.is_empty() {
return Err(
"The genesis state is not present in the binary and there are no known download URLs. \
Please use --checkpoint-sync-url or --genesis-state-url."
.to_string(),
);
}
let mut errors = vec![];
for url in urls {
// URLs are always expected to be the base URL of a server that supports
// the beacon-API.
let url = parse_state_download_url(url)?;
let redacted_url = SensitiveUrl::new(url.clone())
.map(|url| url.to_string())
.unwrap_or_else(|_| "<REDACTED>".to_string());
info!(
log,
"Downloading genesis state";
"server" => &redacted_url,
"timeout" => ?timeout,
"info" => "this may take some time on testnets with large validator counts"
);
let client = Client::new();
let response = client
.get(url)
.header("Accept", "application/octet-stream")
.timeout(timeout)
.send()
.and_then(|r| r.error_for_status().and_then(|r| r.bytes()));
match response {
Ok(bytes) => {
// Check the server response against our local checksum.
if Sha256::digest(bytes.as_ref())[..] == checksum[..] {
return Ok(bytes.into());
} else {
warn!(
log,
"Genesis state download failed";
"server" => &redacted_url,
"timeout" => ?timeout,
);
errors.push(format!(
"Response from {} did not match local checksum",
redacted_url
))
}
}
Err(e) => errors.push(PrettyReqwestError::from(e).to_string()),
}
}
Err(format!(
"Unable to download a genesis state from {} source(s): {}",
errors.len(),
errors.join(",")
))
}
/// Parses the `url` and joins the necessary state download path.
fn parse_state_download_url(url: &str) -> Result<Url, String> {
Url::parse(url)
.map_err(|e| format!("Invalid genesis state URL: {:?}", e))?
.join("eth/v2/debug/beacon/states/genesis")
.map_err(|e| format!("Failed to append genesis state path to URL: {:?}", e))
}
#[cfg(test)]
mod tests {
use super::*;
@@ -260,7 +466,9 @@ mod tests {
#[test]
fn mainnet_genesis_state() {
let config = Eth2NetworkConfig::from_hardcoded_net(&MAINNET).unwrap();
config.beacon_state::<E>().expect("beacon state can decode");
config
.genesis_state::<E>(None, Duration::from_secs(1), &logging::test_logger())
.expect("beacon state can decode");
}
#[test]
@@ -285,10 +493,25 @@ mod tests {
assert_eq!(
config.genesis_state_bytes.is_some(),
net.genesis_is_known,
net.genesis_state_source == GenesisStateSource::IncludedBytes,
"{:?}",
net.name
);
if let GenesisStateSource::Url {
urls,
checksum,
genesis_validators_root,
} = net.genesis_state_source
{
Hash256::from_str(checksum).expect("the checksum must be a valid 32-byte value");
Hash256::from_str(genesis_validators_root)
.expect("the GVR must be a valid 32-byte value");
for url in urls {
parse_state_download_url(url).expect("url must be valid");
}
}
assert_eq!(config.config.config_name, Some(net.config_dir.to_string()));
}
}
@@ -324,10 +547,20 @@ mod tests {
let base_dir = temp_dir.path().join("my_testnet");
let deposit_contract_deploy_block = 42;
let genesis_state_source = if genesis_state.is_some() {
GenesisStateSource::IncludedBytes
} else {
GenesisStateSource::Unknown
};
let testnet: Eth2NetworkConfig = Eth2NetworkConfig {
deposit_contract_deploy_block,
boot_enr,
genesis_state_bytes: genesis_state.as_ref().map(Encode::as_ssz_bytes),
genesis_state_source,
genesis_state_bytes: genesis_state
.as_ref()
.map(Encode::as_ssz_bytes)
.map(Into::into),
config,
};