Files
lighthouse/Cargo.toml
Michael Sproul 9fdd53df56 Hierarchical state diffs (#5978)
* Start extracting freezer changes for tree-states

* Remove unused config args

* Add comments

* Remove unwraps

* Subjective more clear implementation

* Clean up hdiff

* Update xdelta3

* Tree states archive metrics (#6040)

* Add store cache size metrics

* Add compress timer metrics

* Add diff apply compute timer metrics

* Add diff buffer cache hit metrics

* Add hdiff buffer load times

* Add blocks replayed metric

* Move metrics to store

* Future proof some metrics

---------

Co-authored-by: Michael Sproul <michael@sigmaprime.io>

* Port and clean up forwards iterator changes

* Add and polish hierarchy-config flag

* Merge remote-tracking branch 'origin/unstable' into tree-states-archive

* Cleaner errors

* Fix beacon_chain test compilation

* Merge remote-tracking branch 'origin/unstable' into tree-states-archive

* Patch a few more freezer block roots

* Fix genesis block root bug

* Fix test failing due to pending updates

* Beacon chain tests passing

* Merge remote-tracking branch 'origin/unstable' into tree-states-archive

* Merge remote-tracking branch 'origin/unstable' into tree-states-archive

* Fix doc lint

* Implement DB schema upgrade for hierarchical state diffs (#6193)

* DB upgrade

* Add flag

* Delete RestorePointHash

* Update docs

* Update docs

* Implement hierarchical state diffs config migration (#6245)

* Implement hierarchical state diffs config migration

* Review PR

* Remove TODO

* Set CURRENT_SCHEMA_VERSION correctly

* Fix genesis state loading

* Re-delete some PartialBeaconState stuff

---------

Co-authored-by: Michael Sproul <michael@sigmaprime.io>

* Merge remote-tracking branch 'origin/unstable' into tree-states-archive

* Fix test compilation

* Update schema downgrade test

* Fix tests

* Fix null anchor migration

* Merge remote-tracking branch 'origin/unstable' into tree-states-archive

* Fix tree states upgrade migration (#6328)

* Towards crash safety

* Fix compilation

* Move cold summaries and state roots to new columns

* Rename StateRoots chunked field

* Update prune states

* Clean hdiff CLI flag and metrics

* Fix "staged reconstruction"

* Merge remote-tracking branch 'origin/unstable' into tree-states-archive

* Fix alloy issues

* Fix staged reconstruction logic

* Prevent weird slot drift

* Remove "allow" flag

* Update CLI help

* Remove FIXME about downgrade

* Merge remote-tracking branch 'origin/unstable' into tree-states-archive

* Remove some unnecessary error variants

* Fix new test

* Tree states archive - review comments and metrics (#6386)

* Review PR comments and metrics

* Comments

* Add anchor metrics

* drop prev comment

* Update metadata.rs

* Apply suggestions from code review

---------

Co-authored-by: Michael Sproul <micsproul@gmail.com>

* Update beacon_node/store/src/hot_cold_store.rs

Co-authored-by: Lion - dapplion <35266934+dapplion@users.noreply.github.com>

* Merge remote-tracking branch 'origin/unstable' into tree-states-archive

* Clarify comment and remove anchor_slot garbage

* Simplify database anchor (#6397)

* Simplify database anchor

* Update beacon_node/store/src/reconstruct.rs

* Add migration for anchor

* Fix and simplify light_client store tests

* Fix incompatible config test

* Merge remote-tracking branch 'origin/unstable' into tree-states-archive

* Merge remote-tracking branch 'origin/unstable' into tree-states-archive

* More metrics

* Merge remote-tracking branch 'origin/unstable' into tree-states-archive

* New historic state cache (#6475)

* New historic state cache

* Add more metrics

* State cache hit rate metrics

* Fix store metrics

* More logs and metrics

* Fix logger

* Ensure cached states have built caches :O

* Replay blocks in preference to diffing

* Two separate caches

* Distribute cache build time to next slot

* Re-plumb historic-state-cache flag

* Clean up metrics

* Update book

* Update beacon_node/store/src/hdiff.rs

Co-authored-by: Lion - dapplion <35266934+dapplion@users.noreply.github.com>

* Update beacon_node/store/src/historic_state_cache.rs

Co-authored-by: Lion - dapplion <35266934+dapplion@users.noreply.github.com>

---------

Co-authored-by: Lion - dapplion <35266934+dapplion@users.noreply.github.com>

* Update database docs

* Update diagram

* Merge remote-tracking branch 'origin/unstable' into tree-states-archive

* Update lockbud to work with bindgen/etc

* Correct pkg name for Debian

* Remove vestigial epochs_per_state_diff

* Merge remote-tracking branch 'origin/unstable' into tree-states-archive

* Markdown lint

* Merge remote-tracking branch 'origin/unstable' into tree-states-archive

* Address Jimmy's review comments

* Simplify ReplayFrom case

* Fix and document genesis_state_root

* Typo

Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>

* Merge branch 'unstable' into tree-states-archive

* Compute diff of validators list manually (#6556)

* Split hdiff computation

* Dedicated logic for historical roots and summaries

* Benchmark against real states

* Mutated source?

* Version the hdiff

* Add lighthouse DB config for hierarchy exponents

* Tidy up hierarchy exponents flag

* Apply suggestions from code review

Co-authored-by: Michael Sproul <micsproul@gmail.com>

* Address PR review

* Remove hardcoded paths in benchmarks

* Delete unused function in benches

* lint

---------

Co-authored-by: Michael Sproul <michael@sigmaprime.io>

* Test hdiff binary format stability (#6585)

* Merge remote-tracking branch 'origin/unstable' into tree-states-archive

* Add deprecation warning for SPRP

* Update xdelta to get rid of duplicate deps

* Document test
2024-11-18 01:51:44 +00:00

277 lines
8.9 KiB
TOML

[workspace]
members = [
"account_manager",
"beacon_node",
"beacon_node/beacon_chain",
"beacon_node/beacon_processor",
"beacon_node/builder_client",
"beacon_node/client",
"beacon_node/eth1",
"beacon_node/lighthouse_network",
"beacon_node/lighthouse_network/gossipsub",
"beacon_node/execution_layer",
"beacon_node/http_api",
"beacon_node/http_metrics",
"beacon_node/network",
"beacon_node/store",
"beacon_node/timer",
"boot_node",
"common/account_utils",
"common/clap_utils",
"common/compare_fields",
"common/compare_fields_derive",
"common/deposit_contract",
"common/directory",
"common/eth2",
"common/eth2_config",
"common/eth2_interop_keypairs",
"common/eth2_network_config",
"common/eth2_wallet_manager",
"common/metrics",
"common/lighthouse_version",
"common/lockfile",
"common/logging",
"common/lru_cache",
"common/malloc_utils",
"common/oneshot_broadcast",
"common/pretty_reqwest_error",
"common/sensitive_url",
"common/slot_clock",
"common/system_health",
"common/task_executor",
"common/target_check",
"common/test_random_derive",
"common/unused_port",
"common/validator_dir",
"common/warp_utils",
"common/monitoring_api",
"database_manager",
"consensus/int_to_bytes",
"consensus/fixed_bytes",
"consensus/fork_choice",
"consensus/proto_array",
"consensus/safe_arith",
"consensus/state_processing",
"consensus/swap_or_not_shuffle",
"crypto/bls",
"crypto/kzg",
"crypto/eth2_key_derivation",
"crypto/eth2_keystore",
"crypto/eth2_wallet",
"lcli",
"lighthouse",
"lighthouse/environment",
"slasher",
"slasher/service",
"testing/ef_tests",
"testing/eth1_test_rig",
"testing/execution_engine_integration",
"testing/node_test_rig",
"testing/simulator",
"testing/test-test_logger",
"testing/state_transition_vectors",
"testing/web3signer_tests",
"validator_client",
"validator_client/beacon_node_fallback",
"validator_client/doppelganger_service",
"validator_client/graffiti_file",
"validator_client/http_api",
"validator_client/http_metrics",
"validator_client/initialized_validators",
"validator_client/signing_method",
"validator_client/slashing_protection",
"validator_client/validator_metrics",
"validator_client/validator_services",
"validator_client/validator_store",
"validator_manager",
"watch",
]
resolver = "2"
[workspace.package]
edition = "2021"
[workspace.dependencies]
alloy-primitives = { version = "0.8", features = ["rlp", "getrandom"] }
alloy-rlp = "0.3.4"
alloy-consensus = "0.3.0"
anyhow = "1"
arbitrary = { version = "1", features = ["derive"] }
async-channel = "1.9.0"
axum = "0.7.7"
bincode = "1"
bitvec = "1"
byteorder = "1"
bytes = "1"
clap = { version = "4.5.4", features = ["derive", "cargo", "wrap_help"] }
# Turn off c-kzg's default features which include `blst/portable`. We can turn on blst's portable
# feature ourselves when desired.
c-kzg = { version = "1", default-features = false }
compare_fields_derive = { path = "common/compare_fields_derive" }
criterion = "0.5"
delay_map = "0.3"
derivative = "2"
dirs = "3"
either = "1.9"
rust_eth_kzg = "0.5.1"
discv5 = { version = "0.7", features = ["libp2p"] }
env_logger = "0.9"
error-chain = "0.12"
ethereum_hashing = "0.7.0"
ethereum_serde_utils = "0.7"
ethereum_ssz = "0.7"
ethereum_ssz_derive = "0.7"
ethers-core = "1"
ethers-providers = { version = "1", default-features = false }
exit-future = "0.2"
fnv = "1"
fs2 = "0.4"
futures = "0.3"
graffiti_file = { path = "validator_client/graffiti_file" }
hex = "0.4"
hashlink = "0.9.0"
hyper = "1"
itertools = "0.10"
libsecp256k1 = "0.7"
log = "0.4"
lru = "0.12"
maplit = "1"
milhouse = "0.3"
num_cpus = "1"
parking_lot = "0.12"
paste = "1"
prometheus = "0.13"
quickcheck = "1"
quickcheck_macros = "1"
quote = "1"
r2d2 = "0.8"
rand = "0.8"
rayon = "1.7"
regex = "1"
reqwest = { version = "0.11", default-features = false, features = ["blocking", "json", "stream", "rustls-tls", "native-tls-vendored"] }
ring = "0.16"
rpds = "0.11"
rusqlite = { version = "0.28", features = ["bundled"] }
serde = { version = "1", features = ["derive"] }
serde_json = "1"
serde_repr = "0.1"
serde_yaml = "0.9"
sha2 = "0.9"
slog = { version = "2", features = ["max_level_debug", "release_max_level_debug", "nested-values"] }
slog-async = "2"
slog-term = "2"
sloggers = { version = "2", features = ["json"] }
smallvec = { version = "1.11.2", features = ["arbitrary"] }
snap = "1"
ssz_types = "0.8"
strum = { version = "0.24", features = ["derive"] }
superstruct = "0.8"
syn = "1"
sysinfo = "0.26"
tempfile = "3"
tokio = { version = "1", features = ["rt-multi-thread", "sync", "signal", "macros"] }
tokio-stream = { version = "0.1", features = ["sync"] }
tokio-util = { version = "0.7", features = ["codec", "compat", "time"] }
tracing = "0.1.40"
tracing-appender = "0.2"
tracing-core = "0.1"
tracing-log = "0.2"
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
tree_hash = "0.8"
tree_hash_derive = "0.8"
url = "2"
uuid = { version = "0.8", features = ["serde", "v4"] }
warp = { version = "0.3.7", default-features = false, features = ["tls"] }
zeroize = { version = "1", features = ["zeroize_derive"] }
zip = "0.6"
# Local crates.
account_utils = { path = "common/account_utils" }
beacon_chain = { path = "beacon_node/beacon_chain" }
beacon_node = { path = "beacon_node" }
beacon_node_fallback = { path = "validator_client/beacon_node_fallback" }
beacon_processor = { path = "beacon_node/beacon_processor" }
bls = { path = "crypto/bls" }
clap_utils = { path = "common/clap_utils" }
compare_fields = { path = "common/compare_fields" }
deposit_contract = { path = "common/deposit_contract" }
directory = { path = "common/directory" }
doppelganger_service = { path = "validator_client/doppelganger_service" }
validator_services = { path = "validator_client/validator_services" }
environment = { path = "lighthouse/environment" }
eth1 = { path = "beacon_node/eth1" }
eth1_test_rig = { path = "testing/eth1_test_rig" }
eth2 = { path = "common/eth2" }
eth2_config = { path = "common/eth2_config" }
eth2_key_derivation = { path = "crypto/eth2_key_derivation" }
eth2_keystore = { path = "crypto/eth2_keystore" }
eth2_network_config = { path = "common/eth2_network_config" }
eth2_wallet = { path = "crypto/eth2_wallet" }
execution_layer = { path = "beacon_node/execution_layer" }
fixed_bytes = { path = "consensus/fixed_bytes" }
filesystem = { path = "common/filesystem" }
fork_choice = { path = "consensus/fork_choice" }
genesis = { path = "beacon_node/genesis" }
gossipsub = { path = "beacon_node/lighthouse_network/gossipsub/" }
http_api = { path = "beacon_node/http_api" }
initialized_validators = { path = "validator_client/initialized_validators" }
int_to_bytes = { path = "consensus/int_to_bytes" }
kzg = { path = "crypto/kzg" }
metrics = { path = "common/metrics" }
lighthouse_network = { path = "beacon_node/lighthouse_network" }
lighthouse_version = { path = "common/lighthouse_version" }
lockfile = { path = "common/lockfile" }
logging = { path = "common/logging" }
lru_cache = { path = "common/lru_cache" }
malloc_utils = { path = "common/malloc_utils" }
merkle_proof = { path = "consensus/merkle_proof" }
monitoring_api = { path = "common/monitoring_api" }
network = { path = "beacon_node/network" }
operation_pool = { path = "beacon_node/operation_pool" }
pretty_reqwest_error = { path = "common/pretty_reqwest_error" }
proto_array = { path = "consensus/proto_array" }
safe_arith = { path = "consensus/safe_arith" }
sensitive_url = { path = "common/sensitive_url" }
signing_method = { path = "validator_client/signing_method" }
slasher = { path = "slasher", default-features = false }
slashing_protection = { path = "validator_client/slashing_protection" }
slot_clock = { path = "common/slot_clock" }
state_processing = { path = "consensus/state_processing" }
store = { path = "beacon_node/store" }
swap_or_not_shuffle = { path = "consensus/swap_or_not_shuffle" }
system_health = { path = "common/system_health" }
task_executor = { path = "common/task_executor" }
types = { path = "consensus/types" }
unused_port = { path = "common/unused_port" }
validator_client = { path = "validator_client" }
validator_dir = { path = "common/validator_dir" }
validator_http_api = { path = "validator_client/http_api" }
validator_http_metrics = { path = "validator_client/http_metrics" }
validator_metrics = { path = "validator_client/validator_metrics" }
validator_store= { path = "validator_client/validator_store" }
warp_utils = { path = "common/warp_utils" }
xdelta3 = { git = "http://github.com/sigp/xdelta3-rs", rev = "50d63cdf1878e5cf3538e9aae5eed34a22c64e4a" }
zstd = "0.13"
[profile.maxperf]
inherits = "release"
lto = "fat"
codegen-units = 1
incremental = false
[patch.crates-io]
quick-protobuf = { git = "https://github.com/sigp/quick-protobuf.git", rev = "681f413312404ab6e51f0b46f39b0075c6f4ebfd" }