Compare commits

...

1076 Commits

Author SHA1 Message Date
Pawan Dhananjay
b6ff8241c8 Add newPayloadWithWitness methods 2024-06-26 16:21:45 +05:30
ethDreamer
c52c598f69 Electra: Remaining Consensus Data Structures (#5712)
* Attestation superstruct changes for EIP 7549 (#5644)

* update

* experiment

* superstruct changes

* revert

* superstruct changes

* fix tests

* indexed attestation

* indexed attestation superstruct

* updated TODOs

* `superstruct` the `AttesterSlashing` (#5636)

* `superstruct` Attester Fork Variants

* Push a little further

* Deal with Encode / Decode of AttesterSlashing

* not so sure about this..

* Stop Encode/Decode Bounds from Propagating Out

* Tons of Changes..

* More Conversions to AttestationRef

* Add AsReference trait (#15)

* Add AsReference trait

* Fix some snafus

* Got it Compiling! :D

* Got Tests Building

* Get beacon chain tests compiling

---------

Co-authored-by: Michael Sproul <micsproul@gmail.com>

* Merge remote-tracking branch 'upstream/unstable' into electra_attestation_changes

* Make EF Tests Fork-Agnostic (#5713)

* Finish EF Test Fork Agnostic (#5714)

* Superstruct `AggregateAndProof` (#5715)

* Upgrade `superstruct` to `0.8.0`

* superstruct `AggregateAndProof`

* Merge remote-tracking branch 'sigp/unstable' into electra_attestation_changes

* cargo fmt

* Merge pull request #5726 from realbigsean/electra_attestation_changes

Merge unstable into Electra attestation changes

* EIP7549 `get_attestation_indices` (#5657)

* get attesting indices electra impl

* fmt

* get tests to pass

* fmt

* fix some beacon chain tests

* fmt

* fix slasher test

* fmt got me again

* fix more tests

* fix tests

* Some small changes (#5739)

* cargo fmt (#5740)

* Sketch op pool changes

* fix get attesting indices (#5742)

* fix get attesting indices

* better errors

* fix compile

* only get committee index once

* Ef test fixes (#5753)

* attestation related ef test fixes

* delete commented out stuff

* Fix Aggregation Pool for Electra (#5754)

* Fix Aggregation Pool for Electra

* Remove Outdated Interface

* fix ssz (#5755)

* Get `electra_op_pool` up to date (#5756)

* fix get attesting indices (#5742)

* fix get attesting indices

* better errors

* fix compile

* only get committee index once

* Ef test fixes (#5753)

* attestation related ef test fixes

* delete commented out stuff

* Fix Aggregation Pool for Electra (#5754)

* Fix Aggregation Pool for Electra

* Remove Outdated Interface

* fix ssz (#5755)

---------

Co-authored-by: realbigsean <sean@sigmaprime.io>

* Revert "Get `electra_op_pool` up to date (#5756)" (#5757)

This reverts commit ab9e58aa3d.

* Merge branch 'electra_attestation_changes' of https://github.com/sigp/lighthouse into electra_op_pool

* Compute on chain aggregate impl (#5752)

* add compute_on_chain_agg impl to op pool changes

* fmt

* get op pool tests to pass

* update the naive agg pool interface (#5760)

* Fix bugs in cross-committee aggregation

* Add comment to max cover optimisation

* Fix assert

* Merge pull request #5749 from sigp/electra_op_pool

Optimise Electra op pool aggregation

* update committee offset

* Fix Electra Fork Choice Tests (#5764)

* Subscribe to the correct subnets for electra attestations (#5782)

* subscribe to the correct att subnets for electra

* subscribe to the correct att subnets for electra

* cargo fmt

* fix slashing handling

* Merge remote-tracking branch 'upstream/unstable'

* Send unagg attestation based on fork

* Publish all aggregates

* just one more check bro plz..

* Merge pull request #5832 from ethDreamer/electra_attestation_changes_merge_unstable

Merge `unstable` into `electra_attestation_changes`

* Merge pull request #5835 from realbigsean/fix-validator-logic

Fix validator logic

* Merge pull request #5816 from realbigsean/electra-attestation-slashing-handling

Electra slashing handling

* Electra attestation changes rm decode impl (#5856)

* Remove Crappy Decode impl for Attestation

* Remove Inefficient Attestation Decode impl

* Implement Schema Upgrade / Downgrade

* Update beacon_node/beacon_chain/src/schema_change/migration_schema_v20.rs

Co-authored-by: Michael Sproul <micsproul@gmail.com>

---------

Co-authored-by: Michael Sproul <micsproul@gmail.com>

* Fix failing attestation tests and misc electra attestation cleanup (#5810)

* - get attestation related beacon chain tests to pass
- observed attestations are now keyed off of data + committee index
- rename op pool attestationref to compactattestationref
- remove unwraps in agg pool and use options instead
- cherry pick some changes from ef-tests-electra

* cargo fmt

* fix failing test

* Revert dockerfile changes

* make committee_index return option

* function args shouldnt be a ref to attestation ref

* fmt

* fix dup imports

---------

Co-authored-by: realbigsean <seananderson33@GMAIL.com>

* fix some todos (#5817)

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into electra_attestation_changes

* add consolidations to merkle calc for inclusion proof

* Remove Duplicate KZG Commitment Merkle Proof Code (#5874)

* Remove Duplicate KZG Commitment Merkle Proof Code

* s/tree_lists/fields/

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into electra_attestation_changes

* fix compile

* Fix slasher tests (#5906)

* Fix electra tests

* Add electra attestations to double vote tests

* Update superstruct to 0.8

* Merge remote-tracking branch 'origin/unstable' into electra_attestation_changes

* Small cleanup in slasher tests

* Clean up Electra observed aggregates (#5929)

* Use consistent key in observed_attestations

* Remove unwraps from observed aggregates

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into electra_attestation_changes

* De-dup attestation constructor logic

* Remove unwraps in Attestation construction

* Dedup match_attestation_data

* Remove outdated TODO

* Use ForkName Ord in fork-choice tests

* Use ForkName Ord in BeaconBlockBody

* Make to_electra not fallible

* Remove TestRandom impl for IndexedAttestation

* Remove IndexedAttestation faulty Decode impl

* Drop TestRandom impl

* Add PendingAttestationInElectra

* Indexed att on disk (#35)

* indexed att on disk

* fix lints

* Update slasher/src/migrate.rs

Co-authored-by: ethDreamer <37123614+ethDreamer@users.noreply.github.com>

---------

Co-authored-by: Lion - dapplion <35266934+dapplion@users.noreply.github.com>
Co-authored-by: ethDreamer <37123614+ethDreamer@users.noreply.github.com>

* add electra fork enabled fn to ForkName impl (#36)

* add electra fork enabled fn to ForkName impl

* remove inadvertent file

* Update common/eth2/src/types.rs

Co-authored-by: ethDreamer <37123614+ethDreamer@users.noreply.github.com>

* Dedup attestation constructor logic in attester cache

* Use if let Ok for committee_bits

* Dedup Attestation constructor code

* Diff reduction in tests

* Fix beacon_chain tests

* Diff reduction

* Use Ord for ForkName in pubsub

* Resolve into_attestation_and_indices todo

* Remove stale TODO

* Fix beacon_chain tests

* Test spec invariant

* Use electra_enabled in pubsub

* Remove get_indexed_attestation_from_signed_aggregate

* Use ok_or instead of if let else

* committees are sorted

* remove dup method `get_indexed_attestation_from_committees`

* Merge pull request #5940 from dapplion/electra_attestation_changes_lionreview

Electra attestations #5712 review

* update default persisted op pool deserialization

* ensure aggregate and proof uses serde untagged on ref

* Fork aware ssz static attestation tests

* Electra attestation changes from Lions review (#5971)

* dedup/cleanup and remove unneeded hashset use

* remove irrelevant TODOs

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into electra_attestation_changes

* Electra attestation changes sean review (#5972)

* instantiate empty bitlist in unreachable code

* clean up error conversion

* fork enabled bool cleanup

* remove a couple todos

* return bools instead of options in `aggregate` and use the result

* delete commented out code

* use map macros in simple transformations

* remove signers_disjoint_from

* get ef tests compiling

* get ef tests compiling

* update intentionally excluded files

* Avoid changing slasher schema for Electra

* Delete slasher schema v4

* Fix clippy

* Fix compilation of beacon_chain tests

* Update database.rs

* Add electra lightclient types

* Update slasher/src/database.rs

* fix imports

* Merge pull request #5980 from dapplion/electra-lightclient

Add electra lightclient types

* Merge pull request #5975 from michaelsproul/electra-slasher-no-migration

Avoid changing slasher schema for Electra

* Update beacon_node/beacon_chain/src/attestation_verification.rs

* Update beacon_node/beacon_chain/src/attestation_verification.rs
2024-06-24 21:08:07 +00:00
chonghe
758b58c9e9 Update slasher DB size and Lighthouse book (#5934)
* Update book

* Fix

* mdlint

* Revise

* Update slasher doc

* Revise max db size

* change blob to file

* Add checkpoint-blobs

* Thanks Jimmy for the command

* Update schema docs
2024-06-23 23:52:15 +00:00
Age Manning
f69ccc3b70 Ensure all handler events are covered (#5945)
* Ensure handler events are covered
2024-06-19 05:02:37 +00:00
João Oliveira
1503f7d652 report failed requests from RPC after being clear from rate limiter but not yet sent. (#5942)
* report failed rpc requests from RPC
2024-06-19 05:02:34 +00:00
Michael Sproul
adb3f865ef Lower tolerance of stale blobs on gossip (#5935)
* Lower tolerance of stale blobs on gossip

* Drop to debug
2024-06-19 05:02:31 +00:00
kevaundray
806c3ce9e9 chore: simplify method to generate a random valid blob (#5946)
* chore: simplify method to generate a random blob

* chore: remove now unused import
2024-06-19 05:02:29 +00:00
kevaundray
a87f19d801 chore: change impl Into<T> for U to impl From<U> for T (#5948)
* chore: Change Into trait impl for KzgProof to From trait impl

* chore: change `impl Into <T> for U` to `impl From<U> for T`

* chore: remove `from-over-into` clippy lint exception
2024-06-19 05:02:26 +00:00
Jimmy Chen
9f8aa963b1 Update CI nethermind and LLVM versions (#5933)
* Update nethermind and llvm versions.

* Fix nethermind binary path
2024-06-19 05:02:24 +00:00
Jimmy Chen
d06e3894de Fix cargo audit: update curve25519-dalek (#5959)
* Patch curve25519-dalek.
2024-06-19 05:02:22 +00:00
chonghe
ab7db7ca9c Update local testnet documentation (#5896)
* Update local testnet doc

* Update doc

* Minor revision

* Fix directory path

* Update scripts/local_testnet/README.md

Co-authored-by: Pawan Dhananjay <pawandhananjay@gmail.com>

* Add kurtosis web

* Add save logs command

* Log of a service

* Update scripts/local_testnet/README.md

Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>

* Update scripts/local_testnet/README.md

Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>

* Minor revision

* Merge branch 'local-testnet-doc' of https://github.com/chong-he/lighthouse into local-testnet-doc
2024-06-18 05:53:09 +00:00
Akihito Nakano
c2838ba2bd Fix SelfRateLimiter breaks the sequence of delayed_requests. (#5903)
* Add test for next_peer_request_ready

* Use push_front to requeue
2024-06-18 04:49:45 +00:00
Lion - dapplion
474c1b4486 Verify inclusion proof should not be fallible (#5787)
* Verify inclusion proof should not be fallible

* Add blob sidecar inclusion test (#33)

* Add blob sidecar inclusion test.

* Fix lint
2024-06-17 15:05:24 +00:00
Michael Sproul
21f3a191c5 Remove extern crate (#5922)
* Remove extern crate
2024-06-17 15:05:21 +00:00
Paul Hauner
44c03d5d17 Add FUNDING.json for DRIPS (#5932)
* Add FUNDING.json
2024-06-17 15:05:18 +00:00
Jimmy Chen
bc044ed275 Fix CommandLineTest port conflicts on CI (#5908)
* Fix port conflicts on CI.
2024-06-17 15:05:15 +00:00
Michael Sproul
1eb8694a86 Remove some easy Electra TODOs (#5928)
* Remove some easy Electra TODOs
2024-06-17 13:32:39 +00:00
realbigsean
a74098044a Rust 1.79 lints (#5927)
* max_value -> MAX

* remove unnecesary closures

* a couple more max_value -> MAX

* a couple more max_value -> MAX

* Revert "a couple more max_value -> MAX"

This reverts commit 807fe7cae9.

* unused spec field -> phantom data

* ignore some dead code warnings

* update kurtosis repo location
2024-06-13 23:04:30 +00:00
Akihito Nakano
5789db042d Fix panic on startup in debug build (#5917)
* Fix panic in debug build

* make cli-local to update the book
2024-06-13 05:22:07 +00:00
Jimmy Chen
1bcd1f15b1 Fix skip-ci on stable (#5909)
* Fix skip-ci on `stable`.
2024-06-13 02:38:31 +00:00
realbigsean
c24b2f39b3 Beta compiler fixes (#5916)
* remove or update unused cfg
2024-06-13 01:32:05 +00:00
Michael Sproul
f1d88ba4b1 Release v5.2.0 (#5664)
* Bump version to v5.2.0
2024-06-10 13:28:11 +00:00
Michael Sproul
947e2e8db1 Undo some CLI flag breakages (#5902)
* Undo some CLI breakages

* Update CLI book docs
2024-06-07 12:40:40 +00:00
Age Manning
22fe0a6622 Allow all RPC messages on disconnect (#5876)
* Permit rpc messages on disconnect

* Use or instead of and
2024-06-07 00:56:40 +00:00
Age Manning
7b48b0b4a7 Errors for all RPC Requests (#5867)
* Return and error if peer has disconnected

* Report errors for rate limited requests

* Code improvement

* Bump rust version to 1.78

* Downgrade to 1.77

* Update beacon_node/lighthouse_network/src/service/mod.rs

Co-authored-by: João Oliveira <hello@jxs.pt>

* fix fmt

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into rpc-peer-disconnect-error

* update lockfile
2024-06-07 00:12:45 +00:00
Michael Sproul
7a7fc82cbd Update criterion (#5886)
* Start updating criterion

* Update consensus benches
2024-06-04 15:12:56 +00:00
Michael Sproul
2c72bb8fc6 Fix getClientVersionV1 for Geth 1.13 (#5884)
* Fix getClientVersionV1 for Geth 1.13
2024-06-04 04:29:12 +00:00
Armağan Yıldırak
8a247eb7ed Investigate UnknownTargetRoot slasher errors (#5006)
* Fix the UnknownTargetRoot
2024-06-04 04:29:08 +00:00
Jimmy Chen
5fc01454dc Replace local testnet script with Kurtosis (#5865)
* Kurtosis local testnet.

* Remove unused `lcli` subcommands

* Migrate doppelganger_protection test to kurtosis and further cleanup.

* Fix lint

* Add missing download image step and remove unused `lcli` dependencies.

* doppelganger success case working

* Run tests on hosted runner and improve error handling.

* Start the dp vc only after epoch 1

* Add more logging to test results.

* Fix exit code and speed up docker build.

* Fix incorrect exit codes and split doppelganger tests on CI.

* Missing the escape for double quotes 😫

* Remove unnecessary vc params in kurtosis config.
2024-06-04 03:03:26 +00:00
Michael Sproul
1b7c4a4523 Increase slog buffer size (#5882)
* Increase slog buffer size
2024-06-04 03:03:22 +00:00
Lion - dapplion
cb328072c1 Keep PendingComponents in da_checker during import_block (#5845)
* Ensure lookup sync checks caches correctly

* Simplify BlockProcessStatus

* Keep PendingComponents in da_checker during import_block

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into time_in_da_checker

* Fix tests with DA checker new eviction policy (#34)
2024-06-03 20:37:53 +00:00
Lion - dapplion
2c971fa9a1 Make beacon processor queue sizes dynamic (#5573)
* Make beacon processor queue sizes dynamic

* Update tests

* lint

* Review PR
2024-06-03 07:17:11 +00:00
Pawan Dhananjay
fb790decd6 Fix attestations not getting added to the aggregation pool (#5863)
* Remove from delay_map 2 slots after duty

* Cleanup
2024-05-31 04:08:43 +00:00
Eitan Seri-Levi
fbc230e118 Add builder header timeout flag (#5857)
* fix bitvector test random impl

* add a new flag that allows for configuring the timeout for get builder header api calls

* make cli

* add upper limit check, changes based on feedback

* update cli

* capitalization

* cli fix
2024-05-31 03:35:57 +00:00
João Oliveira
bbe9242811 replace instant with web-time (#5800)
* replace instant with web_time

* Merge branch 'unstable' into replace-instant-gossipsub

# Conflicts:
#	Cargo.lock
#	beacon_node/lighthouse_network/Cargo.toml
2024-05-30 19:18:35 +00:00
Lion - dapplion
17dc978760 Add peers to parent lookups (#5858)
* Add peers to parent lookups

* Add test

* lint

* Do not attempt to continue

* add_peers_to_lookup_and_ancestors can't drop
2024-05-30 17:07:32 +00:00
Eitan Seri-Levi
df6e1c9add Fix BitVectors TestRandom implementation (#5854)
* fix bitvector test random impl
2024-05-30 14:09:22 +00:00
chonghe
ffe29c087d Add bls_to_execution_change topic to eth1/v1/events (#5823)
* Add bls event

* Update events and types

* Add bls in event

* Event bls

* tests..rs

* change order

* another tests.rs

* Signed BLS

* Revert "another tests.rs"

This reverts commit 7f54e9c1ce.

* Revert "Signed BLS"

This reverts commit 1146bc734b.

* withdrawal_keyparis

* Fix genesis
2024-05-30 05:48:29 +00:00
Pawan Dhananjay
6daeec31e2 Rename deploy_block in network config (#5850)
* Rename deploy_block.txt to deposit_contract_block.txt

* fmt

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into rename-deply-block
2024-05-29 18:56:11 +00:00
Jimmy Chen
48789c7c16 Always build lcli on CI (#5860)
* Always build `lcli` on CI.
2024-05-29 15:47:32 +00:00
Eitan Seri-Levi
df983a83e1 upgrade clap to v4.5 (#5273)
* upgrade clap to v4.5

* cli fixes

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into upgrade-clap-cli

* value parser for mnemonic

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into upgrade-clap-cli

* merge unstable

* default --format val

* fix eth sim

* fix eth sim

* merge conflicts

* resolve beta compiler issue

* add num args, version

* add custom flag parser, make rate limiter flags clap friendly

* remove unneeded check

* fmt

* update

* alphabetic order

* resolve merge conflict

* fix test

* resolve conflicts

* fix test

* revert removed if statement

* fmt got me again

* fix broken flag

* make cli

* make cli

* update

* remove -e files

* update

* cli help updates

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into upgrade-clap-cli

* cli help updates

* md files

* merge conflict

* merge conflicts

* md

* help text, text width, and a few flag fixes

* fmt

* merge

* revert

* revert

* resolve merge conflicts

* merge conflicts

* revert simulator changes

* require at least one arg

* fix eth sim cli

* resolve merge conflicts

* book changes

* md changes

* cli check

* cli check

* retry cli check

* retry cli check

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into upgrade-clap-cli

* cli

* Merge remote-tracking branch 'origin/unstable' into upgrade-clap-cli

* Update CLI docs for Goerli removal

* Fix cargo lock
2024-05-28 05:46:39 +00:00
Lion - dapplion
6a7305a487 Add more info about why lookup is in AwaitingDownload (#5838)
* Add more info about why lookup is in AwaitingDownload

* Review comments

* Merge branch 'unstable' into awaiting-download-context

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into awaiting-download-context

* cargo fmt
2024-05-28 02:48:37 +00:00
Michael Sproul
7f8b600f2a Remove Goerli support (#5770)
* Delete Goerli

* Generate validator manager test vectors

* Fix newlines in CLI docs

* Fix deposit-cli tests

* Run web3signer tests for Holesky from Bellatrix

* Fix mainnet bellatrix web3signer test

* Merge remote-tracking branch 'origin/unstable' into rm-goerli

* Fix snafu
2024-05-27 07:59:10 +00:00
Akihito Nakano
6f05863007 Remove unused dependencies (#5812)
* Remove unused dependencies from lighthouse_network

* Remove unused dependencies from gossipsub

* Remove unused dependencies from network

* Remove unused dependencies from consensus/types

* Remove unused dependencies from account_manager

* Remove unused dependencies from boot_node

* Remove unused dependencies from database_manager

* Remove unused dependencies from validator_manager

* Remove unused dependencies from testing/ef_tests

* Remove unused dependencies from testing/simulator

* Remove unused dependencies from testing/execution_engine_integration

* Remove unused dependencies from common/system_health

* Remove unused dependencies from common/eth2

* Remove unused dependencies from common/lighthouse_metrics

* Remove unused dependencies from common/eth2_network_config

* Remove unused dependencies from common/logging

* Remove unused dependencies from watch

* Remove unused dependencies from slasher

* Remove unused dependencies from lighthouse

* Remove unused dependencies from lcli

* Remove unused dependencies from beacon_node

* Remove unused dependencies from beacon_node/builder_client

* Remove unused dependencies from beacon_node/beacon_processor

* Remove unused dependencies from beacon_node/execution_layer

* Remove unused dependencies from beacon_node/eth1

* Remove unused dependencies from beacon_node/client

* Remove unused dependencies from beacon_node/store

* Fix missing deps which is used only for testing
2024-05-27 07:59:07 +00:00
realbigsean
393c5bcb8a don't block mev boost till genesis finalization (#5834)
* don't block mev boost till genesis finalization
2024-05-27 07:31:44 +00:00
Lion - dapplion
e4984217a6 Ensure lookup sync checks caches correctly (#5840)
* Ensure lookup sync checks caches correctly

* fix tests and remove unused method

* Simplify BlockProcessStatus
2024-05-25 14:56:51 +00:00
Lion - dapplion
f187ad8bb4 Do not drop lookups without peers while awaiting events (#5839)
* Do not drop lookups without peers while awaiting events
2024-05-24 21:28:37 +00:00
Lion - dapplion
7fda18bf49 Report sync of block import via blob import (#5836)
* Report sync of block import via blob import
2024-05-24 19:45:04 +00:00
Lion - dapplion
f1f1250784 Do not double log uknown lookup (#5837)
* Do not double log uknown lookup

* Fix log text, not necessarily a child lookup
2024-05-24 19:45:01 +00:00
Lion - dapplion
a1271bc839 Debounce UnknownBlockHashFromAttestation events (#5706)
* Debounce UnknownBlockHashFromAttestation events

* Merge branch 'unstable' into debounce-sync-block-unknown

* Re-add dropped comment
2024-05-24 19:17:22 +00:00
chonghe
3070cb7c39 Markdown linter (#5494)
* linter

* Add markdown linter

* add env

* only check markdown

* Add token

* Update .github/workflows/test-suite.yml

* Markdown linter

* Exit code

* Update script

* rename

* mdlint

* Add an empty line after end of file

* Testing disable

* add text

* update mdlint.sh

* ori validator inclusion

* Add config yml file

* Remove MD041 and fix advanced-datadir file

* FIx validator inclusion file conflict

* Merge branch 'unstable' into markdown-linter

* change files

* Merge branch 'markdown-linter' of https://github.com/chong-he/lighthouse into markdown-linter

* mdlint

* Remove MD025

* Remove MD036

* Remove MD045

* Removr MD001

* Set MD028 to false

* Remove MD024

* Remove MD055

* Remove MD029

* Remove MD040

* Set MD040 to false

* Set MD033 to false

* Set MD013 to false

* Rearrange yml file

* Update mdlint.sh and test

* Test remove fix

* Test with fix

* Test with space

* Fix summary indentation

* Test mdlint.sh

* Update mdlint.sh

* Test

* Update

* Test fix

* Test again

* Fix

* merge into check-code

* Update scripts/mdlint.sh

Co-authored-by: Mac L <mjladson@pm.me>

* Update scripts/mdlint.sh

Co-authored-by: Mac L <mjladson@pm.me>

* Remove set -e

* Add comment

* Merge pull request #7 from chong-he/unstable

Merge unstable to markdown branch

* mdlint

* Merge branch 'unstable' into markdown-linter

* mdlint
2024-05-24 02:45:19 +00:00
Lion - dapplion
7073242ccc Suppress RPC Error disconnect log (#5802)
* Suppress RPC Error disconnect log
2024-05-23 14:34:49 +00:00
Eitan Seri-Levi
61b29fa361 Update default target peers documentation (#5727)
* the default target peers is 100
2024-05-23 12:46:08 +00:00
Lion - dapplion
17d9086df3 Drop stuck lookups (#5824)
* Drop stuck lookups
2024-05-23 12:46:05 +00:00
Michael Sproul
8762d82adf Fix hot state disk leak (#5768)
* Fix hot state leak

* Don't delete the genesis state when split is 0x0!
2024-05-23 00:17:53 +00:00
Jimmy Chen
52e31121df Reduce frequency of polling unknown validators to avoid overwhelming the Beacon Node (#5628)
* Reduce frequency of polling unknown validators.

* Move slot calculation into for loop.

* Simplify logic.

Co-authored-by: Michael Sproul <micsproul@gmail.com>

* Fix formatting
2024-05-22 00:52:40 +00:00
Lion - dapplion
2a87016d94 Fix lookup disconnect peer (#5815)
* Test lookup peer disconnect modes

* Fix lookup peer disconnected return early
2024-05-20 18:27:57 +00:00
chonghe
b5de925d8f Use JSON header by default for /eth/v1/beacon/deposit_snapshot (#5813)
* Fix with or

* Flip case
2024-05-20 02:17:43 +00:00
Lion - dapplion
8006418d80 Type sync network context send errors (#5808)
* Type sync network context send errors

* Consisntent naming
2024-05-17 11:34:21 +00:00
Lion - dapplion
319b4a2467 Skip creating child lookup if parent is never created (#5803)
* Skip creating child lookup if parent is never created
2024-05-17 10:58:27 +00:00
antondlr
0f49951363 Skip CI's test-suite when the skip-ci label is present (#5790)
* skip `test-suite` if `skip-ci` label present
2024-05-16 08:33:32 +00:00
realbigsean
1d6160549d use electra feature in notifier completeness check (#5786)
* use electra feature in notifier completeness check

* update capella and denab readiness chacks to use fork name rather than random fields
2024-05-15 10:37:59 +00:00
Eitan Seri-Levi
6636167503 Log block import source (#5738)
* the default target peers is 100

* add some comments

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into track-block-import-source

* add block import source

* revert

* update logging text

* fix tests

* lint

* use % instaed of to_string
2024-05-15 09:17:06 +00:00
Lion - dapplion
6f45ad4534 Log stuck lookups (#5778)
* Log stuck lookups every interval

* Implement debug manually

* Add comment

* Do not print peers twice

* Add SYNC_LOOKUPS_STUCK metric

* Skip logging request root

* use derivative

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into log-stuck-lookups

* add req id to debug

* Merge remote-tracking branch 'sigp/unstable' into log-stuck-lookups

* Fix conflict with unstable
2024-05-14 17:34:26 +00:00
Lion - dapplion
683d9df63b Don't request block components until having block (#5774)
* Don't request block components until having block

* Update tests

* Resolve todo comment

* Merge branch 'unstable' into request-blocks-first
2024-05-14 14:50:38 +00:00
Lion - dapplion
ce66ab374e Enforce sync lookup receives a single result (#5777)
* Enforce sync lookup receives a single result
2024-05-14 10:12:48 +00:00
Lion - dapplion
f37ffe4b8d Do not request current child lookup peers (#5724)
* Do not request current child lookup peers

* Update tests
2024-05-13 15:13:32 +00:00
Lion - dapplion
93e0649abc Notify lookup sync of gossip processing results (#5722)
* Notify lookup sync of gossip processing results

* Add tests

* Add GossipBlockProcessResult event

* Re-add dropped comments

* Update beacon_node/network/src/network_beacon_processor/sync_methods.rs

* update test_lookup_disconnection_peer_left
2024-05-13 11:41:29 +00:00
Jimmy Chen
6d792b4280 Revise contributors guide (#5720)
* Revise contributors guide.
2024-05-07 00:36:32 +00:00
Lion - dapplion
5135a77e38 Move sync lookup trait function to its caller (#5704)
* Move sync lookup trait function to its caller

* lint
2024-05-06 17:55:12 +00:00
Lion - dapplion
b87c36ac0e Report RPC Errors to the application on peer disconnections (#5680)
* Report RPC Errors to the application on peer disconnections

Co-authored-by: Age Manning <Age@AgeManning.com>

* Expect RPCError::Disconnect to fail ongoing requests

* Drop lookups after peer disconnect and not awaiting events

* Allow RPCError disconnect through network service

* Update beacon_node/lighthouse_network/src/service/mod.rs

Co-authored-by: Age Manning <Age@AgeManning.com>

* Merge branch 'unstable' into rpc-error-on-disconnect
2024-05-06 17:18:47 +00:00
Lion - dapplion
436d54e4bf Consistent logging of full root in lookup debug logs (#5700)
* Consistent logging of full root in lookup debug logs

* Tag sync log with service

* More logs

* Log when new peers are added

* Don't shortcircuit add_peer
2024-05-06 15:41:38 +00:00
Lukas Rusak
da9d38698d web3Signer: set header "Accept: application/json" as we expect json in the response (#5692)
* web3Signer: set header "Accept: application/json" as we expect json in the response

The web3signer handler in lighthouse requires a json response.
Setting the header "Accept: application/json" indicates to the web3signer that json is an acceptable response.

Signed-off-by: Lukas Rusak <lorusak@gmail.com>

* fixup! web3Signer: set header "Accept: application/json" as we expect json in the response
2024-05-06 03:02:47 +00:00
Michael Sproul
fe20ef955b One less listen address race (#5718)
* One less listen address race
2024-05-06 01:59:47 +00:00
Pawan Dhananjay
1af3f0f9d8 Make modified helpers in electra fork aware (#5698)
* Make modified helpers in electra fork aware

* Make more functions fork aware

* formatting fixes only.
2024-05-03 09:54:01 +00:00
Michael Sproul
d3d429ff5c Improve logging and metrics for block publication (#5699)
* Improve logging and metrics for block publication

* Add better buckets

* Bump SQL connection timeout for tests.
2024-05-03 08:24:49 +00:00
Michael Sproul
ee974db0ba Add metric for current epoch total balance (#5688)
* Add metric for current epoch total balance
2024-05-02 04:50:30 +00:00
João Oliveira
6725837dd7 Update Cargo.lock (#5670)
* update rust-yamux

* update Cargo.lock

* Merge branch 'unstable' of github.com:jxs/lighthouse into update-cargo

* Merge branch 'unstable' of github.com:sigp/lighthouse into update-cargo

* update to new libp2p versions
2024-05-02 03:32:00 +00:00
Lion - dapplion
01e4e3527e Check da_checker before doing a block lookup request (#5681)
* Check da_checker before doing a block lookup request

* Ensure consistent handling of lookup result

* use req resp pre import cache rather than da checker
2024-05-01 14:45:36 +00:00
Age Manning
59753f5fed Improve ENR updates (#5483)
* Improve ENR updates

* forever fmt

* Appease my old friend clippy

* Merge network unstable
2024-05-01 08:40:19 +00:00
realbigsean
b9655b658e Ensure block only range requests don't fail on download (#5675)
* ensure pruned blobs don't fail on download

* Typo
2024-05-01 06:10:15 +00:00
realbigsean
beaa586d98 delete spammy log (#5672)
*  delete spammy log
2024-04-30 21:31:58 +00:00
Lion - dapplion
ce66582c16 Merge parent and current sync lookups (#5655)
* Drop lookup type trait for a simple arg

* Drop reconstructed for processing

* Send parent blocks one by one

* Merge current and parent lookups

* Merge current and parent lookups clean up todos

* Merge current and parent lookups tests

* Merge remote-tracking branch 'origin/unstable' into sync-merged-lookup

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into sync-merged-lookup

* fix compile after merge

* #5655 pr review (#26)

* fix compile after merge

* remove todos, fix typos etc

* fix compile

* stable rng

* delete TODO and unfilled out test

* make download result a struct

* enums instead of bools as params

* fix comment

* Various fixes

* Track ignored child components

* Track dropped lookup reason as metric

* fix test

* add comment describing behavior of avail check error

*  update ordering
2024-04-30 20:12:15 +00:00
antondlr
196d9fd110 Only portable builds (binaries) (#5615)
* release workflow: portable builds by default

* Delete outdated comment

* Merge branch 'unstable' into portable-builds-binaries

# Conflicts:
#	.github/workflows/release.yml
2024-04-30 20:12:11 +00:00
Ærvin
aea02c60d3 Uncomment self_hosted_runner after PR Merge #5137 (#5291)
* Uncomment self_hosted_runner after PR Merge #5137

* Merge branch 'unstable' into fix_todo

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into fix_todo
2024-04-30 19:34:04 +00:00
Jimmy Chen
63fad7e330 Remove snapshot cache related code (#5661)
* Remove snapshot cache and other references.

* Fix default state cache size in docs

* Remove cache miss comment entirely

* Add state cache CLI tests
2024-04-30 16:08:13 +00:00
antondlr
d0602c3207 pin macos release runner to macos-13 (#5665)
* pin macos release runner to `macos-13`

* Update .github/workflows/release.yml
2024-04-30 08:56:49 +00:00
Eitan Seri-Levi
d3bf9a8956 Proposer and attester slashing sse events (#5327)
* default vc to block v3 endpoint and deprecate block-v3 flag

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into unstable

* add proposer and attester event variants

* add TOOOs

* add tests, event triggers

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into proposer-and-attester-slashing-sse-events

* revert

* revert

* remove double event tracking

* Merge branch 'unstable' into proposer-and-attester-slashing-sse-events

* remove todo, fix test

* resolve merge conflicts

* Merge branch 'proposer-and-attester-slashing-sse-events' of https://github.com/eserilev/lighthouse into proposer-and-attester-slashing-sse-events

* leftover debugging

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into proposer-and-attester-slashing-sse-events

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into proposer-and-attester-slashing-sse-events
2024-04-30 08:56:45 +00:00
Roman Krasiuk
c8ffafb3f8 fix(validator_client): raise soft fd limit (#4796)
* fix(validator_client): raise soft fd limit

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into rkrasiuk/raise-vc-fdlimit

* cargo lock

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into rkrasiuk/raise-vc-fdlimit
2024-04-29 22:55:23 +00:00
realbigsean
c33edc82eb Beta compiler fix (#5659)
* fix beta compiler compilation

* remove unused import

* Revert "remove unused import"

This reverts commit 0bef36b05b.

* Revert "fix beta compiler compilation"

This reverts commit 23152cf4cc.

* rename ununsed fields

* allow dead code on some error variants

* remove unused blob download queue

* add back debug to backfill error

* more allow dead code on errors
2024-04-29 20:45:54 +00:00
Lion - dapplion
40d412629e Ignore gossip blob already imported (#5656)
* Ignore gossip blob already imported
2024-04-29 14:30:13 +00:00
Pawan Dhananjay
8b24880df7 Add more electra helpers (#5653)
* Add new helpers

* Fix some stuff

* Fix compilation errors

* lint

* Address review
2024-04-27 01:09:29 +00:00
realbigsean
000a4fdf4d Electra other containers (#5652)
* add new fields to execution payload and header

* beacon state changes

* partial beacon state

* safe arith in upgrade to electra

* initialize balances cache in interop genesis state

* Revert "initialize balances cache in interop genesis state"

This reverts commit c60b522865.

* always initialize balances cache if necessary in electra upgrade

* build cache earlier

* fix block test

* per fork NUM_FIELDS_POW2

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into electra-other-containers

* fix lints

* get fields based on post state, as is spec'd

* fix type and move cache build
2024-04-26 20:17:23 +00:00
Lion - dapplion
a1141ea1ef Deterministic block generation for tests (#5654)
* Deterministic block generation for tests
2024-04-26 13:25:45 +00:00
Mac L
13f94ef0f3 Rename Merge to Bellatrix (#5601)
* Rename Merge to Bellatrix

* Remove tree-hash-cache which got readded from the rebase
2024-04-25 20:19:41 +00:00
realbigsean
320345695d Add electra presets to beacon API (#5630)
* add presets to API

* add extra fields to config spec in beacon API

* remove unused

* add mainnet presets for gnosis and fix minimal preset default values
2024-04-25 18:33:03 +00:00
realbigsean
dd340eebdc Fix execution integration tests (#5647)
* update waiting status

* revert to old nethermind version
2024-04-25 17:11:01 +00:00
ethDreamer
4a48d7b546 Encode Execution Engine Client Version In Graffiti (#5290)
* Add `engine_clientVersionV1` structs

* Implement `engine_clientVersionV1`

* Update to latest spec changes

* Implement GraffitiCalculator Service

* Added Unit Tests for GraffitiCalculator

* Address Mac's Comments

* Remove need to use clap in beacon chain

* Merge remote-tracking branch 'upstream/unstable' into el_client_version_graffiti

* Merge branch 'unstable' into el_client_version_graffiti

# Conflicts:
#	beacon_node/beacon_chain/Cargo.toml
2024-04-24 06:02:48 +00:00
Lion - dapplion
c38b05d640 Drop lookup type trait for a simple arg (#5620)
* Drop lookup type trait for a simple arg
2024-04-24 06:02:45 +00:00
Michael Sproul
1eaaa4a8bd Bump jobserver and fix non-portable builds (#5641)
* Bump jobserver and fix non-portable builds
2024-04-24 06:02:42 +00:00
realbigsean
c4a2bcb9c7 Yaml rust2 (#5635)
* use yaml-rust2
2024-04-24 06:02:10 +00:00
Michael Sproul
61962898e2 In-memory tree states (#5533)
* Consensus changes

* EF tests

* lcli

* common and watch

* account manager

* cargo

* fork choice

* promise cache

* beacon chain

* interop genesis

* http api

* lighthouse

* op pool

* beacon chain misc

* parallel state cache

* store

* fix issues in store

* IT COMPILES

* Remove some unnecessary module qualification

* Revert Arced pubkey optimization (#5536)

* Merge remote-tracking branch 'origin/unstable' into tree-states-memory

* Fix caching, rebasing and some tests

* Remove unused deps

* Merge remote-tracking branch 'origin/unstable' into tree-states-memory

* Small cleanups

* Revert shuffling cache/promise cache changes

* Fix state advance bugs

* Fix shuffling tests

* Remove some resolved FIXMEs

* Remove StateProcessingStrategy

* Optimise withdrawals calculation

* Don't reorg if state cache is missed

* Remove inconsistent state func

* Fix beta compiler

* Rebase early, rebase often

* Fix state caching behaviour

* Update to milhouse release

* Fix on-disk consensus context format

* Merge remote-tracking branch 'origin/unstable' into tree-states-memory

* Squashed commit of the following:

commit 3a16649023
Author: Michael Sproul <michael@sigmaprime.io>
Date:   Thu Apr 18 14:26:09 2024 +1000

    Fix on-disk consensus context format

* Keep indexed attestations, thanks Sean

* Merge branch 'on-disk-consensus-context' into tree-states-memory

* Merge branch 'unstable' into tree-states-memory

* Address half of Sean's review

* More simplifications from Sean's review

* Cache state after get_advanced_hot_state
2024-04-24 01:22:36 +00:00
antondlr
4cad1fcbbe Add missing lcli targets in makefile (#5633)
* Add missing lcli targets to Makefile
2024-04-23 20:12:55 +00:00
antondlr
76460ba838 Only portable builds (docker) (#5614)
* portable builds by default, build multiarch lcli
2024-04-23 17:58:47 +00:00
ethDreamer
05fbbdd840 Electra: Add Preset, Constants, & Config (#5606)
* Electra: Add Presets, Constants, & Config
2024-04-23 14:46:49 +00:00
Michael Sproul
72a33604b3 Add timing for block availability (#5510)
* Add timing for block availability

* Attestation metrics analysis

* Prettier printing

* Add some metrics and timings to track late blocks

* Update to latest unstable

* fmt

* Merge latest unstable

* Small tweaks

* Try pushing blob timing down into verification

* Simplify for clippy
2024-04-23 13:13:34 +00:00
ethDreamer
82b131d37f Electra: Add New Containers (#5607)
* Electra: Add New Containers
2024-04-23 00:21:21 +00:00
chonghe
ad7f0e0cdb Delete repetitive execute command in local testnet scripts (#5611)
* Delete repetitive execute

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into local-testnet
2024-04-22 16:07:43 +00:00
Jimmy Chen
532206e008 Fix stuck backfill when scheduled work queue is at capacity (#5575)
* Fix stuck backfill and add regression test.

* Remove unnecessary `yield_now`

* Merge branch 'unstable' into fix-stuck-backfill

* Revert previous change and add extra comment.

* Merge branch 'unstable' into fix-stuck-backfill

* Update tests to use configured event schedule instead of hard coded values.

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into fix-stuck-backfill
2024-04-22 16:06:46 +00:00
Lion - dapplion
f7aca97a55 Handle sync lookup request streams in network context (#5583)
* by-root-stream-terminator

* Fix tests

* Resolve merge conflicts

* Log report reason

* Some lints and bugfixes (#23)

* fix lints

* bug fixes

* Fix tests

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into handle-sync-lookup-requests

* Pr 5583 review (#24)

* add bad state warn log

* add rust docs to new fields in `SyncNetworkContext`

* remove timestamp todo

* add back lookup verify error

* remove TODOs
2024-04-22 16:06:39 +00:00
Mac L
67f8405921 Update Simulator tests (#5520)
* Rewrite Simulator

* Add fallback simulator

* Try Sean's test fix

* More fixes

* Cleanup

* Merge branch 'unstable' into update-simulator

* Update cli.rs

* Add sync sim to basic sim

* Formatting

* Add fixes and new block production check

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into update-simulator

* fix compile
2024-04-22 15:08:36 +00:00
João Oliveira
9b5895ca89 Fix cargo audit RUSTSEC-2024-0336 (#5612)
* replace unmaintained rust_yaml with serde_yml

* update warp
2024-04-22 13:01:06 +00:00
Michael Sproul
5a9e973f04 Fix on-disk consensus context format (#5598)
* Fix on-disk consensus context format

* Keep indexed attestations, thanks Sean
2024-04-19 08:18:30 +00:00
chonghe
5c30afbc7c Revise secrets-dir flag in the VC (#5480)
* Update docs on secrets-dir

* Hidden secrets-dir flag

* Remove conflicts_with

* Restore description

* make cli

* Update book/src/validator-management.md
2024-04-18 07:14:59 +00:00
AMIT SINGH
62e4abfbff Fix execution layer redundancy (#5588)
* remove execution layer url redundancy

* fix typo

* fix tests

* fix formatting
2024-04-18 07:14:56 +00:00
Michael Sproul
49617f3e82 Set web3signer keep-alive to 20s by default (#5587)
* Set web3signer keep-alive to 20s by default

* add tests
2024-04-17 15:09:09 +00:00
ethDreamer
cda926ce1b Rename Functions to More Closely Match Spec (#5591)
* Rename Functions to More Closely Match Spec
2024-04-16 19:03:08 +00:00
ethDreamer
f68989815c Restore Log on Error & Spawn Blocking in Streamer (#5585)
* Restore Logging in Error Cases

* Use Spawn Blocking for Loading Blocks in Streamer

* Merge remote-tracking branch 'upstream/unstable' into request_logging_spawn_blocking

* Address Sean's Comments

* save a clone
2024-04-16 14:56:00 +00:00
Lion - dapplion
e5b8d1237d Use Action in single_block_component_processed (#5564)
* Use Action in single_block_component_processed

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into single_block_component_processed-action

* fix compile after merge

* add continue action for when we are awaiting other parts of missing components
2024-04-16 12:59:53 +00:00
Lion - dapplion
f5e0404fb8 Ensure proper ReqResp blocks_by_* response stream termination (#5582)
* Ensure proper ReqResp blocks_by_* response stream termination

* retrigger CI
2024-04-15 21:20:50 +00:00
Lion - dapplion
fee2ee9c08 Make SingleLookupRequestState fields private (#5563)
* Make SingleLookupRequestState fields private

* Fix tests
2024-04-15 13:47:08 +00:00
Lion - dapplion
b6a1c863a2 Use spawn_async in ByRoot handling workers (#5557)
* Use spawn_async in ByRoot handling workers

* box large variants
2024-04-12 19:30:04 +00:00
Lion - dapplion
116a55e8a5 Ensure proper ReqResp response stream termination (#5556)
* Ensure proper ReqResp response stream termination

* Update beacon_node/network/src/network_beacon_processor/rpc_methods.rs

* Update beacon_node/network/src/network_beacon_processor/rpc_methods.rs

* cargo fmt
2024-04-12 16:15:04 +00:00
Lion - dapplion
6fb0b2ed78 Sync lookup dedup range and blobs (#5561)
* Handle sync range blocks as blocks and blobs

* Merge range sync and backfill sync handling

* Update tests

* Add no_blobs_into_responses test

* Address @realbigsean comments

* Merge remote-tracking branch 'origin/unstable' into sync-lookup-dedup-range-and-blobs
2024-04-12 15:39:11 +00:00
Lion - dapplion
5fdd3b39bb Allow 1 count block request to return 0 blocks (#5554)
* Allow 1 count block request to return 0 blocks

* Address @pawanjay176 review
2024-04-12 14:22:20 +00:00
Eitan Seri-Levi
6bac5ce12b Deprecate http-spec-fork and http-allow-sync-stalled (#5500)
* deprecate flags

* fmt

* remove backslash

* remove hidden flags from the book

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into deprecate-http-spec-fork-and-http-allow-sync-stalled

* add warn, re-add tests

* Apply suggestions from code review

* Merge remote-tracking branch 'origin/unstable' into deprecate-http-spec-fork-and-http-allow-sync-stalled

* Fix imports
2024-04-12 04:21:00 +00:00
Jimmy Chen
d30ba976a1 Use the mesh_n value from NetworkLoad for PeerScoreSettings (#5013)
* Build `gs_config` and use the `mesh_n` config for `PeerScoreSettings`

* Remove unused imports

* Merge branch 'unstable' into peer-score-settings

# Conflicts:
#	beacon_node/lighthouse_network/src/config.rs
#	beacon_node/lighthouse_network/src/service/gossipsub_scoring_parameters.rs
#	beacon_node/lighthouse_network/src/service/mod.rs
2024-04-11 20:14:15 +00:00
Jimmy Chen
7e49f82726 Only load Kzg in tests if necessary and only load it once (#5555)
* Only load KZG once if necessary  in tests.
2024-04-11 20:14:11 +00:00
Lion - dapplion
34dbb32610 Nest lookup type into request id SingleBlock and SingleBlob (#5562)
* Nest lookup type into block lookup RequestId
2024-04-11 16:32:06 +00:00
realbigsean
3d4e6e263e Remove availability view trait (#5544)
* Move processing cache out of DA

* Merge branch 'sigp/unstable' into non-da-processing-cach

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into non-da-processing-cache

* remove unused file, remove outdated TODO, add is_deneb check to missing blob id calculations

* remove availability view trait

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into remove-availability-view-trait

* fix lints
2024-04-11 15:33:54 +00:00
Pawan Dhananjay
c0177687e1 Bump ckzg to use the crates.io version (#5542)
* Bump ckzg to use the crates.io version

* Update crypto/kzg/Cargo.toml
2024-04-10 23:25:14 +00:00
Lion - dapplion
54fbdda4d2 Add logging in UnknownBlockHashFromAttestation handling (#5546)
* Use should_search_for_block in unknown block root event

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into should_search_for_block
2024-04-10 20:52:22 +00:00
Lion - dapplion
30dc260472 Move processing cache out of DA (#5420)
* Move processing cache out of DA

* Merge branch 'sigp/unstable' into non-da-processing-cach

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into non-da-processing-cache

* remove unused file, remove outdated TODO, add is_deneb check to missing blob id calculations

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into non-da-processing-cache

* fix lints
2024-04-10 17:19:40 +00:00
Lion - dapplion
b1f9751a69 Event-based block lookup tests (#5534)
* WIP

* Initial working version of new sync tests.

* Remove sync traits and fix lints.

* Reduce internal method visibility and make test method instead. Remove extra beacon chain harness instance created in tests.

* Improve `SyncTester` api.

* Fix lint.

* Test example

* Lookup tests using rig

* Tests should interface with events only

* lint

* Skip deneb test pre-deneb

* Add more assertions

* Remove logging changes

* Address @jimmygchen comments

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into bn-p2p-tests

* remove unused assertions

* fix lint
2024-04-10 12:05:18 +00:00
pahor167
72af6fb83a Skip calculation of boosted_relay_value when builder_boost_factor pro… (#5352)
* Skip calculation of boosted_relay_value when builder_boost_factor provided

* PR comments
2024-04-10 12:05:13 +00:00
Lion - dapplion
ced653873e Impl Ord on ForkName for ChainSpec usage (#5531)
* Impl Ord on ForkName for ChainSpec usage

* add fork ord consistent test
2024-04-10 09:02:23 +00:00
realbigsean
d527d124dd Beta compiler fixes (#5543)
* remove duplicate imports in gossip tests. get rid of duplicate trait bound locations

* more double import fixes

* cargo fmt
2024-04-10 07:47:05 +00:00
Lion - dapplion
b74da14261 Improve block lookup logging (#5535)
* Improve block lookup logging
2024-04-09 14:25:18 +00:00
GeemoCandama
32be063f0f Support LightClientFinalityUpdate and LightClientOptimisticUpdate rpcs (#3849)
* add light client optimistic and finality update rpc

* Arc the updates in the response

* add conditional advertisement for both LightClientOptimisticUpdate and LightClientFinalityUpdate

* alter display for inboundrequest light client optimistic and finality updates

* remove LightClientOptimistic/FinalityReuest struct and some minor fixes

* rebase

* failing rpc_test for LightClientBootstrap and beginning of MockLib2pLightClient

* minor change

* added MockRPCHandler by importing everything except OutboundRequest. Need to implement the ConnectionHandler trait now should be copy pastable

* almost there but ran into issue where needed to implement BaseOutboundRequest.

* failing but running with a light client service of sorts

* small test change

* changed Protocol::LightClientBootstrap response limit

* deleted some stuff from ConnectionHandler Implementation for the mock light client if you need to make something with multiple requests work maybe check here

* deleted purging expired inbound/outbound streams code

* deleted drive inbound streams that need to be processed

* removed unused imports

* made things private again

* deleted inject_fully_negotiated_inbound

* made more things private again

* more

* turned the logger off in the test

* added failing test for new rpc

* add rate limit for new rpcs

* change InboundUpgrade function to use new rpcs. fmt. add test for LightClientFinalityUpdate

* rebasing fix

* add LightClientUpdate to handle_rpc functions

* added context bytes

* fmt

* use correct unsed_tcp4_port function

* fix for recent config changes and adding context_bytes for the light client protocols

* fix clippy complaint

* Merge branch 'unstable' into lc-reqresp

# Conflicts:
#	beacon_node/beacon_processor/src/lib.rs
#	beacon_node/lighthouse_network/src/peer_manager/mod.rs
#	beacon_node/lighthouse_network/src/rpc/codec/ssz_snappy.rs
#	beacon_node/lighthouse_network/src/rpc/config.rs
#	beacon_node/lighthouse_network/src/rpc/methods.rs
#	beacon_node/lighthouse_network/src/rpc/mod.rs
#	beacon_node/lighthouse_network/src/rpc/outbound.rs
#	beacon_node/lighthouse_network/src/rpc/protocol.rs
#	beacon_node/lighthouse_network/src/rpc/rate_limiter.rs
#	beacon_node/lighthouse_network/src/rpc/self_limiter.rs
#	beacon_node/lighthouse_network/src/service/api_types.rs
#	beacon_node/lighthouse_network/tests/common/mod.rs
#	beacon_node/lighthouse_network/tests/rpc_tests.rs
#	beacon_node/network/src/network_beacon_processor/rpc_methods.rs
#	beacon_node/network/src/router.rs

* Error handling updates and various cleanups.

* Moar minor clean ups.

* Do not ban peer for rate limiting light client requests

* Merge branch 'unstable' into lc-reqresp. Also removed the mock light client tests to make it compile (See #4940).

# Conflicts:
#	beacon_node/lighthouse_network/src/rpc/codec/ssz_snappy.rs
#	beacon_node/lighthouse_network/src/rpc/methods.rs
#	beacon_node/lighthouse_network/src/rpc/mod.rs
#	beacon_node/lighthouse_network/src/rpc/protocol.rs
#	beacon_node/lighthouse_network/src/service/api_types.rs
#	beacon_node/lighthouse_network/tests/common/mod.rs
#	beacon_node/network/src/network_beacon_processor/rpc_methods.rs
#	beacon_node/network/src/router.rs
#	consensus/types/src/light_client_bootstrap.rs
#	consensus/types/src/light_client_finality_update.rs
#	consensus/types/src/light_client_optimistic_update.rs

* Remove unnecessary changes

* Add missing light client queue handling.

* Merge branch 'unstable' into lc-reqresp

* Merge branch 'unstable' into lc-reqresp

# Conflicts:
#	beacon_node/lighthouse_network/src/rpc/codec/ssz_snappy.rs
#	beacon_node/lighthouse_network/src/service/api_types.rs
#	consensus/types/src/light_client_finality_update.rs
#	consensus/types/src/light_client_optimistic_update.rs

* Add context bytes for light client RPC responses.

* Add RPC limits for light client object.

* Fix lint

* Fix incorrect light client max size computation.

* Merge branch 'unstable' into lc-reqresp

# Conflicts:
#	beacon_node/lighthouse_network/src/rpc/codec/ssz_snappy.rs
#	beacon_node/lighthouse_network/src/rpc/protocol.rs
#	beacon_node/lighthouse_network/src/service/api_types.rs

* Remove unwanted local changes.

* Merge branch 'unstable' into lc-reqresp

* Replace `unimplemented` electra code path with deneb values.
2024-04-09 07:23:39 +00:00
0xalex88
1b88d29807 Use hashset to filter validators ids in http_api (#5468)
* Use hashset to filter validators ids in http_api

* Update beacon_node/http_api/src/validators.rs
2024-04-09 03:49:03 +00:00
Michael Sproul
06eb181b42 Delete ParticipationCache (#5525)
* Delete `ParticipationCache`
2024-04-05 03:15:42 +00:00
Eitan Seri-Levi
b65daac907 Add missing header to eth/v1/builder/blinded_blocks (#5407)
* add missing header

* read header in mock builder

* Merge branch 'unstable' into builder-blinded-blocks-missing-header
2024-04-04 19:38:09 +00:00
Eitan Seri-Levi
ee69e14db9 Add is_parent_strong proposer re-org check (#5417)
* initial fork choice additions

* add helper fns

* add is_parent_strong

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into add_is_parent_strong_check

* disabling proposer reorg should set parent_threshold to u64 max

* add new flag, is_parent_strong check in override fcu params

* cherry-pick changes

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into add_is_parent_strong_check

* cleanup

* fmt

* Minor review tweaks
2024-04-04 19:38:06 +00:00
Lion - dapplion
053525e281 Remove DataAvailabilityView trait from ChildComponents (#5421)
* Remove DataAvailabilityView trait from ChildComponents

* PR reviews

* Update beacon_node/network/src/sync/block_lookups/common.rs

Co-authored-by: realbigsean <seananderson33@GMAIL.com>

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into child_components_independent
2024-04-04 18:49:09 +00:00
Michael Sproul
feb531f85b Single-pass epoch processing and optimised block processing (#5279)
* Single-pass epoch processing (#4483, #4573)

Co-authored-by: Michael Sproul <michael@sigmaprime.io>

* Delete unused epoch processing code (#5170)

* Delete unused epoch processing code

* Compare total deltas

* Remove unnecessary apply_pending

* cargo fmt

* Remove newline

* Use epoch cache in block packing (#5223)

* Remove progressive balances mode (#5224)

* inline inactivity_penalty_quotient_for_state

* drop previous_epoch_total_active_balance

* fc lint

* spec compliant process_sync_aggregate (#15)

* spec compliant process_sync_aggregate

* Update consensus/state_processing/src/per_block_processing/altair/sync_committee.rs

Co-authored-by: Michael Sproul <micsproul@gmail.com>

---------

Co-authored-by: Michael Sproul <micsproul@gmail.com>

* Delete the participation cache (#16)

* update help

* Fix op_pool tests

* Fix fork choice tests

* Merge remote-tracking branch 'sigp/unstable' into epoch-single-pass

* Simplify exit cache (#5280)

* Fix clippy on exit cache

* Clean up single-pass a bit (#5282)

* Address Mark's review of single-pass (#5386)

* Merge remote-tracking branch 'origin/unstable' into epoch-single-pass

* Address Sean's review comments (#5414)

* Address most of Sean's review comments

* Simplify total balance cache building

* Clean up unused junk

* Merge remote-tracking branch 'origin/unstable' into epoch-single-pass

* More self-review

* Merge remote-tracking branch 'origin/unstable' into epoch-single-pass

* Merge branch 'unstable' into epoch-single-pass

* Fix imports for beta compiler

* Fix tests, probably
2024-04-04 13:14:36 +00:00
Eitan Seri-Levi
f4cdcea7b1 Return not synced errors for endpoints that require syncing (#5136)
* add not synced filter into then blocks

* refactor
2024-04-04 01:36:23 +00:00
Michael Sproul
7825af4a6e Bump h2 for RUSTSEC-2024-0332 (#5514)
* Bump `h2` for RUSTSEC-2024-0332
2024-04-04 01:36:20 +00:00
Mac L
969d12dc6f Use E for EthSpec globally (#5264)
* Use `E` for `EthSpec` globally

* Fix tests

* Merge branch 'unstable' into e-ethspec

* Merge branch 'unstable' into e-ethspec

# Conflicts:
#	beacon_node/execution_layer/src/engine_api.rs
#	beacon_node/execution_layer/src/engine_api/http.rs
#	beacon_node/execution_layer/src/engine_api/json_structures.rs
#	beacon_node/execution_layer/src/test_utils/handle_rpc.rs
#	beacon_node/store/src/partial_beacon_state.rs
#	consensus/types/src/beacon_block.rs
#	consensus/types/src/beacon_block_body.rs
#	consensus/types/src/beacon_state.rs
#	consensus/types/src/config_and_preset.rs
#	consensus/types/src/execution_payload.rs
#	consensus/types/src/execution_payload_header.rs
#	consensus/types/src/light_client_optimistic_update.rs
#	consensus/types/src/payload.rs
#	lcli/src/parse_ssz.rs
2024-04-02 15:12:25 +00:00
Mac L
f8fdb71f50 Add Electra fork boilerplate (#5122)
* Add Electra fork boilerplate

* Remove electra from spec tests

* Fix tests

* Remove sneaky log file

* Fix more tests

* Fix even more tests and add suggestions

* Remove unrelated lcli addition

* Update more tests

* Merge branch 'unstable' into electra

* Add comment for test-suite lcli override

* Merge branch 'unstable' into electra

* Cleanup

* Merge branch 'unstable' into electra

* Apply suggestions

* Merge branch 'unstable' into electra

* Merge sigp/unstable into electra

* Merge branch 'unstable' into electra
2024-04-02 12:35:02 +00:00
realbigsean
3058b96f25 Release v5.1.3 (#5497)
* Release v5.1.3
2024-03-28 05:22:30 +00:00
Pawan Dhananjay
c909941ca1 Bump duplicate cache time (#5493)
* Bump seen_ttl for gossipsub duplicate cache
2024-03-27 17:31:51 +00:00
realbigsean
9d24844f50 Lookup log improvements (#5491)
* log improvements
2024-03-27 13:24:04 +00:00
realbigsean
334aa2eabd Single lookup improvements (#5488)
* Fix unexpected `UnrequestedBlobId` and `ExtraBlocksReturned` errors due to race conditions.

* Continue chain segment processing and skip any blocks that are already known, rather than returning an error.

* more de-dup checking

* ensure we don't reset `requested_ids` during rpc download

* better fix

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into more-dup-lookup-fixes

* remove chain hash check

* Merge branch 'fix-block-lookup-race' of https://github.com/jimmygchen/lighthouse into sean-test-lookups

* remove block check

* add back tests

* Log and CI fixes

* undue extra check

* Merge branch 'sean-test-lookups' of https://github.com/realbigsean/lighthouse into sean-test-lookups

* log improvements

* Improve logging
2024-03-27 10:01:27 +00:00
Michael Sproul
250a5bdc27 Run fork choice after RPC blob import (#5475)
* Run fork choice after RPC blob import
2024-03-26 21:36:08 +00:00
João Oliveira
59ef564b1d Move gossipsub into a separate crate (#5401)
* move gossipsub into a separate crate

* Merge branch 'unstable' of github.com:sigp/lighthouse into separate-gossipsub

* address review 2

* clippy beta

* update logging to log gossipsub logs
2024-03-26 03:10:59 +00:00
Akihito Nakano
8cec8a6793 Fix double counted metrics (#5476)
* Fix double counted metrics
2024-03-26 03:10:26 +00:00
Eitan Seri-Levi
e4d4e439cb Add Capella & Deneb light client support (#4946)
* rebase and add comment

* conditional test

* test

* optimistic chould be working now

* finality should be working now

* try again

* try again

* clippy fix

* add lc bootstrap beacon api

* add lc optimistic/finality update to events

* fmt

* That error isn't occuring on my computer but I think this should fix it

* Merge branch 'unstable' into light_client_beacon_api_1

# Conflicts:
#	beacon_node/beacon_chain/src/events.rs
#	beacon_node/http_api/src/lib.rs
#	beacon_node/http_api/src/test_utils.rs
#	beacon_node/http_api/tests/main.rs
#	beacon_node/lighthouse_network/src/rpc/codec/ssz_snappy.rs
#	beacon_node/lighthouse_network/src/rpc/methods.rs
#	beacon_node/lighthouse_network/src/service/api_types.rs
#	beacon_node/network/src/beacon_processor/worker/rpc_methods.rs
#	beacon_node/tests/test.rs
#	common/eth2/src/types.rs
#	lighthouse/src/main.rs

* Add missing test file

* Update light client types to comply with Altair light client spec.

* Fix test compilation

* Merge branch 'unstable' into light_client_beacon_api_1

* Support deserializing light client structures for the Bellatrix fork

* Move `get_light_client_bootstrap` logic to `BeaconChain`. `LightClientBootstrap` API to return `ForkVersionedResponse`.

* Misc fixes.
- log cleanup
- move http_api config mutation to `config::get_config` for consistency
- fix light client API responses

* Add light client bootstrap API test and fix existing ones.

* Merge branch 'unstable' into light_client_beacon_api_1

* Fix test for `light-client-server` http api config.

* Appease clippy

* Add Altair light client SSZ tests

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into light_client_beacon_api_1

* updates to light client header

* light client header from signed beacon block

* using options

* implement helper functions

* placeholder conversion from vec hash256 to exec branch

* add deneb

* using fixed vector

* remove unwraps

* by epoch

* compute merkle proof

* merkle proof

* update comments

* resolve merge conflicts

* linting

* Merge branch 'unstable' into light-client-ssz-tests

# Conflicts:
#	beacon_node/beacon_chain/src/beacon_chain.rs
#	consensus/types/src/light_client_bootstrap.rs
#	consensus/types/src/light_client_header.rs

* superstruct attempt

* superstruct changes

* lint

* altair

* update

* update

* changes to light_client_optimistic_ and finality

* merge unstable

* refactor

* resolved merge conflicts

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into capella_deneb_light_client_types

* block_to_light_client_header fork aware

* fmt

* comment fix

* comment fix

* include merge fork, update deserialize_by_fork, refactor

* fmt

* pass by ref to prevent clone

* rename merkle proof fn

* add FIXME

* LightClientHeader TestRandom

* fix comments

* fork version deserialize

* merge unstable

* move fn arguments, fork name calc

* use task executor

* remove unneeded fns

* remove dead code

* add manual ssz decoding/encoding and add ssz_tests_by_fork macro

* merge deneb types with tests

* merge ssz tests, revert code deletion, cleanup

* move chainspec

* update ssz tests

* fmt

* light client ssz tests

* change to superstruct

* changes from feedback

* linting

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into capella_deneb_light_client_types

* test fix

* cleanup

* Remove unused `derive`.

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into capella_deneb_light_client_types

* beta compiler fix

* merge
2024-03-25 11:01:56 +00:00
chonghe
19b0db2cdf Delete PRE_CAPELLA_ENGINE_CAPABILITIES (#5406)
* Adjust width

* Commit changes

* Delete PRE_CAPELLA_ENGINE_CAPABILITIES

* Revert "Adjust width"

This reverts commit 6fea81b897.

* Revert "Commit changes"

This reverts commit d00859a63e.

* Simplify

* Merge branch 'delete-pre-capella' of https://github.com/chong-he/lighthouse into delete-pre-capella
2024-03-23 21:30:36 +00:00
chonghe
a0e64d0652 Built-in documentation text width in Lighthouse book (#5394)
* Adjust width

* Commit changes

* Trigger Build
2024-03-23 20:52:09 +00:00
dknopik
0a6e4a11d7 Verify whether validators really are unknown during sync committee duty API request (#5174)
* Verify whether validators really are unknown during sync committee duty API request

* Merge branch 'unstable' into fix-4717

* Merge branch 'unstable' into fix-4717

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into fix-4717
2024-03-23 00:01:11 +00:00
Daniel Ramírez-Chiquillo
035c378c61 Make sure all geth processes are killed when stopping a local testnet (#5383)
* Fix geth processes not being killed when stopping a local testnet

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into fix_stop_testnet
2024-03-23 00:01:08 +00:00
Lion - dapplion
306d3eb0eb Delete unused incomplete_processing_components (#5418)
* Delete unused incomplete_processing_components

* lint
2024-03-23 00:00:35 +00:00
Michael Sproul
332e1bb9e6 Fix one and hide all beacon-processor flags (#5397)
* Fix `beacon-processor-work-queue-len`

* Hide beacon-processor flags
2024-03-22 23:24:17 +00:00
zhiqiangxu
5121d655f7 chore: reduce scope of commitment (#5426)
* reduce scope of commitment

* avoid clone for last reference

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into mod_merge_single_blob
2024-03-22 18:16:52 +00:00
zhiqiangxu
5bfe6a8ae3 chore: remove stale comment (#5440)
* rm stale comment

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into rm_irelevant_comment
2024-03-22 18:16:49 +00:00
Afanti
003bb0ae3c fix: tail command typo (#5456)
* fix: tail command typo

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into patch-1
2024-03-22 18:16:46 +00:00
realbigsean
21cdc64bfe Improve parent lookup logging (#5451)
* upgrade parent lookup result processing logs to debug, use display instead of debug for BlockError in case a blob parent unknown error is hit, add block root to BlockIsAlreadyKnown

* fix compile

* fix compile

* fix compile
2024-03-22 18:16:13 +00:00
realbigsean
5ce16192c7 Release v5.1.2 (#5453)
* Release v5.1.2
2024-03-21 18:04:00 +00:00
João Oliveira
6edf031490 disable libp2p upnp (#5449)
* disable libp2p upnp

when passing --disable-upnp cli flag
2024-03-21 01:13:03 +00:00
Michael Sproul
65a6118c53 Fix gossip verification of duplicate attester slashings (#5385)
* Fix gossip verification of duplicate attester slashings
2024-03-20 20:47:38 +00:00
João Oliveira
f33ce8cc34 fix NAT nat_open metrics report (#5427)
* fix nat reporting
2024-03-20 06:45:18 +00:00
Akihito Nakano
7117772fb3 Fix peer count metrics (#5404)
* Set the peers_per_client metrics directly, rather than using increment/decrement

* Move PEERS_CONNECTED related update to the same place

* Move PEERS_CONNECTED_MULTI related update to the same place

* Rename

* Remove unused variables
2024-03-20 06:45:16 +00:00
joao
aa592853df Improvements and Fixes in Documentation, Including Corrected Command Usage (#4998)
* Fix typo: change 'periodical' to 'periodic' in progress updates description

* Fix wrong command in Usage section

* fix typo in Development Environment section

* Fix typo: change 'Explictly' to 'Explicitly'

* Fix typos in Lighthouse UI and Contributing sections

* Fix typo: replace 'confirms' with 'conforms' in Beacon Node API description

* fix minor typographical error: change 'advice' to 'advise' in SIGILL warning message

* Fix spelling error in Detailed Guide section

* Revert "Fix typo: change 'Explictly' to 'Explicitly'"

This reverts commit 6b0781682b4f9d129d9af48d428353f596b8b852.

* Revert "fix minor typographical error: change 'advice' to 'advise' in SIGILL warning message"

This reverts commit a4904a0afd2d531caf4585e1f0726a6dd65081ea.

* compiled

* Revert "compiled"

This reverts commit 425a553bd93af93340858050cc45041ae86b3733.

* Revert "Revert "compiled""

This reverts commit b1f871cb1bd41822f4c7bd696f77ba38cdd32f9d.

* Empty commit to trigger CI.
2024-03-20 06:44:44 +00:00
Eitan Seri-Levi
01ec42e75a Fix Rust beta compiler errors 1.78.0-beta.1 (#5439)
* remove redundant imports

* fix test

* contains key

* fmt

* Merge branch 'unstable' into fix-beta-compiler
2024-03-20 05:17:02 +00:00
realbigsean
4449627c7c Dedup parent blob requests (#5432)
* de dup parent blob requests

* add new line
2024-03-20 00:19:26 +00:00
chonghe
eab3672c6d Add attestation simulator, blobs info and some updates to Lighthouse Book (#5364)
* Apply suggestions from code review

* Revise attestation simulator doc

* Revise blobs.md

* Summary

* Add blobs

* Simulator docs

* Revise attestation simulator

* minor formatting

* Revise vm node

* Update faq

* Update faq

* Add link to v4.6.0

* Remove minification in the docs

* Update Goerli to Holesky

* Add a note on moved vm validator monitor

* Update Rpi 4 note

* Revise attestation simulator doc

* Add docs for attestation simulator

* update database table

* Update faq on resources used

* Fix and update table
2024-03-14 06:12:25 +00:00
Michael Sproul
2a3c709f8c Release v5.1.1 (#5396)
* Release v5.1.1
2024-03-12 03:42:15 +00:00
Michael Sproul
1d7223fadf Fix macOS build by updating cc (#5393)
* Fix macOS build by bumping `cc`
2024-03-12 02:47:54 +00:00
Pawan Dhananjay
10a38a8aae Release v5.1.0 (#5372)
* Bump versions
2024-03-10 23:44:29 +00:00
Pawan Dhananjay
54b1c229e1 Downgrade rate limited log (#5381)
* Address review comments

* Downgrade log for by_root requests
2024-03-10 23:05:24 +00:00
Michael Sproul
f93844e63b Optimise concurrent block production (#5368)
* Optimise concurrent block production
2024-03-08 05:15:28 +00:00
Jimmy Chen
762dab23b8 Fix AddrInUse error in cli tests (#5266)
* Fix `AddrInUse` error in cli tests.
2024-03-08 05:15:25 +00:00
Jonas Bostoen
641f6be3f0 Explicit peers (#5333)
* Merge branch 'unstable' into feature/explicit-peers

* Merge latest unstable

* refactor: remove explicit-peers flag, mark trusted peers as explicit instead

* feat(beacon_node): add explicit peers to GossipSub, mark as trusted

* feat(beacon_node): add explicit peers cli + config
2024-03-07 22:22:39 +00:00
Age Manning
de91c77cb2 Improve peer performance for NAT'd nodes (#5345)
* Merge latest unstable

* Reduce diff

* Reduce logic, discover up to max peers

* Peer discovery for Natd peers
2024-03-07 12:32:30 +00:00
Pawan Dhananjay
84a902a589 Reduce load on validator subscription channels (#5311)
* Fix tests

* Merge branch 'unstable' into unclog-channels

* Avoid reallocations

* Reduce subscription load on beacon node
2024-03-07 12:32:27 +00:00
antondlr
8cd2b1ca87 Update CI actions to alleviate deprecation warnings (#5321)
* Update and pin all actions to a modern release
2024-03-07 12:32:24 +00:00
Age Manning
85c3204d70 Correct the metrics for topic subscriptions (#5344)
* Handle fork boundaries

* Merge latest unstable

* Topic subscription fix
2024-03-07 12:32:21 +00:00
Age Manning
fc8f1a4ca7 Attempt to publish to at least mesh_n peers (#5357)
* Code improvements

* Fix gossipsub tests

* Merge latest unstable

* Differentiate errors and better scoring

* Attempt to publish to mesh_n peers
2024-03-07 09:48:51 +00:00
Krishang Shah
b9614571a3 Fix 5288: Doesn't POST if attestations is empty. (#5318)
* changed to is_empty() and removed WARN

* added log argument

* fix: issue 5288
2024-03-07 08:49:18 +00:00
Michael Sproul
bf118a17d4 Fix block v3 header decoding (#5366)
* Fix block v3 header decoding
2024-03-07 03:31:06 +00:00
chonghe
258eeb5f09 Delete milagro library (#5298)
* fix lib.rs and tests.rs

* update decode.rs

* auto-delete in Cargo.lock

* delete milagro in cargo.toml

* remove milagro from makefile

* remove milagro from the name

* delete milagro in comment

* delete milagro in cargo.toml

* delete in /testing/ef_tests/cargo.toml

* delete milagro in the logical OR

* delete milagro in /lighthouse/src/main.rs

* delete milagro in /crypto/bls/tests/tests.rs

* delete milagro in comment

* delete milagro in /testing//ef_test/src//cases/bls_eth_aggregate_pubkeys.rs

* delete milagro

* delete more in lib.rs

* delete more in lib.rs

* delete more in lib.rs

* delete milagro in /crypto/bls/src/lib.rs

* delete milagro in crypto/bls/src/mod.rs

* delete milagro.rs
2024-03-06 23:17:42 +00:00
Jimmy Chen
6aebb49718 Update dependency whoami (#5351)
* Update dependency
2024-03-06 17:29:05 +00:00
Michael Sproul
cff6258bb1 Fix duties override bug in VC (#5305)
* Fix duties override bug in VC

* Use initial request efficiently

* Prevent expired subscriptions by construction

* Clean up selection proof logic

* Add test
2024-03-04 23:15:05 +00:00
Age Manning
f919f82e4f Improve logging around peer scoring (#5325)
* Improve logging around score states

* Improve logs also
2024-03-04 19:39:23 +00:00
zhiqiangxu
ee6f667702 bump ethereum_serde_utils (#5341)
* bump ethereum_serde_utils

* cargo update
2024-03-02 19:14:03 +00:00
Lion - dapplion
7b65d385b3 Drop address_change_broadcast (#5287)
* Drop address_change_broadcast
2024-02-29 01:51:11 +00:00
Michael Sproul
88b37a10df Optimise no-op PATCH ops in VC HTTP API (#5064)
* Optimise no-op changes in VC API

* Handle another no-op case

* Merge remote-tracking branch 'origin/unstable' into opt-vc-patch-api
2024-02-29 01:51:07 +00:00
Age Manning
64e563f5e9 Recognize the Caplin consensus client (#5304)
* Caplin has joined the party

* Fix typo
2024-02-28 05:38:24 +00:00
João Oliveira
a89ff100af improve libp2p connected peer metrics (#5314)
* patch rust-yamux dep

* improve libp2p connected peer metrics
2024-02-28 03:52:55 +00:00
João Oliveira
65c4ff0775 remove exit-future (#5183)
* remove exit-future usage,

as it is non maintained, and replace with async-channel which is already in the repo.

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into remove-exit-future

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into remove-exit-future
2024-02-27 22:12:44 +00:00
João Oliveira
abd99652b4 remove nat module and use libp2p upnp (#4840)
* remove nat module and use libp2p upnp

* update Cargo.lock

* remove no longer used dependencies

* restore nat module refactored

* log successful mapping

* only activate upnp if config enabled

reduce logs to debug!

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into libp2p-nat

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into libp2p-nat

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into libp2p-nat

* address review

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into libp2p-nat

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into libp2p-nat

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into libp2p-nat

* address review
2024-02-27 07:29:18 +00:00
Age Manning
d36241b4a1 Track multiaddr in connection status (#5308)
* Record the multiaddr for connected peers
2024-02-27 06:32:48 +00:00
Michael Sproul
744c598b1c Fix typos and make block hash calculation public (#5275)
* Fix typo in `verify_payload_block_hash`

* Make calculate_execution_block_hash public again
2024-02-26 07:45:17 +00:00
Pawan Dhananjay
3ab9d3a84e Add a cli option for the snapshot cache size (#5270)
* Add a cli option for snapshot cache size

* Remove junk

* Make snapshot_cache module public

* lint

* Update docs
2024-02-26 05:19:39 +00:00
Michael Sproul
de6ede163c Delete ancient, unused HTTP docs (#5281)
* Delete ancient, unused HTTP docs
2024-02-26 05:19:35 +00:00
Jimmy Chen
f08e8f5633 Run apt update before install. (#5295)
* Run `apt update` before install.
2024-02-26 03:10:59 +00:00
chonghe
13956a0741 Add build instructions for Fedora/RHEL/CentOS (#5225)
* Add dependencies
2024-02-23 15:31:55 +00:00
Paul Hauner
b5bae6e7a2 Release v5.0.0 (#5254)
* Bump versions
2024-02-20 22:12:52 +00:00
Jimmy Chen
50c423ad88 Revert libp2p metrics (#4870) (#5265)
* Revert "improve libp2p connected peer metrics (#4870)"

This reverts commit 0c3fef59b3.
2024-02-20 04:19:17 +00:00
João Oliveira
a229b52723 Deactivate RPC Connection Handler after goodbye message is sent (#5250)
* Deactivate RPC Connection Handler

after goodbye message is sent

* nit: use to_string instead of format

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into rpc-shutdown-improvement

* clippy

* Fix cargo.lock

* Merge latest unstable
2024-02-19 07:16:01 +00:00
Eitan Seri-Levi
4d625951b8 Deprecate env_log flag in tracing layer (#5228)
* deprecate terminal logs file in tracing layer

* sink writer
2024-02-19 05:17:58 +00:00
Michael Sproul
c9702cb0a1 Download checkpoint blobs during checkpoint sync (#5252)
* MVP implementation (untested)

* update store checkpoint sync test

* update cli help

* Merge pull request #5253 from realbigsean/checkpoint-blobs-sean

Checkpoint blobs sean

* Warn only if blobs are missing from server

* Merge remote-tracking branch 'origin/unstable' into checkpoint-blobs

* Verify checkpoint blobs

* Move blob verification earlier
2024-02-19 02:22:23 +00:00
Paul Hauner
e22c9eed8f Add Deneb fork epoch for Gnosis (#5242)
* Add Deneb fork epoch for Gnosis

* Add deneb constants

* Update common/eth2_network_config/built_in_network_configs/gnosis/config.yaml

Co-authored-by: realbigsean <seananderson33@GMAIL.com>

* Adjust `min_epochs_for_block_requests`

* Change `MAX_PER_EPOCH_ACTIVATION_CHURN_LIMIT`

* Fix chain spec `max_per_epoch_activation_churn_limit`

* Fix chain spec values again
2024-02-19 02:22:19 +00:00
realbigsean
f21472991d check the da cache and the attester cache in responding to RPC requests (#5138)
* check the da cache and the attester cache in responding to RPC requests

* use the processing cache instead

* update comment

* add da cache metrics

* rename early attester cache method

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into check-da-cache-in-rpc-response

* make rustup update run on the runners

* Revert "make rustup update run on the runners"

This reverts commit d097e9bfa8.
2024-02-19 02:22:15 +00:00
ethDreamer
a264afd19f Verify Versioned Hashes During Optimistic Sync (#4832)
* Convert NewPayloadRequest to use Reference

* Refactor for Clarity

* Verify Versioned Hashes

* Added Tests for Version Hash Verification

* Added Moar Tests

* Fix Problems Caused By  Merge

* Update to use Alloy Instead of Reth Crates (#14)

* Update beacon_node/execution_layer/src/engine_api/new_payload_request.rs

Co-authored-by: realbigsean <seananderson33@GMAIL.com>

* Faster Versioned Hash Extraction

* Update to rust 1.75 & Pin alloy-consensus
2024-02-18 12:40:45 +00:00
realbigsean
1711b80779 enable doppelganger tests for deneb (#5137)
* enable doppelganger tests for deneb

* comment out lcli install skip

* Add sanity check

* Merge remote-tracking branch 'origin/unstable' into deneb-doppelganger
2024-02-15 12:25:02 +00:00
Michael Sproul
f17fb291b7 Handle unknown head during attestation publishing (#5010)
* Handle unknown head during attestation publishing

* Merge remote-tracking branch 'origin/unstable' into queue-http-attestations

* Simplify task spawner

* Improve logging

* Add a test

* Improve error logging

* Merge remote-tracking branch 'origin/unstable' into queue-http-attestations

* Fix beta compiler warnings
2024-02-15 12:24:47 +00:00
Age Manning
49536ff103 Add distributed flag to VC to enable support for DVT (#4867)
* Initial flag building

* Update validator_client/src/cli.rs

Co-authored-by: Abhishek Kumar <43061995+xenowits@users.noreply.github.com>

* Merge latest unstable

* Per slot aggregates

* One slot lookahead for sync committee aggregates

* Update validator_client/src/duties_service.rs

Co-authored-by: Abhishek Kumar <43061995+xenowits@users.noreply.github.com>

* Rename selection_look_ahead

* Merge branch 'unstable' into vc-distributed

* Merge remote-tracking branch 'origin/unstable' into vc-distributed

* Update CLI text
2024-02-15 12:23:58 +00:00
Michael Sproul
0e819fa785 Schedule Deneb on mainnet (#5233)
* Schedule Deneb on mainnet

* Fix trusted setup roundtrip test

* Fix BN CLI tests for insecure genesis sync
2024-02-15 12:23:51 +00:00
Michael Sproul
7c23625193 Quieten gossip republish logs (#5235)
* Quieten gossip republish logs
2024-02-15 04:18:23 +00:00
João Oliveira
256d9042d3 Drop gossipsub stale messages when polling ConnectionHandler. (#5175)
* drop gossipsub stale messages

* convert async-channel::Receiver to Peekable,

to be able to Peek next message without dropping it
2024-02-15 00:41:52 +00:00
Eitan Seri-Levi
e7ef2a3a54 validator liveness endpoint should accept string encoded indices (#5184)
* deserialize string indices as u64

* client should send quoted indices
2024-02-09 04:59:39 +00:00
realbigsean
4172d9f75c Update to consensus spec v1.4.0-beta.6 (#5094)
* get latest ef tests passing

* fix tests

* Fix invalid payload recovery tests

* Merge branch 'unstable' into update-to-spec-v1.4.0-beta.6

* Revert "fix tests"

This reverts commit 0c875b02e0.

* Fix fork choice def. tests

* Update beacon_node/beacon_chain/tests/payload_invalidation.rs
2024-02-08 18:08:21 +00:00
Akihito Nakano
e353358484 Update to warp v0.3.6 (#5172)
* Update to warp 0.3.6

* Update Cargo.lock
2024-02-08 16:57:26 +00:00
Jimmy Chen
4b19eac8ce Remove curve25519-dalek patch (#5214)
* Remove patch dependencies
2024-02-08 16:57:19 +00:00
Michael Sproul
6f442f2bb8 Improve database compaction and prune-states (#5142)
* Fix no-op state prune check

* Compact freezer DB after pruning

* Refine DB compaction

* Add blobs-db options to inspect/compact

* Better key size

* Fix compaction end key
2024-02-08 10:05:08 +00:00
zilayo
e470596715 chore(docs): amend port guidance to enable QUIC support (#5029)
* chore(docs): amend port guidance to enable QUIC support
2024-02-08 02:40:58 +00:00
João Oliveira
0c3fef59b3 improve libp2p connected peer metrics (#4870)
* improve libp2p connected peer metrics

* separate discv5 port from libp2p for NAT open

* use metric family for DISCOVERY_BYTES

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into improve-metrics
2024-02-08 02:40:54 +00:00
Pawan Dhananjay
0b59d10ab6 Fix backfill stalling (#5192)
* Prevent early short circuit in `peer_disconnected`

* lint
2024-02-08 02:40:51 +00:00
Age Manning
4db84de563 Improve network parameters (#5177)
* Modify network parameters for current mainnet conditions
2024-02-08 02:40:47 +00:00
João Oliveira
675a231b45 increment peer per topic count on graft messages (#5212)
* increment peer per topic count on graft messages
2024-02-08 02:00:49 +00:00
Age Manning
853042746b Downgrade gossipsub duplicate logs (#5163)
* Downgrade duplicate publish logs

* Maintain backwards compatiblity, deprecate flag

* The tests had to go, because there's no config to test against

* Update help_bn.md
2024-02-06 07:24:01 +00:00
Jimmy Chen
5cc29e47c5 Fix failing cargo-udeps (#5203)
* Attempt to fix nightly build.

* Update Cargo.toml to pin a commit

* Update Cargo.lock
2024-02-06 07:23:57 +00:00
Jimmy Chen
4b62a024d7 Update external LLVM version in preparation for Rust 1.76. (#5179)
* Update external LLVM version in preparation for Rust 1.76.
2024-02-06 07:23:53 +00:00
Age Manning
ebe77bbad2 Small typo in testing config (#5178)
* Small typo
2024-02-06 07:23:49 +00:00
Michael Sproul
7bec3f9b59 Optional slashing protection for remote keys (#4981)
* Optional slashing protection for remote keys

* Merge remote-tracking branch 'origin/unstable' into disable-slashing-protection-web3signer

* Start writing tests

* Merge remote-tracking branch 'origin/unstable' into disable-slashing-protection-web3signer

* Merge remote-tracking branch 'michael/disable-slashing-protection-web3signer' into disable-slashing-protection-web3signer

* Make half-written tests compile

* Make tests work

* Update help text

* Update book CLI text

* Merge remote-tracking branch 'origin/unstable' into disable-slashing-protection-web3signer

* More logging & CLI tests

* CLI tweaks
2024-02-06 01:30:31 +00:00
Jimmy Chen
795c5778e1 Remove unused js file in Lighthouse book (#5164)
* Remove unused js file in Lighthouse book.
2024-02-06 01:30:26 +00:00
Jimmy Chen
8fa11aa792 Fix incorrect value set for blobs_by_root_request rpc limit. (#5181)
* Fix incorrect value set for `blobs_by_root_request` rpc limit.
2024-02-05 18:37:28 +00:00
Jimmy Chen
39e9f7dc6b Fix Rust beta compiler errors (1.77) (#5180)
* Lint fixes

* More fixes for beta compiler.

* Format fixes

* Move `#[allow(dead_code)]` to field level.

* Remove old comment.

* Update beacon_node/execution_layer/src/test_utils/mod.rs

Co-authored-by: João Oliveira <hello@jxs.pt>

* remove duplicate line
2024-02-05 17:54:11 +00:00
Michael Sproul
8fb6989801 Config for web3signer keep-alive (#5007)
* Allow tweaking connection pool settings

* Build docker image

* Fix imports

* Merge tag 'v4.6.0' into web3signer-keep-alive

v4.6.0

* Delete temp docker build stuff

* Fix tests

* Merge remote-tracking branch 'origin/unstable' into web3signer-keep-alive

* Update CLI text
2024-02-01 08:35:14 +00:00
Michael Sproul
0b6416c444 Re-disable block verification tests in debug (#5155)
* Re-disable block verification tests in debug
2024-02-01 08:35:10 +00:00
4rgon4ut
1d4ee5d150 Update gnosis bootnodes (#4986)
* chore(eth2_network_config): update gnosis bootnodes

* Merge remote-tracking branch 'origin/unstable' into update_gnosis_bootnodes
2024-02-01 08:35:06 +00:00
4rgon4ut
b7ba5a087c feat(config): add chiado bootnodes ENRs (#4727)
* feat(config): add chiado bootnodes erns

* Merge remote-tracking branch 'origin/unstable' into feat/add-chiado-bootnodes
2024-02-01 08:35:00 +00:00
João Oliveira
dada5750ee Properly log panics with slog (#5075)
* log panics with slog

* update set_hook location

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into slog-panics
2024-01-31 19:20:09 +00:00
chonghe
ab6a6e0741 Some updates in Lighthouse Book (#5000)
* Add jq in api-bn

* Update beaconstate size

* Add fields to web3signer API

* Link web3signer API

* Update /lighthouse/logs in table

* plural

* update slasher doc

* update FAQ

* Add link in validator section

* Add more info on state pruning

* Update database size

* Merge branch 'unstable' into book-update

* Revise Siren for vc to connect bn

* Merge branch 'book-update' of https://github.com/chong-he/lighthouse into book-update

* Corrections to siren faq

* Fix typos

* Update release date for 4.6.0

* Merge branch 'unstable' into book-update
2024-01-31 18:11:54 +00:00
Age Manning
f02189c86a Prevent QUIC logs when quic is disabled (#5071)
* Prevent logs and dialing quic multiaddrs when not supported

* Merge latest unstable
2024-01-31 18:11:49 +00:00
Age Manning
7582da7855 Test backfill (#5109)
* Test backfill

* Revert cargo.toml

* Update beacon_node/beacon_chain/src/builder.rs

Co-authored-by: João Oliveira <hello@jxs.pt>

* Remove redundant code
2024-01-31 18:11:45 +00:00
chonghe
a257a12110 Add a note about validator monitoring in Lighthouse book (#5143)
* Add validator monitor
2024-01-31 17:32:38 +00:00
Age Manning
4273004bd9 Add gossipsub as a Lighthouse behaviour (#5066)
* Move gossipsub as a lighthouse behaviour

* Update dependencies, pin to corrected libp2p version

* Merge latest unstable

* Fix test

* Remove unused dep

* Fix cargo.lock

* Re-order behaviour, pin upstream libp2p

* Pin discv5 to latest version
2024-01-31 17:32:31 +00:00
Akihito Nakano
b9c519d565 Move from igd to igd-next (#5068)
* Move from igd to igd-next

* Fix clippy error

warning: useless conversion to the same type: `std::net::IpAddr`
2024-01-31 15:48:01 +00:00
Divma
8353ec9785 Update deterministic subnets upgrade to allow prefix computation (#4959)
* add new spec field to spec

* add spec changes

* fix bug and update expected subnets with validation from pyspec

* fix values and make test prettier (?)

* fix style

---------

Co-authored-by: Lion - dapplion <35266934+dapplion@users.noreply.github.com>
2024-01-31 09:30:26 +00:00
Michael Sproul
d2aef1b35c Fix bug in --builder-proposals (#5151)
* Fix bug in `--builder-proposals`

* Add tests

* More sensible test order

* Fix duplicate builder-boost test case

* Cargo fmt and rename
2024-01-31 05:25:55 +00:00
Lion - dapplion
b035638f9b Compute recent lightclient updates (#4969)
* Compute recent lightclient updates

* Review PR

* Merge remote-tracking branch 'upstream/unstable' into lc-prod-recent-updates

* Review PR

* consistent naming

* add metrics

* revert dropping reprocessing queue

* Update light client optimistic update re-processing logic. (#7)

* Add light client server simulator tests. Co-authored by @dapplion.

* Merge branch 'unstable' into fork/dapplion/lc-prod-recent-updates

* Fix lint

* Enable light client server in simulator test.

* Fix test for light client optimistic updates and finality updates.
2024-01-31 05:25:51 +00:00
Eitan Seri-Levi
1d87edb03d Assume Content-Type is json for endpoints that require json (#4575)
* added default content type filter

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into unstable

* create custom warp json filter that ignores content type header

* cargo fmt and linting

* updated test

* updated test

* merge unstable

* merge conflicts

* workspace=true

* use Bytes instead of Buf

* resolve merge conflict

* resolve merge conflicts

* add extra error message context

* merge conflicts

* lint
2024-01-31 01:11:54 +00:00
Michael Sproul
b8db3e4f08 Delete temporary mitigation for equivocation via RPC (#5037)
* Delete unnecessary RPC equivocation check
2024-01-31 01:11:49 +00:00
Sylvain Bossut
0f345c7e0a Make lcli docker image portable (#5069)
* Set lcli docker build to use portable feature (#4370)

* Merge remote-tracking branch 'origin/unstable' into unstable
2024-01-30 03:03:48 +00:00
chonghe
020702f8eb Update to the docs (#5106)
* Perform transaction

* Remove lighthouse/beacon/states/{state_id}/ssz

* Update database api example

* Update checkpoint sync

* minor update

* single beacon node

* add info in mev

* Merge remote-tracking branch 'origin/unstable' into testnet

* add builder_boost_factor example

* change to 50 for consistency
2024-01-30 03:03:37 +00:00
Peter Straus
47f05ac60d fix: update outdated links to external resources (#5018)
* fix: update outdated links to external resources

* Update style guide link
2024-01-30 00:33:10 +00:00
Jack McPherson
abeb358f0b Remove custom SSZ beacon states route (#5065)
* Remove SSZ state root route

* Remove SSZ states route from client impl

* Patch tests

* Merge branch 'unstable' into 5063-delete-ssz-state-route

* Further remove dead code
2024-01-30 00:33:05 +00:00
Michael Sproul
6f3af67362 Fix off-by-one in backfill sig verification (#5120)
* Fix off-by-one in backfill sig verification

* Add self-referential PR link
2024-01-30 00:33:01 +00:00
realbigsean
a4fcf60bcc Increase attestation cache sizes (#5135)
* increase observed attesters and aggregates cache sizes

* fix comment
2024-01-30 00:32:57 +00:00
Sergey Kisel
64efdaf39a #5102 Fix load_state_for_block_production metric mapping (#5103)
* #5102 Fix load_state_for_block_production metric mapping
2024-01-30 00:32:52 +00:00
Jimmy Chen
c7e5dd1098 Update Mergify commit message template (#5126) 2024-01-27 00:43:44 +00:00
Mac L
0b6c898213 Replace backticks with single quotes (#5121) 2024-01-25 03:57:48 +00:00
Paul Hauner
1be5253610 Bump versions (#5123) 2024-01-25 10:02:00 +11:00
Eitan Seri-Levi
f9e36c94ed Expose additional builder booster related flags in the vc (#5086)
* expose builder booster flags in vc, enable options in validator endpoints, update tests

* resolve failing test

* fix issues related to CreateConfig and MoveConfig

* remove unneeded val, change how boost factor flag logic in the vc, add some additional documentation

* fix typos

* fix typos

* assume builder-proosals flag if one of other two vc builder flags are present

* fmt

* typo

* typo

* Fix CLI help text

* Prioritise per validator builder boost configurations over CLI flags.

* Add http test for builder boost factor with process defaults.

* Fix issue with PATCH request

* Add prefer builder proposals

* Add more builder boost factor tests.

---------

Co-authored-by: Mac L <mjladson@pm.me>
Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>
Co-authored-by: Paul Hauner <paul@paulhauner.com>
2024-01-25 09:09:47 +11:00
Eitan Seri-Levi
612eaf2d41 Prevent rolling file appender panic (#5117)
* rolling file appender panic removal and max log file count

* max log file
2024-01-24 10:12:48 +11:00
Age Manning
a36a12a8d2 Correct multiple dial bug (#5113)
* Dialing the same peer-id error fix

* Improve dialing logging

* Update beacon_node/lighthouse_network/src/peer_manager/mod.rs

Co-authored-by: Lion - dapplion <35266934+dapplion@users.noreply.github.com>

---------

Co-authored-by: Lion - dapplion <35266934+dapplion@users.noreply.github.com>
2024-01-24 10:10:05 +11:00
Pawan Dhananjay
b55b58b3c6 Fix indices filter in blobs_sidecar http endpoint (#5118)
* add get blobs unit test

* Use a multi_key_query for blob_sidecar indices

* Fix test

* Remove env_logger

---------

Co-authored-by: realbigsean <seananderson33@GMAIL.com>
2024-01-24 09:35:24 +11:00
realbigsean
1cebf41452 Backfill blob storage fix (#5119)
* store blobs in the correct db in backfill

* add database migration

* add migration file

* remove log info suggesting deneb isn't schedule

* add batching in blob migration
2024-01-24 09:35:02 +11:00
Age Manning
5851027bfd Correct discovery logic (#5111) 2024-01-23 16:58:24 +11:00
Age Manning
b7c9b27872 Gossipsub fanout correction (#5110)
* Correct fanout in gossipsub

* Upgrade discv5 to pin new libp2p version

* Update cargo.lock
2024-01-23 16:38:00 +11:00
Michael Sproul
a403138ed0 Reduce size of futures in HTTP API to prevent stack overflows (#5104)
* Box::pin a few big futures

* Arc the blocks early in publication

* Fix more tests
2024-01-23 15:32:07 +11:00
ethDreamer
02d1f36090 Small Readability Improvement in Networking Code (#5098) 2024-01-23 12:08:57 +08:00
ethDreamer
712f5aba73 Upgrade shlex to 1.3.0 (#5108) 2024-01-23 14:16:36 +11:00
ethDreamer
9a630e4dbb Stop Penalizing Peers in Parent SingleBlobLookup (#5096)
* Stop Penalizing Peers in Parent SingleBlobLookup

* Add test for parent lookup bug (#13)

---------

Co-authored-by: realbigsean <seananderson33@GMAIL.com>
2024-01-22 05:35:06 +01:00
realbigsean
70b0528f36 set deneb fork on testnets (#5089)
* set deneb fork on testnets

* re-order fields in holesky
2024-01-22 15:14:26 +11:00
Lion - dapplion
585124fb2f Hold HeadTracker lock until persisting to disk (#5084)
* Fix head tracker drop order on un-ordered shutdown

* lint

---------

Co-authored-by: Michael Sproul <michael@sigmaprime.io>
2024-01-22 15:14:11 +11:00
Michael Sproul
185646acb2 Fix PublishBlockRequest SSZ decoding (#5078)
* Fix PublishBlockRequest SSZ decoding

* Send header for block v1 ssz client

* Delete unused function
2024-01-19 12:56:30 +11:00
realbigsean
2a161cef73 Bump h2 (#5085)
* bump h2

* Empty commit for CI

* bump the other h2 version
2024-01-19 12:41:28 +11:00
Mac L
47b28c4935 Remove blobs_db when purge-db (#5081) 2024-01-18 15:59:08 -05:00
Mac L
a0b407c15d Add Deneb readiness logging (#5074) 2024-01-18 15:21:38 -05:00
Eitan Seri-Levi
f22e5b0f85 Critical dependency logging (#4988)
* add metrics layer

* add metrics

* simplify getting the target

* make clippy happy

* fix typos

* unify deps under workspace

* make import statement shorter, fix typos

* enable warn by default, mark flag as deprecated

* do not exit on error when initializing logging fails

* revert exit on error

* adjust bootnode logging

* add logging layer

* non blocking file writer

* non blocking file writer

* add tracing visitor

* use target as is by default

* make libp2p events register correctly

* adjust repilcated cli help

* refactor tracing layer

* linting

* filesize

* log gossipsub, dont filter by log level

* turn on debug logs by default, remove deprecation warning

* truncate file, add timestamp, add unit test

* suppress output (#5)

* use tracing appender

* cleanup

* Add a task to remove old log files and upgrade to warn level

* Add the time feature for tokio

* Udeps and fmt

---------

Co-authored-by: Diva M <divma@protonmail.com>
Co-authored-by: Divma <26765164+divagant-martian@users.noreply.github.com>
Co-authored-by: Age Manning <Age@AgeManning.com>
2024-01-16 19:44:26 +11:00
Divma
a68b701807 add tracing metrics layer for dependency logging (#4979)
* add metrics layer

* add metrics

* simplify getting the target

* make clippy happy

* fix typos

* unify deps under workspace

* make import statement shorter, fix typos

* enable warn by default, mark flag as deprecated

* do not exit on error when initializing logging fails

* revert exit on error

* adjust bootnode logging

* use target as is by default

* make libp2p events register correctly

* adjust repilcated cli help

* turn on debug logs by default, remove deprecation warning

* suppress output (#5)

---------

Co-authored-by: Age Manning <Age@AgeManning.com>
2024-01-16 12:22:09 +11:00
Akihito Nakano
e10e4b7811 Fix zero port error (#5021)
* Fix zero port error and add tests

* Tweak indentation

* assign 0 to quic_port if tcp_port == 0

* Remove unnecessary deps
2024-01-16 12:19:49 +11:00
vuittont60
0613eb7a21 docs: fix typos (#5059)
* docs: fix typos

* docs: fix typo

* docs: update by 'make cli'
2024-01-16 12:18:54 +11:00
Enrico Del Fante
31b797ed83 Update teku's bootnodes (#5052) 2024-01-16 12:18:06 +11:00
Eitan Seri-Levi
72bcf47dd0 add content-type octet stream helper fn (#5062) 2024-01-15 16:23:39 +11:00
Michael Sproul
00af017582 Fix asset paths in release CI upload job (#5056) 2024-01-15 11:19:47 +11:00
Paul Hauner
2e8e160679 v4.6.0-rc.0 (#5042) 2024-01-11 10:46:29 +11:00
João Oliveira
38df87c3c5 Switch libp2p sigp gossipsub fork (#4999)
* switch libp2p source to sigp fork

* Shift the connection closing inside RPC behaviour

* Tag specific commits

* Add slow peer scoring

* Fix test

* Use default yamux config

* Pin discv5 to our libp2p fork and cargo update

* Upgrade libp2p to enable yamux gains

* Add a comment specifying the branch being used

* cleanup build output from within container
(prevents CI warnings related to fs permissions)

* Remove revision tags add branches for testing, will revert back once we're happy

* Update to latest rust-libp2p version

* Pin forks

* Update cargo.lock

* Re-pin to panic-free rust

---------

Co-authored-by: Age Manning <Age@AgeManning.com>
Co-authored-by: Pawan Dhananjay <pawandhananjay@gmail.com>
Co-authored-by: antondlr <anton@delaruelle.net>
Co-authored-by: Michael Sproul <michael@sigmaprime.io>
2024-01-10 16:26:52 +11:00
Paul Hauner
be79f74c6d Fix IncorrectAttestationSource error in attn. simulator (#5048)
* Don't produce attestations when syncing

* Handle `IncorrectAttestationSource`
2024-01-10 10:47:00 +11:00
Michael Sproul
7e948eec9d Fix block v3 reward encodings (#5049)
* Fix block v3 reward encodings

* Use crates.io version
2024-01-10 10:44:07 +11:00
zhiqiangxu
4a65d28b3b remove needless Into (#5026) 2024-01-10 08:38:16 +11:00
Paul Hauner
12d3d237cd Disallow genesis sync outside blob pruning window (#5038)
* Disallow Syncing From Genesis By Default

* Fix CLI Tests

* Perform checks in the `ClientBuilder`

* Tidy, fix tests

* Return an error based on the Deneb fork

* Fix typos

* Fix failing test

* Add missing CLI flag

* Fix CLI flags

* Add suggestion from Sean

* Fix conflict with blob sidecars epochs

---------

Co-authored-by: Mark Mackey <mark@sigmaprime.io>
2024-01-09 14:17:01 -05:00
realbigsean
b47e3f252e Runtime rpc request sizes (#4841)
* add runtime variable list type

* add configs to ChainSpec

* git rid of max request blocks type

* fix tests and lints

* remove todos

* git rid of old const usage

* fix decode impl

* add new config to `Config` api struct

* add docs fix compilt

* move methods for per-fork-spec to chainspec

* get values off chain spec

* fix compile

* remove min by root size

* add tests for runtime var list

---------

Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>
2024-01-09 10:23:47 +11:00
Eitan Seri-Levi
5c8c8da8b1 Use blocks v3 endpoint in the VC (#4813)
* block v3 endpoint init

* block v3 flow

* block v3 flow

* continue refactor

* the full flow...

* add api logic

* add api logic

* add new endpoint version

* added v3 endpoint

* some debugging

* merge v2 flow with v3

* debugging

* tests passing

* tests passing

* revert cargo lock

* initial v3 test

* blinded payload test case passing

* fix clippy issues

* cleanup

* cleanup

* remove dead code

* fixed logs

* add block value

* block value fix

* linting

* merge unstable

* refactor

* add consensus block value

* lint

* update header name to consensus block value

* prevent setting the participation flag

* clone get_epoch_participation result

* fmt

* clone epoch participation outside of the loop

* add block v3 to vc

* add v3 logic into vc

* add produce-block-v3

* refactor based on feedback

* update

* remove comments

* refactor

* header bugfix

* fmt

* resolve merge conflicts

* fix merge

* fix merge

* refactor

* refactor

* cleanup

* lint

* changes based on feedback

* revert

* remove block v3 fallback to v2

* publish_block_v3 should return irrecoveerable errors

* comments

* comments

* fixed issues from merge

* merge conflicts

* Don't activate at fork; support builder_proposals

* Update CLI flags & book

* Remove duplicate `current_slot` parameter in `publish_block` function, and remove unnecessary clone.

* Revert changes on making block errors irrecoverable.

---------

Co-authored-by: Michael Sproul <michael@sigmaprime.io>
Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>
2024-01-08 16:12:39 -05:00
realbigsean
f70c32ec70 create unified slashing cache (#5033)
* create unified slashing cache

* add observed slashable file

* fix broadcast validation tests

* revert block seen cache changes

* clean up slashable cache test

* check header signatures for RPC blobs

* don't throw error on RPC signature invalie
2024-01-08 10:30:57 -05:00
Paul Hauner
f62cfc6475 Add hint for solving CLI tests failure (#5041)
* Add hint for solving CLI test failure.

* Add `make cli-local`
2024-01-08 10:30:37 -05:00
realbigsean
db05d37675 update goerli config (#5036) 2024-01-08 11:24:12 +11:00
Eitan Seri-Levi
9c1505d082 Block v3 builder boost factor (#5035)
* builder boost factor

* default boost factor

* revert

* deprecate always_prefer_builder_payload, builder-profit-threshold, ignore_builder_override_suggestion_threshold and builder_comparison_factor flags

* revert

* set deprecated flags to no op, revert should_override_builder

* fix test, calc boosted relay value correctly, dont calculate if none

* Add deprecation warnings and restore CLI docs
2024-01-08 11:10:32 +11:00
Jimmy Chen
0c97762032 Track shared memory in metrics (#5023)
* Track shared memory so we can exclude them form resident memory.

* Bump psutil version to 3.3.0 to get shared memory metrics.
2024-01-05 14:48:11 -05:00
Akihito Nakano
01994c438e Bump zerocopy (#5024) 2023-12-22 09:57:39 -05:00
Michael Sproul
af11e78ae1 Clean up blockv3 metadata and client (#5015)
* Improve block production v3 client

* Delete wayward line

* Overhaul JSON endpoint as well

* Rename timeout param

* Update tests

* I broke everything

* Ah this is an insane fix

* Remove unnecessary optionals

* Doc fix
2023-12-22 09:39:17 -05:00
Age Manning
a7e5926a1f Information for network testing (#4961)
* Documentation for network testing

* Update doc
2023-12-22 09:02:58 +11:00
realbigsean
c55608be10 suppress error on duplicate blobs (#4995) 2023-12-18 12:15:12 -05:00
Jimmy Chen
dfc3b3714a Fix incorrect blob queue metrics (#5014)
* Fix blob queue metrics.

* Update blob metric description
2023-12-18 12:04:57 -05:00
Jimmy Chen
b0c374c1ca Update dependencies to get rid of the yanked deps (#4994)
* Initial attempt to upgrade hashbrown to latest to get rid of the crate warnings.

* Replace `.expect()` usage with use of `const`.

* Update ahash 0.7 as well

* Remove unsafe code.

* Update `lru` to 0.12 and fix release test errors.

* Set non-blocking socket

* Bump testcontainers to 0.15.

* Fix lint

---------

Co-authored-by: Michael Sproul <michael@sigmaprime.io>
2023-12-15 18:31:59 +11:00
Michael Sproul
f1113540d8 Fix off-by-one bug in missed block detection (#5012)
* Regression test

* Fix the bug
2023-12-15 14:23:54 +11:00
Jimmy Chen
366f0d7ac2 Add flag to disable warning logs for duplicate gossip messages (#5009)
* Add flag to disable warning logs for duplicate gossip messages.

* Update Lighthouse book.
2023-12-15 09:26:51 +11:00
Pawan Dhananjay
ae4a296089 Convert a FullPayload to a BlindedPayload correctly (#5005) 2023-12-14 08:44:14 -05:00
Joel Rousseau
189430a45c Add attestation simulator (#4880)
* basic scaffold

* remove unnecessary ?

* check if committee cache is init

* typed ValidatorMonitor with ethspecs + store attestations within

* nits

* process unaggregated attestation

* typo

* extract in func

* add tests

* better naming

* better naming 2

* less verbose

* use same naming as validator monitor

* use attestation_simulator

* add metrics

* remove cache

* refacto flag_indices process

* add lag

* remove copying state

* clean and lint

* extract metrics

* nits

* compare prom metrics in tests

* implement lag

* nits

* nits

* add attestation simulator service

* fmt

* return beacon_chain as arc

* nit: debug

* sed s/unaggregated/unagg.//

* fmt

* fmt

* nit: remove unused comments

* increase max unaggregated attestation hashmap to 64

* nit: sed s/clone/copied//

* improve perf: remove unecessary hashmap copy

* fix flag indices comp

* start service in client builder

* remove //

* cargo fmt

* lint

* cloned keys

* fmt

* use Slot value instead of pointer

* Update beacon_node/beacon_chain/src/attestation_simulator.rs

Co-authored-by: Paul Hauner <paul@paulhauner.com>

---------

Co-authored-by: Paul Hauner <paul@paulhauner.com>
2023-12-14 11:44:56 +11:00
Pawan Dhananjay
a3a370302a Use the block header to compute the canonical_root (#5003) 2023-12-14 09:24:36 +11:00
Gua00va
a3fb27c99b add forK_choice_read_lock as parameter (#4978) 2023-12-13 16:06:00 +11:00
Jimmy Chen
7f64738085 Update Rust version in lcli Dockerfile. (#5002) 2023-12-12 10:20:49 -05:00
Jimmy Chen
153aaa1679 Remove bors.toml 🫡 (#5001) 2023-12-12 16:54:22 +11:00
Jimmy Chen
4cf4819497 Add mergify merge queue configuration file (#4917)
* Add mergify.yml.

* Abstract out CI jobs to a "success" job so that a change in the CI jobs don't require modification to mergify configuration on stable branch.

* Add new jobs to the `needs` list of  `test-suite-success`.

* Set `batch_max_wait_time` to 60 s.
2023-12-12 16:04:29 +11:00
Age Manning
69f1b7afec Disable flood publishing (#4383)
* Disable flood publish

* Change default configuration
2023-12-12 09:45:54 +11:00
ethDreamer
78ffa378b4 Batch Verify RPC Blobs (#4934) 2023-12-08 16:48:03 -05:00
realbigsean
46184e5ce4 Remove delayed lookups (#4992)
* initial rip out

* fix unused imports

* delete tests and fix lint

* fix peers scoring for blobs
2023-12-08 15:42:55 -05:00
Michael Sproul
b882519d2f Implement POST validators/validator_balances APIs (#4872)
* Add POST for fetching validators from state

* Implement POST for balances

* Tests
2023-12-08 12:09:36 +11:00
Michael Sproul
e02adbf7bf Update docs for v4.6.0 (#4982)
* Update DB migration docs

* Document VC broadcast modes

* Update downgrade example (#6)

* update downgrade example

* Add period

* Add v4.1.0

---------

Co-authored-by: chonghe <44791194+chong-he@users.noreply.github.com>
2023-12-08 10:45:05 +11:00
Divma
6c0c41c7ac upgrade libp2p to v0.53.* (#4935)
* update libp2p and address compiler errors

* remove bandwidth logging from transport

* use libp2p registry

* make clippy happy

* use rust 1.73

* correct rpc keep alive

* remove comments and obsolte code

* remove libp2p prefix

* make clippy happy

* use quic under facade

* remove fast msg id

* bubble up close statements

* fix wrong comment
2023-12-07 20:39:59 +11:00
Jimmy Chen
67e0569d9b Fix corrupted DB on networks where the first slot is skipped (Holesky) (#4985)
* Fix zero block roots on skip slots.

* Remove temporary comment, println code and unused imports.

* Remove `println!` in test.
2023-12-07 15:12:06 +11:00
Eitan Seri-Levi
8ba39cbf2c Implement graffiti management API (#4951)
* implement get graffiti

* add set graffiti

* add set graffiti

* delete graffiti

* set graffiti

* set graffiti

* fmt

* added tests

* add graffiti file check

* update

* fixed delete req

* remove unused code

* changes based on feedback

* changes based on feedback

* invalid auth test plus lint

* fmt

* remove unneeded async
2023-12-07 12:02:46 +11:00
chonghe
d9d84242a7 CLI in Lighthouse Book (#4571)
* Add cli.sh file

* update bash script

* update Makefile

* update

* modified test-suite

* fix path

* Fix cli.sh permissions

* update cmd

* cli_manual

* Revise to update

* Update directory in Github

* Correct cli.txt directory

* test old cli_manual

* change exit 1

* Update cli and makefile

* Move cli.sh

* remove files

* fix permission

* Indentation and revision

* Fixed permission

* Create new cli folder

* remove dummy

* put a dummy file

* Revise cli.sh

* comment

* function

* remove vm.md

* test make cli

* test

* testing

* testing

* update

* update

* test

* test

* add vm

* change back non-debug mode

* add exist and update for future debug

* revise

* remove troubleshooting part

* update

* add summary.md

* test

* test

* Update Makefile

Co-authored-by: Mac L <mjladson@pm.me>

* Update Makefile

Co-authored-by: Mac L <mjladson@pm.me>

* Remove help-cli.md and rearrange

* Remove help-cli.md

* Update scripts/cli.sh

Co-authored-by: Mac L <mjladson@pm.me>

* Update scripts/cli.sh

Co-authored-by: Mac L <mjladson@pm.me>

* remove maxperf

* move then to same line as if

* Fix indent and echo file not found

* To be explicit in replacing the old file

* Add logging when there are changes

* Add local variables

* spacing

* remove cargo fmt

* update .md files

* Edit exit message to avoid confusion

* Remove am and add vm subcommands

* Add cargo-fmt

* Revise test-suite.yml

* Update SUMMARY.md

* Add set -e

* Add vm

* Fix

* Add vm

* set -e

* Remove return 1 and add :

* Small revision

* Fix typo

* Update scripts/cli.sh

Co-authored-by: Mac L <mjladson@pm.me>

* Indent

* Update scripts/cli.sh

Co-authored-by: Mac L <mjladson@pm.me>

* Remove .exe in Windows

* Fix period with \.

* test

* check diff

* linux commit

* Add cli.sh file

* update bash script

* update Makefile

* update

* modified test-suite

* fix path

* Fix cli.sh permissions

* update cmd

* cli_manual

* Revise to update

* Update directory in Github

* Correct cli.txt directory

* test old cli_manual

* change exit 1

* Update cli and makefile

* Move cli.sh

* remove files

* fix permission

* Indentation and revision

* Fixed permission

* Create new cli folder

* remove dummy

* put a dummy file

* Revise cli.sh

* comment

* function

* remove vm.md

* test make cli

* test

* testing

* testing

* update

* update

* test

* test

* add vm

* change back non-debug mode

* add exist and update for future debug

* revise

* remove troubleshooting part

* update

* add summary.md

* test

* test

* Update Makefile

Co-authored-by: Mac L <mjladson@pm.me>

* Update Makefile

Co-authored-by: Mac L <mjladson@pm.me>

* Remove help-cli.md and rearrange

* Remove help-cli.md

* Update scripts/cli.sh

Co-authored-by: Mac L <mjladson@pm.me>

* Update scripts/cli.sh

Co-authored-by: Mac L <mjladson@pm.me>

* remove maxperf

* move then to same line as if

* Fix indent and echo file not found

* To be explicit in replacing the old file

* Add logging when there are changes

* Add local variables

* spacing

* remove cargo fmt

* Edit exit message to avoid confusion

* update .md files

* Remove am and add vm subcommands

* Add cargo-fmt

* Revise test-suite.yml

* Update SUMMARY.md

* Add set -e

* Add vm

* Fix

* Add vm

* set -e

* Remove return 1 and add :

* Small revision

* Fix typo

* Update scripts/cli.sh

Co-authored-by: Mac L <mjladson@pm.me>

* Indent

* Update scripts/cli.sh

Co-authored-by: Mac L <mjladson@pm.me>

* Remove .exe in Windows

* Fix period with \.

* test

* check diff

* Revert "Merge branch 'book-cli' of https://github.com/chong-he/lighthouse into book-cli"

This reverts commit 314005d3f8, reversing
changes made to a007f61378.

* update

* update

* Remove echo diff

* Dockerize

* Remove `-ti`

* take ownership inside container

* fix mistake

* proper escaping, restore ownership afterwards

* try without taking ownership of repo

* update

* add diff for troubleshooting

* binary

* update using linux

* binary

* make file

* remove diff

* add diff

* update progressive balance help text

* Remove diff

---------

Co-authored-by: Mac L <mjladson@pm.me>
Co-authored-by: antondlr <anton@delaruelle.net>
2023-12-07 10:39:22 +11:00
ethDreamer
52117f43ba Small Improvements (#4980)
* initial changes

* use arc<blobsidecar> in vector

* Utilize new pattern for KzgVerifiedBlob

* fmt

* Update beacon_node/beacon_chain/src/blob_verification.rs

Co-authored-by: realbigsean <seananderson33@GMAIL.com>

* forgot to save..

* lint

* fmt.. again

---------

Co-authored-by: realbigsean <seananderson33@GMAIL.com>
2023-12-06 15:51:40 -05:00
Pawan Dhananjay
31044402ee Sidecar inclusion proof (#4900)
* Refactor BlobSidecar to new type

* Fix some compile errors

* Gossip verification compiles

* Fix http api types take 1

* Fix another round of compile errors

* Beacon node crate compiles

* EF tests compile

* Remove all blob signing from VC

* fmt

* Tests compile

* Fix some tests

* Fix more http tests

* get compiling

* Fix gossip conditions and tests

* Add basic proof generation and verification

* remove unnecessary ssz decode

* add back build_sidecar

* remove default at fork for blobs

* fix beacon chain tests

* get relase tests compiling

* fix lints

* fix existing spec tests

* add new ef tests

* fix gossip duplicate rule

* lints

* add back sidecar signature check in gossip

* add finalized descendant check to blob sidecar gossip

* fix error conversion

* fix release tests

* sidecar inclusion self review cleanup

* Add proof verification and computation metrics

* Remove accidentally committed file

* Unify some block and blob errors; add slashing conditions for sidecars

* Address review comment

* Clean up re-org tests (#4957)

* Address more review comments

* Add Comments & Eliminate Unnecessary Clones

* update names

* Update beacon_node/beacon_chain/src/metrics.rs

Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>

* Update beacon_node/network/src/network_beacon_processor/tests.rs

Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>

* pr feedback

* fix test compile

* Sidecar Inclusion proof small refactor and updates (#4967)

* Update some comments, variables and small cosmetic fixes.

* Couple blobs and proofs into a tuple in `PayloadAndBlobs` for simplicity and safety.

* Update function comment.

* Update testing/ef_tests/src/cases/merkle_proof_validity.rs

Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>

* Rename the block and blob wrapper types used in the beacon API interfaces.

* make sure gossip invalid blobs are passed to the slasher (#4970)

* Add blob headers to slasher before adding to DA checker

* Replace Vec with HashSet in BlockQueue

* fmt

* Rename gindex -> index

* Simplify gossip condition

---------

Co-authored-by: realbigsean <seananderson33@gmail.com>
Co-authored-by: realbigsean <sean@sigmaprime.io>
Co-authored-by: Michael Sproul <michael@sigmaprime.io>
Co-authored-by: Mark Mackey <mark@sigmaprime.io>
Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>
2023-12-05 11:19:59 -05:00
Manu NALEPA
ec8edfb89a EIP-3076 tests - complete database (#4958)
* EIP-3076 interchange tests: Add `should_succeed_complete` boolean.

This new added `should_succeed_complete` boolean means the test should succeed using
complete anti-slashing DB, while the untouched `should_succeed` boolean is
still implicitely reserved for minimal anti-slashing DB.

Note:
This commit only adds the new `should_succeed_complete` boolean, and copy the value from
`should_succeed` without any meaning regarding the value this boolean should
take.

* `TestEIP3076SpecTests`: Modify two tests

Those two tests are modified in a way they both comply with minimal AND
complete anti-slashing DB.

* Disallow false positives and differentiate more minimal vs complete cases

* Fix my own typos

* Update to v5.3.0 tag

---------

Co-authored-by: Michael Sproul <michael@sigmaprime.io>
2023-12-05 10:49:50 +11:00
Jimmy Chen
4250385ae1 Fix stuck linkcheck on CI (#4977)
* Fix linkcheck CI job timeouts. Use linkcheck 3.0.0 without Docker.

* Add sleep to wait for the mdbook server to start serving
2023-12-05 10:32:10 +11:00
Michael Sproul
c8b2324880 Enable progressive balances fast mode by default (#4971)
* Enable progressive balances fast mode by default

* Fix default in chain_config
2023-12-04 14:42:49 +11:00
Gua00va
44aaf13ff0 Standard Liveness Endpoint (#4853)
* Changes to use required Endpoint

* Format

* fixed doppleganger service

* minor fix

* efficiency changes

* fixed tests

* remove commented line

---------

Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>
2023-11-30 17:41:22 +11:00
Michael Sproul
547ed1de63 Clone state ahead of block production (#4925)
* Clone state ahead of block production

* Add pruning and fix logging

* Don't hold 2 states in mem
2023-11-30 13:49:35 +11:00
ethDreamer
43d98153d6 Refactor & Fix Bugs in Payload Selection Logic (#4950)
* Refactor & Fix Bugs in Payload Selection Logic

* Fix lint

* Update beacon_node/execution_layer/src/lib.rs

* Finish renaming function
2023-11-29 15:20:12 +11:00
Paul Hauner
86163d94f2 Add lcli mock-el (#4587)
* Track time to early attester cache

* Add debug log for early attester cache

* Add mock-el command to lcli

* Revert beacon chain changes

* Tidy, fix compilation errors

* Add required flag

* Default to SYNCING response

* Hide flag
2023-11-29 10:04:29 +11:00
GeemoCandama
8a599ec7dc API for LightClientBootstrap, LightClientFinalityUpdate, LightClientOptimisticUpdate and light client events (#3954)
* rebase and add comment

* conditional test

* test

* optimistic chould be working now

* finality should be working now

* try again

* try again

* clippy fix

* add lc bootstrap beacon api

* add lc optimistic/finality update to events

* fmt

* That error isn't occuring on my computer but I think this should fix it

* Add missing test file

* Update light client types to comply with Altair light client spec.

* Fix test compilation

* Support deserializing light client structures for the Bellatrix fork

* Move `get_light_client_bootstrap` logic to `BeaconChain`. `LightClientBootstrap` API to return `ForkVersionedResponse`.

* Misc fixes.
- log cleanup
- move http_api config mutation to `config::get_config` for consistency
- fix light client API responses

* Add light client bootstrap API test and fix existing ones.

* Fix test for `light-client-server` http api config.

* Appease clippy

* Efficiency improvement when retrieving beacon state.

---------

Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>
2023-11-28 17:14:29 +11:00
Michael Sproul
44c1817c2b Remove block-delay-ms (#4956) 2023-11-28 16:00:28 +11:00
Jimmy Chen
c88cb371a0 Remove MAX_BLOBS_PER_BLOCK from Holesky config. (#4955) 2023-11-27 23:07:40 +11:00
Alexander Uvizhev
b4556a3d62 Broadcast various requests as per #4684 (#4920)
* multiple broadcast flags

* rewrite with single --broadcast option

* satisfy cargo fmt

* shorten sync-committee-messages

* fix a doc comment and a test

* use strum

* Add broadcast test to simulator

* bring --disable-run-on-all flag back with deprecation notice
2023-11-27 15:39:37 +11:00
realbigsean
e856a904ef Merge pull request #4939 from eserilev/block-v3-general-tests
Add more block v3 tests
2023-11-21 14:52:52 -05:00
Eitan Seri-Levi
98f159cc18 fmt 2023-11-20 21:37:58 -08:00
Eitan Seri-Levi
24a0a7ffd0 remove cache check 2023-11-20 21:37:04 -08:00
Eitan Seri-Levi
228180bb35 add some more block v3 tests 2023-11-20 20:46:13 -08:00
Jimmy Chen
6b63d18420 Fix Rust beta compiler warnings (rustc 1.75.0-beta.1 (782883f60 2023-11-12)) (#4932) 2023-11-18 03:55:11 +11:00
Mac L
68e076d60a Remove legacy database migrations (#4919)
* Remove legacy db migrations

* Fix tests

* Remove migrations to/from v15

* Remove v16 migrations
2023-11-16 22:09:05 +11:00
Michael Sproul
d04e361129 Fix missed block logs (#4922) 2023-11-16 22:07:58 +11:00
Zackary Scott
e181741d38 Fix for issue 4860 - Added in process_justification_and_finalization (#4877)
* Added in process_justification_and_finalization

Added in process_justification_and_finalization to compute_attestation_rewards_altair to take into account justified attestations when coming out of inactivity leak. Also added in test to check for this edge case.

* Added in justification and finalization for compute_attestation_rewards_base

* Added in test for altair rewards without inactivity leak
2023-11-16 22:07:48 +11:00
Michael Sproul
051c3e842f Always use a separate database for blobs (#4892)
* Always use a separate blobs DB

* Add + update tests
2023-11-09 16:51:36 +11:00
chonghe
1b8c0ed987 Update local testnet doc and parameters (#4749)
* update

* revise link

* update parameters

* update doc

* Update doc

* add quic port

* remove debug log

* Fix el_bootnode not being killed

* Fix time

* Fix doc in manual creation of testnet

* Update file

* update api doc

* Revert "update api doc"

This reverts commit ed695743de.

* add git clone

* Fix path

* Fix path

* Update scripts/local_testnet/setup_time.sh

Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>

* Update scripts/local_testnet/README.md

Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>

* Fix SLOT_PER_EPOCH that changes with mainnet or minimal

* Embedded setup_time.sh in start_local_testnet.sh

* fix slot per epoch constant

* Add comment

* Add CANCUN_TIME

* Fix CANCUN_TIME constant 32 slots

* Correct typo

* chmod +x ./setup_time.sh

---------

Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>
2023-11-09 15:06:10 +11:00
chonghe
7fd9389a8c Delete deprecated cli flags (#4906)
* Delete BN spec flag and VC beacon-node flag

* Remove warn

* slog

* add warn

* delete eth1-endpoint

* delete server from vc cli.rs

* delete server flag in config.rs

* delete delete-lockfiles in vc

* delete allow-unsynced flag in VC

* delete strict-fee-recipient in VC and warn log

* delete merge flag in bn (hidden)

* delete count-unrealized and count-unrealized-full in bn (hidden)

* delete http-disable-legacy-spec in bn (hidden)

* delete eth1-endpoint in lcli

* delete warn message lcli

* delete eth1-endpoints

* delete minify in slashing protection

* delete minify related

* Remove mut

* add back warn! log

* Indentation

* Delete count-unrealized

* Delete eth1-endpoints

* Delete eth1-endpoint test

* delete eth1-endpints test

* delete allow-unsynced test

* Add back lcli eth1-endpoint

---------

Co-authored-by: Michael Sproul <michael@sigmaprime.io>
2023-11-09 15:05:55 +11:00
ethDreamer
7818100777 Verify KZG in Bulk During Block Sync (#4903) 2023-11-09 15:05:44 +11:00
Eitan Seri-Levi
a380f6ef1f Add block-v3 SSZ tests (#4902)
* create block-v3 ssz tests

* fmt
2023-11-09 15:05:35 +11:00
Joel Rousseau
ac8811afac Add missed blocks to monitored validators (#4731)
* add missed_block metric

* init missed_block in constructor

* declare beaconproposercache in ValidatorMonitor

* refacto proposer_shuffling_decision_root to use epoch instead of current.epoch

* imple new proposer_shuffling_decision_root in callers

* push missed_blocks

* prune missed_blocks

* only add to hashmap if it's a monitored validator

* remove current_epoch dup + typos

* extract in func

* add prom metrics

* checkpoint is not only epoch but slot as well

* add safeguard if we start a new chain at slot 0

* clean

* remove unnecessary negative value for a slot

* typo in comment

* remove unused current_epoch

* share beacon_proposer_cache between validator_monitor and beacon_chain

* pass Hash256::zero()

* debug objects

* fix loop: lag is at the head

* sed s/get_slot/get_epoch

* fewer calls to cache.get_epoch

* fix typos

* remove cache first call

* export TYPICAL_SLOTS_PER_EPOCH and use it in validator_monitor

* switch to gauge & loop over missed_blocks hashset

* fix subnet_service tests

* remove unused var

* clean + fix nits

* add beacon_proposer_cache + validator_monitor in builder

* fix store_tests

* fix builder tests

* add tests

* add validator monitor set of tests

* clean tests

* nits

* optimise imports

* lint

* typo

* added self.aggregatable

* duplicate proposer_shuffling_decision_root

* remove duplication in passing beacon_proposer_cache

* remove duplication in passing beacon_proposer_cache

* using indices

* fmt

* implement missed blocks total

* nits

* avoid heap allocation

* remove recursion limit

* fix lint

* Fix valdiator monitor builder pattern

Unify validator monitor config struct

* renaming metrics

* renaming metrics in validator monitor

* add log if there's a missing validator index

* consistent log

* fix loop

* better loop

* move gauge to counter

* fmt

* add error message

* lint

* fix prom metrics

* set gauge to 0 when non-finalized epochs

* better wording

* remove hash256::zero in favour of block_root

* fix gauge total label

* fix last missed block validator

* Add `MissedBlock` struct

* Fix comment

* Refactor non-finalized block loop

* Fix off-by-one

* Avoid string allocation

* Fix compile error

* Remove non-finalized blocks metric

* fix func clojure

* remove unused variable

* remove unused DEFAULT_INDIVIDUAL_TRACKING_THRESHOLD

* remove unused DEFAULT_INDIVIDUAL_TRACKING_THRESHOLD in builder

* add validator index depending on the fork name

* typos

---------

Co-authored-by: Paul Hauner <paul@paulhauner.com>
2023-11-09 15:05:14 +11:00
realbigsean
bcca88a150 more logging improvements (#4885)
## Issue Addressed

-downgrades `Missing components over rpc` to debug because this isn't unusual and just results in a re-try
-removes the result from  `Block component processed for lookup` because this prints the full block on an unknown parent error

Co-authored-by: realbigsean <seananderson33@gmail.com>
2023-11-03 13:31:27 +00:00
Justin Traglia
f2aabe915b Fix comment for blob sidecar observation pruning (#4893)
## Issue Addressed

The comment implies that observations for the given slot would be retained but they are not.

## Proposed Changes

I'm pretty sure the functionality is correct and the comment is slightly incorrect, so just update the comment. The comment needs to say something along the lines of "less than or equal to" rather than just "less than."

## Additional Info

It doesn't make sense to keep finalized observations since those are no longer accepted.
2023-11-03 12:38:13 +00:00
Jimmy Chen
36d8849813 Add commmand for pruning states (#4835)
## Issue Addressed

Closes #4481. 

(Continuation of #4648)

## Proposed Changes

- [x] Add `lighthouse db prune-states`
- [x] Make it work
- [x] Ensure block roots are handled correctly (to be addressed in 4735)
- [x] Check perf on mainnet/Goerli/Gnosis (takes a few seconds max)
- [x] Run block root healing logic (#4875 ) at the beginning
- [x] Add some tests
- [x] Update docs
- [x] Add `--freezer` flag and other improvements to `lighthouse db inspect`

Co-authored-by: Michael Sproul <michael@sigmaprime.io>
Co-authored-by: Jimmy Chen <jimmy@sigmaprime.io>
Co-authored-by: Michael Sproul <micsproul@gmail.com>
2023-11-03 00:12:19 +00:00
Eitan Seri-Levi
07f53b18fc Block v3 endpoint (#4629)
## Issue Addressed

#4582

## Proposed Changes

Add a new v3 block fetching flow that can decide to return a Full OR Blinded payload

## Additional Info



Co-authored-by: Michael Sproul <micsproul@gmail.com>
2023-11-03 00:12:18 +00:00
realbigsean
42da392edc fix deneb sync bug (#4869)
## Issue Addressed

I observed our forward sync on devnet 9 would stall when we would hit this log:
```
250425:Oct 19 00:54:17.133 WARN Blocks and blobs request for range received invalid data, error: KzgCommitmentMismatch, batch_id: 4338, peer_id: 16Uiu2HAmHbmkEQFDrJfNuy1aYyAfHkNUwSD9FN7EVAqGJ8YTF9Mh, service: sync, module: network::sync::manager:1036
```

## Proposed Changes

`range_sync_block_and_blob_response` [here](1cb02a13a5/beacon_node/network/src/sync/manager.rs (L1013)) removes the request from the sync manager. later, however if there's an error, `inject_error` [here](1cb02a13a5/beacon_node/network/src/sync/manager.rs (L1055)) expects the request to exist so we can handle retry logic. So this PR just re-inserts the request (withthout any accumulated blobs or blocks) when we hit an error here.

The issue is unique to block+blob sync because the error here is only possible from mismatches between blocks + blobs after we've downloaded both, there's no equivalent error in block sync



Co-authored-by: realbigsean <seananderson33@gmail.com>
2023-10-31 21:04:18 +00:00
Eitan Seri-Levi
4ce01ddd11 Activate clippy::manual_let_else lint (#4889)
## Issue Addressed

#4888

## Proposed Changes

Enabled `clippy::manual_let_else` lint and resolved the warning messages.
2023-10-31 10:31:02 +00:00
realbigsean
a9f9dc241d restore cargo vendor in test suite (#4886)
## Issue Addressed

resolves https://github.com/sigp/lighthouse/issues/4440

## Proposed Changes

restore our `cargo vendor` test in CI

changes to `c-kzg` here mean we no longer have to compile it twice and get duplicate source errors: https://github.com/sigp/lighthouse/pull/4862

Co-authored-by: realbigsean <seananderson33@gmail.com>
2023-10-27 10:21:47 +00:00
Michael Sproul
c574f8136e Fix block backfill with genesis skip slots (#4820)
## Issue Addressed

Closes #4817.

## Proposed Changes

- Fill in the linear block roots array between 0 and the slot of the first block (e.g. slots 0 and 1 on Holesky).
- Backport the `--freezer`, `--skip` and `--limit` options for `lighthouse db inspect` from tree-states. This allows us to easily view the database corruption of 4817 using `lighthouse db inspect --network holesky --freezer --column bbr --output values --limit 2`.
- Backport the `iter_column_from` change and `MemoryStore` overhaul from tree-states. These are required to enable `lighthouse db inspect`.
- Rework `freezer_upper_limit` to allow state lookups for slots below the `state_lower_limit`. Currently state lookups will fail until state reconstruction completes entirely.

There is a new regression test for the main bug, but no test for the `freezer_upper_limit` fix because we don't currently support running state reconstruction partially (see #3026). This will be fixed once we merge `tree-states`! In lieu of an automated test, I've tested manually on a Holesky node while it was reconstructing.

## Additional Info

Users who backfilled Holesky to slot 0 (e.g. using `--reconstruct-historic-states`) need to either:

- Re-sync from genesis.
- Re-sync using checkpoint sync and the changes from this PR.

Due to the recency of the Holesky genesis, writing a custom pass to fix up broken databases (which would require its own thorough testing) was deemed unnecessary. This is the primary reason for this PR being marked `backwards-incompat`.

This will create few conflicts with Deneb, which I've already resolved on `tree-states-deneb` and will be happy to backport to Deneb once this PR is merged to unstable.
2023-10-27 05:08:49 +00:00
zhiqiangxu
b82d1a993c fix docs about --builder (#4754)
Align the document with the [implementation](2841f60686/beacon_node/execution_layer/src/lib.rs (L740)) regarding the `--builder` flag.


The `tokio::join!` macro takes a list of async expressions and evaluates them concurrently on the same task.
2023-10-26 05:23:50 +00:00
realbigsean
ba891e1fed deneb related logging improvements (#4859)
1. Add commitments to logs and update the `Display` implementation of `KzgCommitment` to become truncated similarly to block root.

I've been finding it difficult to debug scenarios involving multiple blobs for the same `(index, block_root)`. Logging the commitment will help with this, we can match it to what exists in the block. 

Example output:

```
Oct 20 21:13:36.700 DEBG Successfully verified gossip blob       commitment: 0xa3c1…1cd8, index: 0, root: 0xf31e…f9de, slot: 154568
Oct 20 21:13:36.785 DEBG Successfully verified gossip block      commitments: [0xa3c1…1cd8, 0x8655…02ff, 0x8d6a…955a, 0x84ac…3a1b, 0x9752…629b, 0xb9fc…20fb], root: 0xf31eeb732702e429e89057b15e1c0c631e8452e09e03cb1924353f536ef4f9de, slot: 154568, graffiti: teku/besu, service: beacon
```

Example output in a block with no blobs (this will show up pre-deneb):
```
426734:Oct 20 21:15:24.113 DEBG Successfully verified gossip block, commitments: [], root: 0x619db1360ba0e8d44ae2a0f2450ebca47e167191feecffcfac0e8d7b6c39623c, slot: 154577, graffiti: teku/nethermind, service: beacon, module: beacon_chain::beacon_chain:2765
```

2. Remove `strum::IntoStaticStr` from `AvailabilityCheckError`. This is because `IntoStaticStr` end up dropping information inside the enum. So kzg commitments in this error are dropped, making it more difficult to debug
```
AvailabilityCheckError::KzgCommitmentMismatch {
        blob_commitment: KzgCommitment,
        block_commitment: KzgCommitment,
    },
```
which is output as just `AvailabilityCheckError`

3. Some additional misc sync logs I found useful in debugging https://github.com/sigp/lighthouse/pull/4869

4. This downgrades ”Block returned for single block lookup not present” to debug because I don’t think we can fix the scenario that causes this unless we can cancel inflight rpc requests

Co-authored-by: realbigsean <seananderson33@gmail.com>
2023-10-25 15:30:17 +00:00
Jimmy Chen
d4f26ee123 Add block roots heal logic in v18 schema migration. (#4875)
## Issue Addressed

Fixes #4697. 

This also unblocks the state pruning PR (#4835). Because self healing breaks if state pruning is applied to a database with missing block roots.

## Proposed Changes

- Fill in the missing block roots between last restore point slot and split slot when upgrading to latest database version.
2023-10-25 03:42:24 +00:00
antondlr
a228e61773 don't make lcli on self-hosted runners (#4874)
## Issue Addressed

Our self-hosted runners now have a modern (Deneb-ready) version of `lcli` preinstalled so we no longer need to compile it.
2023-10-25 03:42:23 +00:00
Pawan Dhananjay
6315a81260 Upgrade to v1.4.0-beta.3 (#4862)
## Issue Addressed

Makes lighthouse compliant with new kzg changes in https://github.com/ethereum/consensus-specs/releases/tag/v1.4.0-beta.3

## Proposed Changes

1. Adds new official trusted setup
2. Refactors kzg to match upstream changes in https://github.com/ethereum/c-kzg-4844/pull/377
3. Updates pre-generated `BlobBundle` to work with official trusted setup. ~~Using json here instead of ssz to account for different value of `MaxBlobCommitmentsPerBlock` in minimal and mainnet. By using json, we can just use one pre generated bundle for both minimal and mainnet. Size of 2 separate ssz bundles is approximately equal to one json bundle cc @jimmygchen~~ 
Dunno what I was doing, ssz works without any issues  
4. Stores trusted_setup as just bytes in eth2_network_config so that we don't have kzg dependency in that lib and in lcli. 


Co-authored-by: realbigsean <seananderson33@gmail.com>
Co-authored-by: realbigsean <seananderson33@GMAIL.com>
2023-10-21 13:49:27 +00:00
Pawan Dhananjay
074c4951fc Reduce calls to network channel (#4863)
## Issue Addressed

N/A

## Proposed Changes

Sends blocks and blobs from http_api to the network channel for publishing in a single network channel send. This is to avoid overhead of multiple calls.
Also adds a metric for rpc blob retrieval duration.
2023-10-20 19:42:47 +00:00
Jimmy Chen
e8fba8d3a7 Enable BLS portable feature on all CI tests (#4868)
## Issue Addressed

Addresses the recent CI failures caused by caching `blst` for the wrong CPU type. 

## Proposed Changes

- Use `FEATURES: jemalloc,portable` when building Lighthouse & `lcli` in tests
- Add a new `TEST_FEATURES` and set to `portable` for all CI test jobs.
- Updated Makefiles to read the `TEST_FEATURES` environment variable, and default to none.
2023-10-20 07:30:27 +00:00
Jimmy Chen
8880675eda Add make lint to development environment section in Book (#4866)
## Issue Addressed

We run `clippy` as part of our CI, so it would help new devs if we add the `make lint` command to the dev setup section in the Lighthouse book.
2023-10-20 06:23:29 +00:00
Zackary Scott
b11988223f #4512 inactivity calculation for Altair (#4807)
## Issue Addressed
#4512 
Which issue # does this PR address?

## Proposed Changes
Add inactivity calculation for Altair

Please list or describe the changes introduced by this PR.
Add inactivity calculation for Altair

## Additional Info

Please provide any additional information. For example, future considerations
or information useful for reviewers.


Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>
2023-10-20 06:23:28 +00:00
GoodDaisy
90f78d141f fix typos (#4838) 2023-10-19 22:05:15 +00:00
João Oliveira
c6583bb5fa update libp2p (#4864)
## Issue Addressed

updates libp2p to the latest version and uses the new `SwarmBuilder`. Superseeds https://github.com/sigp/lighthouse/pull/4695/
CC @mxinden I don't think we can use both `bandwidth_loggers` with the new syntax right?
2023-10-19 21:22:55 +00:00
Michael Sproul
8c28d175b8 Fix: write post state in lcli skip-slots (#4843)
## Issue Addressed

Fix a bug in `lcli skip-slots` that resulted in it always writing the pre-state to the output file.

## Proposed Changes

Correctly keep track of the post-state, and write it.
2023-10-19 05:19:25 +00:00
Dustin Brickwood
1c6356f8f3 chore: replace deprecated hub with gh for releases (#4839)
## Issue Addressed

- The tool hub that is used to create draft releases has been removed as of October 2. See: https://github.com/actions/runner-images/issues/8362
- This change replaces `hub` usage in favour of [`gh`](https://cli.github.com/manual/gh_release_create) 

## Proposed Changes

- This change replaces `hub` usage in favour of [`gh`](https://cli.github.com/manual/gh_release_create) 

## Additional Info

Please provide any additional information. For example, future considerations
or information useful for reviewers.


Co-authored-by: Michael Sproul <michael@sigmaprime.io>
2023-10-19 05:19:24 +00:00
Joe Clapis
1de02f731b Add CARGO_USE_GIT_CLI to the Dockerfile to work around an OOM bug during cross-compiling (#4828)
## Issue Addressed
#4827 

## Proposed Changes

This PR introduces a new build-arg to the Lighthouse Dockerfile: `CARGO_USE_GIT_CLI`. This arg will be passed into the `CARGO_NET_GIT_FETCH_WITH_CLI` [environment variable](https://doc.rust-lang.org/cargo/reference/config.html#netgit-fetch-with-cli), which instructs `cargo` to use the git CLI during `fetch` operations instead of the git library. Doing so works around [a bug](https://github.com/rust-lang/cargo/issues/10583) with the git library that causes it to go OOM during `fetch` operations on `arm64` platforms.

The default value is `false` so this doesn't affect Lighthouse builds or the CI pipeline. Running a build with `--build-arg CARGO_USE_GIT_CLI=true` will activate it, which is necessary to cross-compile the `arm64` binary when not using `cross` (i.e., when building via the Dockerfile instead of natively if you don't have a rust environment ready to go).

Special thanks to @michaelsproul for helping me repro the initial problem.

Co-authored-by: Michael Sproul <micsproul@gmail.com>
2023-10-19 05:19:23 +00:00
João Oliveira
f06391717c collect bandwidth metrics per transport (#4805)
## Issue Addressed

Following the conversation on https://github.com/libp2p/rust-libp2p/pull/3666 the changes introduced in this PR will allow us to give more insights if the bandwidth limitations happen at the transport level, namely if quic helps vs yamux and it's [window size limitation](https://github.com/libp2p/rust-yamux/issues/162) or if the bottleneck is at the gossipsub level.
## Proposed Changes

introduce new quic and tcp bandwidth metric gauges.

cc @mxinden (turned out to be easier, Thomas gave me a hint)
2023-10-19 05:19:22 +00:00
Jimmy Chen
98cac2bc6b Deneb review .github (CI cleanup) (#4696)
## Issue Addressed

Related to https://github.com/sigp/lighthouse/issues/4676.

Deneb-specifc CI code to be removed before merging to `unstable`. Dot not merge until we're ready to merge into `unstable`, as we may need to release deneb docker images before merging.

Keep in mind that most of the changes in the below PR (to `unstable`) have already 
been merged to `deneb-free-blobs`, so merging `deneb-free-blobs` into `unstable` would include those changes - it would be ok if the release runners are ready, otherwise we may want to exclude them before merging.
- https://github.com/sigp/lighthouse/pull/4592
2023-10-18 15:23:31 +00:00
Michael Sproul
5bbeedb5b7 Reduce nextest threads to 8 (#4846)
## Issue Addressed

Fix OOMs caused by too many concurrent tests. The runner machine is currently liable to run `32 * 5 = 160` tests in parallel. If each test uses say 300MB max, this is 48GB of RAM!

## Proposed Changes

Reduce the number of threads per runner job to 8. This should cap the memory at 4x lower than the current limit, i.e. around 12GB. If we continue to run out of RAM, we should consider more sophisticated limits.
2023-10-18 14:19:41 +00:00
Michael Sproul
192d442718 Fix Rayon deadlock in test utils (#4837)
## Issue Addressed

Fix a deadlock in the tests that was causing tests on tree-states to run for hours without finishing: https://github.com/sigp/lighthouse/actions/runs/6491194654/job/17628138360.

## Proposed Changes

Avoid using a Mutex under the Rayon `par_iter`. Instead, use an `AtomicUsize`. I've run the new version several times in a loop and it hasn't deadlocked (it was deadlocking consistently on tree-states).

## Additional Info

The same bug exists in unstable and tree-states, but I'm not sure why it was triggering so consistently on the tree-states branch.
2023-10-18 13:36:42 +00:00
Michael Sproul
463e62e833 Generalise compare_fields to work with iterators (#4823)
## Proposed Changes

Add `compare_fields(as_iter)` as a field attribute to `compare_fields_derive`. This allows any iterable type to be compared in the same as a slice (by index). 

This is forwards-compatible with tree-states types like `List` and `Vector` which can not be cast to slices.
2023-10-18 12:59:53 +00:00
Jimmy Chen
1b4545cd9d Remove blob clones in KZG verification (#4852)
## Issue Addressed

This PR removes two instances of blob clones during blob verification that may not be necessary.
2023-10-18 06:52:54 +00:00
Mac L
a7c46bf7ed Fix Homebrew link (#4822)
## Issue Addressed

N/A

## Proposed Changes

I saw a false positive on the link-check CI run and while investigating I noticed that this link technically 404's but is not "dead" in the strict sense. I have updated it to the correct path.
2023-10-18 06:52:53 +00:00
João Oliveira
f10d3d07c3 remove crit! logging from ListenerClosed event on Ok() (#4821)
## Issue Addressed

Since adding Quic support on https://github.com/sigp/lighthouse/pull/4577, and due to `quinn`s api nature LH now triggers the [`ListenerClosed`](https://docs.rs/libp2p/0.52.3/libp2p/swarm/struct.ListenerClosed.html) event.. @michaelsproul noticed we are logging this event as `crit!` independently of the reason. This PR matches the reason, logging with `debug!` and `error!` (instead of `crit!`) according to its `Result`  
## Additional Info
LH will still log `crit!` until https://github.com/libp2p/rust-libp2p/pull/4621 has been merged
2023-10-18 06:52:52 +00:00
Jimmy Chen
18f3edff0a Add vendor directory to .gitignore (#4819)
## Issue Addressed

The vendor directory gets populated after running `cargo vendor`. This directory should be ignored by VCS.
2023-10-18 06:52:51 +00:00
Michael Sproul
5cc0f1097b Fix metric for total block production time (#4794)
## Proposed Changes

Fix the misplacement of the total block production time metric, which occurred during a previous refactor.

Total block production times are no longer skewed low (data from Holesky + blockdreamer):

```
# HELP beacon_block_production_seconds Full runtime of block production
# TYPE beacon_block_production_seconds histogram
beacon_block_production_seconds_bucket{le="0.005"} 0
beacon_block_production_seconds_bucket{le="0.01"} 0
beacon_block_production_seconds_bucket{le="0.025"} 0
beacon_block_production_seconds_bucket{le="0.05"} 0
beacon_block_production_seconds_bucket{le="0.1"} 0
beacon_block_production_seconds_bucket{le="0.25"} 0
beacon_block_production_seconds_bucket{le="0.5"} 37
beacon_block_production_seconds_bucket{le="1"} 65
beacon_block_production_seconds_bucket{le="2.5"} 66
beacon_block_production_seconds_bucket{le="5"} 66
beacon_block_production_seconds_bucket{le="10"} 66
beacon_block_production_seconds_bucket{le="+Inf"} 66
beacon_block_production_seconds_sum 34.225780452
beacon_block_production_seconds_count 66
```

## Additional Info

Cheers to @jimmygchen for helping spot this.
2023-10-18 06:52:50 +00:00
Jimmy Chen
64c156c0c1 Pre-generate test blobs bundle to improve test time. (#4829)
## Issue Addressed

Addresses #4778, and potentially fixes the flaky deneb builder test `builder_works_post_deneb`.

The [deneb builder test](c5c84f1213/beacon_node/http_api/tests/tests.rs (L5371)) has been quite flaky on our CI (`release-tests`) since it was introduced. I'm guessing that it might be timing out on the builder `get_header` call (1 second), and therefore the local payload is used, while the test expects builder payload to be used. 

On my machine the [`get_header` ](c5c84f1213/beacon_node/execution_layer/src/test_utils/mock_builder.rs (L367)) call takes about 550ms, which could easily go over 1s on slower environments (our windows CI runner is much slower than the ubuntu one).

I did a profile on the test and it showed that `blob_to_kzg_commiment` and `compute_kzg_proof` was taking a large chunk of time, so perhaps pre-generating the blobs could help stablise this test.

## Proposed Changes

Pre-generate blobs bundle for Mainnet and Minimal presets.

Before the change `get_header` took about **550ms**, and it's now reduced to **50-55ms** after the change. If timeout was indeed the cause of the flaky test, this fix should stablise it. This also brings the flaky `builder_works_post_deneb` test time from 50s to 10s. (8s if we only use a single blob)
2023-10-18 04:40:29 +00:00
Mac L
369b624b19 Fix broken Nethermind integration tests (#4836)
## Issue Addressed

CI is currently blocked by persistently failing integration tests.

## Proposed Changes

Use latest Nethermind release and apply the appropriate fixes as there have been breaking changes.
Also increase the timeout since I had some local timeouts.


Co-authored-by: Michael Sproul <michael@sigmaprime.io>
Co-authored-by: antondlr <anton@delaruelle.net>
Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>
2023-10-18 04:08:55 +00:00
Michael Sproul
8b0545da12 Merge Deneb (#4054) 2023-10-17 10:58:11 +11:00
realbigsean
283ec8cf24 Deneb pr updates 2 (#4851)
* use workspace deps in kzg crate

* delete unused blobs dp path field

* full match on fork name in engine api get payload v3

* only accept v3 payloads on get payload v3 endpoint in mock el

* remove FIXMEs related to merge transition tests

* move static tx to test utils

* default max_per_epoch_activation_churn_limit to mainnet value

* remove unnecessary async

* remove comment

* use task executor in `blob_sidecars` endpoint
2023-10-17 09:53:46 +11:00
Michael Sproul
ba0567d3ef Merge remote-tracking branch 'origin/unstable' into deneb-free-blobs 2023-10-16 16:33:37 +11:00
Age Manning
cf544b3996 Very minor own nitpicks (#4845) 2023-10-16 16:30:14 +11:00
Michael Sproul
2d662f78ae Use Deneb fork in generate_genesis_header
Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>
2023-10-16 16:24:27 +11:00
Jimmy Chen
38e7172508 Add blob_sidecar event to SSE (#4790)
* Add `blob_sidecar` event to SSE.

* Return 202 if a block is published but failed blob validation when validation level is `Gossip`.

* Move `BlobSidecar` event to `process_gossip_blob` and add test.

* Emit `BlobSidecar` event when blobs are received over rpc.

* Improve test assertions on `SseBlobSidecar`s.

* Add quotes to blob index serialization in `SseBlobSidecar`

Co-authored-by: realbigsean <seananderson33@GMAIL.com>

---------

Co-authored-by: realbigsean <seananderson33@GMAIL.com>
2023-10-12 10:13:08 -04:00
realbigsean
4555e33048 Remove serde derive references (#4830)
* remove remaining uses of serde_derive

* fix lockfile

---------

Co-authored-by: João Oliveira <hello@jxs.pt>
2023-10-11 13:01:30 -04:00
Michael Sproul
d9acee5a72 Delete unused ssz_types file (#4824) 2023-10-11 12:49:08 -04:00
ethDreamer
8660043024 Prevent Overflow LRU Cache from Exploding (#4801)
* Initial Commit of State LRU Cache

* Build State Caches After Reconstruction

* Cleanup Duplicated Code in OverflowLRUCache Tests

* Added Test for State LRU Cache

* Prune Cache of Old States During Maintenance

* Address Michael's Comments

* Few More Comments

* Removed Unused impl

* Last touch up

* Fix Clippy
2023-10-10 23:51:00 -05:00
Jimmy Chen
4ad7e15732 Address Clippy 1.73 lints on Deneb branch (#4810)
* Address Clippy 1.73 lints (#4809)

## Proposed Changes

Fix Clippy lints enabled by default in Rust 1.73.0, released today.

* Address Clippy 1.73 lints.

---------

Co-authored-by: Michael Sproul <michael@sigmaprime.io>
2023-10-06 12:23:57 +05:30
Michael Sproul
c3321dddb7 Reduce attestation subscription spam from VC (#4806)
## Proposed Changes

Instead of sending every attestation subscription every slot to every BN:

- Send subscriptions 32, 16, 8, 7, 6, 5, 4, 3 slots before they occur.
- Track whether each subscription is sent successfully and retry it in subsequent slots if necessary.

## Additional Info

- [x] Add unit tests for `SubscriptionSlots`.
- [x] Test on Holesky.
- [x] Based on #4774 for testing.
2023-10-06 06:26:18 +00:00
chonghe
accb56e4fb Revise doc API section (#4798)
## Issue Addressed

Partially #4788 

## Proposed Changes

Remove documentation on `/lighthouse/database/reconstruct` API to avoid confusion as the calling the API during historical block download will show an error in the beacon log

Add Events API about `payload_attributes`

## Additional Info

Please provide any additional information. For example, future considerations
or information useful for reviewers.


Co-authored-by: chonghe <44791194+chong-he@users.noreply.github.com>
Co-authored-by: Michael Sproul <micsproul@gmail.com>
2023-10-06 04:34:47 +00:00
Michael Sproul
9769a247b2 Address Clippy 1.73 lints (#4809)
## Proposed Changes

Fix Clippy lints enabled by default in Rust 1.73.0, released today.
2023-10-06 03:05:47 +00:00
realbigsean
203ac65041 Merge pull request #4808 from jimmygchen/merge-unstable-to-deneb-20231005
Merge `unstable` branch to deneb 20231005
2023-10-05 11:17:48 -04:00
Jimmy Chen
a96963fd5f Re-commit corrupted key files 2023-10-06 00:24:09 +11:00
Jimmy Chen
3692622339 Merge branch 'unstable' into merge-unstable-to-deneb-20231005 2023-10-06 00:20:54 +11:00
Michael Sproul
b82f7843ff Use peeking_take_while in BlockReplayer (#4803)
## Issue Addressed

While reviewing #4801 I noticed that our use of `take_while` in the block replayer means that if a state root iterator _with gaps_ is provided, some additonal state roots will be dropped unnecessarily. In practice the impact is small, because once there's _one_ state root miss, the whole tree hash cache needs to be built anyway, and subsequent misses are less costly. However this was still a little inefficient, so I figured it's better to fix it.

## Proposed Changes

Use [`peeking_take_while`](https://docs.rs/itertools/latest/itertools/trait.Itertools.html#method.peeking_take_while) to avoid consuming the next element when checking whether it satisfies the slot predicate.

## Additional Info

There's a gist here that shows the basic dynamics in isolation: https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=40b623cc0febf9ed51705d476ab140c5. Changing the `peeking_take_while` to a `take_while` causes the assert to fail. Similarly I've added a new test `block_replayer_peeking_state_roots` which fails if the same change is applied inside `get_state_root`.
2023-10-05 06:03:24 +00:00
Jimmy Chen
72563ffb41 Fix CI tests 2023-10-05 16:38:06 +11:00
Jimmy Chen
c5c84f1213 Merge branch 'unstable' into merge-unstable-to-deneb-20231005
# Conflicts:
#	.github/workflows/test-suite.yml
#	Cargo.lock
#	beacon_node/execution_layer/Cargo.toml
#	beacon_node/execution_layer/src/test_utils/mock_builder.rs
#	beacon_node/execution_layer/src/test_utils/mod.rs
#	beacon_node/network/src/service/tests.rs
#	consensus/types/src/builder_bid.rs
2023-10-05 15:54:44 +11:00
Nico Flaig
4b619c63d7 Exit aggregation step early if no validator is aggregator (#4774)
## Issue Addressed

Closes https://github.com/sigp/lighthouse/issues/4712

## Proposed Changes

Exit aggregation step early if no validator is aggregator. This avoids an unnecessary request to the beacon node and more importantly fixes noisy errors if Lighthouse VC is used with other clients such as Lodestar and Prysm.

## Additional Info

Related issue https://github.com/ChainSafe/lodestar/issues/5553
2023-10-05 02:14:55 +00:00
duguorong009
7d537214b7 fix(validator_client): return http 404 rather than 405 in http api (#4758)
## Issue Addressed
- Close #4596 

## Proposed Changes
- Add `Filter::recover` to handle rejections specifically as 404 NOT FOUND

Please list or describe the changes introduced by this PR.

## Additional Info

Similar to PR #3836
2023-10-04 00:43:29 +00:00
Akihito Nakano
ba8bcf4bd3 Remove deficit gossipsub scoring during topic transition (#4486)
## Issue Addressed

This PR closes https://github.com/sigp/lighthouse/issues/3237

## Proposed Changes

Remove topic weight of old topics when the fork happens.

## Additional Info

- Divided `NetworkService::start()` into `NetworkService::build()` and `NetworkService::start()` for ease of testing.
2023-10-04 00:43:28 +00:00
Michael Sproul
6ec649a4e2 Optimise head block root API (#4799)
## Issue Addressed

We've had a report of sync committee performance suffering with the beacon processor HTTP API prioritisations.

## Proposed Changes

Increase the priority of `/eth/v1/beacon/blocks/head/root` requests, which are used by the validator client to form sync committee messages, here:

441fc1691b/validator_client/src/sync_committee_service.rs (L181-L188)

Additionally, avoid loading the blinded block in all but the `block_id=block_root` case. I'm not sure why we were doing this previously, I suspect it was just an oversight during the implementation of the `finalized` status on API requests.

## Additional Info

I think this change should have minimal negative impact as:

- The block root endpoint is quick to compute (a few ms max).
- Only the priority of `head` requests is increased. Analytical processes that are making lots of block root requests for past slots are unable to DoS the beacon processor, as their requests will still be processed after attestations.
2023-10-03 23:59:37 +00:00
Pawan Dhananjay
5bab9b866e Don't downscore peers on duplicate blocks (#4791)
## Issue Addressed

N/A

## Proposed Changes

We were currently downscoring a peer for sending us a block that we already have in fork choice. This is unnecessary as we get duplicates in lighthouse only when
1. We published the block, so the block is already in fork choice
2. We imported the same block over rpc

In both scenarios, the peer who sent us the block over gossip is not at fault.

This isn't exploitable as valid duplicates will get dropped by the gossipsub duplicate filter
2023-10-03 23:59:35 +00:00
Lucas Saldanha
f7daf82430 Removed old Teku mainnet bootnode ENRs (#4786)
## Issue Addressed

N/A

## Proposed Changes

Removing the two Teku mainnet bootnodes that are being sunset.

## Additional Info

We are leaving only these two bootnodes: https://github.com/eth-clients/eth2-networks/blob/master/shared/mainnet/bootstrap_nodes.txt#L10-L11
2023-10-03 23:59:34 +00:00
Divma
f11884ccdb enforce non zero enr ports (#4776)
## Issue Addressed

Right now lighthouse accepts zero as enr ports. Since enr ports should be reachable, zero ports should be rejected here

## Proposed Changes

- update the config to use `NonZerou16` as an ENR port for all enr-related fields.
- the enr builder from config now sets the enr to the listening port only if the enr port is not already set (prev behaviour) and the listening port is not zero (new behaviour)
- reject zero listening ports when used with `enr-match`. 
- boot node now rejects listening port as zero, since those are advertised.
- generate-bootnode-enr also rejected zero listening ports for the same reason.
- update local network scripts

## Additional Info

Unrelated, but why do we overwrite `enr-x-port` values with listening ports if `enr-match` is present? we prob should only do this for enr values that are not already set.
2023-10-03 23:59:34 +00:00
João Oliveira
0dc95a1d37 PeerManager: move the check for banned peers from connection_established (#4569)
## Issue Addressed
https://github.com/sigp/lighthouse/issues/4543
## Proposed Changes
- Removes `NotBanned` from `BanResult`, implements `Display` and `std::error::Error` for `BanResult` and changes `ban_result` return type to `Option<BanResult>` which helps returning `BanResult` on `handle_established_inbound_connection`  
- moves the check from for banned peers from `on_connection_established` to `handle_established_inbound_connection` to start addressing #4543.
- Removes `allow_block_list` as it's now redundant? Not sure about this one but if `PeerManager` keeps track of the banned peers, no need to send a `Swarm` event for `alow_block_list` to also keep that list right? 
 
## Questions

-  #4543 refers:
>  More specifically, implement the connection limit behaviour inside the peer manager.

@AgeManning do you mean copying `libp2p::connection_limits::Behaviour`'s code into `PeerManager`/ having it as an inner `NetworkBehaviour` of `PeerManager`/other? If it's the first two, I think it probably makes more sense to have it as it is as it's less code to maintain.

> Also implement the banning of peers inside the behaviour, rather than passing messages back up to the swarm.

I tried to achieve this, but we still need to pass the `PeerManagerEvent::Banned` swarm event as `DiscV5` handles it's node and ip management internally and I did not find a method to query if a peer is banned. Is there anything else we can do from here?

3397612160/beacon_node/lighthouse_network/src/discovery/mod.rs (L931-L940)

Same as the question above, I did not find a way to check if `DiscV5` has the peer banned, so that we could check here and avoid sending `Swarm` events

3397612160/beacon_node/lighthouse_network/src/peer_manager/network_behaviour.rs (L168-L178)

Is there a chance we try to dial a peer that has been banned previously? 

Thanks!
2023-10-03 23:59:32 +00:00
realbigsean
7605494791 Use only lighthouse types in the mock builder (#4793)
## Proposed Changes

- only use LH types to avoid build issues
- use warp instead of axum for the server to avoid importing the dep

## Additional Info

- wondering if we can move the `execution_layer/test_utils` to its own crate and import it as a dev dependency
- this would be made easier by separating out our engine API types into their own crate so we can use them in the test crate
- or maybe we can look into using reth types for the engine api if they are in their own crate


Co-authored-by: realbigsean <seananderson33@gmail.com>
2023-10-03 17:59:28 +00:00
realbigsean
c7ddf1f0b1 add processing and processed caching to the DA checker (#4732)
* add processing and processed caching to the DA checker

* move processing cache out of critical cache

* get it compiling

* fix lints

* add docs to `AvailabilityView`

* some self review

* fix lints

* fix beacon chain tests

* cargo fmt

* make availability view easier to implement, start on testing

* move child component cache and finish test

* cargo fix

* cargo fix

* cargo fix

* fmt and lint

* make blob commitments not optional, rename some caches, add missing blobs struct

* Update beacon_node/beacon_chain/src/data_availability_checker/processing_cache.rs

Co-authored-by: ethDreamer <37123614+ethDreamer@users.noreply.github.com>

* marks review feedback and other general cleanup

* cargo fix

* improve availability view docs

* some renames

* some renames and docs

* fix should delay lookup logic

* get rid of some wrapper methods

* fix up single lookup changes

* add a couple docs

* add single blob merge method and improve process_... docs

* update some names

* lints

* fix merge

* remove blob indices from lookup creation log

* remove blob indices from lookup creation log

* delayed lookup logging improvement

* check fork choice before doing any blob processing

* remove unused dep

* Update beacon_node/beacon_chain/src/data_availability_checker/availability_view.rs

Co-authored-by: Michael Sproul <micsproul@gmail.com>

* Update beacon_node/beacon_chain/src/data_availability_checker/availability_view.rs

Co-authored-by: Michael Sproul <micsproul@gmail.com>

* Update beacon_node/beacon_chain/src/data_availability_checker/availability_view.rs

Co-authored-by: Michael Sproul <micsproul@gmail.com>

* Update beacon_node/beacon_chain/src/data_availability_checker/availability_view.rs

Co-authored-by: Michael Sproul <micsproul@gmail.com>

* Update beacon_node/network/src/sync/block_lookups/delayed_lookup.rs

Co-authored-by: Michael Sproul <micsproul@gmail.com>

* remove duplicate deps

* use gen range in random blobs geneartor

* rename processing cache fields

* require block root in rpc block construction and check block root consistency

* send peers as vec in single message

* spawn delayed lookup service from network beacon processor

* fix tests

---------

Co-authored-by: ethDreamer <37123614+ethDreamer@users.noreply.github.com>
Co-authored-by: Michael Sproul <micsproul@gmail.com>
2023-10-03 09:59:33 -04:00
Age Manning
8a1b77bf89 Ultra Fast Super Slick CI (#4755)
Attempting to improve our CI speeds as its recently been a pain point.

Major changes:

 - Use a github action to pull stable/nightly rust rather than building it each run
 - Shift test suite to `nexttest` https://github.com/nextest-rs/nextest for CI
 
 UPDATE:

So I've iterated on some changes, and although I think its still not optimal I think this is a good base to start from. Some extra things in this PR:
- Shifted where we pull rust from. We're now using this thing: https://github.com/moonrepo/setup-rust . It's got some interesting cache's built in, but was not seeing the gains that Jimmy managed to get. In either case tho, it can pull rust, cargofmt, clippy, cargo nexttest all in < 5s. So I think it's worthwhile. 
- I've grouped a few of the check-like tests into a single test called `code-test`. Although we were using github runners in parallel which may be faster, it just seems wasteful. There were like 4-5 tests, where we would pull lighthouse, compile it, then run an action, like clippy, cargo-audit or fmt. I've grouped these into a single action, so we only compile lighthouse once, then in each step we run the checks. This avoids compiling lighthouse like 5 times.
- Ive made doppelganger tests run on our local machines to avoid pulling foundry, building and making lcli which are all now baked into the images. 
- We have sccache and do not incremental compile lighthouse

Misc bonus things:
- Cargo update
- Fix web3 signer openssl keys which is required after a cargo update
- Use mock_instant in an LRU cache test to avoid non-deterministic test
- Remove race condition in building web3signer tests

There's still some things we could improve on. Such as downloading the EF tests every run and the web3-signer binary, but I've left these to be out of scope of this PR. I think the above are meaningful improvements.



Co-authored-by: Paul Hauner <paul@paulhauner.com>
Co-authored-by: realbigsean <seananderson33@gmail.com>
Co-authored-by: antondlr <anton@delaruelle.net>
2023-10-03 06:33:15 +00:00
Jack McPherson
1c98806b6f Allow libp2p to determine listening addresses (#4700)
## Issue Addressed

#4675 

## Proposed Changes

 - Update local ENR (**only port numbers**) with local addresses received from libp2p (via `SwarmEvent::NewListenAddr`)
 - Only use the zero port for CLI tests

## Additional Info

### See Also ###

 - #4705 
 - #4402 
 - #4745
2023-10-03 04:57:20 +00:00
realbigsean
a935daebd5 Clean bors.toml (#4795)
unblock https://github.com/sigp/lighthouse/pull/4755

Co-authored-by: Paul Hauner <paul@paulhauner.com>
Co-authored-by: realbigsean <seananderson33@gmail.com>
2023-10-03 03:37:12 +00:00
realbigsean
67aeb6bf6b insert cached child at the front of a chain of parent lookups (#4780)
* insert cached child at the front of a chain of parent lookups

* use vecdeque in parent lookup chain of blocks
2023-09-29 12:55:12 -04:00
Jimmy Chen
57edc0f3ce Add serde(default) to max_per_epoch_activation_churn_limit in spec config so that VC is compatible to older BN versions. (#4783) 2023-09-26 08:48:32 -04:00
realbigsean
9f37d6df77 reduce blob prune logging in forward sync (#4779) 2023-09-26 08:22:26 -04:00
realbigsean
a642bd7de7 Merge pull request #4781 from jimmygchen/merge-unstable-to-deneb-20230926
Merge `unstable` into `deneb-free-blobs`
2023-09-26 08:19:37 -04:00
Jimmy Chen
8f07a96b88 Fix failing tests. 2023-09-26 12:39:58 +10:00
Jimmy Chen
1458394cd9 Fix compilation issues after merging unstable. 2023-09-26 11:46:20 +10:00
Jimmy Chen
7a3cb135d4 Fix tests and add BlockContents decoding. Remove unused builder_threshold field in ApiTesterConfig. 2023-09-26 10:57:21 +10:00
Jimmy Chen
c0b6b92f27 Merge unstable 20230925 into deneb-free-blobs. 2023-09-26 10:32:18 +10:00
Michael Sproul
9244f7f7bc Improvements to Deneb store upon review (#4693)
* Start testing blob pruning

* Get rid of unnecessary orphaned blob column

* Make random blob tests deterministic

* Test for pruning being blocked by finality

* Fix bugs and test fork boundary

* A few more tweaks to pruning conditions

* Tweak oldest_blob_slot semantics

* Test margin pruning

* Clean up some terminology and lints

* Schema migrations for v18

* Remove FIXME

* Prune blobs on finalization not every slot

* Fix more bugs + tests

* Address review comments
2023-09-25 14:21:54 -04:00
Lion - dapplion
5c5afafc0d Update Deneb to 1.4.0-beta.2 (devnet-9) (#4735)
* Add MAX_PER_EPOCH_ACTIVATION_CHURN_LIMIT

* Update tests to 1.4.0-beta.2

* Implement equivocation check for proposer boost

* Use hotfix tests and fix minimal config

* Start updating fork choice tests for Deneb

* Finish implementing fork choice blob handling

---------

Co-authored-by: Michael Sproul <michael@sigmaprime.io>
2023-09-25 15:05:31 +10:00
Jimmy Chen
665334e936 Implement SSZ decoding for SignedBlockContents (#4744)
* Implement `SignedBlockContent` decoding and fixed bug in `SignedBlockContent::new`

* Update Cargo.lock file

* Use `make_genesis_spec` to simplify test setup.

* Fix syntax errors.
2023-09-19 18:18:05 -04:00
realbigsean
5f98a7b8ad Deneb PR feedback updates (#4716)
* move length update outside of if let in LRU cache

* add comment and use hex for G1_POINT_AT_INFINITY

* remove some misleading comments from `ssz_snappy`

* make sure we can't overflow on blobs by range requests with large counts

* downgrade gossip verification internal availability check error

* change blob rpc responses from BlockingFnWithManualSendOnIdle to BlockingFn

* remove unnecessary collect in blobs by range response

* add a comment to blobs by range response start slot logic

* typo persist_data_availabilty_checker -> persist_data_availability_checker

* unify cheap_state_advance_to_obtain_committees
2023-09-15 15:05:48 +10:00
Pawan Dhananjay
2cfcb51207 Blob references in ckzg (#4723)
* Move to using references in ckzg functions

* cleanup TrustedSetup a bit

* Remove BYTES_PER_FIELD_ELEMENT from KzgPreset
2023-09-14 12:14:30 +10:00
Stefan
4a31e369bf increase the max topic subscriptions #4581 (#4588)
* increase the max topic subscriptions #4581

* make the max_subscription limitation based off constants / configuration

* format

* wording & add deneb topic array

* reduce max_subscriptions_per_request to 2x

* format

* update comment
2023-09-13 17:06:33 +10:00
realbigsean
8e7b57a794 Merge pull request #4719 from jimmygchen/deneb-merge-from-unstable-20230911
Deneb merge from unstable 20230911
2023-09-12 17:41:42 -04:00
realbigsean
58cd50681c fix the missing components response code for block publish (#4721) 2023-09-11 09:54:14 -04:00
Jimmy Chen
d4aedab21f Fix some compilation errors in tests 2023-09-11 14:07:08 +10:00
Jimmy Chen
6771954c5f Merge unstable 20230911 into deneb-free-blobs. 2023-09-11 12:09:58 +10:00
Jimmy Chen
1db739490e Update BlindedBlobsBundle SSZ list max length and update builder tests (#4710)
* Update mev-rs and ethereum-consensus

* Fix mock buidler open bid to return fork versioned response

* Update `mev-rs` and `ethereum-consensus`

* Remove BuilderKzgCommitments and use BlockBodyKzgCommitments everywhere.

* Update testnet scripts to support builder testing and update README.md.

* Add comment on `mev-rs` version.

* Add `BN_ARGS` config to `./scripts/tests/vars.env`

* Update builder testing command in README.md

* Reject zero block hash payloads after Bellatrix.

* Update scripts/local_testnet/README.md

Co-authored-by: realbigsean <seananderson33@GMAIL.com>

---------

Co-authored-by: realbigsean <seananderson33@GMAIL.com>
2023-09-09 16:10:15 +10:00
Jimmy Chen
50bf40b4bc Update GET config/spec endpoint to support Deneb config (#4708)
* Update `GET config/spec` endpoint to support Deneb config.

* Update config spec `http_api` test
2023-09-09 16:09:50 +10:00
Jimmy Chen
87ec7599fe Fix duplicate geth startup command in doppelganger_protection.sh (#4713) 2023-09-08 12:25:57 +10:00
Michael Sproul
8db44decb7 Fix parent_beacon_block_root during proposer prep (#4703)
* Fix `parent_beacon_block_root` during prep/reorg

* Fix another bug and add tests

* Remove overzealous payload attributes check
2023-09-07 15:43:23 +10:00
Jimmy Chen
f9bea3c174 Deneb review common/eth2 (#4698)
* Update comments and small cleanup.

* Deserialize into `SsePayloadAttributesV3` for Deneb fork. Update `SignedBlockContents::blobs_cloned` to return blobs for `BlindedBlockAndBlobSidecars`.

* Improve code readability and error handling when converting blinded block into full block.
2023-09-06 14:39:25 +10:00
Michael Sproul
13606533b5 Simplifications for ./crypto (#4677) 2023-09-05 17:22:43 -04:00
Paul Hauner
2550170337 Deneb review suggestions (3) (#4694)
* Fix typos

* Avoid consuming/cloning blob

* Tidy comments
2023-09-05 15:50:57 -04:00
Paul Hauner
0bfc933c50 Deneb review suggestions (2) (#4680)
* Serde deny unknown fields

* Require 0x prefix
2023-09-05 15:49:57 -04:00
Paul Hauner
e783a40e01 Deneb review suggestions (#4678)
* Increase resolution for single lookup delay

* Remove code duplication

* Remove trusted setup code duplication
2023-09-05 15:49:11 -04:00
Jimmy Chen
5ea38d90cc Pin foundry toolchain version to fix stuck CI jobs (#4682) 2023-09-05 09:41:11 -04:00
realbigsean
ce824e00a3 Merge pull request #4658 from realbigsean/merge-unstable-deneb-aug-24
Merge unstable deneb aug 24
2023-08-24 17:33:49 -04:00
realbigsean
42b34dbbe4 cargo fmt 2023-08-24 14:35:06 -04:00
realbigsean
f90b190d9a Merge branch 'unstable' of https://github.com/sigp/lighthouse into merge-unstable-deneb-aug-24 2023-08-24 14:34:32 -04:00
realbigsean
8ed77d4550 Merge branch 'unstable' of https://github.com/sigp/lighthouse into merge-unstable-deneb-aug-24 2023-08-24 10:54:43 -04:00
realbigsean
255394419c Merge pull request #4649 from jimmygchen/merge-unstable-to-deneb-20230822
Merge `unstable` to deneb 20230822
2023-08-23 14:41:31 -04:00
Jimmy Chen
4bd527546b Fix failing beacon chain tests and remove unnecessary blob clone 2023-08-23 11:23:12 +10:00
Jimmy Chen
609c2c2250 Update ckzg to latest version with blst 0.3.11 2023-08-23 06:59:23 +10:00
Jimmy Chen
03c610ed77 Fix release test compilation error 2023-08-22 23:20:58 +10:00
Jimmy Chen
7138763299 Empty commit 2023-08-22 22:59:49 +10:00
Jimmy Chen
8a6f171b2a Merge branch 'unstable' into merge-unstable-to-deneb-20230822
# Conflicts:
#	beacon_node/beacon_chain/src/builder.rs
#	beacon_node/beacon_chain/tests/store_tests.rs
#	beacon_node/client/src/builder.rs
#	beacon_node/src/config.rs
#	beacon_node/store/src/hot_cold_store.rs
#	lighthouse/tests/beacon_node.rs
2023-08-22 21:20:47 +10:00
Jimmy Chen
49d7fdfed1 Fix proposer cache miss in blob verification (#4646) 2023-08-21 18:35:49 -04:00
realbigsean
63b876b87e fix beacon chain tests (#4645) 2023-08-21 11:42:50 -04:00
realbigsean
7d468cb487 More deneb cleanup (#4640)
* remove protoc and token from network tests github action

* delete unused beacon chain methods

* downgrade writing blobs to store log

* reduce diff in block import logic

* remove some todo's and deneb built in network

* remove unnecessary error, actually use some added metrics

* remove some metrics, fix missing components on publish funcitonality

* fix status tests

* rename sidecar by root to blobs by root

* clean up some metrics

* remove unnecessary feature gate from attestation subnet tests, clean up blobs by range response code

* pawan's suggestion in `protocol_info`, peer score in matching up batch sync block and blobs

* fix range tests for deneb

* pub block and blob db cache behind the same mutex

* remove unused errs and an empty file

* move sidecar trait to new file

* move types from payload to eth2 crate

* update comment and add flag value name

* make function private again, remove allow unused

* use reth rlp for tx decoding

* fix compile after merge

* rename kzg commitments

* cargo fmt

* remove unused dep

* Update beacon_node/execution_layer/src/lib.rs

Co-authored-by: Pawan Dhananjay <pawandhananjay@gmail.com>

* Update beacon_node/beacon_processor/src/lib.rs

Co-authored-by: Pawan Dhananjay <pawandhananjay@gmail.com>

* pawan's suggestiong for vec capacity

* cargo fmt

* Revert "use reth rlp for tx decoding"

This reverts commit 5181837d81.

* remove reth rlp

---------

Co-authored-by: Pawan Dhananjay <pawandhananjay@gmail.com>
2023-08-20 21:17:17 -04:00
Jimmy Chen
4898430330 Add Deneb builder test & update mock builder (#4607)
* Update mock builder, mev-rs dependencies, eth2 lib to support deneb builder flow

* Replace `sharingForkTime` with `cancunTime`

* Patch `ethereum-consensus` to include some deneb-devnet-8 changes

* Add deneb builder test and fix block contents deserialization

* Fix builder bid encoding issue and passing deneb builder test \o/

* Fix test compilation

* Revert `cancunTime` change in genesis to pass doppelganger tests
2023-08-18 20:12:09 -04:00
Jimmy Chen
f031a570ce Merge pull request #4632 from jimmygchen/revive-mplex-deneb
Revive mplex (Deneb)
2023-08-17 16:37:53 +10:00
Pawan Dhananjay
c280b4849c Add back mplex 2023-08-17 16:09:03 +10:00
ethDreamer
0e04f36a20 Add Test for deneb Block Hash Calculation (#4621) 2023-08-16 11:14:51 -04:00
Jimmy Chen
ba6662344b Update docs to remove lighthouse/database/historical_blocks (removed in #4307) (#4627) 2023-08-16 11:13:44 -04:00
realbigsean
e1991e7e81 Merge pull request #4628 from jimmygchen/merge-unstable-to-deneb-20230816
Merge unstable to deneb 20230816
2023-08-16 11:12:27 -04:00
Jimmy Chen
ff792d950c Merge branch 'unstable' into merge-unstable-to-deneb-20230816
# Conflicts:
#	beacon_node/http_api/src/lib.rs
2023-08-16 14:31:59 +10:00
realbigsean
7f7ad799b3 Fix manifest lists (#4605)
* self hosted docker builds attempted fix

* use imagetools instead of docker manifest
2023-08-10 15:51:44 -04:00
realbigsean
ab37f02ddc Fix env var references (#4604)
* self hosted docker builds attempted fix

* fix env var references in docker builds
2023-08-10 13:09:18 -04:00
realbigsean
11027e3487 self hosted docker builds attempted fix (#4603) 2023-08-10 12:42:56 -04:00
realbigsean
87b165c304 bump clang in cross.toml (#4602) 2023-08-10 10:58:57 -04:00
realbigsean
1440dc0cd2 Merge pull request #4599 from realbigsean/self-hosted-docker-deneb
Self hosted docker deneb
2023-08-10 09:44:04 -04:00
realbigsean
754ce5ec61 remove self hosted runner check where it might not be needed 2023-08-10 09:42:50 -04:00
realbigsean
8d81f1bee7 self hosted docker builds 2023-08-10 09:42:36 -04:00
Jimmy Chen
0b7a426946 Builder flow for Deneb & Blobs (#4428)
* Add Deneb builder flow types with generics

* Update validator client `get_blinded_blocks` call to support Deneb

* `produceBlindedBlock` endpoint updates:
- Handle new Deneb BuilderBid response from builder endpoint (new BlindedBlobsBundle type)
- Build BlockContents response (containing kzg_commitments, proof and blinded_blob_sidecars)

* Appease Clippy lint

* Partial implementation of submit blinded block & blobs. Refactor existing `BlobSidecar` related types to support blinded blobs.

* Add associated types for BlockProposal

* Rename `AbstractSidecar` to `Sidecar`

* Remove blob cache as it's no longer necessary

* Remove unnecessary enum variant

* Clean up

* Hanlde unblinded blobs and publish full block contents

* Fix tests

* Add local EL blobs caching in blinded flow

* Remove BlockProposal and move associated Sidecar trait to AbstractExecPayload to simplify changes

* add blob roots associated type

* move raw blobs associated type to sidecar trait

* Fix todos and improve error handling

* Consolidate BlobsBundle from `execution_layer` into `consensus/types`

* Rename RawBlobs, Blobs, and BlobRoots

* Use `BlobRoots` type alias

* Update error message.

Co-authored-by: realbigsean <seananderson33@GMAIL.com>

* update builder bid type

# Conflicts:
#	consensus/types/src/builder_bid.rs

* Fix lint

* remove generic from builder bid

---------

Co-authored-by: realbigsean <seananderson33@gmail.com>
2023-08-10 09:32:49 -04:00
realbigsean
fddd4e4c87 Merge pull request #4591 from realbigsean/merge-unstable-deneb-aug-9
Merge unstable deneb aug 9
2023-08-09 16:24:18 -04:00
ethDreamer
2b5385fb46 Changes for devnet-8 (#4518)
* Addressed #4487

Add override threshold flag
Added tests for Override Threshold Flag
Override default shown in decimal

* Addressed #4445

Addressed Jimmy's Comments
No need for matches
Fix Mock Execution Engine Tests
Fix clippy
fix fcuv3 bug

* Fix Block Root Calculation post-Deneb

* Addressed #4444

Attestation Verification Post-Deneb
Fix Gossip Attestation Verification Test

* Addressed #4443

Fix Exit Signing for EIP-7044
Fix cross exit test
Move 7044 Logic to signing_context()

* Update EF Tests

* Addressed #4560

* Added Comments around EIP7045

* Combine Altair Deneb to Eliminate Duplicated Code
2023-08-09 15:44:47 -04:00
realbigsean
9e8a289d21 fix blobs by range test 2023-08-09 15:36:39 -04:00
realbigsean
4da6ca73d7 fix imports 2023-08-09 14:12:45 -04:00
realbigsean
c3ced28095 cargo fmt 2023-08-09 10:45:21 -04:00
realbigsean
12b5e9ad3d Merge branch 'unstable' of https://github.com/sigp/lighthouse into merge-unstable-deneb-aug-9 2023-08-09 10:42:51 -04:00
realbigsean
02c7a2eaf5 Improve single block/blob logging (#4579)
* remove closure from `check_availability_mayb_import`

* impove logging, add wrapper struct to requested ids

* improve logging

* only log if we're in deneb. Only delay lookup if we're in deneb

* fix bug in missing components check
2023-08-08 18:45:11 -04:00
realbigsean
efbf906094 Merge pull request #4578 from jimmygchen/merge-unstable-to-deneb-20230808
Merge unstable to deneb 20230808
2023-08-08 10:42:14 -04:00
Jimmy Chen
3ba9047437 Fix release tests 2023-08-08 17:40:40 +10:00
Jimmy Chen
d401633100 Add same error handling for blob signing when pubkey is missing 2023-08-08 17:25:29 +10:00
Jimmy Chen
ec416df061 Merge branch 'unstable' into merge-unstable-to-deneb-20230808
# Conflicts:
#	Cargo.lock
#	beacon_node/beacon_chain/src/lib.rs
#	beacon_node/execution_layer/src/engine_api.rs
#	beacon_node/execution_layer/src/engine_api/http.rs
#	beacon_node/execution_layer/src/test_utils/mod.rs
#	beacon_node/lighthouse_network/src/rpc/codec/ssz_snappy.rs
#	beacon_node/lighthouse_network/src/rpc/handler.rs
#	beacon_node/lighthouse_network/src/rpc/protocol.rs
#	beacon_node/lighthouse_network/src/service/utils.rs
#	beacon_node/lighthouse_network/tests/rpc_tests.rs
#	beacon_node/network/Cargo.toml
#	beacon_node/network/src/network_beacon_processor/tests.rs
#	lcli/src/parse_ssz.rs
#	scripts/cross/Dockerfile
#	validator_client/src/block_service.rs
#	validator_client/src/validator_store.rs
2023-08-08 17:02:51 +10:00
realbigsean
731b7e7af5 Refactor deneb networking (#4561)
* Revert "fix merge"

This reverts commit 405e95b0ce.

* refactor deneb block processing

* cargo fmt

* make block and blob single lookups generic

* get tests compiling

* clean up everything add child component, fix peer scoring and retry logic

* smol cleanup and a bugfix

* remove ParentLookupReqId

* Update beacon_node/network/src/sync/manager.rs

Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>

* Update beacon_node/network/src/sync/manager.rs

Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>

* update unreachables to crits

* Revert "update unreachables to crits"

This reverts commit 064bf64dff.

* update make request/build request to make more sense

* pr feedback

* Update beacon_node/network/src/sync/block_lookups/mod.rs

Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>

* Update beacon_node/network/src/sync/block_lookups/mod.rs

Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>

* more pr feedback, fix availability check error handling

* improve block component processed log

---------

Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>
2023-08-07 14:16:21 -04:00
ethDreamer
c8ea3e1c86 Fix small bug in test utils (#4570) 2023-08-07 11:49:52 -04:00
Pawan Dhananjay
a36e34eec4 Fix todos in deneb code (#4547)
* Low hanging fruits

* Remove unnecessary todo

I think it's fine to not handle this since the calling functions handle the error.
No specific reason imo to handle it in the function as well.

* Rename BlobError to GossipBlobError

I feel this signified better what the error is for. The BlobError was only for failures when gossip
verifying a blob. We cannot get this error when doing rpc validation

* Remove the BlockError::BlobValidation variant

This error was only there to appease gossip verification before publish.
It's unclear how to peer score this error since this cannot actually occur during any
block verification flows.
This commit introuduces an additional error type BlockContentsError to better represent the
Error type

* Add docs for peer scoring (or lack thereof) of AvailabilityCheck errors

* I do not see a non-convoluted way of doing this. Okay to have some redundant code here

* Removing this to catch the failure red handed

* Fix compilation

* Cannot be deleted because some tests assume the trait impl

Also useful to have around for testing in the future imo

* Add some metrics and logs

* Only process `Imported` variant in sync_methods

The only additional thing for other variants that might be useful is logging. We can do that
later if required

* Convert to TryFrom

Not really sure where this would be used, but just did what the comment says.
Could consider just returning the Block variant for a deneb block in the From version

* Unlikely to change now

* This is fine as this is max_rpc_size per rpc chunk (for blobs, it would be 128kb max)

* Log count instead of individual blobs, can delete log later if it becomes too annoying.

* Add block production blob verification timer

* Extend block_straemer test to deneb

* Remove dbg statement

* Fix tests
2023-08-03 20:27:03 -04:00
Jimmy Chen
9c75d8088d Merge pull request #4535 from qu0b/blob/empty
API - Handle blocks without blobs gracefully
2023-07-28 20:00:26 +10:00
qu0b
e021575d8a formatting 2023-07-28 10:27:38 +02:00
qu0b
c991516419 fix CI errors 2023-07-27 14:32:58 +02:00
qu0b
28de041527 cargo fmt & lint-fix 2023-07-26 11:22:06 +02:00
qu0b
1be4d54035 remove option & remove block check 2023-07-26 10:12:58 +02:00
qu0b
97bffd03d0 handle empty blocks gracefully 2023-07-26 10:11:38 +02:00
realbigsean
8c341bb9cc cargo fmt (#4541) 2023-07-25 14:40:04 -04:00
realbigsean
33dd13c798 Refactor deneb block processing (#4511)
* Revert "fix merge"

This reverts commit 405e95b0ce.

* refactor deneb block processing

* cargo fmt

* fix ci
2023-07-25 10:51:10 -04:00
realbigsean
3735450749 Merge pull request #4533 from jimmygchen/deneb-free-blobs
Merge `unstable` into `deneb-free-blobs`
2023-07-25 10:45:22 -04:00
Jimmy Chen
fe94a05dd1 Fix lint 2023-07-24 23:02:29 +10:00
Jimmy Chen
54c6e1dd3d Fix compilation 2023-07-24 21:09:07 +10:00
Jimmy Chen
4ca101e085 Merge branch 'unstable' into deneb-free-blobs 2023-07-24 20:55:27 +10:00
Gua00va
f1f04bc68a Add Changes to BlobSidecars Endpoint (#4455)
* changed name

* Fix sidecars

* Added query type and parrameter

* added query struct and function

* added method

* improved filtering method

* added blob_sidecar_list_indexed to block_id

* minor blobqueryindex fix

* function and formatting fix

* minor function and naming fix

* minor changes
2023-07-21 10:56:57 -04:00
realbigsean
f98671f5ab Merge pull request #4477 from realbigsean/merge-unstable-deneb-june-6th
Merge unstable deneb june 6th
2023-07-17 16:32:26 -04:00
realbigsean
cffa562384 cargo fmt 2023-07-17 16:32:09 -04:00
Pawan Dhananjay
e1d0724abf Fix more beta compiler warnings 2023-07-17 13:29:12 -07:00
Pawan Dhananjay
382b5abbee Merge branch 'deneb-free-blobs' into merge-unstable-deneb-june-6th 2023-07-17 11:47:46 -07:00
realbigsean
27a99015f5 Merge pull request #38 from realbigsean/merge-unstable-deneb-jul-14
Merge unstable deneb jul 14
2023-07-17 12:41:49 -04:00
realbigsean
597389cea5 Merge branch 'merge-unstable-deneb-june-6th' of https://github.com/realbigsean/lighthouse into merge-unstable-deneb-jul-14 2023-07-17 12:41:30 -04:00
realbigsean
aeee5beac2 smol fixes 2023-07-17 10:46:54 -04:00
realbigsean
a618830f8f fix smol lint 2023-07-17 10:28:37 -04:00
realbigsean
0f514cbb36 fixes after merge 2023-07-17 09:50:32 -04:00
realbigsean
b96db45090 Merge branch 'unstable' of https://github.com/sigp/lighthouse into merge-unstable-deneb-jul-14 2023-07-17 09:33:37 -04:00
Pawan Dhananjay
18760822fe Fix beta compiler warnings 2023-07-14 13:16:48 -07:00
realbigsean
405e95b0ce fix merge 2023-07-14 16:15:28 -04:00
realbigsean
42f54ee561 fix merge conflict issues 2023-07-14 16:01:57 -04:00
realbigsean
2b93c0eb0d remove spec minimal feature gating in tests (#4468)
* remove spec minimal feature gating in tests

* do merge transition in overflow cache test
2023-07-14 12:59:15 -07:00
realbigsean
a6f48f5ecb Merge branch 'unstable' of https://github.com/sigp/lighthouse into merge-unstable-deneb-june-6th 2023-07-12 13:05:30 -04:00
realbigsean
c016f5d787 gossip validate blobs prior to publish 2023-07-12 12:32:14 -04:00
realbigsean
1599487933 fix tests 2023-07-12 10:13:25 -04:00
realbigsean
57bb1d931e fix progressive balance slashing tests 2023-07-11 17:07:52 -04:00
realbigsean
782a53ad9d fix compile 2023-07-11 16:05:05 -04:00
realbigsean
6fd2ef49e4 Revert "remove into gossip verified block"
This reverts commit 246d52d209.
2023-07-10 10:22:28 -04:00
realbigsean
c4da1ba450 resolve merge issues 2023-07-07 10:17:04 -04:00
realbigsean
cfe2452533 Merge branch 'remove-into-gossip-verified-block' of https://github.com/realbigsean/lighthouse into merge-unstable-deneb-june-6th 2023-07-06 16:51:35 -04:00
realbigsean
246d52d209 remove into gossip verified block 2023-07-06 11:40:58 -04:00
realbigsean
c3ef84bda2 remove uninlined arg lint suppression (#4469) 2023-07-05 16:54:08 -04:00
realbigsean
ba65812972 remove patched dependencies (#4470) 2023-07-05 15:53:35 -04:00
realbigsean
af4a66846e remove clang from Dockerfile (#4471) 2023-07-05 15:53:22 -04:00
realbigsean
d41193c318 fix failing network tests (#4472) 2023-07-05 15:52:59 -04:00
realbigsean
d9254b7ded update get blobs endpoint name from blobs to blob_sidecars (#4467)
* changed name

* Fix sidecars

* update get blobs endpoint name from blobs to blob_sidecars

---------

Co-authored-by: Rahul Dogra <rahulcooldogra@gmail.com>
2023-07-05 12:04:12 -04:00
realbigsean
4a79328055 update clang in cross docker (#4451) 2023-06-29 22:40:53 -04:00
realbigsean
adbb62f7f3 Devnet6 (#4404)
* some blob reprocessing work

* remove ForceBlockLookup

* reorder enum match arms in sync manager

* a lot more reprocessing work

* impl logic for triggerng blob lookups along with block lookups

* deal with rpc blobs in groups per block in the da checker. don't cache missing blob ids in the da checker.

* make single block lookup generic

* more work

* add delayed processing logic and combine some requests

* start fixing some compile errors

* fix compilation in main block lookup mod

* much work

* get things compiling

* parent blob lookups

* fix compile

* revert red/stevie changes

* fix up sync manager delay message logic

* add peer usefulness enum

* should remove lookup refactor

* consolidate retry error handling

* improve peer scoring during certain failures in parent lookups

* improve retry code

* drop parent lookup if either req has a peer disconnect during download

* refactor single block processed method

* processing peer refactor

* smol bugfix

* fix some todos

* fix lints

* fix lints

* fix compile in lookup tests

* fix lints

* fix lints

* fix existing block lookup tests

* renamings

* fix after merge

* cargo fmt

* compilation fix in beacon chain tests

* fix

* refactor lookup tests to work with multiple forks and response types

* make tests into macros

* wrap availability check error

* fix compile after merge

* add random blobs

* start fixing up lookup verify error handling

* some bug fixes and the start of deneb only tests

* make tests work for all forks

* track information about peer source

* error refactoring

* improve peer scoring

* fix test compilation

* make sure blobs are sent for processing after stream termination, delete copied tests

* add some tests and fix a bug

* smol bugfixes and moar tests

* add tests and fix some things

* compile after merge

* lots of refactoring

* retry on invalid block/blob

* merge unknown parent messages before current slot lookup

* get tests compiling

* penalize blob peer on invalid blobs

* Check disk on in-memory cache miss

* Update beacon_node/beacon_chain/src/data_availability_checker/overflow_lru_cache.rs

* Update beacon_node/network/src/sync/network_context.rs

Co-authored-by: Divma <26765164+divagant-martian@users.noreply.github.com>

* fix bug in matching blocks and blobs in range sync

* pr feedback

* fix conflicts

* upgrade logs from warn to crit when we receive incorrect response in range

* synced_and_connected_within_tolerance -> should_search_for_block

* remove todo

* add data gas used and update excess data gas to u64

* Fix Broken Overflow Tests

* payload verification with commitments

* fix merge conflicts

* restore payload file

* Restore payload file

* remove todo

* add max blob commitments per block

* c-kzg lib update

* Fix ef tests

* Abstract over minimal/mainnet spec in kzg crate

* Start integrating new KZG

* checkpoint sync without alignment

* checkpoint sync without alignment

* add import

* add import

* query for checkpoint state by slot rather than state root (teku doesn't serve by state root)

* query for checkpoint state by slot rather than state root (teku doesn't serve by state root)

* loosen check

* get state first and query by most recent block root

* Revert "loosen check"

This reverts commit 069d13dd63.

* get state first and query by most recent block root

* merge max blobs change

* simplify delay logic

* rename unknown parent sync message variants

* rename parameter, block_slot -> slot

* add some docs to the lookup module

* use interval instead of sleep

* drop request if blocks and blobs requests both return `None` for `Id`

* clean up `find_single_lookup` logic

* add lookup source enum

* clean up `find_single_lookup` logic

* add docs to find_single_lookup_request

* move LookupSource our of param where unnecessary

* remove unnecessary todo

* query for block by `state.latest_block_header.slot`

* fix lint

* fix merge transition ef tests

* fix test

* fix test

* fix observed  blob sidecars test

* Add some metrics (#33)

* fix protocol limits for blobs by root

* Update Engine API for 1:1 Structure Method

* make beacon chain tests to fix devnet 6 changes

* get ckzg working and fix some tests

* fix remaining tests

* fix lints

* Fix KZG linking issues

* remove unused dep

* lockfile

* test fixes

* remove dbgs

* remove unwrap

* cleanup tx generator

* small fixes

* fixing fixes

* more self reivew

* more self review

* refactor genesis header initialization

* refactor mock el instantiations

* fix compile

* fix network test, make sure they run for each fork

* pr feedback

* fix last test (hopefully)

---------

Co-authored-by: Pawan Dhananjay <pawandhananjay@gmail.com>
Co-authored-by: Mark Mackey <mark@sigmaprime.io>
Co-authored-by: Divma <26765164+divagant-martian@users.noreply.github.com>
Co-authored-by: Michael Sproul <michael@sigmaprime.io>
2023-06-29 15:35:43 -04:00
realbigsean
4c9fcf1e83 Merge pull request #4432 from jimmygchen/deneb-merge-from-unstable-20230627
Deneb merge from unstable 20230627
2023-06-28 10:20:59 -04:00
Jimmy Chen
5b85aeca5f Add BlobSidecar encode & decode test and fix RpcLimit for BlobsByRoot 2023-06-28 22:42:30 +10:00
Jimmy Chen
03a17a84da Update handle_rpc_response blobs match arms to be consistent with block v2 protocols. 2023-06-28 16:56:52 +10:00
Jimmy Chen
68140fa036 Update max block request limit to MAX_REQUEST_BLOCKS_DENEB to ensure this doesn't cause incompatibilities with other clients. 2023-06-28 16:46:20 +10:00
Jimmy Chen
5c4485e45e Merge branch 'deneb-merge-from-unstable-20230627' of github.com:jimmygchen/lighthouse into deneb-merge-from-unstable-20230627 2023-06-28 16:26:38 +10:00
Jimmy Chen
d1146ec8b5 Sync finalized sync to 2 epochs + 1 slot past our peer's finalized slot in order to finalize the chain locally 2023-06-28 16:15:37 +10:00
Jimmy Chen
dfbe4b1add Add missing Cargo.lock changes (ssz_types patch) 2023-06-28 16:11:38 +10:00
Jimmy Chen
56caccbac0 Added a few fixes from merge 2023-06-27 17:48:50 +10:00
Jimmy Chen
cc03ba430c Merge branch 'unstable' into deneb-merge-from-unstable-20230627
# Conflicts:
#	Cargo.lock
#	common/eth2_network_config/built_in_network_configs/gnosis/config.yaml
2023-06-27 15:30:44 +10:00
Jimmy Chen
d062f61125 Fix failing tests after merge 2023-06-27 15:27:42 +10:00
Jimmy Chen
97c4660761 Merge branch 'unstable' into deneb-merge-from-unstable-20230627
# Conflicts:
#	beacon_node/beacon_chain/src/beacon_chain.rs
#	beacon_node/beacon_chain/src/block_verification.rs
#	beacon_node/beacon_chain/src/lib.rs
#	beacon_node/beacon_chain/src/test_utils.rs
#	beacon_node/beacon_chain/tests/block_verification.rs
#	beacon_node/beacon_chain/tests/store_tests.rs
#	beacon_node/beacon_chain/tests/tests.rs
#	beacon_node/http_api/src/publish_blocks.rs
#	beacon_node/lighthouse_network/src/rpc/codec/ssz_snappy.rs
#	beacon_node/lighthouse_network/src/rpc/methods.rs
#	beacon_node/lighthouse_network/src/rpc/outbound.rs
#	beacon_node/lighthouse_network/src/rpc/protocol.rs
#	beacon_node/lighthouse_network/src/service/api_types.rs
#	beacon_node/network/src/beacon_processor/worker/gossip_methods.rs
#	beacon_node/network/src/beacon_processor/worker/rpc_methods.rs
#	beacon_node/network/src/beacon_processor/worker/sync_methods.rs
#	beacon_node/network/src/sync/block_lookups/single_block_lookup.rs
#	beacon_node/network/src/sync/network_context.rs
#	beacon_node/network/src/sync/range_sync/batch.rs
#	beacon_node/network/src/sync/range_sync/chain.rs
#	common/eth2/src/types.rs
#	consensus/fork_choice/src/fork_choice.rs
2023-06-27 08:40:24 +10:00
ethDreamer
e1af24b470 Add more comments to overflow LRU cache (#4406) 2023-06-16 09:27:36 -04:00
realbigsean
a62e52f319 Single blob lookups (#4152)
* some blob reprocessing work

* remove ForceBlockLookup

* reorder enum match arms in sync manager

* a lot more reprocessing work

* impl logic for triggerng blob lookups along with block lookups

* deal with rpc blobs in groups per block in the da checker. don't cache missing blob ids in the da checker.

* make single block lookup generic

* more work

* add delayed processing logic and combine some requests

* start fixing some compile errors

* fix compilation in main block lookup mod

* much work

* get things compiling

* parent blob lookups

* fix compile

* revert red/stevie changes

* fix up sync manager delay message logic

* add peer usefulness enum

* should remove lookup refactor

* consolidate retry error handling

* improve peer scoring during certain failures in parent lookups

* improve retry code

* drop parent lookup if either req has a peer disconnect during download

* refactor single block processed method

* processing peer refactor

* smol bugfix

* fix some todos

* fix lints

* fix lints

* fix compile in lookup tests

* fix lints

* fix lints

* fix existing block lookup tests

* renamings

* fix after merge

* cargo fmt

* compilation fix in beacon chain tests

* fix

* refactor lookup tests to work with multiple forks and response types

* make tests into macros

* wrap availability check error

* fix compile after merge

* add random blobs

* start fixing up lookup verify error handling

* some bug fixes and the start of deneb only tests

* make tests work for all forks

* track information about peer source

* error refactoring

* improve peer scoring

* fix test compilation

* make sure blobs are sent for processing after stream termination, delete copied tests

* add some tests and fix a bug

* smol bugfixes and moar tests

* add tests and fix some things

* compile after merge

* lots of refactoring

* retry on invalid block/blob

* merge unknown parent messages before current slot lookup

* get tests compiling

* penalize blob peer on invalid blobs

* Check disk on in-memory cache miss

* Update beacon_node/beacon_chain/src/data_availability_checker/overflow_lru_cache.rs

* Update beacon_node/network/src/sync/network_context.rs

Co-authored-by: Divma <26765164+divagant-martian@users.noreply.github.com>

* fix bug in matching blocks and blobs in range sync

* pr feedback

* fix conflicts

* upgrade logs from warn to crit when we receive incorrect response in range

* synced_and_connected_within_tolerance -> should_search_for_block

* remove todo

* Fix Broken Overflow Tests

* fix merge conflicts

* checkpoint sync without alignment

* add import

* query for checkpoint state by slot rather than state root (teku doesn't serve by state root)

* get state first and query by most recent block root

* simplify delay logic

* rename unknown parent sync message variants

* rename parameter, block_slot -> slot

* add some docs to the lookup module

* use interval instead of sleep

* drop request if blocks and blobs requests both return `None` for `Id`

* clean up `find_single_lookup` logic

* add lookup source enum

* clean up `find_single_lookup` logic

* add docs to find_single_lookup_request

* move LookupSource our of param where unnecessary

* remove unnecessary todo

* query for block by `state.latest_block_header.slot`

* fix lint

* fix test

* fix test

* fix observed  blob sidecars test

* PR updates

* use optional params instead of a closure

* create lookup and trigger request in separate method calls

* remove `LookupSource`

* make sure duplicate lookups are not dropped

---------

Co-authored-by: Pawan Dhananjay <pawandhananjay@gmail.com>
Co-authored-by: Mark Mackey <mark@sigmaprime.io>
Co-authored-by: Divma <26765164+divagant-martian@users.noreply.github.com>
2023-06-15 12:59:10 -04:00
realbigsean
5428e68943 Merge pull request #4392 from jimmygchen/deneb-merge-from-unstable-20230613
Deneb merge from unstable 20230613
2023-06-13 12:23:04 -04:00
Jimmy Chen
801bf5951c Merge branch 'unstable' into deneb-merge-from-unstable-20230613 2023-06-13 09:44:18 +10:00
Pawan Dhananjay
0a2a00a527 Update max blobs per block (#4391)
* Change max blobs to 6 in code

* Rename

* fmt
2023-06-12 16:33:48 -04:00
realbigsean
ec1b36474b update automatic docker builds for the correct branch (#4375) 2023-06-05 10:41:11 -04:00
realbigsean
72d6cda11b Revert "Temporarily allow Rust check warnings on 4844 branch. (#4088)" (#4373)
This reverts commit fa9baab0f7.
2023-06-05 09:54:19 -04:00
ethDreamer
aef232cb20 Remove Unnecessary Option in Blob Pruning (#4363) 2023-06-05 09:11:18 -04:00
ethDreamer
ceaa740841 Validate & Store Blobs During Backfill (#4307)
* Verify and Store Blobs During Backfill

* Improve logs

* Eliminated Clone

* Fix Inital Vector Capacity

* Addressed Sean's Comments
2023-06-05 09:09:42 -04:00
realbigsean
7a4be59884 Merge pull request #4367 from realbigsean/merge-unstable-to-deneb
Merge unstable to deneb
2023-06-02 12:12:36 -04:00
realbigsean
6adb68c17a fix compile after merge 2023-06-02 12:10:01 -04:00
realbigsean
a227959298 Merge branch 'unstable' of https://github.com/sigp/lighthouse into deneb-free-blobs 2023-06-02 11:57:15 -04:00
realbigsean
e8f1d533fb Merge pull request #4349 from jimmygchen/deneb-merge-from-unstable-20230530
Deneb merge from unstable 2023/05/30
2023-05-31 09:49:29 -04:00
Jimmy Chen
00ffd186c2 Empty commit. 2023-05-31 21:17:46 +10:00
Jimmy Chen
65a2ae38fe Fix failing tests (workaround) 2023-05-31 12:32:31 +10:00
Jimmy Chen
c4063056ac Update testnet ETH1_BLOCK_HASH value 2023-05-31 11:54:56 +10:00
Jimmy Chen
18347231b1 Fix failing tests 2023-05-30 23:31:09 +10:00
Jimmy Chen
81c9af5aaf Use patched versions of common libraries 2023-05-30 22:46:22 +10:00
Jimmy Chen
70c4ae35ab Merge branch 'unstable' into deneb-free-blobs
# Conflicts:
#	.github/workflows/docker.yml
#	.github/workflows/local-testnet.yml
#	.github/workflows/test-suite.yml
#	Cargo.lock
#	Cargo.toml
#	beacon_node/beacon_chain/src/beacon_chain.rs
#	beacon_node/beacon_chain/src/builder.rs
#	beacon_node/beacon_chain/src/test_utils.rs
#	beacon_node/execution_layer/src/engine_api/json_structures.rs
#	beacon_node/network/src/beacon_processor/mod.rs
#	beacon_node/network/src/beacon_processor/worker/gossip_methods.rs
#	beacon_node/network/src/sync/backfill_sync/mod.rs
#	beacon_node/store/src/config.rs
#	beacon_node/store/src/hot_cold_store.rs
#	common/eth2_network_config/Cargo.toml
#	consensus/ssz/src/decode/impls.rs
#	consensus/ssz_derive/src/lib.rs
#	consensus/ssz_derive/tests/tests.rs
#	consensus/ssz_types/src/serde_utils/mod.rs
#	consensus/tree_hash/src/impls.rs
#	consensus/tree_hash/src/lib.rs
#	consensus/types/Cargo.toml
#	consensus/types/src/beacon_state.rs
#	consensus/types/src/chain_spec.rs
#	consensus/types/src/eth_spec.rs
#	consensus/types/src/fork_name.rs
#	lcli/Cargo.toml
#	lcli/src/main.rs
#	lcli/src/new_testnet.rs
#	scripts/local_testnet/el_bootnode.sh
#	scripts/local_testnet/genesis.json
#	scripts/local_testnet/geth.sh
#	scripts/local_testnet/setup.sh
#	scripts/local_testnet/start_local_testnet.sh
#	scripts/local_testnet/vars.env
#	scripts/tests/doppelganger_protection.sh
#	scripts/tests/genesis.json
#	scripts/tests/vars.env
#	testing/ef_tests/Cargo.toml
#	validator_client/src/block_service.rs
2023-05-30 22:44:05 +10:00
realbigsean
9b55d74c1c fix db startup (#4298) 2023-05-16 09:43:26 -04:00
realbigsean
c4b2f1c8ac fix count usage in blobs by range (#4289) 2023-05-15 12:37:51 -04:00
ethDreamer
46db30416d Implement Overflow LRU Cache for Pending Blobs (#4203)
* All Necessary Objects Implement Encode/Decode

* Major Components for LRUOverflowCache Implemented

* Finish Database Code

* Add Maintenance Methods

* Added Maintenance Service

* Persist Blobs on Shutdown / Reload on Startup

* Address Clippy Complaints

* Add (emum_behaviour = "tag") to ssz_derive

* Convert Encode/Decode Implementations to "tag"

* Started Adding Tests

* Added a ton of tests

* 1 character fix

* Feature Guard Minimal Spec Tests

* Update beacon_node/beacon_chain/src/data_availability_checker.rs

Co-authored-by: realbigsean <seananderson33@GMAIL.com>

* Address Sean's Comments

* Add iter_raw_keys method

* Remove TODOs

---------

Co-authored-by: realbigsean <seananderson33@GMAIL.com>
2023-05-12 10:08:24 -04:00
ethDreamer
a22e4bf636 Implement KZG EF Tests (#4274) 2023-05-08 15:58:23 -04:00
realbigsean
9db6b39dc3 fix check on max request size (#4250) 2023-05-02 19:14:02 -04:00
ethDreamer
c1d47da02d Update engine_api to latest version (#4223)
* Update Engine API to Latest

* Get Mock EE Working

* Fix Mock EE

* Update Engine API Again

* Rip out get_blobs_bundle Stuff

* Fix Test Harness

* Fix Clippy Complaints

* Fix Beacon Chain Tests
2023-04-27 14:18:21 -04:00
Justin Traglia
aa34339298 Rename to MIN_EPOCHS_FOR_BLOB_SIDECARS_REQUESTS (#4206)
Co-authored-by: realbigsean <sean@sigmaprime.io>
2023-04-26 14:53:06 -04:00
Pawan Dhananjay
cbe4880490 Fix deneb doppelganger tests (#4124)
* Temp hack to compile

* Fix doppelganger tests

* Kill in groups instead of storing pid

* Install geth in CI

* Install geth first

* Fix eth1_block_hash

* Fix directory paths and block hash

* Fix workflow for local testnets; reset genesis.json after running script

* Disable capella and deneb forks for doppelganger tests

* oops not capella

* Spin up a spare bn for the doppelganger validator

* testing

* Revert "testing"

This reverts commit 14eb178bca.

* Modify beacon_node script to take trusted peers

* Set doppelganger bn as a trusted peer

* Update var

* update another

* Fix port

* Add a flag to disable peer scoring

* Disable peer scoring in local testnet bn script

* Revert trusted peers hack

* fmt

* Fix proposer boost score
2023-04-26 13:26:00 -04:00
Pawan Dhananjay
a632969695 Gossip verification cleanup (#4219)
* Add ObservedBlobSidecar tests

* Add logging for tricky verification cases

* Update beacon_node/beacon_chain/src/blob_verification.rs

---------

Co-authored-by: realbigsean <seananderson33@GMAIL.com>
2023-04-26 10:44:58 -04:00
realbigsean
cbe2e47931 update blobs by range protocol name (#4229) 2023-04-24 09:03:23 -04:00
Pawan Dhananjay
7a36d004e4 Subscribe blob topics (#4224) 2023-04-22 09:21:09 -04:00
Pawan Dhananjay
b6c0e91c05 Merge branch 'eip4844' into deneb-free-blobs 2023-04-21 14:34:50 -07:00
Pawan Dhananjay
b2ccc822d8 Fix compiler issues 2023-04-21 14:14:57 -07:00
Pawan Dhananjay
689c0f76d3 Merge branch 'unstable' into eip4844 2023-04-21 14:13:25 -07:00
Pawan Dhananjay
895bbd6c03 Gossip conditions deneb (#4164)
* Add all gossip conditions

* Handle some gossip errors

* Update beacon_node/beacon_chain/src/blob_verification.rs

Co-authored-by: Divma <26765164+divagant-martian@users.noreply.github.com>

* Add an ObservedBlobSidecars cache

---------

Co-authored-by: Divma <26765164+divagant-martian@users.noreply.github.com>
2023-04-20 18:26:20 -04:00
realbigsean
e6b033aefd update blob transaction (#4218)
* update blob transaction

* update blob transaction

* rename in JSON deserializing
2023-04-20 18:23:59 -04:00
realbigsean
a6335eb27e bump ef tests version (#4217) 2023-04-20 14:00:25 -04:00
Jimmy Chen
2d083436c8 Switch blob tx type to 0x03 (#4186) 2023-04-20 13:58:49 -04:00
Jimmy Chen
9dee718153 Remove unused blob endpoint and types (#4209) 2023-04-20 13:40:10 -04:00
Divma
fca8559acc Update kzg to get windows going, expose blst features (#4177)
* fmt

* update kzg

* use commit from main repo
2023-04-10 19:05:01 -05:00
Diva M
f6f63b18ed Merge branch 'eip4844' into deneb-free-blobs 2023-04-10 10:47:43 -05:00
Diva M
df1da104fd Merge branch 'unstable' into eip4844 2023-04-10 10:46:41 -05:00
Diva M
911a63559b Merge branch 'eip4844' into deneb-free-blobs 2023-04-05 13:33:33 -05:00
Pawan Dhananjay
1b8225c76d Revert upgrade to tokio utils to reprocessing queue (#4167) 2023-04-05 11:43:39 -05:00
Diva M
32f9ba04d7 fix merge conflict 2023-04-04 12:10:51 -05:00
Diva M
cb818152f3 Merge branch 'unstable' into eip4844 2023-04-04 12:07:09 -05:00
Diva M
3c1a22ceaf Merge commit '1e029ce5384e911390a513e2d1885532f34a8b2b' into eip4844 2023-04-04 11:56:54 -05:00
Diva M
9558c18dc5 Merge commit 'c5383e393acee152e92641ce4699d05913953e70' into eip4844 2023-04-04 11:56:01 -05:00
Diva M
905322394b Merge commit '036b797b2c1831352f937356576b3c78c65220ad' into eip4844 2023-04-04 11:53:55 -05:00
ethDreamer
3a21317600 Unified Availability Cache into One (#4161)
* Unified Availability Cache into One

* Update beacon_node/beacon_chain/src/data_availability_checker.rs

Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>

---------

Co-authored-by: realbigsean <seananderson33@GMAIL.com>
Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>
2023-04-04 09:50:35 -04:00
Pawan Dhananjay
ffefd20137 Block processing cleanup (#4153)
* Implements Ord for BlobSidecar based on index

* Use BTreeMap for gossip cache to maintain blob order by index

* fmt

* Another panic fix
2023-04-03 15:07:11 +05:30
realbigsean
deec9c51ba clean up blob by root response (#4136) 2023-03-28 12:49:32 -04:00
realbigsean
d24e5cc22a clean up blobs by range response (#4137) 2023-03-28 12:49:19 -04:00
realbigsean
da7fab5188 Ef test version update (#4142)
* Revert "revert change to ef_tests"

This reverts commit 1093ba1a27.

* Revert "Revert "Use consensus-spec-tests `v1.3.0-rc.3` (#4021)""

This reverts commit 20be7024e1.

* update tests
2023-03-28 11:47:30 -04:00
realbigsean
f580863337 Add simple pruning to data availability checker (#4132)
* use block wrapper in sync pairing

* add pruning to the data availability checker

* remove unused function, rename function
2023-03-27 10:09:53 -04:00
realbigsean
af974dc0b8 use block wrapper in sync pairing (#4131) 2023-03-26 19:18:54 -04:00
realbigsean
a5addf661c Rename eip4844 to deneb (#4129)
* rename 4844 to deneb

* rename 4844 to deneb

* move excess data gas field

* get EF tests working

* fix ef tests lint

* fix the blob identifier ef test

* fix accessed files ef test script

* get beacon chain tests passing
2023-03-26 11:49:16 -04:00
ethDreamer
d84117c0d0 Removed TODO (#4128) 2023-03-24 17:33:49 -04:00
Pawan Dhananjay
b276af98b7 Rework block processing (#4092)
* introduce availability pending block

* add intoavailableblock trait

* small fixes

* add 'gossip blob cache' and start to clean up processing and transition types

* shard memory blob cache

* Initial commit

* Fix after rebase

* Add gossip verification conditions

* cache cleanup

* general chaos

* extended chaos

* cargo fmt

* more progress

* more progress

* tons of changes, just tryna compile

* everything, everywhere, all at once

* Reprocess an ExecutedBlock on unavailable blobs

* Add sus gossip verification for blobs

* Merge stuff

* Remove reprocessing cache stuff

* lint

* Add a wrapper to allow construction of only valid `AvailableBlock`s

* rename blob arc list to blob list

* merge cleanuo

* Revert "merge cleanuo"

This reverts commit 5e98326878.

* Revert "Revert "merge cleanuo""

This reverts commit 3a4009443a.

* fix rpc methods

* move beacon block and blob to eth2/types

* rename gossip blob cache to data availability checker

* lots of changes

* fix some compilation issues

* fix compilation issues

* fix compilation issues

* fix compilation issues

* fix compilation issues

* fix compilation issues

* cargo fmt

* use a common data structure for block import types

* fix availability check on proposal import

* refactor the blob cache and split the block wrapper into two types

* add type conversion for signed block and block wrapper

* fix beacon chain tests and do some renaming, add some comments

* Partial processing (#4)

* move beacon block and blob to eth2/types

* rename gossip blob cache to data availability checker

* lots of changes

* fix some compilation issues

* fix compilation issues

* fix compilation issues

* fix compilation issues

* fix compilation issues

* fix compilation issues

* cargo fmt

* use a common data structure for block import types

* fix availability check on proposal import

* refactor the blob cache and split the block wrapper into two types

* add type conversion for signed block and block wrapper

* fix beacon chain tests and do some renaming, add some comments

* cargo update (#6)

---------

Co-authored-by: realbigsean <sean@sigmaprime.io>
Co-authored-by: realbigsean <seananderson33@gmail.com>
2023-03-24 17:30:41 -04:00
Diva M
25a2d8f078 Merge branch 'eip4844' into deneb-free-blobs 2023-03-24 14:38:29 -05:00
Diva M
1093ba1a27 revert change to ef_tests 2023-03-24 14:32:58 -05:00
Diva M
1b9cfcc11b Merge branch 'unstable' into eip4844 2023-03-24 13:32:50 -05:00
Diva M
7fad926b65 Merge commit '65a5eb829264cb279ed66814c961991ae3a0a04b' into eip4844 2023-03-24 13:24:21 -05:00
ethDreamer
d1e653cfdb Update Blob Storage Structure (#4104)
* Initial Changes to Blob Storage

* Add Arc to SignedBlobSidecar Definition
2023-03-21 15:33:06 -04:00
Jimmy Chen
b40dceaae9 Update get blobs endpoint to return a list of BlobSidecars (#4109)
* Update get blobs endpoint to return BlobSidecarList

* Update code comment

* Update blob retrieval to return BlobSidecarList without Arc

* Remove usage of BlobSidecarList type alias to avoid code conflicts

* Add clippy allow exception
2023-03-21 09:56:32 -05:00
Diva M
78414333a2 Merge branch 'eip4844' into deneb-free-blobs 2023-03-17 16:39:17 -05:00
Diva M
607242c127 Merge branch 'unstable' into eip4844 2023-03-17 16:26:51 -05:00
Jimmy Chen
1301c62436 Validator blob signing for the unblinded flow (#4096)
* Implement validator blob signing (full block and full blob)

* Fix compilation error and remove redundant slot check

* Fix clippy error
2023-03-17 09:29:25 -04:00
Divma
3c18e1a3a4 thread blocks and blobs to sync (#4100)
* thread blocks and blobs to sync

* satisfy dead code analysis
2023-03-16 19:20:39 -05:00
realbigsean
1a87222641 Merge pull request #4094 from realbigsean/free-blob-lints
Free blob lints
2023-03-15 16:56:53 -04:00
realbigsean
cf4285e1d4 compile tests 2023-03-15 16:34:00 -04:00
realbigsean
fb7d729d92 migrate types to API crate 2023-03-15 16:03:36 -04:00
realbigsean
b303d2fb7e lints 2023-03-15 15:32:22 -04:00
Diva M
34cea6d1c3 fmt 2023-03-15 12:37:53 -05:00
realbigsean
91a06ba484 Merge pull request #4083 from jimmygchen/post-block-and-blobs
Implement POST beacon block for EIP-4844
2023-03-15 13:35:37 -04:00
Diva M
4a39e43f96 Merge branch 'eip4844' into deneb-free-blobs 2023-03-15 12:26:30 -05:00
realbigsean
2e075c0a80 Merge pull request #4091 from realbigsean/get-validator-block-and-blobs
Get validator block and blobs
2023-03-15 12:22:03 -04:00
realbigsean
3d99e1f14d move block contents to api crate, rename blob sidecars list 2023-03-15 12:15:08 -04:00
Divma
2c9477de43 Fix block and blob coupling in the network context (#4086)
* update docs

* introduce a temp enum to model an adjusted `BlockWrapper` and fix blob coupling

* fix compilation issue

* fix blob coupling in the network context

* review comments
2023-03-15 11:04:45 -05:00
Jimmy Chen
2ef3ebbef3 Update SignedBlobSidecar container (#4078) 2023-03-15 11:03:56 -05:00
Jimmy Chen
fa9baab0f7 Temporarily allow Rust check warnings on 4844 branch. (#4088) 2023-03-15 09:43:31 -05:00
Jimmy Chen
775ca89801 Add blobs publishing 2023-03-15 15:57:30 +11:00
Jimmy Chen
5887c8fe92 Update commented code to use todo! 2023-03-15 15:29:16 +11:00
Jimmy Chen
02a88f0704 Add KZG proof and blob validation 2023-03-15 15:15:46 +11:00
Jimmy Chen
62627d984c Comment out code that fails to compile 2023-03-15 12:30:59 +11:00
Jimmy Chen
e4608d44ea Merge branch 'deneb-free-blobs' into get-validator-block-and-blobs 2023-03-15 10:55:00 +11:00
Diva M
9974dfe87b fix merge mistake 2023-03-14 14:45:17 -05:00
Diva M
7f2e9b80bb Merge branch 'unstable' into eip4844 2023-03-14 12:00:32 -05:00
Pawan Dhananjay
76f49bdb44 Update kzg interface (#4077)
* Update kzg interface

* Update utils

* Update dependency

* Address review comments
2023-03-14 12:13:15 +05:30
Jimmy Chen
a8978a5f69 Implement publish block and blobs endpoint (WIP) 2023-03-14 17:03:45 +11:00
Jimmy Chen
9ba390f7c5 Merge branch 'update-signed-blob-sidecar' into post-block-and-blobs 2023-03-14 11:52:25 +11:00
Jimmy Chen
e81a5029b8 Update SignedBlobSidecar container 2023-03-14 11:48:58 +11:00
Jimmy Chen
6ec0ce6070 Implement get validator block endpoint for EIP-4844 2023-03-13 16:50:08 +11:00
Diva M
0ae3078988 bump up recursion limits in the sim bin 2023-03-10 17:51:35 -05:00
Diva M
ea23d55b72 more lints 2023-03-10 15:05:08 -05:00
Diva M
9e30c3c169 new lints 2023-03-10 14:35:43 -05:00
Diva M
61b12c2eff bump up recursion limits by a lot 2023-03-10 14:26:15 -05:00
Diva M
ae3e5f73d6 fmt 2023-03-10 11:24:22 -05:00
Diva M
fe421f7987 Merge branch 'eip4844' into deneb-free-blobs 2023-03-10 11:21:23 -05:00
Diva M
203a9e5f5e Merge branch 'unstable' into eip4844 2023-03-10 11:19:56 -05:00
Divma
140bdd370d update code paths in the network crate (#4065)
* wip

* fix router

* arc the byroot responses we send

* add placeholder for blob verification

* respond to blobs by range and blobs by root request in the most horrible and gross way ever

* everything in sync is now unimplemented

* fix compiation issues

* http_pi change is very small, just add it

* remove ctrl-c ctrl-v's docs
2023-03-10 16:52:31 +05:30
Divma
3898cf7be8 Modify lighthouse_network gossip types to free the blobs (#4064)
* Modify blob topics

* add signedblol type

pubsun messages are signed

* improve subnet topic index

* improve display code

* fix parse code

---------

Co-authored-by: Pawan Dhananjay <pawandhananjay@gmail.com>
2023-03-10 16:23:36 +05:30
Divma
545532a883 fix rpc types to free the blobs (#4059)
* rename to follow name in spec

* use roots and indexes

* wip

* fix req/resp types

* move blob identifier to consensus types
2023-03-07 16:28:45 -05:00
Diva M
63011c5bca fmt 2023-03-06 18:09:31 -05:00
Diva M
75a0d52ee0 get back the Blobs pub use to fix compilation issues 2023-03-06 17:58:00 -05:00
Diva M
bf40acd9df adjust constant to spec values and names 2023-03-06 17:32:40 -05:00
Pawan Dhananjay
39d4f0a1f3 Add BlobSidecar type 2023-03-06 17:10:12 -05:00
Diva M
91a8d4575d add missing eip4844 core topics placeholder 2023-03-06 14:35:52 -05:00
Diva M
56841bcfcf fix gitignore 2023-03-03 14:31:33 -05:00
Diva M
f16e82ab2c Merge branch 'unstable' into eip4844 2023-03-03 14:14:18 -05:00
Diva M
6fb1681ee7 Merge branch 'unstable-up-to-clean-capella' into eip4844 2023-03-03 14:13:29 -05:00
Diva M
a3862e42b1 Revert "Clean capella (#4019)"
This reverts commit 047c7544e3.
2023-03-03 14:13:03 -05:00
Diva M
f405feb2cd define missing vars 2023-03-03 13:16:27 -05:00
Diva M
e602b94b8d define CAPELLA_FORK_EPOCH in scripts/tests/vars.env with same value as ALTAIR_FORK_EPOCH 2023-03-03 12:41:36 -05:00
Diva M
74542843a7 define BELLATRIX_FORK_EPOCH in scripts/tests/vars.env with same value as ALTAIR_FORK_EPOCH 2023-03-03 12:07:04 -05:00
Diva M
20be7024e1 Revert "Use consensus-spec-tests v1.3.0-rc.3 (#4021)"
This reverts commit caa6190d4a.
2023-03-03 11:07:26 -05:00
Diva M
a9be1eae40 solve couple of merge conflicts 2023-03-02 15:46:14 -05:00
Diva M
d93753cc88 Merge branch 'unstable' into off-4844 2023-03-02 15:38:00 -05:00
realbigsean
55753f8bc8 bump recursion limit 2023-02-15 16:32:50 -05:00
realbigsean
ca8e341649 fix compilation after merge 2023-02-15 14:30:39 -05:00
realbigsean
8320b918ae merge self limiter 2023-02-15 14:26:18 -05:00
realbigsean
4d0b0f681d merge self limiter 2023-02-15 14:25:58 -05:00
realbigsean
b805fa6279 merge with upstream 2023-02-15 14:20:12 -05:00
realbigsean
87d1fbeb21 Merge pull request #3905 from emhane/beacon_chain_tests
Debug CI
2023-02-15 11:53:41 -05:00
Emilia Hane
aaf6404d4f Remove unused generic 2023-02-15 17:45:22 +01:00
Emilia Hane
2672cf40bb Better fix for debug tests 2023-02-15 11:47:56 +01:00
Emilia Hane
9fea440ae6 Fix lint 2023-02-15 09:54:36 +01:00
Michael Sproul
fd379ae2e2 Upgrade sqlite3 2023-02-15 09:44:37 +01:00
realbigsean
44dbccfeae add v3 to capabilities 2023-02-15 09:23:59 +01:00
Emilia Hane
13efd47238 fixup! Disable use of system time in tests 2023-02-15 09:20:30 +01:00
Emilia Hane
9e4abc79fb Comment out tests that use system time 2023-02-14 14:12:50 +01:00
Emilia Hane
73c7ad73b8 Disable use of system time in tests 2023-02-14 13:33:38 +01:00
Emilia Hane
148385eb70 Remove unused error 2023-02-14 12:43:13 +01:00
Emilia Hane
810d875b02 Fix merge conflicts with eip4844 2023-02-14 12:42:41 +01:00
Emilia Hane
8200d37045 Merge pull request #2 from realbigsean/sean-debug-ci
Debug CI Sean's PR feedback
2023-02-14 12:24:15 +01:00
realbigsean
d2ecbd942e fix a couple new lints 2023-02-13 17:13:47 -05:00
realbigsean
cd8757de1c Revert "make batch size check compile time panic"
This reverts commit 68f2484efc.
2023-02-13 16:51:55 -05:00
realbigsean
68f2484efc make batch size check compile time panic 2023-02-13 16:51:46 -05:00
realbigsean
4c3561dcaf make batch size check compile time panic 2023-02-13 16:50:33 -05:00
realbigsean
8f9c5cfca9 remove unused structs 2023-02-13 16:47:36 -05:00
realbigsean
ad9af6d8b1 complete match for has_context_bytes 2023-02-13 16:44:54 -05:00
realbigsean
fc2d07b4e3 allow unused 2023-02-13 16:36:38 -05:00
realbigsean
28702c9d5d merge upstream, add back get_blobs logic 2023-02-13 16:29:21 -05:00
realbigsean
e58d7e85bf Merge pull request #3951 from realbigsean/fix-blob-tx-ssz
Encode blob transactions as signed blob transactions
2023-02-13 14:50:44 -05:00
realbigsean
e1cb4b8a11 Merge pull request #3869 from emhane/blobs_freezer
Blobs freezer
2023-02-13 14:40:02 -05:00
Emilia Hane
c6dfa7a1ac fixup! Add tests for blob pruning flags 2023-02-10 21:15:57 +01:00
Emilia Hane
d3a09af7f7 Run cargo update 2023-02-10 16:29:36 +01:00
Emilia Hane
28e9f07746 Fix lint for prune blobs pr 2023-02-10 16:23:04 +01:00
Emilia Hane
d9eed481b7 fixup! Add tests for blob pruning flags 2023-02-10 16:18:39 +01:00
Emilia Hane
a1768b16b9 fixup! Update dependencies (#3946) 2023-02-10 15:43:40 +01:00
Michael Sproul
d890f2bf6b Update dependencies (#3946)
Resolves the cargo-audit failure caused by https://rustsec.org/advisories/RUSTSEC-2023-0010.

I also removed the ignore for `RUSTSEC-2020-0159` as we are no longer using a vulnerable version of `chrono`. We still need the other ignore for `time 0.1` because we depend on it via `sloggers -> chrono -> time 0.1`.
2023-02-10 15:35:01 +01:00
Emilia Hane
3dd42e5723 Remove unused dependencies 2023-02-10 15:35:01 +01:00
Emilia Hane
0104d6143c fixup! Fix latest clippy lints 2023-02-10 15:35:01 +01:00
Emilia Hane
02cca3478b Fix conflicts rebasing eip4844 2023-02-10 15:35:01 +01:00
Michael Sproul
1f3eef2c5f Unpin fixed-hash (#3917)
## Proposed Changes
Remove the `[patch]` for `fixed-hash`.

We pinned it years ago in #2710 to fix `arbitrary` support. Nowadays the 0.7 version of `fixed-hash` is only used by the `web3` crate and doesn't need `arbitrary`.

~~Blocked on #3916 but could be merged in the same Bors batch.~~
2023-02-10 15:35:01 +01:00
Emilia Hane
615402abcf fixup! Fix conflicts rebasing eip4844 2023-02-10 15:35:00 +01:00
Emilia Hane
db36eb978b Fix latest clippy lints 2023-02-10 15:35:00 +01:00
Emilia Hane
2653f88b5f Fix conflicts rebasing eip4844 2023-02-10 15:35:00 +01:00
Emilia Hane
c7b49feb9e fixup! Change CI clippy 2023-02-10 15:35:00 +01:00
Emilia Hane
e0a9cd6b84 Change CI clippy 2023-02-10 15:35:00 +01:00
Emilia Hane
43bf908e7a Fix release tests 2023-02-10 15:34:59 +01:00
Emilia Hane
481718856c Fix clippy 2023-02-10 15:34:59 +01:00
Emilia Hane
4d3ff347a3 Fixes after rebasing eip4844 2023-02-10 15:34:58 +01:00
Emilia Hane
5437dcae9c Fix conflicts rebasing eip4844 2023-02-10 15:34:58 +01:00
Emilia Hane
7545ae9e9b fixup! Fix block lookup debug tests 2023-02-10 15:34:46 +01:00
realbigsean
a68e3eac2c pr feedback 2023-02-10 08:25:42 -05:00
Emilia Hane
6beca6defc Fix range sync tests 2023-02-10 09:41:24 +01:00
Emilia Hane
e9e198a2b6 Fix conflicts rebasing eip4844 2023-02-10 09:41:23 +01:00
Emilia Hane
d292a3a6a8 Fix conflicts rebasing eip4844 2023-02-10 09:41:23 +01:00
Emilia Hane
994990063a Fix weak_subjectivity_sync test 2023-02-10 09:41:23 +01:00
Emilia Hane
546d63f83c Fix rebase conflicts 2023-02-10 09:41:23 +01:00
Emilia Hane
50e01bef1f Add eip4844 fork to tests 2023-02-10 09:41:22 +01:00
Emilia Hane
09370e70d9 Fix rebase conflicts 2023-02-10 09:41:19 +01:00
Emilia Hane
69c30bb6eb Fix release test 2023-02-10 09:39:22 +01:00
Emilia Hane
8365d76277 fixup! Debug tests 2023-02-10 09:39:22 +01:00
Emilia Hane
16cb9cfca2 fixup! Debug tests 2023-02-10 09:39:22 +01:00
Emilia Hane
7220f35ff6 Debug tests 2023-02-10 09:39:21 +01:00
Emilia Hane
995b2715f2 Fix network block_lookups test 2023-02-10 09:39:21 +01:00
Emilia Hane
3676ce78b5 Fix rebase conflicts 2023-02-10 09:39:21 +01:00
Emilia Hane
12720f9ac5 fixup! Help user choose blobs db 2023-02-09 10:37:53 +01:00
Emilia Hane
1300fb7ffa Fix conflicts from rebasing eip4844 2023-02-09 10:37:11 +01:00
Emilia Hane
290e1d2ff7 fixup! Complete making blocks and blobs db atomic 2023-02-09 07:50:57 +01:00
Emilia Hane
38fe2dce3f fixup! Complete making blocks and blobs db atomic 2023-02-09 07:50:55 +01:00
Emilia Hane
ca934b7cb5 Fix rebase conflicts 2023-02-09 07:50:30 +01:00
Emilia Hane
72cd68c0a4 Complete making blocks and blobs db atomic 2023-02-09 07:46:27 +01:00
Emilia Hane
89cccfc397 Fix rebase conflicts 2023-02-09 07:46:25 +01:00
Emilia Hane
ba882958ed Delete blobs along with block 2023-02-09 07:42:46 +01:00
Emilia Hane
04fafebfa6 fixup! Throw error when params don't match with previous run 2023-02-09 07:42:46 +01:00
Emilia Hane
00ce8d9572 Throw error when params don't match with previous run 2023-02-09 07:42:46 +01:00
Emilia Hane
d8e501d3ab Add todos 2023-02-09 07:42:43 +01:00
Emilia Hane
f971f3a3a2 Fix rebase conflicts 2023-02-09 07:41:38 +01:00
Emilia Hane
f8c3e7fc91 Lint fix 2023-02-09 07:36:11 +01:00
Emilia Hane
7f91dd803c Help user choose blobs db 2023-02-09 07:36:11 +01:00
Emilia Hane
22915c2d7e fixup! Store blobs in correct db for atomic ops 2023-02-09 07:36:10 +01:00
Emilia Hane
dcb5495745 Store blobs in correct db for atomic ops 2023-02-09 07:36:10 +01:00
Emilia Hane
625980e484 Fix rebase conflicts 2023-02-09 07:36:07 +01:00
Emilia Hane
04f635c0ac Remove IDE file 2023-02-09 07:35:47 +01:00
Emilia Hane
3679a0f1cb Improve syntax 2023-02-09 07:35:47 +01:00
Emilia Hane
3c0aa201e3 fixup! Help user configure blobs freezer correctly between start ups 2023-02-09 07:35:47 +01:00
Emilia Hane
0ba0775812 Help user configure blobs freezer correctly between start ups 2023-02-09 07:35:45 +01:00
Emilia Hane
05c51b37b1 fix rebase conflicts 2023-02-09 07:35:10 +01:00
Emilia Hane
e0b1a0841c fixup! Store blobs in separate freezer or historical state freezer 2023-02-09 07:35:10 +01:00
Emilia Hane
f9737628fc Store blobs in separate freezer or historical state freezer 2023-02-09 07:34:59 +01:00
realbigsean
41567194e9 Merge pull request #3852 from emhane/prune_blobs
Prune blobs
2023-02-08 13:39:26 -05:00
realbigsean
99da11e9f4 fix lints 2023-02-08 11:03:34 -05:00
realbigsean
902f64a946 remove clone of access lists 2023-02-08 10:59:48 -05:00
realbigsean
dd40adc5c0 check byte length when converting to uint256 and hash256 from bytes. Add comments 2023-02-08 10:38:45 -05:00
Emilia Hane
6a37e84399 fixup! Fix regression in DB write atomicity 2023-02-08 11:44:46 +01:00
Emilia Hane
bc468b4ce5 fixup! Improve use of whitespace 2023-02-08 11:44:45 +01:00
Michael Sproul
ac4b5b580c Fix regression in DB write atomicity 2023-02-08 11:44:45 +01:00
Emilia Hane
9d919917f5 Removed unused code 2023-02-08 11:44:45 +01:00
Emilia Hane
d7eb9441cf Reorder loading of db metadata from disk to allow for future changes to schema 2023-02-08 11:44:45 +01:00
Emilia Hane
d599e41f3d Remove debug comment
Co-authored-by: Michael Sproul <micsproul@gmail.com>
2023-02-08 11:44:44 +01:00
Emilia Hane
577262ccbf Improve use of whitespace
Co-authored-by: Michael Sproul <micsproul@gmail.com>
2023-02-08 11:44:44 +01:00
Emilia Hane
56c84178f2 Fix conflicts rebasing eip4844 2023-02-08 11:44:44 +01:00
Emilia Hane
b2abec5d35 Verify StoreConfig 2023-02-08 11:44:44 +01:00
Emilia Hane
00ca21e84c Make implementation of BlobInfo more coder friendly 2023-02-08 11:44:43 +01:00
Emilia Hane
8f137df02e fixup! Allow user to set an epoch margin for pruning 2023-02-08 11:44:43 +01:00
Emilia Hane
a2eda76291 Correct comment 2023-02-08 11:44:43 +01:00
Emilia Hane
1e59cb9dea Add tests for blob pruning flags 2023-02-08 11:44:43 +01:00
Emilia Hane
9ee9b6df76 Remove unused stuff 2023-02-08 11:44:42 +01:00
Emilia Hane
6dff69bde9 Atomically update blob info with pruned blobs 2023-02-08 11:44:42 +01:00
Emilia Hane
5d2480c762 Improve naming 2023-02-08 11:44:42 +01:00
Emilia Hane
9c2e623555 Reflect use of prune margin epochs at import 2023-02-08 11:44:42 +01:00
Emilia Hane
d4795601f2 fixup! Prune from highest data availability boundary 2023-02-08 11:44:41 +01:00
Emilia Hane
43c3c74a48 fixup! Fix blobs store bug 2023-02-08 11:44:41 +01:00
Emilia Hane
63ca3bfb29 Prune from highest data availability boundary 2023-02-08 11:44:41 +01:00
Emilia Hane
c50f83116e Fix wording
Co-authored-by: Michael Sproul <micsproul@gmail.com>
2023-02-08 11:44:41 +01:00
Emilia Hane
f6346f89c1 Clarify comment
Co-authored-by: Michael Sproul <micsproul@gmail.com>
2023-02-08 11:44:41 +01:00
Emilia Hane
e4b447395a Clarify wording
Co-authored-by: Michael Sproul <micsproul@gmail.com>
2023-02-08 11:44:40 +01:00
Emilia Hane
756c881857 Keep uniform size small keys
Co-authored-by: Michael Sproul <micsproul@gmail.com>
2023-02-08 11:44:40 +01:00
Emilia Hane
4de523fb75 fixup! Allow user to set an epoch margin for pruning 2023-02-08 11:44:40 +01:00
Emilia Hane
1812301c9c Allow user to set an epoch margin for pruning 2023-02-08 11:44:40 +01:00
Emilia Hane
d7fc24a9d5 Plug in running blob pruning in migrator, related bug fixes and add todos 2023-02-08 11:44:40 +01:00
Emilia Hane
d1b75e281f Fix typo 2023-02-08 11:44:39 +01:00
Emilia Hane
0bdc291490 Only store non-empty orphaned blobs 2023-02-08 11:44:39 +01:00
Emilia Hane
caa04db58a Run prune blobs on migrator thread 2023-02-08 11:44:39 +01:00
Emilia Hane
a875bec5f2 Fix blobs store bug 2023-02-08 11:44:39 +01:00
Emilia Hane
3bede06c9b Fix typo 2023-02-08 11:44:38 +01:00
Emilia Hane
54699f808c fixup! Clarify hybrid blob prune solution and fix error handling 2023-02-08 11:44:38 +01:00
Emilia Hane
83a9520761 Clarify hybrid blob prune solution and fix error handling 2023-02-08 11:44:38 +01:00
Emilia Hane
74172ed160 Ignore IDE file 2023-02-08 11:44:38 +01:00
Emilia Hane
3d93dad0e2 Fix type bug
Co-authored-by: realbigsean <seananderson33@GMAIL.com>
2023-02-08 11:44:37 +01:00
Emilia Hane
44ec331452 fixup! Simplify conceptual design 2023-02-08 11:44:37 +01:00
Emilia Hane
20567750c1 fixup! Simplify conceptual design 2023-02-08 11:44:37 +01:00
Emilia Hane
7103a257ce Simplify conceptual design 2023-02-08 11:44:37 +01:00
Emilia Hane
0d13932663 Fix epoch constructor misconception 2023-02-08 11:44:37 +01:00
Emilia Hane
b5abfe620a Convert epochs_per_blob_prune to Epoch once 2023-02-08 11:44:36 +01:00
Emilia Hane
fb2ce909f6 Avoid repeteadly updating blob info for multiple head candidates 2023-02-08 11:44:36 +01:00
Emilia Hane
d58a30b3de fixup! Store orphan block roots 2023-02-08 11:44:36 +01:00
Emilia Hane
6346c30158 Enable skipping blob pruning at each epoch 2023-02-08 11:44:35 +01:00
Emilia Hane
2f565d25b2 Prune blobs in bg after canonical head update 2023-02-08 11:44:35 +01:00
Emilia Hane
8752deeced Store orphan block roots 2023-02-08 11:44:35 +01:00
Emilia Hane
c7f53a9062 Delete blobs that conflict with finalization 2023-02-08 11:44:34 +01:00
Emilia Hane
94aa2cef67 Log info loaded from disk 2023-02-08 11:44:34 +01:00
Emilia Hane
a2b8c6ee69 Save fetching state for blobs pruning 2023-02-08 11:44:33 +01:00
Emilia Hane
6f5ca02ac9 Improve syntax
Co-authored-by: Michael Sproul <micsproul@gmail.com>
2023-02-08 11:44:33 +01:00
Emilia Hane
667cca5cf2 Fix try_prune_blobs to use state root 2023-02-08 11:44:33 +01:00
Emilia Hane
d67468d737 Prune blobs on migration in addition to start-up 2023-02-08 11:44:32 +01:00
Emilia Hane
82ffec378a Fix typo
Co-authored-by: realbigsean <seananderson33@GMAIL.com>
2023-02-08 11:44:32 +01:00
Emilia Hane
ce2db355de Fix rebase conflict 2023-02-08 11:44:32 +01:00
Emilia Hane
a211e6afee Fix rebase conflict 2023-02-08 11:44:31 +01:00
Emilia Hane
28e1e635c3 Fix rebase conflict 2023-02-08 11:44:31 +01:00
Emilia Hane
d3b94d8617 fixup! Prune blobs before data availability breakpoint 2023-02-08 11:44:31 +01:00
Emilia Hane
934f3ab587 Remove inaccurate guess for db index 2023-02-08 11:44:31 +01:00
Emilia Hane
d21c66ddf4 fixup! Plug in pruning of blobs into app 2023-02-08 11:44:31 +01:00
Emilia Hane
b88d888145 fixup! Plug in pruning of blobs into app 2023-02-08 11:44:30 +01:00
Emilia Hane
2a41f25d68 fixup! Prune blobs before data availability breakpoint 2023-02-08 11:44:30 +01:00
Emilia Hane
fe0c911402 Plug in pruning of blobs into app 2023-02-08 11:44:30 +01:00
Emilia Hane
7bf88c2336 Prune blobs before data availability breakpoint 2023-02-08 11:44:30 +01:00
Emilia Hane
e2a6da4274 Boiler plate code for blobs pruning 2023-02-08 11:44:20 +01:00
realbigsean
8661477675 use hex decode instead of parse 2023-02-07 21:51:19 -05:00
Diva M
493784366f self rate limiting 2023-02-07 13:00:35 -05:00
realbigsean
a42d07592c fix compilation issues after merge 2023-02-07 12:33:29 -05:00
realbigsean
26a296246d Merge branch 'capella' of https://github.com/sigp/lighthouse into eip4844
# Conflicts:
#	beacon_node/beacon_chain/src/beacon_chain.rs
#	beacon_node/beacon_chain/src/block_verification.rs
#	beacon_node/beacon_chain/src/test_utils.rs
#	beacon_node/execution_layer/src/engine_api.rs
#	beacon_node/execution_layer/src/engine_api/http.rs
#	beacon_node/execution_layer/src/lib.rs
#	beacon_node/execution_layer/src/test_utils/handle_rpc.rs
#	beacon_node/http_api/src/lib.rs
#	beacon_node/http_api/tests/fork_tests.rs
#	beacon_node/network/src/beacon_processor/mod.rs
#	beacon_node/network/src/beacon_processor/work_reprocessing_queue.rs
#	beacon_node/network/src/beacon_processor/worker/sync_methods.rs
#	beacon_node/operation_pool/src/bls_to_execution_changes.rs
#	beacon_node/operation_pool/src/lib.rs
#	beacon_node/operation_pool/src/persistence.rs
#	consensus/serde_utils/src/u256_hex_be_opt.rs
#	testing/antithesis/Dockerfile.libvoidstar
2023-02-07 12:12:56 -05:00
realbigsean
21e5b7fecd Merge pull request #3941 from realbigsean/blob-decoding
Blob decoding
2023-02-07 08:38:20 -05:00
realbigsean
3533ed418e pr feedback and bugfixes 2023-02-07 08:36:35 -05:00
sean
e5896d9b71 re-order methods 2023-02-05 18:14:24 -05:00
sean
1315098c15 variable list from -> new 2023-02-05 17:56:03 -05:00
sean
38db8d7952 add back in 4844 tx consistencycheck during payload reconstruction 2023-02-05 17:31:20 -05:00
sean
f22aac1603 improve error handling 2023-02-05 17:26:36 -05:00
sean
90e25dc6cf from to new 2023-02-05 16:55:55 -05:00
sean
94a369b9df blob tx decoding 2023-02-05 16:39:15 -05:00
realbigsean
d9e83e6cec blob decoding 2023-02-03 17:06:33 -05:00
realbigsean
6d2dff66a0 Merge pull request #3926 from divagant-martian/stream-timeout-blob-bug
send stream terminators
2023-01-27 19:05:04 +01:00
realbigsean
b7e20fb87a Update beacon_node/lighthouse_network/src/rpc/protocol.rs 2023-01-27 19:03:43 +01:00
Diva M
9976d3bbbc send stream terminators 2023-01-27 18:11:26 +01:00
realbigsean
c0bdc1d120 Merge pull request #3920 from realbigsean/fix-and-loosen-execution-block-decoding
Fix and loosen execution block decoding
2023-01-27 18:10:47 +01:00
realbigsean
37e7c1d5c7 keep verification of payloads pre 4844 2023-01-27 17:59:40 +01:00
realbigsean
2645249eb5 Merge branch 'eip4844' of https://github.com/sigp/lighthouse into fix-and-loosen-execution-block-decoding 2023-01-27 11:54:30 +01:00
realbigsean
ee25c21463 Merge pull request #3921 from realbigsean/cap-4844
Eip4844 capella rebase
2023-01-27 11:51:11 +01:00
realbigsean
2c12200a1a fix compile 2023-01-27 11:41:59 +01:00
realbigsean
dd512cd82a stub out tx root check, fix block hash calculation 2023-01-27 11:39:26 +01:00
realbigsean
7c8d97c06e remove unused import 2023-01-25 14:26:01 +01:00
Michael Sproul
17b6a6094f Update another test broken by the shuffling change 2023-01-25 14:23:35 +01:00
Michael Sproul
494a270279 Fix the new BLS to execution change test 2023-01-25 14:23:35 +01:00
Michael Sproul
550d63f3fe Update sync rewards API for abstract exec payload 2023-01-25 14:23:35 +01:00
GeemoCandama
f857811e5f light client optimistic update reprocessing (#3799)
Currently there is a race between receiving blocks and receiving light client optimistic updates (in unstable), which results in processing errors. This is a continuation of PR #3693 and seeks to progress on issue #3651

Add the parent_root to ReprocessQueueMessage::BlockImported so we can remove blocks from queue when a block arrives that has the same parent root. We use the parent root as opposed to the block_root because the LightClientOptimisticUpdate does not contain the block_root.

If light_client_optimistic_update.attested_header.canonical_root() != head_block.message().parent_root() then we queue the update. Otherwise we process immediately.
michaelsproul came up with this idea.
The code was heavily based off of the attestation reprocessing.
I have not properly tested this to see if it works as intended.
2023-01-25 14:23:33 +01:00
naviechan
9b5c2eefd5 Implement sync_committee_rewards API (per-validator reward) (#3903)
[#3661](https://github.com/sigp/lighthouse/issues/3661)

`/eth/v1/beacon/rewards/sync_committee/{block_id}`

```
{
  "execution_optimistic": false,
  "finalized": false,
  "data": [
    {
      "validator_index": "0",
      "reward": "2000"
    }
  ]
}
```

The issue contains the implementation of three per-validator reward APIs:
* `sync_committee_rewards`
* [`attestation_rewards`](https://github.com/sigp/lighthouse/pull/3822)
* `block_rewards`

This PR only implements the `sync_committe_rewards `.

The endpoints can be viewed in the Ethereum Beacon nodes API browser: [https://ethereum.github.io/beacon-APIs/?urls.primaryName=dev#/Rewards](https://ethereum.github.io/beacon-APIs/?urls.primaryName=dev#/Rewards)

The implementation of [consensus client reward APIs](https://github.com/eth-protocol-fellows/cohort-three/blob/master/projects/project-ideas.md#consensus-client-reward-apis) is part of the [EPF](https://github.com/eth-protocol-fellows/cohort-three).

Co-authored-by: navie <naviechan@gmail.com>
Co-authored-by: kevinbogner <kevbogner@gmail.com>
2023-01-25 14:22:15 +01:00
ethDreamer
eb9da6c837 Use eth1_withdrawal_credentials in Test States (#3898)
* Use eth1_withdrawal_credential in Some Test States

* Update beacon_node/genesis/src/interop.rs

Co-authored-by: Michael Sproul <micsproul@gmail.com>

* Update beacon_node/genesis/src/interop.rs

Co-authored-by: Michael Sproul <micsproul@gmail.com>

* Increase validator sizes

* Pick next sync committee message

Co-authored-by: Michael Sproul <micsproul@gmail.com>
Co-authored-by: Paul Hauner <paul@paulhauner.com>
2023-01-25 14:21:54 +01:00
Michael Sproul
a4cfe50ade Import BLS to execution changes before Capella (#3892)
* Import BLS to execution changes before Capella

* Test for BLS to execution change HTTP API

* Pack BLS to execution changes in LIFO order

* Remove unused var

* Clippy
2023-01-25 14:21:54 +01:00
antondlr
9f2baced0b fix multiarch docker builds (#3904)
## Issue Addressed

#3902 
Tested and confirmed working [here](https://github.com/antondlr/lighthouse/actions/runs/3970418322)

## Additional Info

buildx v0.10.0 added provenance attestations to images but they are packed in a way that's incompatible with `docker manifest`
https://github.com/docker/buildx/releases
2023-01-25 14:21:54 +01:00
Michael Sproul
c2f64f8216 Switch allocator to jemalloc (#3697)
## Proposed Changes

Another `tree-states` motivated PR, this adds `jemalloc` as the default allocator, with an option to use the system allocator by compiling with `FEATURES="" make`.

- [x] Metrics
- [x] Test on Windows
- [x] Test on macOS
- [x] Test with `musl`
- [x] Metrics dashboard on `lighthouse-metrics` (https://github.com/sigp/lighthouse-metrics/pull/37)


Co-authored-by: Michael Sproul <micsproul@gmail.com>
2023-01-25 14:21:54 +01:00
Age Manning
528f7181bc Improve block delay metrics (#3894)
We recently ran a large-block experiment on the testnet and plan to do a further experiment on mainnet.

Although the metrics recovered from lighthouse nodes were quite useful, I think we could do with greater resolution in the block delay metrics and get some specific values for each block (currently these can be lost to large exponential histogram buckets). 

This PR increases the resolution of the block delay histogram buckets, but also introduces a new metric which records the last block delay. Depending on the polling resolution of the metric server, we can lose some block delay information, however it will always give us a specific value and we will not lose exact data based on poor resolution histogram buckets.
2023-01-25 14:21:53 +01:00
realbigsean
8e50d316de update antithesis dockerfile (#3883)
Resolves https://github.com/sigp/lighthouse/issues/3879

Co-authored-by: realbigsean <sean@sigmaprime.io>
2023-01-25 14:21:48 +01:00
aliask
63593ef711 Fix some dead links in markdown files (#3885)
## Issue Addressed

No issue has been raised for these broken links.

## Proposed Changes

Update links with the new URLs for the same document.

## Additional Info

~The link for the [Lighthouse Development Updates](https://eepurl.com/dh9Lvb/) mailing list is also broken, but I can't find the correct link.~


Co-authored-by: Paul Hauner <paul@paulhauner.com>
2023-01-25 14:21:16 +01:00
GeemoCandama
12bdde13fe add logging for starting request and receiving block (#3858)
## Issue Addressed

#3853 

## Proposed Changes

Added `INFO` level logs for requesting and receiving the unsigned block.

## Additional Info

Logging for successfully publishing the signed block is already there. And seemingly there is a log for when "We realize we are going to produce a block" in the `start_update_service`: `info!(log, "Block production service started");
`.  Is there anywhere else you'd like to see logging around this event?


Co-authored-by: GeemoCandama <104614073+GeemoCandama@users.noreply.github.com>
2023-01-25 14:21:16 +01:00
David Theodore
1b6d1a9ef9 add better err reporting UnableToOpenVotingKeystore (#3781)
## Issue Addressed

#3780 

## Proposed Changes

Add error reporting that notifies the node operator that the `voting_keystore_path` in their `validator_definitions.yml` file may be incorrect.

## Additional Info

There is more info in issue #3780 


Co-authored-by: Paul Hauner <paul@paulhauner.com>
2023-01-25 14:21:16 +01:00
Mac L
47ade13ef7 Add CLI flag to specify the format of logs written to the logfile (#3839)
## Proposed Changes

Decouple the stdout and logfile formats by adding the `--logfile-format` CLI flag.
This behaves identically to the existing `--log-format` flag, but instead will only affect the logs written to the logfile.
The `--log-format` flag will no longer have any effect on the contents of the logfile.

## Additional Info
This avoids being a breaking change by causing `logfile-format` to default to the value of `--log-format` if it is not provided. 
This means that users who were previously relying on being able to use a JSON formatted logfile will be able to continue to use `--log-format JSON`. 

Users who want to use JSON on stdout and default logs in the logfile, will need to pass the following flags: `--log-format JSON --logfile-format DEFAULT`
2023-01-25 14:21:16 +01:00
Santiago Medina
bd7bd005ee Return HTTP 404 rather than 405 (#3836)
Issue #3112

Add `Filter::recover` to the GET chain to handle rejections specifically as 404 NOT FOUND

Making a request to `http://localhost:5052/not_real` now returns the following:

```
{
    "code": 404,
    "message": "NOT_FOUND",
    "stacktraces": []
}
```

Co-authored-by: Paul Hauner <paul@paulhauner.com>
2023-01-25 14:21:07 +01:00
Madman600
6511d28f0b Update checkpoint-sync.md (#3831)
Remove infura checkpoint sync instructions.


Co-authored-by: Adam Patacchiola <aep600@gmail.com>
2023-01-25 14:19:48 +01:00
realbigsean
1dd9812f62 Merge pull request #3915 from realbigsean/devnetv4-builtin-config
built in devnetv4 config eip4844
2023-01-25 13:41:09 +01:00
realbigsean
86177312f8 update zip 2023-01-25 11:55:05 +01:00
realbigsean
32b0fb13d2 Merge pull request #3912 from realbigsean/sync-error-handling
Sync error handling
2023-01-25 11:46:59 +01:00
realbigsean
5e8d79891b merge conflict resolution 2023-01-25 11:10:44 +01:00
realbigsean
9fde813a7e Merge pull request #3913 from divagant-martian/reject-pre-fork-rpc-upgrades
only support 4844 rpc methods if on 4844
2023-01-25 10:56:56 +01:00
realbigsean
eabe5dc970 Merge pull request #3910 from realbigsean/block-wrapper-refactor
Block wrapper refactor
2023-01-25 10:56:30 +01:00
realbigsean
d3240c1ffb fix common issue across blocks by range and blobs by range 2023-01-24 15:42:28 +01:00
realbigsean
18d4faf611 review updates 2023-01-24 15:30:29 +01:00
realbigsean
2225e6ac89 pass in data availability boundary to the get_blobs method 2023-01-24 14:35:07 +01:00
realbigsean
a4ea1761bb Update beacon_node/beacon_chain/src/beacon_chain.rs 2023-01-24 12:28:58 +01:00
realbigsean
5b4cd997d0 Update beacon_node/lighthouse_network/src/rpc/methods.rs 2023-01-24 12:20:40 +01:00
Diva M
2d2da92132 only support 4844 rpc methods if on 4844 2023-01-24 05:15:23 -05:00
realbigsean
b658cc7aaf simplify checking attester cache for block and blobs. use ResourceUnavailable according to the spec 2023-01-24 10:50:47 +01:00
Emilia Hane
e14550425d Fix mismatched response bug 2023-01-23 13:23:04 +01:00
realbigsean
4a51f65ce2 move available block comment 2023-01-23 09:17:57 +01:00
realbigsean
75320ff8bc cleanup 2023-01-22 05:54:25 +01:00
Emilia Hane
81a754577d fixup! Improve error handling 2023-01-21 15:47:33 +01:00
Emilia Hane
f32f08eec0 Fix typo 2023-01-21 14:47:14 +01:00
Emilia Hane
5fc648217d fixup! Improve error handling 2023-01-21 14:46:24 +01:00
realbigsean
a83fd1afb4 remove unused imports 2023-01-21 04:52:36 -05:00
realbigsean
cbd09dc281 finish refactor 2023-01-21 04:48:25 -05:00
realbigsean
eb9feed784 add new traits 2023-01-20 16:04:35 -05:00
Emilia Hane
f7eb89ddd9 Improve error handling 2023-01-20 21:16:47 +01:00
realbigsean
8e57eef0ed return a BlobsUnavailable error when the block root is a pre-4844 block 2023-01-20 21:16:47 +01:00
realbigsean
c6479444c2 don't send errors when we *correctly* don't have blobs 2023-01-20 21:16:47 +01:00
realbigsean
e1ce4e5b78 make explicity BlobsUnavailable error and handle it directly 2023-01-20 21:16:47 +01:00
realbigsean
f7f64eb007 fix/consolidate some error handling 2023-01-20 21:16:47 +01:00
Emilia Hane
89cb58d17b Fix typo
Co-authored-by: realbigsean <seananderson33@GMAIL.com>
2023-01-20 21:16:47 +01:00
Emilia Hane
9cc25162e2 Send error message if eip4844 fork disabled
Co-authored-by: realbigsean <seananderson33@GMAIL.com>
2023-01-20 21:16:46 +01:00
Emilia Hane
654e59cbba Fix rename fn bug
Co-authored-by: realbigsean <seananderson33@GMAIL.com>
2023-01-20 21:16:46 +01:00
Emilia Hane
a00b355800 fixup! Less strict handling of faulty rpc req params and syntax improvement 2023-01-20 21:16:46 +01:00
Emilia Hane
b4ec4c1ccf Less strict handling of faulty rpc req params and syntax improvement 2023-01-20 21:16:46 +01:00
Emilia Hane
9445ac70d8 Check data availability boundary in rpc request 2023-01-20 21:16:46 +01:00
realbigsean
3cb8fb7973 block wrapper refactor initial commit 2023-01-20 11:50:16 -05:00
realbigsean
e046657b4f Merge pull request #3897 from realbigsean/capella-back-merge
execution API related fixes after capella merge
2023-01-19 14:52:38 -05:00
realbigsean
8f8b7211d0 execution API related fixes 2023-01-19 09:32:08 -05:00
realbigsean
e179be1f0b Merge pull request #3876 from realbigsean/devnetv4
Devnet V4
2023-01-19 09:03:08 -05:00
realbigsean
acbbdd8b9e merge eip4844 2023-01-19 08:46:38 -05:00
Pawan Dhananjay
fd08a2cb0a Fix trusted setup in lcli::new_testnet 2023-01-19 18:16:04 +05:30
Pawan Dhananjay
f04486dc71 Update kzg library to use bytes only interface 2023-01-17 12:12:17 +05:30
realbigsean
ddcd10b194 merge latest capella changes 2023-01-16 09:17:18 -05:00
realbigsean
e9affafb6b get rid of EL endpoint switching at forks 2023-01-13 10:51:45 -05:00
realbigsean
d96d793bfb fix compilation issues 2023-01-12 14:17:14 -05:00
realbigsean
06f71e8cce merge capella 2023-01-12 12:51:09 -05:00
realbigsean
c705be202b run historical_summary ef tests for 4844 2023-01-12 11:15:33 -05:00
realbigsean
63da81de62 check for eip4844 unaccessed files 2023-01-12 08:50:02 -05:00
realbigsean
f7f351784a get ef tests passing after capella rebase 2023-01-11 18:32:15 -05:00
realbigsean
438126f19a merge upstream, fix compile errors 2023-01-11 13:52:58 -05:00
realbigsean
680a5c7fc8 Merge pull request #3861 from paulhauner/ssz-transparent-4844
Allow leading skipped tuple fields
2023-01-09 09:50:00 -05:00
Pawan Dhananjay
ba410c3012 Embed trusted setup in network config (#3851)
* Load trusted setup in network config

* Fix trusted setup serialize and deserialize

* Load trusted setup from hardcoded preset instead of a file

* Truncate after deserialising trusted setup

* Fix beacon node script

* Remove hardcoded setup file

* Add length checks
2023-01-09 12:34:16 +05:30
Paul Hauner
026b5afbb9 Allow leading skipped tuple fields 2023-01-09 15:56:17 +11:00
realbigsean
33ff84743d Merge pull request #3848 from emhane/empty_blobs_db
Don't write empty blobs to db
2023-01-06 10:12:02 -05:00
Emilia Hane
c44738c77b Undo response modification in commit 597363d2f 2023-01-06 12:42:21 +01:00
Emilia Hane
74bca46fc2 Fix bug of early termination of batch send 2023-01-06 11:45:13 +01:00
Emilia Hane
caad492d48 Avoid loading unecessary data 2023-01-06 11:17:26 +01:00
Emilia Hane
88dde8b3bb Remove accidentally added file 2023-01-06 08:17:39 +01:00
Emilia Hane
e9e9fc865b Cargo clean 2023-01-05 16:28:59 +01:00
Emilia Hane
597363d2f9 Don't send empty blobs sidecar for blobs by range request 2023-01-05 16:28:59 +01:00
Emilia Hane
01ac7ad23c Return empty blobs sidecar when no kzg commitments in block 2023-01-05 16:28:58 +01:00
Emilia Hane
92c4e99305 Don't write empty blobs to db 2023-01-05 16:28:58 +01:00
realbigsean
8a77b05c19 Merge pull request #3830 from jimmygchen/blobs-beacon-api
Add Beacon API endpoint (/lighthouse) to download blobs by block ID
2023-01-04 09:04:32 -05:00
Jimmy Chen
11e3902dc8 Add support for blobs_sidecar ssz parsing 2023-01-04 18:51:28 +11:00
realbigsean
e02fcb30ab dont verify the signature of the genesis block in backfill sync (#3846) 2023-01-03 17:23:01 -05:00
Jimmy Chen
7c2a0aeb58 Merge branch 'blobs-beacon-api' of github.com:jimmygchen/lighthouse into blobs-beacon-api 2023-01-04 01:58:08 +11:00
Jimmy Chen
355cab8704 Simplify blob_sidecar query and remove override for Head and Slot 2023-01-04 01:57:15 +11:00
realbigsean
786d9834f5 sym link clang in cross dockerfile 2023-01-03 09:25:02 -05:00
Jimmy Chen
11736b68d3 Return beacon_chain_error if blob query from BeaconStore returns an error 2023-01-04 01:23:31 +11:00
realbigsean
c4a41e8f55 only use withdrawals-processing in docker build on capella 2022-12-29 11:07:22 -05:00
Pawan Dhananjay
2734f3f9db Fix devnet3 genesis.ssz.zip 2022-12-29 20:34:25 +05:30
realbigsean
c28c38b044 Merge pull request #3844 from pawanjay176/minor-fixes
Devnet 3 minor fixes
2022-12-29 09:05:00 -05:00
Pawan Dhananjay
e63cf80040 Update c-kzg version 2022-12-29 16:42:48 +05:30
Pawan Dhananjay
c47a0ade22 Fix config values for devnet 3; add more bootnodes 2022-12-29 16:42:26 +05:30
sean
31ba051879 add clang to dockerfile 2022-12-28 18:34:28 +00:00
sean
e94eb1d2d6 Merge branch 'capella' of https://github.com/sigp/lighthouse into eip4844 2022-12-28 18:29:45 +00:00
realbigsean
d34706b044 Merge pull request #3840 from realbigsean/blob-renamings-and-cleanup
Blob cleanup (renames, remove , wrap BlockWrapper enum)
2022-12-28 13:28:36 -05:00
sean
40c6daa34b add pawan's suggestsion 2022-12-28 18:27:21 +00:00
realbigsean
6e73965450 Merge pull request #3841 from realbigsean/docker-eip4844
create automatic docker build for eip4844
2022-12-28 11:12:37 -05:00
realbigsean
6d85930520 create automatic docker build for eip4844 2022-12-28 11:11:17 -05:00
realbigsean
019467840f Merge pull request #3833 from realbigsean/empty-blobs
Empty blobs validation
2022-12-28 10:32:06 -05:00
realbigsean
8a70d80a2f Revert "Revert "renames, remove , wrap BlockWrapper enum to make descontruction private""
This reverts commit 1931a442dc.
2022-12-28 10:31:18 -05:00
realbigsean
1931a442dc Revert "renames, remove , wrap BlockWrapper enum to make descontruction private"
This reverts commit 5b3b34a9d7.
2022-12-28 10:30:36 -05:00
realbigsean
5b3b34a9d7 renames, remove , wrap BlockWrapper enum to make descontruction private 2022-12-28 10:28:45 -05:00
realbigsean
502b5e5bf0 unused error lint 2022-12-28 09:32:29 -05:00
Jimmy Chen
1b64cbadba Merge remote-tracking branch 'origin/eip4844' into blobs-beacon-api 2022-12-24 00:41:45 +00:00
Diva M
aeb243fe61 fmt 2022-12-23 17:44:50 -05:00
Diva M
6bf439befd Merge branch 'eip4844' into empty-blobs 2022-12-23 17:38:59 -05:00
Divma
240854750c cleanup: remove unused imports, unusued fields (#3834) 2022-12-23 17:16:10 -05:00
realbigsean
dcd5e40fa0 Merge pull request #3832 from divagant-martian/4844-backfill-fixes
Remove sync validation no longer valid + Fix bakcfill batch sizes
2022-12-23 13:43:44 -05:00
realbigsean
adf5f462d5 fix blob validation for empty blobs when using 2022-12-23 12:59:04 -05:00
realbigsean
1dc0759f57 impl hash correctly for the blob wrapper 2022-12-23 12:53:59 -05:00
realbigsean
d09523802b impl hash correctly for the blob wrapper 2022-12-23 12:52:26 -05:00
realbigsean
5e11edc612 fix blob validation for empty blobs 2022-12-23 12:47:38 -05:00
Diva M
24087f104d add the batch type to the Batch's KV 2022-12-23 10:49:46 -05:00
Diva M
901764b8f0 backfill batches need to be of just one epoch 2022-12-23 10:32:59 -05:00
realbigsean
5db0a88d4f fix compilation errors from merge 2022-12-23 10:27:01 -05:00
realbigsean
f45d117e73 merge with capella 2022-12-23 10:21:18 -05:00
realbigsean
4d50fa36bc Merge pull request #3829 from divagant-martian/handle-no-blob-range-response
Handle peers sending no blob when the blob is empty in range responses
2022-12-23 10:15:30 -05:00
Diva M
66f9aa922d clean up and improvements 2022-12-23 09:52:10 -05:00
Jimmy Chen
f6d5e8fea3 Rename variable only 2022-12-23 16:14:53 +11:00
Jimmy Chen
3b9041047a Fix typoe and add blobs endpoint to eth2 lib. 2022-12-23 15:28:08 +11:00
Jimmy Chen
bf7f709b51 Add missing route 2022-12-23 14:52:03 +11:00
Jimmy Chen
0155c94f86 Fix code formatting 2022-12-23 14:47:15 +11:00
Jimmy Chen
70d6e6705e Add Beacon API endpoint to download blobs by block ID 2022-12-23 12:42:00 +11:00
Diva M
3643f5cc19 spelling 2022-12-22 17:47:36 -05:00
Diva M
48ff56d9cb spelling 2022-12-22 17:38:55 -05:00
Diva M
e24f6c93d9 fix ctrl c'd comment 2022-12-22 17:38:16 -05:00
Diva M
fbc147e273 remove unused entry struct 2022-12-22 17:34:01 -05:00
Diva M
847f0de0ea change base 2022-12-22 17:32:33 -05:00
Diva M
cd6655dba9 handle no blobs from peers instead of empty blobs in range requests 2022-12-22 17:30:04 -05:00
realbigsean
61763790d5 Merge pull request #3825 from jimmygchen/small-fixes
Various small fixes to 4844 branch
2022-12-22 17:12:09 -05:00
realbigsean
332db29bf9 Merge pull request #3828 from realbigsean/update-eip4844-testnet-info
Update eip4844 testnet info
2022-12-22 14:20:24 -05:00
realbigsean
12402b309d fix geth binary path 2022-12-22 14:08:38 -05:00
realbigsean
d504d51dd9 merge with upstream add context bytes to error log 2022-12-22 14:06:28 -05:00
realbigsean
adbecb80eb config updates 2022-12-22 10:26:09 -05:00
realbigsean
cb28201f5b update built in 4844 testnet 2022-12-22 08:54:24 -05:00
realbigsean
33d01a7911 miscelaneous fixes on syncing, rpc and responding to peer's sync related requests (#3827)
- there was a bug in responding range blob requests where we would incorrectly label the first slot of an epoch as a non-skipped slot if it were skipped. this bug did not exist in the code for responding to block range request because the logic error was mitigated by defensive coding elsewhere
- there was a bug where a block received during range sync without a corresponding blob (and vice versa) was incorrectly interpreted as a stream termination
- RPC size limit fixes.
- Our blob cache was dead locking so I removed use of it for now.
- Because of our change in finalized sync batch size from 2 to 1 and our transition to using exact epoch boundaries for batches (rather than one slot past the epoch boundary), we need to sync finalized sync to 2 epochs + 1 slot past our peer's finalized slot in order to finalize the chain locally.
- use fork context bytes in rpc methods on both the server and client side
2022-12-21 15:50:51 -05:00
realbigsean
2f4c44fd88 remove my binary path for geth 2022-12-21 14:04:14 -05:00
realbigsean
ff772311fa add context bytes to blob messages, fix rpc limits, sync past finalized checkpoint during finalized sync so we can advance our own finalization, fix stream termination bug in blobs by range 2022-12-21 13:56:52 -05:00
Jimmy Chen
f7bb458c5e Fix incorrect logging 2022-12-22 02:01:11 +11:00
Jimmy Chen
ccfd092845 Fix blob request logging and incorrect enum type 2022-12-22 00:22:37 +11:00
Jimmy Chen
14aa87aff3 Fix code comment 2022-12-22 00:19:38 +11:00
Jimmy Chen
a6b771f265 Add more logging to Error::MaxDistanceExceeded 2022-12-22 00:19:22 +11:00
realbigsean
a67fa516c7 don't expect context bytes for blob messages 2022-12-20 19:32:54 -05:00
realbigsean
cc420caaa5 Merge pull request #3823 from jimmygchen/make-bootnode-binary-var
Use bootnode binary variable in testnet scripts
2022-12-20 19:22:46 -05:00
realbigsean
9c46a1cb21 fix rate limits, and a couple other bugs 2022-12-20 18:56:07 -05:00
Jimmy Chen
c76e371559 Add missing source 2022-12-21 00:29:15 +11:00
Jimmy Chen
db29cb08a6 Add bootnode binary variable in testnet scripts 2022-12-21 00:18:05 +11:00
realbigsean
7d5db8015d correctly respond without skips on first range response 2022-12-19 19:07:21 -05:00
Divma
51e588bdf9 Merge pull request #3820 from realbigsean/sync-fixes
Handle ResourceUnavailable errors and other rpc/sync fixes
2022-12-19 12:38:33 -05:00
realbigsean
3ab0f46077 Update beacon_node/http_api/src/publish_blocks.rs
Co-authored-by: Divma <26765164+divagant-martian@users.noreply.github.com>
2022-12-19 12:27:31 -05:00
realbigsean
eddfb50c58 Revert "Revert "remove json snooper from local testnet scripts""
This reverts commit ba1cabc0c9.
2022-12-19 11:39:54 -05:00
realbigsean
5de4f5b8d0 handle parent blob request edge cases correctly. fix data availability boundary check 2022-12-19 11:39:09 -05:00
realbigsean
22ed36bc6a fix is_empty check 2022-12-16 15:16:17 -05:00
realbigsean
ba1cabc0c9 Revert "remove json snooper from local testnet scripts"
This reverts commit 60d70ca501.
2022-12-16 14:52:37 -05:00
realbigsean
4570ccd233 Merge pull request #3816 from realbigsean/advertise-blobs-supported-protocols
Add blob rpc protocols to `protocol_info`
2022-12-16 14:38:06 -05:00
realbigsean
0349b104bf add blob rpc protocols to 2022-12-16 14:28:14 -05:00
realbigsean
1644978cdb fix compilation 2022-12-15 10:26:10 -05:00
realbigsean
d893706e0e merge with capella 2022-12-15 09:33:18 -05:00
Pawan Dhananjay
52a06231c8 Add support for compile time FIELD_ELEMENTS_PER_BLOB 2022-12-14 21:25:47 +05:30
Pawan Dhananjay
2e89a719b0 Fix auth port 2022-12-13 15:14:07 +05:30
realbigsean
e8322c8a04 Merge pull request #3792 from jimmygchen/eip4844
Fix local testnet for MacOS
2022-12-12 17:47:06 -05:00
sean
2a7a1b31dc Merge branch 'capella' of https://github.com/sigp/lighthouse into eip4844 2022-12-12 22:41:54 +00:00
Jimmy Chen
704cf57de4 add gnu sed & gnu grep to testnet job 2022-12-13 00:14:32 +11:00
realbigsean
9806f54f2d update trusted setup and eth1 balance allocations in testnet genesis 2022-12-09 15:21:10 -05:00
realbigsean
d4004ee050 update MaxBlobsPerBlock to 4 2022-12-09 13:50:10 -05:00
realbigsean
60d70ca501 remove json snooper from local testnet scripts 2022-12-08 13:15:04 -05:00
realbigsean
d59a1c63fb small cleanup 2022-12-08 09:18:56 -05:00
realbigsean
254cad369e Merge pull request #3751 from realbigsean/eip4844-devnet-v3
Eip4844 devnet v3
2022-12-08 09:04:42 -05:00
realbigsean
be8c6349dc Merge pull request #27 from realbigsean/sean-interop-4844-testing
P2P fixes, local post merge testnet, query different EL endpoints at fork boundaries
2022-12-08 08:50:28 -05:00
realbigsean
ef9579602f geth binary location update 2022-12-08 08:49:39 -05:00
realbigsean
715002a615 use is synced for notifier again 2022-12-08 08:42:27 -05:00
realbigsean
14fa1e527f remove unused log and fix EL config serde 2022-12-08 08:32:59 -05:00
realbigsean
8c95ab07a3 remove fCU v3 query 2022-12-08 08:21:39 -05:00
realbigsean
658e9d9bba saturating sub epoch for blob boundary 2022-12-07 15:40:51 -05:00
realbigsean
5a42f6b067 range block or block+blob requests 2022-12-07 15:35:46 -05:00
realbigsean
a0d4aecf30 requests block + blob always post eip4844 2022-12-07 15:30:08 -05:00
realbigsean
b616c0a056 reset genesis.json fork times 2022-12-07 14:04:46 -05:00
realbigsean
6d4fb41b84 fix blob slot validation 2022-12-07 13:49:24 -05:00
realbigsean
dbc57ba2d9 merge with upstream 2022-12-07 13:11:21 -05:00
realbigsean
6c8b1b323b merge upstream 2022-12-07 12:27:21 -05:00
realbigsean
33721d17b2 Revert "fix compilation errors (#25)"
This reverts commit ae054d2663.
2022-12-07 12:25:11 -05:00
realbigsean
e5f26516bd add trusted setup, query different version of EL endpoint at each fork 2022-12-07 12:01:21 -05:00
Jimmy Chen
ae054d2663 fix compilation errors (#25) 2022-12-07 10:07:55 -05:00
realbigsean
2704955b2e local testnet config updates 2022-12-06 08:54:46 -05:00
Pawan Dhananjay
e72d9fb922 Working post bellatrix local testnet 2022-12-02 19:50:45 +05:30
Pawan Dhananjay
df3615664e Embed interop validators into genesis 2022-12-02 12:55:27 +05:30
Pawan Dhananjay
ff24773a5a Add EL scripts 2022-12-02 12:55:27 +05:30
realbigsean
2cd971c7d3 attempt to fix serde for opt hex be u256 2022-12-01 15:17:57 -05:00
realbigsean
c96234f38c fix hex be opt 2022-12-01 14:20:50 -05:00
realbigsean
8102a01085 merge with upstream 2022-12-01 11:13:07 -05:00
Diva M
979a95d62f handle unknown parents for block-blob pairs
wip

handle unknown parents for block-blob pairs
2022-11-30 17:21:54 -05:00
realbigsean
2157d91b43 process single block and blob 2022-11-30 11:51:18 -05:00
realbigsean
fc9d0a512d handle blobs by range requests 2022-11-30 10:02:29 -05:00
realbigsean
422d145902 chain segment processing for blobs 2022-11-30 09:40:15 -05:00
Pawan Dhananjay
8b56446b64 Add more kzg validations 2022-11-29 16:04:18 +05:30
Pawan Dhananjay
0ed3f7474c Remove empty blob handling as it's handled in kzg library 2022-11-29 15:15:05 +05:30
Diva M
e548073602 Merge branch 'blob-syncing' into eip4844-devnet-v3 2022-11-28 15:10:50 -05:00
Diva M
c532f4438c debug impl for wrapper type 2022-11-28 14:54:47 -05:00
Diva M
050acf67a3 Revert "TEMP HACK to get it compiling"
This reverts commit 3c79c33d86.
2022-11-28 14:36:07 -05:00
Diva M
3c79c33d86 TEMP HACK to get it compiling 2022-11-28 14:30:40 -05:00
Diva M
4760dbb078 add wrapper type 2022-11-28 14:22:19 -05:00
Diva M
805df307f6 wip 2022-11-28 14:13:12 -05:00
realbigsean
e962e80bb4 fix compile errors 2022-11-28 12:16:45 -05:00
realbigsean
6d7235f2c8 add blob info 2022-11-28 11:36:48 -05:00
realbigsean
0027cdcd3a Merge branch 'eip4844-devnet-v3' of https://github.com/realbigsean/lighthouse into eip4844-devnet-v3 2022-11-28 11:26:55 -05:00
realbigsean
92cae14409 add blob info 2022-11-28 11:26:46 -05:00
Diva M
f4babdedd5 more hacks 2022-11-28 11:06:12 -05:00
Diva M
25f29256ce hack the thing to get it to compile 2022-11-28 10:42:50 -05:00
Pawan Dhananjay
cb78f2f8df Add more kzg validations 2022-11-28 20:23:18 +05:30
Pawan Dhananjay
9640d420f7 Run kzg operations for block operations 2022-11-28 18:17:10 +05:30
Pawan Dhananjay
3075b82ea0 Add kzg trusted setup file cli param and load into beacon chain 2022-11-28 16:54:20 +05:30
realbigsean
3c9e1abcb7 merge upstream 2022-11-26 10:01:57 -05:00
sean
07a79c8266 add block and blobs sidecar to whitelist 2022-11-25 14:44:57 +00:00
sean
d6838d67dc add clang to dockerfile 2022-11-25 14:38:45 +00:00
sean
f88caa7afc set quota for blobs by root 2022-11-25 14:30:51 +00:00
realbigsean
a61f35272c fix compiling 2022-11-24 08:18:01 -05:00
realbigsean
1222404450 Merge branch 'blob-syncing' of https://github.com/realbigsean/lighthouse into blob-sync-kzg 2022-11-24 07:46:04 -05:00
Divma
bf5005244e Blob syncing (#24)
* add a rt is_blob_batch

* use the mixed type everywhere

* glue

* more glue

* minor fixes

* fix range tests

* filling in the gaps

* moore filling in the gaps
2022-11-24 07:45:38 -05:00
realbigsean
beddcfaac2 get spec tests working and fix json serialization 2022-11-23 18:30:45 -05:00
realbigsean
abc933faa8 Merge branch 'capella-bugfixes' of https://github.com/michaelsproul/lighthouse into blob-sync-kzg 2022-11-23 11:27:41 -05:00
realbigsean
7aa52a4141 ef-test fixes 2022-11-23 11:27:37 -05:00
realbigsean
62f8a5ee10 Merge branch 'blob-syncing' of https://github.com/realbigsean/lighthouse into blob-sync-kzg 2022-11-23 11:22:54 -05:00
realbigsean
ce097ac8d2 Merge branch 'json-rpc-blobs-updates' of https://github.com/realbigsean/lighthouse into blob-syncing 2022-11-23 11:22:15 -05:00
realbigsean
743347cf04 Revert "fix payload default check in fork choice"
This reverts commit e56fefbd05.
2022-11-23 11:18:47 -05:00
realbigsean
e56fefbd05 fix payload default check in fork choice 2022-11-23 09:44:00 -05:00
realbigsean
87ec448b00 Merge branch 'eip4844' of https://github.com/sigp/lighthouse into blob-sync-kzg 2022-11-23 09:19:30 -05:00
Michael Sproul
53a22c2fcb Two Capella bugfixes 2022-11-23 18:51:39 +11:00
realbigsean
f601fb3b7e ef-test updates 2022-11-22 20:13:51 -05:00
Pawan Dhananjay
902055f295 ugly utils 2022-11-22 20:10:32 -05:00
Pawan Dhananjay
e8b5f311aa Add kzg crate functions 2022-11-22 20:10:17 -05:00
Pawan Dhananjay
3288404ec1 Skeleton 2022-11-22 20:09:21 -05:00
realbigsean
48b2efce9f merge with upstream 2022-11-22 18:38:30 -05:00
realbigsean
160b915695 remove coments 2022-11-22 17:30:35 -05:00
realbigsean
51b44290a3 rename excess blobs and fix json serialization/deserialization 2022-11-22 17:22:46 -05:00
realbigsean
a0d712ed8c Merge pull request #23 from divagant-martian/blob-syncing
make it compile to start from here
2022-11-21 14:59:12 -05:00
Diva M
7ed2d35424 get it to compile 2022-11-21 14:53:33 -05:00
realbigsean
e7ee79185b add blobs cache and fix some block production 2022-11-21 14:09:06 -05:00
realbigsean
dc87156641 block and blob handling progress 2022-11-19 16:53:34 -05:00
realbigsean
45897ad4e1 remove blob wrapper 2022-11-19 15:18:42 -05:00
realbigsean
dfd0013eab Merge pull request #22 from divagant-martian/seans-block-blobs
toy skelleton of sync changes
2022-11-16 14:45:45 -05:00
Diva M
78c72158c8 toy skelleton of sync changes 2022-11-16 13:53:38 -05:00
realbigsean
7162e5e23b add a bunch of blob coupling boiler plate, add a blobs by root request 2022-11-15 16:43:56 -05:00
realbigsean
fe04d945cc make signed block + sidecar consensus spec 2022-11-10 14:22:30 -05:00
928 changed files with 85159 additions and 31339 deletions

113
.config/nextest.toml Normal file
View File

@@ -0,0 +1,113 @@
# This is the default config used by nextest. It is embedded in the binary at
# build time. It may be used as a template for .config/nextest.toml.
[store]
# The directory under the workspace root at which nextest-related files are
# written. Profile-specific storage is currently written to dir/<profile-name>.
dir = "target/nextest"
# This section defines the default nextest profile. Custom profiles are layered
# on top of the default profile.
[profile.default]
# "retries" defines the number of times a test should be retried. If set to a
# non-zero value, tests that succeed on a subsequent attempt will be marked as
# non-flaky. Can be overridden through the `--retries` option.
# Examples
# * retries = 3
# * retries = { backoff = "fixed", count = 2, delay = "1s" }
# * retries = { backoff = "exponential", count = 10, delay = "1s", jitter = true, max-delay = "10s" }
retries = 0
# The number of threads to run tests with. Supported values are either an integer or
# the string "num-cpus". Can be overridden through the `--test-threads` option.
test-threads = 8
# The number of threads required for each test. This is generally used in overrides to
# mark certain tests as heavier than others. However, it can also be set as a global parameter.
threads-required = 1
# Show these test statuses in the output.
#
# The possible values this can take are:
# * none: no output
# * fail: show failed (including exec-failed) tests
# * retry: show flaky and retried tests
# * slow: show slow tests
# * pass: show passed tests
# * skip: show skipped tests (most useful for CI)
# * all: all of the above
#
# Each value includes all the values above it; for example, "slow" includes
# failed and retried tests.
#
# Can be overridden through the `--status-level` flag.
status-level = "pass"
# Similar to status-level, show these test statuses at the end of the run.
final-status-level = "flaky"
# "failure-output" defines when standard output and standard error for failing tests are produced.
# Accepted values are
# * "immediate": output failures as soon as they happen
# * "final": output failures at the end of the test run
# * "immediate-final": output failures as soon as they happen and at the end of
# the test run; combination of "immediate" and "final"
# * "never": don't output failures at all
#
# For large test suites and CI it is generally useful to use "immediate-final".
#
# Can be overridden through the `--failure-output` option.
failure-output = "immediate"
# "success-output" controls production of standard output and standard error on success. This should
# generally be set to "never".
success-output = "never"
# Cancel the test run on the first failure. For CI runs, consider setting this
# to false.
fail-fast = true
# Treat a test that takes longer than the configured 'period' as slow, and print a message.
# See <https://nexte.st/book/slow-tests> for more information.
#
# Optional: specify the parameter 'terminate-after' with a non-zero integer,
# which will cause slow tests to be terminated after the specified number of
# periods have passed.
# Example: slow-timeout = { period = "60s", terminate-after = 2 }
slow-timeout = { period = "120s" }
# Treat a test as leaky if after the process is shut down, standard output and standard error
# aren't closed within this duration.
#
# This usually happens in case of a test that creates a child process and lets it inherit those
# handles, but doesn't clean the child process up (especially when it fails).
#
# See <https://nexte.st/book/leaky-tests> for more information.
leak-timeout = "100ms"
[profile.default.junit]
# Output a JUnit report into the given file inside 'store.dir/<profile-name>'.
# If unspecified, JUnit is not written out.
# path = "junit.xml"
# The name of the top-level "report" element in JUnit report. If aggregating
# reports across different test runs, it may be useful to provide separate names
# for each report.
report-name = "lighthouse-run"
# Whether standard output and standard error for passing tests should be stored in the JUnit report.
# Output is stored in the <system-out> and <system-err> elements of the <testcase> element.
store-success-output = false
# Whether standard output and standard error for failing tests should be stored in the JUnit report.
# Output is stored in the <system-out> and <system-err> elements of the <testcase> element.
#
# Note that if a description can be extracted from the output, it is always stored in the
# <description> element.
store-failure-output = true
# This profile is activated if MIRI_SYSROOT is set.
[profile.default-miri]
# Miri tests take up a lot of memory, so only run 1 test at a time by default.
test-threads = 4

19
.github/mergify.yml vendored Normal file
View File

@@ -0,0 +1,19 @@
queue_rules:
- name: default
batch_size: 8
batch_max_wait_time: 60 s
checks_timeout: 10800 s
merge_method: squash
commit_message_template: |
{{ title }} (#{{ number }})
{% for commit in commits %}
* {{ commit.commit_message }}
{% endfor %}
queue_conditions:
- "#approved-reviews-by >= 1"
- "check-success=license/cla"
- "check-success=target-branch-check"
merge_conditions:
- "check-success=test-suite-success"
- "check-success=local-testnet-success"

View File

@@ -13,7 +13,7 @@ jobs:
build-and-upload-to-s3:
runs-on: ubuntu-20.04
steps:
- uses: actions/checkout@master
- uses: actions/checkout@v4
- name: Setup mdBook
uses: peaceiris/actions-mdbook@v1

View File

@@ -15,8 +15,6 @@ concurrency:
env:
DOCKER_PASSWORD: ${{ secrets.DOCKER_PASSWORD }}
DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}
IMAGE_NAME: ${{ github.repository_owner}}/lighthouse
LCLI_IMAGE_NAME: ${{ github.repository_owner }}/lcli
# Enable self-hosted runners for the sigp repo only.
SELF_HOSTED_RUNNERS: ${{ github.repository == 'sigp/lighthouse' }}
@@ -49,19 +47,15 @@ jobs:
VERSION: ${{ env.VERSION }}
VERSION_SUFFIX: ${{ env.VERSION_SUFFIX }}
build-docker-single-arch:
name: build-docker-${{ matrix.binary }}${{ matrix.features.version_suffix }}
name: build-docker-${{ matrix.binary }}-${{ matrix.cpu_arch }}${{ matrix.features.version_suffix }}
# Use self-hosted runners only on the sigp repo.
runs-on: ${{ github.repository == 'sigp/lighthouse' && fromJson('["self-hosted", "linux", "release"]') || 'ubuntu-22.04' }}
strategy:
matrix:
binary: [aarch64,
aarch64-portable,
x86_64,
x86_64-portable]
features: [
{version_suffix: "", env: "gnosis,slasher-lmdb,slasher-mdbx,jemalloc"},
{version_suffix: "-dev", env: "jemalloc,spec-minimal"}
]
binary: [lighthouse,
lcli]
cpu_arch: [aarch64,
x86_64]
include:
- profile: maxperf
@@ -69,36 +63,48 @@ jobs:
env:
VERSION: ${{ needs.extract-version.outputs.VERSION }}
VERSION_SUFFIX: ${{ needs.extract-version.outputs.VERSION_SUFFIX }}
FEATURE_SUFFIX: ${{ matrix.features.version_suffix }}
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
- name: Update Rust
if: env.SELF_HOSTED_RUNNERS == 'false'
run: rustup update stable
- name: Dockerhub login
run: |
echo "${DOCKER_PASSWORD}" | docker login --username ${DOCKER_USERNAME} --password-stdin
- name: Cross build Lighthouse binary
- name: Sets env vars for Lighthouse
if: startsWith(matrix.binary, 'lighthouse')
run: |
echo "CROSS_FEATURES=gnosis,spec-minimal,slasher-lmdb,jemalloc" >> $GITHUB_ENV
- name: Set `make` command for lighthouse
if: startsWith(matrix.binary, 'lighthouse')
run: |
echo "MAKE_CMD=build-${{ matrix.cpu_arch }}-portable" >> $GITHUB_ENV
- name: Set `make` command for lcli
if: startsWith(matrix.binary, 'lcli')
run: |
echo "MAKE_CMD=build-lcli-${{ matrix.cpu_arch }}" >> $GITHUB_ENV
- name: Cross build binaries
run: |
cargo install cross
env CROSS_PROFILE=${{ matrix.profile }} CROSS_FEATURES=${{ matrix.features.env }} make build-${{ matrix.binary }}
env CROSS_PROFILE=${{ matrix.profile }} CROSS_FEATURES=${{ env.CROSS_FEATURES }} make ${{ env.MAKE_CMD }}
- name: Make bin dir
run: mkdir ./bin
- name: Move cross-built binary into Docker scope (if ARM)
if: startsWith(matrix.binary, 'aarch64')
run: mv ./target/aarch64-unknown-linux-gnu/${{ matrix.profile }}/lighthouse ./bin
- name: Move cross-built binary into Docker scope (if x86_64)
if: startsWith(matrix.binary, 'x86_64')
run: mv ./target/x86_64-unknown-linux-gnu/${{ matrix.profile }}/lighthouse ./bin
- name: Move cross-built binary into Docker scope
run: mv ./target/${{ matrix.cpu_arch }}-unknown-linux-gnu/${{ matrix.profile }}/${{ matrix.binary }} ./bin
- name: Map aarch64 to arm64 short arch
if: startsWith(matrix.binary, 'aarch64')
if: startsWith(matrix.cpu_arch, 'aarch64')
run: echo "SHORT_ARCH=arm64" >> $GITHUB_ENV
- name: Map x86_64 to amd64 short arch
if: startsWith(matrix.binary, 'x86_64')
if: startsWith(matrix.cpu_arch, 'x86_64')
run: echo "SHORT_ARCH=amd64" >> $GITHUB_ENV;
- name: Set modernity suffix
if: endsWith(matrix.binary, '-portable') != true
run: echo "MODERNITY_SUFFIX=-modern" >> $GITHUB_ENV;
- name: Install QEMU
if: env.SELF_HOSTED_RUNNERS == 'false'
@@ -106,56 +112,57 @@ jobs:
- name: Set up Docker Buildx
if: env.SELF_HOSTED_RUNNERS == 'false'
uses: docker/setup-buildx-action@v2
uses: docker/setup-buildx-action@v3
- name: Build and push
uses: docker/build-push-action@v4
- name: Build and push (Lighthouse)
if: startsWith(matrix.binary, 'lighthouse')
uses: docker/build-push-action@v5
with:
file: ./Dockerfile.cross
context: .
platforms: linux/${{ env.SHORT_ARCH }}
push: true
tags: ${{ env.IMAGE_NAME }}:${{ env.VERSION }}-${{ env.SHORT_ARCH }}${{ env.VERSION_SUFFIX }}${{ env.MODERNITY_SUFFIX }}${{ env.FEATURE_SUFFIX }}
tags: |
${{ github.repository_owner}}/${{ matrix.binary }}:${{ env.VERSION }}-${{ env.SHORT_ARCH }}${{ env.VERSION_SUFFIX }}
${{ github.repository_owner}}/${{ matrix.binary }}:${{ env.VERSION }}-${{ env.SHORT_ARCH }}${{ env.VERSION_SUFFIX }}-dev
${{ github.repository_owner}}/${{ matrix.binary }}:${{ env.VERSION }}-${{ env.SHORT_ARCH }}${{ env.VERSION_SUFFIX }}-modern
${{ github.repository_owner}}/${{ matrix.binary }}:${{ env.VERSION }}-${{ env.SHORT_ARCH }}${{ env.VERSION_SUFFIX }}-modern-dev
- name: Build and push (lcli)
if: startsWith(matrix.binary, 'lcli')
uses: docker/build-push-action@v5
with:
file: ./lcli/Dockerfile.cross
context: .
platforms: linux/${{ env.SHORT_ARCH }}
push: true
tags: |
${{ github.repository_owner}}/${{ matrix.binary }}:${{ env.VERSION }}-${{ env.SHORT_ARCH }}${{ env.VERSION_SUFFIX }}
build-docker-multiarch:
name: build-docker-multiarch${{ matrix.modernity }}
name: build-docker-${{ matrix.binary }}-multiarch
runs-on: ubuntu-22.04
needs: [build-docker-single-arch, extract-version]
strategy:
matrix:
modernity: ["", "-modern"]
binary: [lighthouse,
lcli]
needs: [build-docker-single-arch, extract-version]
env:
VERSION: ${{ needs.extract-version.outputs.VERSION }}
VERSION_SUFFIX: ${{ needs.extract-version.outputs.VERSION_SUFFIX }}
steps:
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v2
uses: docker/setup-buildx-action@v3
- name: Dockerhub login
run: |
echo "${DOCKER_PASSWORD}" | docker login --username ${DOCKER_USERNAME} --password-stdin
- name: Create and push multiarch manifest
- name: Create and push multiarch manifests
run: |
docker buildx imagetools create -t ${IMAGE_NAME}:${VERSION}${VERSION_SUFFIX}${{ matrix.modernity }} \
${IMAGE_NAME}:${VERSION}-arm64${VERSION_SUFFIX}${{ matrix.modernity }} \
${IMAGE_NAME}:${VERSION}-amd64${VERSION_SUFFIX}${{ matrix.modernity }};
docker buildx imagetools create -t ${{ github.repository_owner}}/${{ matrix.binary }}:${VERSION}${VERSION_SUFFIX} \
${{ github.repository_owner}}/${{ matrix.binary }}:${VERSION}-arm64${VERSION_SUFFIX} \
${{ github.repository_owner}}/${{ matrix.binary }}:${VERSION}-amd64${VERSION_SUFFIX};
build-docker-lcli:
runs-on: ubuntu-22.04
needs: [extract-version]
env:
VERSION: ${{ needs.extract-version.outputs.VERSION }}
VERSION_SUFFIX: ${{ needs.extract-version.outputs.VERSION_SUFFIX }}
steps:
- uses: actions/checkout@v3
- name: Dockerhub login
run: |
echo "${DOCKER_PASSWORD}" | docker login --username ${DOCKER_USERNAME} --password-stdin
- name: Build lcli dockerfile (with push)
run: |
docker build \
--build-arg PORTABLE=true \
--tag ${LCLI_IMAGE_NAME}:${VERSION}${VERSION_SUFFIX} \
--file ./lcli/Dockerfile .
docker push ${LCLI_IMAGE_NAME}:${VERSION}${VERSION_SUFFIX}

View File

@@ -20,16 +20,17 @@ jobs:
steps:
- name: Checkout code
uses: actions/checkout@v3
- name: Create docker network
run: docker network create book
uses: actions/checkout@v4
- name: Run mdbook server
run: docker run -v ${{ github.workspace }}/book:/book --network book --name book -p 3000:3000 -d peaceiris/mdbook:v0.4.20-rust serve --hostname 0.0.0.0
run: |
docker run -v ${{ github.workspace }}/book:/book --name book -p 3000:3000 -d peaceiris/mdbook:latest serve --hostname 0.0.0.0
sleep 5
- name: Print logs
run: docker logs book
- name: Run linkcheck
run: docker run --network book tennox/linkcheck:latest book:3000
run: |
curl -sL https://github.com/filiph/linkcheck/releases/download/3.0.0/linkcheck-3.0.0-linux-x64.tar.gz | tar xvzf - linkcheck/linkcheck --strip 1
./linkcheck localhost:3000 -d

View File

@@ -13,74 +13,155 @@ concurrency:
cancel-in-progress: true
jobs:
run-local-testnet:
strategy:
matrix:
os:
- ubuntu-22.04
- macos-12
runs-on: ${{ matrix.os }}
dockerfile-ubuntu:
runs-on: ${{ github.repository == 'sigp/lighthouse' && fromJson('["self-hosted", "linux", "CI", "large"]') || 'ubuntu-latest' }}
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
- name: Get latest version of stable Rust
run: rustup update stable
- name: Install geth (ubuntu)
if: matrix.os == 'ubuntu-22.04'
- name: Build Docker image
run: |
sudo add-apt-repository -y ppa:ethereum/ethereum
sudo apt-get update
sudo apt-get install ethereum
- name: Install geth (mac)
if: matrix.os == 'macos-12'
run: |
brew tap ethereum/ethereum
brew install ethereum
- name: Install GNU sed & GNU grep
if: matrix.os == 'macos-12'
run: |
brew install gnu-sed grep
echo "$(brew --prefix)/opt/gnu-sed/libexec/gnubin" >> $GITHUB_PATH
echo "$(brew --prefix)/opt/grep/libexec/gnubin" >> $GITHUB_PATH
# https://github.com/actions/cache/blob/main/examples.md#rust---cargo
- uses: actions/cache@v3
id: cache-cargo
docker build --build-arg FEATURES=portable -t lighthouse:local .
docker save lighthouse:local -o lighthouse-docker.tar
- name: Upload Docker image artifact
uses: actions/upload-artifact@v4
with:
path: |
~/.cargo/bin/
~/.cargo/registry/index/
~/.cargo/registry/cache/
~/.cargo/git/db/
target/
key: ${{ runner.os }}-cargo-${{ hashFiles('**/Cargo.lock') }}
name: lighthouse-docker
path: lighthouse-docker.tar
retention-days: 3
- name: Install lighthouse
run: make && make install-lcli
run-local-testnet:
runs-on: ubuntu-22.04
needs: dockerfile-ubuntu
steps:
- uses: actions/checkout@v4
- name: Install dependencies
run: |
sudo add-apt-repository ppa:rmescandon/yq
echo "deb [trusted=yes] https://apt.fury.io/kurtosis-tech/ /" | sudo tee /etc/apt/sources.list.d/kurtosis.list
sudo apt update
sudo apt install -y kurtosis-cli yq
kurtosis analytics disable
- name: Download Docker image artifact
uses: actions/download-artifact@v4
with:
name: lighthouse-docker
path: .
- name: Load Docker image
run: docker load -i lighthouse-docker.tar
- name: Start local testnet
run: ./start_local_testnet.sh genesis.json && sleep 60
run: ./start_local_testnet.sh -e local -c -b false && sleep 60
working-directory: scripts/local_testnet
- name: Print logs
run: ./dump_logs.sh
working-directory: scripts/local_testnet
- name: Stop local testnet
run: ./stop_local_testnet.sh
working-directory: scripts/local_testnet
- name: Clean-up testnet
run: ./clean.sh
- name: Stop local testnet and dump logs
run: ./stop_local_testnet.sh local
working-directory: scripts/local_testnet
- name: Start local testnet with blinded block production
run: ./start_local_testnet.sh -p genesis.json && sleep 60
run: ./start_local_testnet.sh -e local-blinded -c -p -b false && sleep 60
working-directory: scripts/local_testnet
- name: Print logs for blinded block testnet
run: ./dump_logs.sh
- name: Stop local testnet and dump logs
run: ./stop_local_testnet.sh local-blinded
working-directory: scripts/local_testnet
- name: Stop local testnet with blinded block production
run: ./stop_local_testnet.sh
working-directory: scripts/local_testnet
- name: Upload logs artifact
uses: actions/upload-artifact@v4
with:
name: logs-local-testnet
path: |
scripts/local_testnet/logs
retention-days: 3
doppelganger-protection-success-test:
needs: dockerfile-ubuntu
runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@v4
- name: Install dependencies
run: |
sudo add-apt-repository ppa:rmescandon/yq
echo "deb [trusted=yes] https://apt.fury.io/kurtosis-tech/ /" | sudo tee /etc/apt/sources.list.d/kurtosis.list
sudo apt update
sudo apt install -y kurtosis-cli yq
kurtosis analytics disable
- name: Download Docker image artifact
uses: actions/download-artifact@v4
with:
name: lighthouse-docker
path: .
- name: Load Docker image
run: docker load -i lighthouse-docker.tar
- name: Run the doppelganger protection success test script
run: |
./doppelganger_protection.sh success
working-directory: scripts/tests
- name: Upload logs artifact
uses: actions/upload-artifact@v4
with:
name: logs-doppelganger-protection-success
path: |
scripts/local_testnet/logs
retention-days: 3
doppelganger-protection-failure-test:
needs: dockerfile-ubuntu
runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@v4
- name: Install dependencies
run: |
sudo add-apt-repository ppa:rmescandon/yq
echo "deb [trusted=yes] https://apt.fury.io/kurtosis-tech/ /" | sudo tee /etc/apt/sources.list.d/kurtosis.list
sudo apt update
sudo apt install -y kurtosis-cli yq
kurtosis analytics disable
- name: Download Docker image artifact
uses: actions/download-artifact@v4
with:
name: lighthouse-docker
path: .
- name: Load Docker image
run: docker load -i lighthouse-docker.tar
- name: Run the doppelganger protection failure test script
run: |
./doppelganger_protection.sh failure
working-directory: scripts/tests
- name: Upload logs artifact
uses: actions/upload-artifact@v4
with:
name: logs-doppelganger-protection-failure
path: |
scripts/local_testnet/logs
retention-days: 3
# This job succeeds ONLY IF all others succeed. It is used by the merge queue to determine whether
# a PR is safe to merge. New jobs should be added here.
local-testnet-success:
name: local-testnet-success
runs-on: ubuntu-latest
needs: [
'dockerfile-ubuntu',
'run-local-testnet',
'doppelganger-protection-success-test',
'doppelganger-protection-failure-test',
]
steps:
- uses: actions/checkout@v4
- name: Check that success job is dependent on all others
run: ./scripts/ci/check-success-job.sh ./.github/workflows/local-testnet.yml local-testnet-success

View File

@@ -31,44 +31,28 @@ jobs:
strategy:
matrix:
arch: [aarch64-unknown-linux-gnu,
aarch64-unknown-linux-gnu-portable,
x86_64-unknown-linux-gnu,
x86_64-unknown-linux-gnu-portable,
x86_64-apple-darwin,
x86_64-apple-darwin-portable,
x86_64-windows,
x86_64-windows-portable]
x86_64-windows]
include:
- arch: aarch64-unknown-linux-gnu
runner: ${{ github.repository == 'sigp/lighthouse' && fromJson('["self-hosted", "linux", "release", "large"]') || 'ubuntu-latest' }}
profile: maxperf
- arch: aarch64-unknown-linux-gnu-portable
runner: ${{ github.repository == 'sigp/lighthouse' && fromJson('["self-hosted", "linux", "release", "large"]') || 'ubuntu-latest' }}
profile: maxperf
- arch: x86_64-unknown-linux-gnu
runner: ${{ github.repository == 'sigp/lighthouse' && fromJson('["self-hosted", "linux", "release", "large"]') || 'ubuntu-latest' }}
profile: maxperf
- arch: x86_64-unknown-linux-gnu-portable
runner: ${{ github.repository == 'sigp/lighthouse' && fromJson('["self-hosted", "linux", "release", "large"]') || 'ubuntu-latest' }}
profile: maxperf
- arch: x86_64-apple-darwin
runner: macos-latest
profile: maxperf
- arch: x86_64-apple-darwin-portable
runner: macos-latest
runner: macos-13
profile: maxperf
- arch: x86_64-windows
runner: ${{ github.repository == 'sigp/lighthouse' && fromJson('["self-hosted", "windows", "release"]') || 'windows-2019' }}
profile: maxperf
- arch: x86_64-windows-portable
runner: ${{ github.repository == 'sigp/lighthouse' && fromJson('["self-hosted", "windows", "release"]') || 'windows-2019' }}
profile: maxperf
runs-on: ${{ matrix.runner }}
needs: extract-version
steps:
- name: Checkout sources
uses: actions/checkout@v3
uses: actions/checkout@v4
- name: Get latest version of stable Rust
if: env.SELF_HOSTED_RUNNERS == 'false'
run: rustup update stable
@@ -80,7 +64,7 @@ jobs:
- uses: KyleMayes/install-llvm-action@v1
if: env.SELF_HOSTED_RUNNERS == 'false' && startsWith(matrix.arch, 'x86_64-windows')
with:
version: "15.0"
version: "17.0"
directory: ${{ runner.temp }}/llvm
- name: Set LIBCLANG_PATH
if: startsWith(matrix.arch, 'x86_64-windows')
@@ -90,53 +74,29 @@ jobs:
# Builds
# ==============================
- name: Build Lighthouse for aarch64-unknown-linux-gnu-portable
if: matrix.arch == 'aarch64-unknown-linux-gnu-portable'
run: |
cargo install cross
env CROSS_PROFILE=${{ matrix.profile }} make build-aarch64-portable
- name: Build Lighthouse for aarch64-unknown-linux-gnu
if: matrix.arch == 'aarch64-unknown-linux-gnu'
run: |
cargo install cross
env CROSS_PROFILE=${{ matrix.profile }} make build-aarch64
- name: Build Lighthouse for x86_64-unknown-linux-gnu-portable
if: matrix.arch == 'x86_64-unknown-linux-gnu-portable'
run: |
cargo install cross
env CROSS_PROFILE=${{ matrix.profile }} make build-x86_64-portable
env CROSS_PROFILE=${{ matrix.profile }} make build-aarch64-portable
- name: Build Lighthouse for x86_64-unknown-linux-gnu
if: matrix.arch == 'x86_64-unknown-linux-gnu'
run: |
cargo install cross
env CROSS_PROFILE=${{ matrix.profile }} make build-x86_64
env CROSS_PROFILE=${{ matrix.profile }} make build-x86_64-portable
- name: Move cross-compiled binary
if: startsWith(matrix.arch, 'aarch64')
run: mv target/aarch64-unknown-linux-gnu/${{ matrix.profile }}/lighthouse ~/.cargo/bin/lighthouse
if: contains(matrix.arch, 'unknown-linux-gnu')
run: mv target/${{ matrix.arch }}/${{ matrix.profile }}/lighthouse ~/.cargo/bin/lighthouse
- name: Move cross-compiled binary
if: startsWith(matrix.arch, 'x86_64-unknown-linux-gnu')
run: mv target/x86_64-unknown-linux-gnu/${{ matrix.profile }}/lighthouse ~/.cargo/bin/lighthouse
- name: Build Lighthouse for x86_64-apple-darwin portable
if: matrix.arch == 'x86_64-apple-darwin-portable'
run: cargo install --path lighthouse --force --locked --features portable,gnosis --profile ${{ matrix.profile }}
- name: Build Lighthouse for x86_64-apple-darwin modern
- name: Build Lighthouse for x86_64-apple-darwin
if: matrix.arch == 'x86_64-apple-darwin'
run: cargo install --path lighthouse --force --locked --features modern,gnosis --profile ${{ matrix.profile }}
- name: Build Lighthouse for Windows portable
if: matrix.arch == 'x86_64-windows-portable'
run: cargo install --path lighthouse --force --locked --features portable,gnosis --profile ${{ matrix.profile }}
- name: Build Lighthouse for Windows modern
- name: Build Lighthouse for Windows
if: matrix.arch == 'x86_64-windows'
run: cargo install --path lighthouse --force --locked --features modern,gnosis --profile ${{ matrix.profile }}
run: cargo install --path lighthouse --force --locked --features portable,gnosis --profile ${{ matrix.profile }}
- name: Configure GPG and create artifacts
if: startsWith(matrix.arch, 'x86_64-windows') != true
@@ -151,6 +111,11 @@ jobs:
cd artifacts
tar -czf lighthouse-${{ needs.extract-version.outputs.VERSION }}-${{ matrix.arch }}.tar.gz lighthouse
echo "$GPG_PASSPHRASE" | gpg --passphrase-fd 0 --pinentry-mode loopback --batch -ab lighthouse-${{ needs.extract-version.outputs.VERSION }}-${{ matrix.arch }}.tar.gz
for ext in "tar.gz" "tar.gz.asc";\
do for f in *.$ext;\
do cp $f "../${f%.$ext}-portable.$ext";\
done;\
done
mv *tar.gz* ..
- name: Configure GPG and create artifacts Windows
@@ -172,17 +137,35 @@ jobs:
# This is required to share artifacts between different jobs
# =======================================================================
- name: Upload artifact
uses: actions/upload-artifact@v3
- name: Upload artifact
uses: actions/upload-artifact@v4
with:
name: lighthouse-${{ needs.extract-version.outputs.VERSION }}-${{ matrix.arch }}.tar.gz
path: lighthouse-${{ needs.extract-version.outputs.VERSION }}-${{ matrix.arch }}.tar.gz
compression-level: 0
- name: Upload artifact (copy)
if: startsWith(matrix.arch, 'x86_64-windows') != true
uses: actions/upload-artifact@v4
with:
name: lighthouse-${{ needs.extract-version.outputs.VERSION }}-${{ matrix.arch }}-portable.tar.gz
path: lighthouse-${{ needs.extract-version.outputs.VERSION }}-${{ matrix.arch }}-portable.tar.gz
compression-level: 0
- name: Upload signature
uses: actions/upload-artifact@v3
uses: actions/upload-artifact@v4
with:
name: lighthouse-${{ needs.extract-version.outputs.VERSION }}-${{ matrix.arch }}.tar.gz.asc
path: lighthouse-${{ needs.extract-version.outputs.VERSION }}-${{ matrix.arch }}.tar.gz.asc
compression-level: 0
- name: Upload signature (copy)
if: startsWith(matrix.arch, 'x86_64-windows') != true
uses: actions/upload-artifact@v4
with:
name: lighthouse-${{ needs.extract-version.outputs.VERSION }}-${{ matrix.arch }}-portable.tar.gz.asc
path: lighthouse-${{ needs.extract-version.outputs.VERSION }}-${{ matrix.arch }}-portable.tar.gz.asc
compression-level: 0
draft-release:
name: Draft Release
@@ -193,7 +176,7 @@ jobs:
steps:
# This is necessary for generating the changelog. It has to come before "Download Artifacts" or else it deletes the artifacts.
- name: Checkout sources
uses: actions/checkout@v3
uses: actions/checkout@v4
with:
fetch-depth: 0
@@ -202,7 +185,7 @@ jobs:
# ==============================
- name: Download artifacts
uses: actions/download-artifact@v3
uses: actions/download-artifact@v4
# ==============================
# Create release draft
@@ -227,9 +210,9 @@ jobs:
## Testing Checklist (DELETE ME)
- [ ] Run on synced Prater Sigma Prime nodes.
- [ ] Run on synced Holesky Sigma Prime nodes.
- [ ] Run on synced Canary (mainnet) Sigma Prime nodes.
- [ ] Resync a Prater node.
- [ ] Resync a Holesky node.
- [ ] Resync a mainnet node.
## Release Checklist (DELETE ME)
@@ -282,9 +265,6 @@ jobs:
| <img src="https://simpleicons.org/icons/docker.svg" style="width: 32px;"/> | Docker | [${{ env.VERSION }}](https://hub.docker.com/r/${{ env.IMAGE_NAME }}/tags?page=1&ordering=last_updated&name=${{ env.VERSION }}) | [${{ env.IMAGE_NAME }}](https://hub.docker.com/r/${{ env.IMAGE_NAME }}) |
ENDBODY
)
assets=()
for asset in ./lighthouse-*.tar.gz*; do
assets+=("-a" "$asset/$asset")
done
assets=(./lighthouse-*.tar.gz*/lighthouse-*.tar.gz*)
tag_name="${{ env.VERSION }}"
echo "$body" | hub release create --draft "${assets[@]}" -F "-" "$tag_name"
echo "$body" | gh release create --draft -F "-" "$tag_name" "${assets[@]}"

View File

@@ -18,15 +18,42 @@ env:
# Deny warnings in CI
# Disable debug info (see https://github.com/sigp/lighthouse/issues/4005)
RUSTFLAGS: "-D warnings -C debuginfo=0"
# The Nightly version used for cargo-udeps, might need updating from time to time.
PINNED_NIGHTLY: nightly-2023-04-16
# Prevent Github API rate limiting.
LIGHTHOUSE_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
# Enable self-hosted runners for the sigp repo only.
SELF_HOSTED_RUNNERS: ${{ github.repository == 'sigp/lighthouse' }}
# Self-hosted runners need to reference a different host for `./watch` tests.
WATCH_HOST: ${{ github.repository == 'sigp/lighthouse' && 'host.docker.internal' || 'localhost' }}
# Disable incremental compilation
CARGO_INCREMENTAL: 0
# Enable portable to prevent issues with caching `blst` for the wrong CPU type
TEST_FEATURES: portable
jobs:
check-labels:
runs-on: ubuntu-latest
name: Check for 'skip-ci' label
outputs:
skip_ci: ${{ steps.set-output.outputs.SKIP_CI }}
steps:
- name: check for skip-ci label
id: set-output
env:
LABELS: ${{ toJson(github.event.pull_request.labels) }}
run: |
SKIP_CI="false"
if [ -z "${LABELS}" ] || [ "${LABELS}" = "null" ]; then
LABELS="none";
else
LABELS=$(echo ${LABELS} | jq -r '.[].name')
fi
for label in ${LABELS}; do
if [ "$label" = "skip-ci" ]; then
SKIP_CI="true"
break
fi
done
echo "skip_ci=$SKIP_CI" >> $GITHUB_OUTPUT
target-branch-check:
name: target-branch-check
runs-on: ubuntu-latest
@@ -34,312 +61,305 @@ jobs:
steps:
- name: Check that the pull request is not targeting the stable branch
run: test ${{ github.base_ref }} != "stable"
extract-msrv:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Extract Minimum Supported Rust Version (MSRV)
run: |
metadata=$(cargo metadata --no-deps --format-version 1)
msrv=$(echo $metadata | jq -r '.packages | map(select(.name == "lighthouse")) | .[0].rust_version')
echo "MSRV=$msrv" >> $GITHUB_OUTPUT
id: extract_msrv
outputs:
MSRV: ${{ steps.extract_msrv.outputs.MSRV }}
cargo-fmt:
name: cargo-fmt
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Get latest version of stable Rust
run: rustup update stable
- name: Check formatting with cargo fmt
run: make cargo-fmt
release-tests-ubuntu:
name: release-tests-ubuntu
needs: [check-labels]
if: needs.check-labels.outputs.skip_ci != 'true'
# Use self-hosted runners only on the sigp repo.
runs-on: ${{ github.repository == 'sigp/lighthouse' && fromJson('["self-hosted", "linux", "CI", "large"]') || 'ubuntu-latest' }}
needs: cargo-fmt
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
- name: Get latest version of stable Rust
if: env.SELF_HOSTED_RUNNERS == false
run: rustup update stable
if: env.SELF_HOSTED_RUNNERS == 'false'
uses: moonrepo/setup-rust@v1
with:
channel: stable
cache-target: release
bins: cargo-nextest
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- name: Install Foundry (anvil)
if: env.SELF_HOSTED_RUNNERS == 'false'
uses: foundry-rs/foundry-toolchain@v1
with:
version: nightly-ca67d15f4abd46394b324c50e21e66f306a1162d
- name: Run tests in release
run: make test-release
run: make nextest-release
- name: Show cache stats
if: env.SELF_HOSTED_RUNNERS == 'true'
run: sccache --show-stats
release-tests-windows:
name: release-tests-windows
needs: [check-labels]
if: needs.check-labels.outputs.skip_ci != 'true'
runs-on: ${{ github.repository == 'sigp/lighthouse' && fromJson('["self-hosted", "windows", "CI"]') || 'windows-2019' }}
needs: cargo-fmt
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
- name: Get latest version of stable Rust
if: env.SELF_HOSTED_RUNNERS == false
run: rustup update stable
if: env.SELF_HOSTED_RUNNERS == 'false'
uses: moonrepo/setup-rust@v1
with:
channel: stable
cache-target: release
bins: cargo-nextest
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- name: Install Foundry (anvil)
if: env.SELF_HOSTED_RUNNERS == 'false'
uses: foundry-rs/foundry-toolchain@v1
with:
version: nightly-ca67d15f4abd46394b324c50e21e66f306a1162d
- name: Install make
if: env.SELF_HOSTED_RUNNERS == 'false'
run: choco install -y make
- uses: KyleMayes/install-llvm-action@v1
if: env.SELF_HOSTED_RUNNERS == false
with:
version: "15.0"
directory: ${{ runner.temp }}/llvm
- name: Set LIBCLANG_PATH
run: echo "LIBCLANG_PATH=$((gcm clang).source -replace "clang.exe")" >> $env:GITHUB_ENV
- name: Run tests in release
run: make test-release
run: make nextest-release
- name: Show cache stats
if: env.SELF_HOSTED_RUNNERS == 'true'
run: sccache --show-stats
beacon-chain-tests:
name: beacon-chain-tests
needs: [check-labels]
if: needs.check-labels.outputs.skip_ci != 'true'
# Use self-hosted runners only on the sigp repo.
runs-on: ${{ github.repository == 'sigp/lighthouse' && fromJson('["self-hosted", "linux", "CI", "large"]') || 'ubuntu-latest' }}
needs: cargo-fmt
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
- name: Get latest version of stable Rust
if: env.SELF_HOSTED_RUNNERS == false
run: rustup update stable
if: env.SELF_HOSTED_RUNNERS == 'false'
uses: moonrepo/setup-rust@v1
with:
channel: stable
cache-target: release
bins: cargo-nextest
- name: Run beacon_chain tests for all known forks
run: make test-beacon-chain
- name: Show cache stats
if: env.SELF_HOSTED_RUNNERS == 'true'
run: sccache --show-stats
op-pool-tests:
name: op-pool-tests
needs: [check-labels]
if: needs.check-labels.outputs.skip_ci != 'true'
runs-on: ubuntu-latest
needs: cargo-fmt
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
- name: Get latest version of stable Rust
run: rustup update stable
uses: moonrepo/setup-rust@v1
with:
channel: stable
cache-target: release
bins: cargo-nextest
- name: Run operation_pool tests for all known forks
run: make test-op-pool
network-tests:
name: network-tests
needs: [check-labels]
if: needs.check-labels.outputs.skip_ci != 'true'
runs-on: ubuntu-latest
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
steps:
- uses: actions/checkout@v4
- name: Get latest version of stable Rust
uses: moonrepo/setup-rust@v1
with:
channel: stable
cache-target: release
bins: cargo-nextest
- name: Run network tests for all known forks
run: make test-network
slasher-tests:
name: slasher-tests
needs: [check-labels]
if: needs.check-labels.outputs.skip_ci != 'true'
runs-on: ubuntu-latest
needs: cargo-fmt
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
- name: Get latest version of stable Rust
run: rustup update stable
uses: moonrepo/setup-rust@v1
with:
channel: stable
cache-target: release
bins: cargo-nextest
- name: Run slasher tests for all supported backends
run: make test-slasher
debug-tests-ubuntu:
name: debug-tests-ubuntu
needs: [check-labels]
if: needs.check-labels.outputs.skip_ci != 'true'
# Use self-hosted runners only on the sigp repo.
runs-on: ${{ github.repository == 'sigp/lighthouse' && fromJson('["self-hosted", "linux", "CI", "large"]') || 'ubuntu-latest' }}
needs: cargo-fmt
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
- name: Get latest version of stable Rust
if: env.SELF_HOSTED_RUNNERS == false
run: rustup update stable
if: env.SELF_HOSTED_RUNNERS == 'false'
uses: moonrepo/setup-rust@v1
with:
channel: stable
bins: cargo-nextest
- name: Install Foundry (anvil)
if: env.SELF_HOSTED_RUNNERS == 'false'
uses: foundry-rs/foundry-toolchain@v1
with:
version: nightly-ca67d15f4abd46394b324c50e21e66f306a1162d
- name: Run tests in debug
run: make test-debug
run: make nextest-debug
- name: Show cache stats
if: env.SELF_HOSTED_RUNNERS == 'true'
run: sccache --show-stats
state-transition-vectors-ubuntu:
name: state-transition-vectors-ubuntu
needs: [check-labels]
if: needs.check-labels.outputs.skip_ci != 'true'
runs-on: ubuntu-latest
needs: cargo-fmt
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
- name: Get latest version of stable Rust
run: rustup update stable
uses: moonrepo/setup-rust@v1
with:
channel: stable
cache-target: release
- name: Run state_transition_vectors in release.
run: make run-state-transition-tests
ef-tests-ubuntu:
name: ef-tests-ubuntu
needs: [check-labels]
if: needs.check-labels.outputs.skip_ci != 'true'
# Use self-hosted runners only on the sigp repo.
runs-on: ${{ github.repository == 'sigp/lighthouse' && fromJson('["self-hosted", "linux", "CI", "small"]') || 'ubuntu-latest' }}
needs: cargo-fmt
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
- name: Get latest version of stable Rust
if: env.SELF_HOSTED_RUNNERS == false
run: rustup update stable
- name: Run consensus-spec-tests with blst, milagro and fake_crypto
run: make test-ef
dockerfile-ubuntu:
name: dockerfile-ubuntu
runs-on: ubuntu-latest
needs: cargo-fmt
steps:
- uses: actions/checkout@v3
- name: Get latest version of stable Rust
run: rustup update stable
- name: Build the root Dockerfile
run: docker build --build-arg FEATURES=portable -t lighthouse:local .
- name: Test the built image
run: docker run -t lighthouse:local lighthouse --version
eth1-simulator-ubuntu:
name: eth1-simulator-ubuntu
runs-on: ubuntu-latest
needs: cargo-fmt
steps:
- uses: actions/checkout@v3
- name: Get latest version of stable Rust
run: rustup update stable
- name: Install Foundry (anvil)
uses: foundry-rs/foundry-toolchain@v1
if: env.SELF_HOSTED_RUNNERS == 'false'
uses: moonrepo/setup-rust@v1
with:
version: nightly-ca67d15f4abd46394b324c50e21e66f306a1162d
- name: Run the beacon chain sim that starts from an eth1 contract
run: cargo run --release --bin simulator eth1-sim
merge-transition-ubuntu:
name: merge-transition-ubuntu
channel: stable
cache-target: release
bins: cargo-nextest
- name: Run consensus-spec-tests with blst and fake_crypto
run: make nextest-ef
- name: Show cache stats
if: env.SELF_HOSTED_RUNNERS == 'true'
run: sccache --show-stats
basic-simulator-ubuntu:
name: basic-simulator-ubuntu
needs: [check-labels]
if: needs.check-labels.outputs.skip_ci != 'true'
runs-on: ubuntu-latest
needs: cargo-fmt
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
- name: Get latest version of stable Rust
run: rustup update stable
- name: Install Foundry (anvil)
uses: foundry-rs/foundry-toolchain@v1
uses: moonrepo/setup-rust@v1
with:
version: nightly-ca67d15f4abd46394b324c50e21e66f306a1162d
- name: Run the beacon chain sim and go through the merge transition
run: cargo run --release --bin simulator eth1-sim --post-merge
no-eth1-simulator-ubuntu:
name: no-eth1-simulator-ubuntu
channel: stable
cache-target: release
- name: Run a basic beacon chain sim that starts from Bellatrix
run: cargo run --release --bin simulator basic-sim
fallback-simulator-ubuntu:
name: fallback-simulator-ubuntu
needs: [check-labels]
if: needs.check-labels.outputs.skip_ci != 'true'
runs-on: ubuntu-latest
needs: cargo-fmt
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
- name: Get latest version of stable Rust
run: rustup update stable
- name: Run the beacon chain sim without an eth1 connection
run: cargo run --release --bin simulator no-eth1-sim
syncing-simulator-ubuntu:
name: syncing-simulator-ubuntu
runs-on: ubuntu-latest
needs: cargo-fmt
steps:
- uses: actions/checkout@v3
- name: Get latest version of stable Rust
run: rustup update stable
- name: Install Foundry (anvil)
uses: foundry-rs/foundry-toolchain@v1
uses: moonrepo/setup-rust@v1
with:
version: nightly-ca67d15f4abd46394b324c50e21e66f306a1162d
- name: Run the syncing simulator
run: cargo run --release --bin simulator syncing-sim
doppelganger-protection-test:
name: doppelganger-protection-test
runs-on: ubuntu-latest
needs: cargo-fmt
steps:
- uses: actions/checkout@v3
- name: Get latest version of stable Rust
run: rustup update stable
- name: Install geth
run: |
sudo add-apt-repository -y ppa:ethereum/ethereum
sudo apt-get update
sudo apt-get install ethereum
- name: Install lighthouse and lcli
run: |
make
make install-lcli
- name: Run the doppelganger protection failure test script
run: |
cd scripts/tests
./doppelganger_protection.sh failure genesis.json
- name: Run the doppelganger protection success test script
run: |
cd scripts/tests
./doppelganger_protection.sh success genesis.json
channel: stable
cache-target: release
- name: Run a beacon chain sim which tests VC fallback behaviour
run: cargo run --release --bin simulator fallback-sim
execution-engine-integration-ubuntu:
name: execution-engine-integration-ubuntu
runs-on: ubuntu-latest
needs: cargo-fmt
needs: [check-labels]
if: needs.check-labels.outputs.skip_ci != 'true'
runs-on: ${{ github.repository == 'sigp/lighthouse' && fromJson('["self-hosted", "linux", "CI", "small"]') || 'ubuntu-latest' }}
steps:
- uses: actions/checkout@v3
- uses: actions/setup-go@v3
with:
go-version: '1.20'
- uses: actions/setup-dotnet@v3
with:
dotnet-version: '6.0.201'
- uses: actions/checkout@v4
- name: Get latest version of stable Rust
run: rustup update stable
if: env.SELF_HOSTED_RUNNERS == 'false'
uses: moonrepo/setup-rust@v1
with:
channel: stable
cache-target: release
cache: false
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- name: Add go compiler to $PATH
if: env.SELF_HOSTED_RUNNERS == 'true'
run: echo "/usr/local/go/bin" >> $GITHUB_PATH
- name: Run exec engine integration tests in release
run: make test-exec-engine
check-benchmarks:
name: check-benchmarks
check-code:
name: check-code
runs-on: ubuntu-latest
needs: cargo-fmt
env:
CARGO_INCREMENTAL: 1
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
- name: Get latest version of stable Rust
run: rustup update stable
- name: Typecheck benchmark code without running it
run: make check-benches
clippy:
name: clippy
runs-on: ubuntu-latest
needs: cargo-fmt
steps:
- uses: actions/checkout@v3
- name: Get latest version of stable Rust
run: rustup update stable
uses: moonrepo/setup-rust@v1
with:
channel: stable
cache-target: release
components: rustfmt,clippy
bins: cargo-audit
- name: Check formatting with cargo fmt
run: make cargo-fmt
- name: Lint code for quality and style with Clippy
run: make lint
- name: Certify Cargo.lock freshness
run: git diff --exit-code Cargo.lock
- name: Typecheck benchmark code without running it
run: make check-benches
- name: Validate state_processing feature arbitrary-fuzz
run: make arbitrary-fuzz
- name: Run cargo audit
run: make audit-CI
- name: Run cargo vendor to make sure dependencies can be vendored for packaging, reproducibility and archival purpose
run: CARGO_HOME=$(readlink -f $HOME) make vendor
- name: Markdown-linter
run: make mdlint
check-msrv:
name: check-msrv
runs-on: ubuntu-latest
needs: [cargo-fmt, extract-msrv]
steps:
- uses: actions/checkout@v3
- name: Install Rust @ MSRV (${{ needs.extract-msrv.outputs.MSRV }})
run: rustup override set ${{ needs.extract-msrv.outputs.MSRV }}
- uses: actions/checkout@v4
- name: Install Rust at Minimum Supported Rust Version (MSRV)
run: |
metadata=$(cargo metadata --no-deps --format-version 1)
msrv=$(echo $metadata | jq -r '.packages | map(select(.name == "lighthouse")) | .[0].rust_version')
rustup override set $msrv
- name: Run cargo check
run: cargo check --workspace
arbitrary-check:
name: arbitrary-check
runs-on: ubuntu-latest
needs: cargo-fmt
steps:
- uses: actions/checkout@v3
- name: Get latest version of stable Rust
run: rustup update stable
- name: Validate state_processing feature arbitrary-fuzz
run: make arbitrary-fuzz
cargo-audit:
name: cargo-audit
runs-on: ubuntu-latest
needs: cargo-fmt
steps:
- uses: actions/checkout@v3
- name: Get latest version of stable Rust
run: rustup update stable
- name: Run cargo audit to identify known security vulnerabilities reported to the RustSec Advisory Database
run: make audit
cargo-vendor:
name: cargo-vendor
runs-on: ubuntu-latest
needs: cargo-fmt
steps:
- uses: actions/checkout@v3
- name: Run cargo vendor to make sure dependencies can be vendored for packaging, reproducibility and archival purpose
run: CARGO_HOME=$(readlink -f $HOME) make vendor
cargo-udeps:
name: cargo-udeps
needs: [check-labels]
if: needs.check-labels.outputs.skip_ci != 'true'
runs-on: ubuntu-latest
needs: cargo-fmt
steps:
- uses: actions/checkout@v3
- name: Install Rust (${{ env.PINNED_NIGHTLY }})
run: rustup toolchain install $PINNED_NIGHTLY
- name: Install cargo-udeps
run: cargo install cargo-udeps --locked --force
- uses: actions/checkout@v4
- name: Get latest version of nightly Rust
uses: moonrepo/setup-rust@v1
with:
channel: nightly
bins: cargo-udeps
cache: false
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- name: Create Cargo config dir
run: mkdir -p .cargo
- name: Install custom Cargo config
@@ -351,12 +371,59 @@ jobs:
RUSTFLAGS: ""
compile-with-beta-compiler:
name: compile-with-beta-compiler
needs: [check-labels]
if: needs.check-labels.outputs.skip_ci != 'true'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
- name: Install dependencies
run: sudo apt install -y git gcc g++ make cmake pkg-config llvm-dev libclang-dev clang
run: sudo apt update && sudo apt install -y git gcc g++ make cmake pkg-config llvm-dev libclang-dev clang
- name: Use Rust beta
run: rustup override set beta
- name: Run make
run: make
cli-check:
name: cli-check
needs: [check-labels]
if: needs.check-labels.outputs.skip_ci != 'true'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Get latest version of stable Rust
uses: moonrepo/setup-rust@v1
with:
channel: stable
cache-target: release
- name: Run Makefile to trigger the bash script
run: make cli
# This job succeeds ONLY IF all others succeed. It is used by the merge queue to determine whether
# a PR is safe to merge. New jobs should be added here.
test-suite-success:
name: test-suite-success
if: needs.check-labels.outputs.skip_ci != 'true'
runs-on: ubuntu-latest
needs: [
'check-labels',
'target-branch-check',
'release-tests-ubuntu',
'release-tests-windows',
'beacon-chain-tests',
'op-pool-tests',
'network-tests',
'slasher-tests',
'debug-tests-ubuntu',
'state-transition-vectors-ubuntu',
'ef-tests-ubuntu',
'basic-simulator-ubuntu',
'fallback-simulator-ubuntu',
'execution-engine-integration-ubuntu',
'check-code',
'check-msrv',
'cargo-udeps',
'compile-with-beta-compiler',
'cli-check',
]
steps:
- uses: actions/checkout@v4
- name: Check that success job is dependent on all others
run: ./scripts/ci/check-success-job.sh ./.github/workflows/test-suite.yml test-suite-success

7
.gitignore vendored
View File

@@ -1,4 +1,5 @@
target/
vendor/
**/*.rs.bk
*.pk
*.sk
@@ -9,7 +10,11 @@ perf.data*
/bin
genesis.ssz
/clippy.toml
/.cargo
# IntelliJ
/*.iml
.idea
.idea
# VSCode
/.vscode

View File

@@ -1,12 +1,14 @@
# Contributors Guide
[![GitPOAP badge](https://public-api.gitpoap.io/v1/repo/sigp/lighthouse/badge)](https://www.gitpoap.io/gh/sigp/lighthouse)
Lighthouse is an open-source Ethereum 2.0 client. We're community driven and
Lighthouse is an open-source Ethereum consensus client. We're community driven and
welcome all contribution. We aim to provide a constructive, respectful and fun
environment for collaboration.
We are active contributors to the [Ethereum 2.0 specification](https://github.com/ethereum/eth2.0-specs) and attend all [Eth
2.0 implementers calls](https://github.com/ethereum/eth2.0-pm).
We are active contributors to
the [Ethereum Proof-of-Stake Consensus specification](https://github.com/ethereum/consensus-specs) and attend
all [Ethereum implementers calls](https://github.com/ethereum/pm/).
This guide is geared towards beginners. If you're an open-source veteran feel
free to just skim this document and get straight into crushing issues.
@@ -41,7 +43,7 @@ We recommend the following work-flow for contributors:
1. **Find an issue** to work on, either because it's interesting or suitable to
your skill-set. Use comments to communicate your intentions and ask
questions.
questions.
2. **Work in a feature branch** of your personal fork
(github.com/YOUR_NAME/lighthouse) of the main repository
(github.com/sigp/lighthouse).
@@ -49,13 +51,13 @@ questions.
`unstable` as the base branch to merge your changes into the main repository.
4. Wait for the repository maintainers to **review your changes** to ensure the
issue is addressed satisfactorily. Optionally, mention your PR on
[discord](https://discord.gg/cyAszAh).
[discord](https://discord.gg/cyAszAh).
5. If the issue is addressed the repository maintainers will **merge your
pull-request** and you'll be an official contributor!
Generally, you find an issue you'd like to work on and announce your intentions
to start work in a comment on the issue. Then, do your work on a separate
branch (a "feature branch") in your own fork of the main repository. Once
branch (a "feature branch") in your own fork of the main repository. Once
you're happy and you think the issue has been addressed, create a pull request
into the main repository.
@@ -66,18 +68,20 @@ steps:
1. [Create a
fork](https://help.github.com/articles/fork-a-repo/#fork-an-example-repository)
and [clone
it](https://help.github.com/articles/fork-a-repo/#step-2-create-a-local-clone-of-your-fork)
to your local machine.
and [clone
it](https://help.github.com/articles/fork-a-repo/#step-2-create-a-local-clone-of-your-fork)
to your local machine.
2. [Add an _"upstream"_
branch](https://help.github.com/articles/fork-a-repo/#step-3-configure-git-to-sync-your-fork-with-the-original-spoon-knife-repository)
that tracks github.com/sigp/lighthouse using `$ git remote add upstream
https://github.com/sigp/lighthouse.git` (pro-tip: [use SSH](https://help.github.com/articles/connecting-to-github-with-ssh/) instead of HTTPS).
that tracks github.com/sigp/lighthouse using `$ git remote add upstream
https://github.com/sigp/lighthouse.git` (
pro-tip: [use SSH](https://help.github.com/articles/connecting-to-github-with-ssh/) instead of HTTPS).
3. Create a new feature branch with `$ git checkout -b your_feature_name`. The
name of your branch isn't critical but it should be short and instructive.
E.g., if you're fixing a bug with serialization, you could name your branch
`fix_serialization_bug`.
4. Make sure you sign your commits. See [relevant doc](https://help.github.com/en/github/authenticating-to-github/about-commit-signature-verification).
E.g., if you're fixing a bug with serialization, you could name your branch
`fix_serialization_bug`.
4. Make sure you sign your commits.
See [relevant doc](https://help.github.com/en/github/authenticating-to-github/about-commit-signature-verification).
5. Commit your changes and push them to your fork with `$ git push origin
your_feature_name`.
6. Go to your fork on github.com and use the web interface to create a pull
@@ -92,22 +96,28 @@ by Rob Allen that provides much more detail on each of these steps, if you're
having trouble. As always, jump on [discord](https://discord.gg/cyAszAh)
if you get stuck.
Additionally,
the ["Contributing to Lighthouse" section](https://lighthouse-book.sigmaprime.io/contributing.html#contributing-to-lighthouse)
of the Lighthouse Book provides more details on the setup.
## FAQs
### I don't think I have anything to add
There's lots to be done and there's all sorts of tasks. You can do anything
from correcting typos through to writing core consensus code. If you reach out,
from enhancing documentation through to writing core consensus code. If you reach out,
we'll include you.
Please note, to maintain project quality, we may not accept PRs for small typos or changes
with minimal impact.
### I'm not sure my Rust is good enough
We're open to developers of all levels. If you create a PR and your code
doesn't meet our standards, we'll help you fix it and we'll share the reasoning
with you. Contributing to open-source is a great way to learn.
### I'm not sure I know enough about Ethereum 2.0
### I'm not sure I know enough about Ethereum
No problems, there's plenty of tasks that don't require extensive Ethereum
knowledge. You can learn about Ethereum as you go.

4126
Cargo.lock generated

File diff suppressed because it is too large Load Diff

View File

@@ -9,6 +9,7 @@ members = [
"beacon_node/client",
"beacon_node/eth1",
"beacon_node/lighthouse_network",
"beacon_node/lighthouse_network/gossipsub",
"beacon_node/execution_layer",
"beacon_node/http_api",
"beacon_node/http_metrics",
@@ -59,6 +60,7 @@ members = [
"consensus/swap_or_not_shuffle",
"crypto/bls",
"crypto/kzg",
"crypto/eth2_key_derivation",
"crypto/eth2_keystore",
"crypto/eth2_wallet",
@@ -93,23 +95,29 @@ resolver = "2"
edition = "2021"
[workspace.dependencies]
anyhow = "1"
arbitrary = { version = "1", features = ["derive"] }
async-channel = "1.9.0"
bincode = "1"
bitvec = "1"
byteorder = "1"
bytes = "1"
clap = "2"
clap = { version = "4.5.4", features = ["cargo", "wrap_help"] }
# Turn off c-kzg's default features which include `blst/portable`. We can turn on blst's portable
# feature ourselves when desired.
c-kzg = { version = "1", default-features = false }
compare_fields_derive = { path = "common/compare_fields_derive" }
criterion = "0.3"
criterion = "0.5"
delay_map = "0.3"
derivative = "2"
dirs = "3"
discv5 = { version = "0.3", features = ["libp2p"] }
either = "1.9"
discv5 = { version = "0.4.1", features = ["libp2p"] }
env_logger = "0.9"
error-chain = "0.12"
ethereum-types = "0.14"
ethereum_hashing = "1.0.0-beta.2"
ethereum_serde_utils = "0.5"
ethereum_hashing = "0.6.0"
ethereum_serde_utils = "0.5.2"
ethereum_ssz = "0.5"
ethereum_ssz_derive = "0.5"
ethers-core = "1"
@@ -119,13 +127,14 @@ fnv = "1"
fs2 = "0.4"
futures = "0.3"
hex = "0.4"
hyper = "0.14"
hyper = "1"
itertools = "0.10"
lazy_static = "1"
libsecp256k1 = "0.7"
log = "0.4"
lru = "0.7"
lru = "0.12"
maplit = "1"
milhouse = "0.1"
num_cpus = "1"
parking_lot = "0.12"
paste = "1"
@@ -136,35 +145,40 @@ r2d2 = "0.8"
rand = "0.8"
rayon = "1.7"
regex = "1"
reqwest = { version = "0.11", default-features = false, features = ["blocking", "json", "stream", "rustls-tls"] }
reqwest = { version = "0.11", default-features = false, features = ["blocking", "json", "stream", "rustls-tls", "native-tls-vendored"] }
ring = "0.16"
rpds = "0.11"
rusqlite = { version = "0.28", features = ["bundled"] }
serde = { version = "1", features = ["derive"] }
serde_json = "1"
serde_repr = "0.1"
serde_yaml = "0.8"
serde_yaml = "0.9"
sha2 = "0.9"
slog = { version = "2", features = ["max_level_trace", "release_max_level_trace"] }
slog = { version = "2", features = ["max_level_trace", "release_max_level_trace", "nested-values"] }
slog-async = "2"
slog-term = "2"
sloggers = { version = "2", features = ["json"] }
smallvec = "1"
smallvec = "1.11.2"
snap = "1"
ssz_types = "0.5"
ssz_types = "0.6"
strum = { version = "0.24", features = ["derive"] }
superstruct = "0.6"
superstruct = "0.8"
syn = "1"
sysinfo = "0.26"
tempfile = "3"
tokio = { version = "1", features = ["rt-multi-thread", "sync"] }
tokio = { version = "1", features = ["rt-multi-thread", "sync", "signal"] }
tokio-stream = { version = "0.1", features = ["sync"] }
tokio-util = { version = "0.6", features = ["codec", "compat", "time"] }
tree_hash = "0.5"
tree_hash_derive = "0.5"
tokio-util = { version = "0.7", features = ["codec", "compat", "time"] }
tracing = "0.1.40"
tracing-appender = "0.2"
tracing-core = "0.1"
tracing-log = "0.2"
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
tree_hash = "0.6"
tree_hash_derive = "0.6"
url = "2"
uuid = { version = "0.8", features = ["serde", "v4"] }
# TODO update to warp 0.3.6 after released.
warp = { git = "https://github.com/seanmonstar/warp.git", default-features = false, features = ["tls"] }
warp = { version = "0.3.7", default-features = false, features = ["tls"] }
zeroize = { version = "1", features = ["zeroize_derive"] }
zip = "0.6"
@@ -172,7 +186,7 @@ zip = "0.6"
account_utils = { path = "common/account_utils" }
beacon_chain = { path = "beacon_node/beacon_chain" }
beacon_node = { path = "beacon_node" }
beacon_processor = { path = "beacon_node/beacon_processor" }
beacon_processor = { path = "beacon_node/beacon_processor" }
bls = { path = "crypto/bls" }
cached_tree_hash = { path = "consensus/cached_tree_hash" }
clap_utils = { path = "common/clap_utils" }
@@ -192,8 +206,10 @@ execution_layer = { path = "beacon_node/execution_layer" }
filesystem = { path = "common/filesystem" }
fork_choice = { path = "consensus/fork_choice" }
genesis = { path = "beacon_node/genesis" }
gossipsub = { path = "beacon_node/lighthouse_network/gossipsub/" }
http_api = { path = "beacon_node/http_api" }
int_to_bytes = { path = "consensus/int_to_bytes" }
kzg = { path = "crypto/kzg" }
lighthouse_metrics = { path = "common/lighthouse_metrics" }
lighthouse_network = { path = "beacon_node/lighthouse_network" }
lighthouse_version = { path = "common/lighthouse_version" }
@@ -207,7 +223,7 @@ network = { path = "beacon_node/network" }
operation_pool = { path = "beacon_node/operation_pool" }
pretty_reqwest_error = { path = "common/pretty_reqwest_error" }
proto_array = { path = "consensus/proto_array" }
safe_arith = {path = "consensus/safe_arith"}
safe_arith = { path = "consensus/safe_arith" }
sensitive_url = { path = "common/sensitive_url" }
slasher = { path = "slasher" }
slashing_protection = { path = "validator_client/slashing_protection" }
@@ -218,7 +234,7 @@ swap_or_not_shuffle = { path = "consensus/swap_or_not_shuffle" }
task_executor = { path = "common/task_executor" }
types = { path = "consensus/types" }
unused_port = { path = "common/unused_port" }
validator_client = { path = "validator_client/" }
validator_client = { path = "validator_client" }
validator_dir = { path = "common/validator_dir" }
warp_utils = { path = "common/warp_utils" }

View File

@@ -1,5 +1,5 @@
[target.x86_64-unknown-linux-gnu]
pre-build = ["apt-get install -y cmake clang-3.9"]
pre-build = ["apt-get install -y cmake clang-5.0"]
[target.aarch64-unknown-linux-gnu]
pre-build = ["apt-get install -y cmake clang-3.9"]
pre-build = ["apt-get install -y cmake clang-5.0"]

View File

@@ -1,10 +1,12 @@
FROM rust:1.69.0-bullseye AS builder
FROM rust:1.78.0-bullseye AS builder
RUN apt-get update && apt-get -y upgrade && apt-get install -y cmake libclang-dev
COPY . lighthouse
ARG FEATURES
ARG PROFILE=release
ARG CARGO_USE_GIT_CLI=true
ENV FEATURES $FEATURES
ENV PROFILE $PROFILE
ENV CARGO_NET_GIT_FETCH_WITH_CLI=$CARGO_USE_GIT_CLI
RUN cd lighthouse && make
FROM ubuntu:22.04
@@ -13,4 +15,4 @@ RUN apt-get update && apt-get -y upgrade && apt-get install -y --no-install-reco
ca-certificates \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
COPY --from=builder /usr/local/cargo/bin/lighthouse /usr/local/bin/lighthouse
COPY --from=builder /usr/local/cargo/bin/lighthouse /usr/local/bin/lighthouse

7
FUNDING.json Normal file
View File

@@ -0,0 +1,7 @@
{
"drips": {
"ethereum": {
"ownedBy": "0x25c4a76E7d118705e7Ea2e9b7d8C59930d8aCD3b"
}
}
}

View File

@@ -14,7 +14,7 @@ BUILD_PATH_AARCH64 = "target/$(AARCH64_TAG)/release"
PINNED_NIGHTLY ?= nightly
CLIPPY_PINNED_NIGHTLY=nightly-2022-05-19
# List of features to use when building natively. Can be overriden via the environment.
# List of features to use when building natively. Can be overridden via the environment.
# No jemalloc on Windows
ifeq ($(OS),Windows_NT)
FEATURES?=
@@ -31,12 +31,15 @@ CROSS_PROFILE ?= release
# List of features to use when running EF tests.
EF_TEST_FEATURES ?=
# List of features to use when running CI tests.
TEST_FEATURES ?=
# Cargo profile for regular builds.
PROFILE ?= release
# List of all hard forks. This list is used to set env variables for several tests so that
# they run for different forks.
FORKS=phase0 altair merge capella
FORKS=phase0 altair bellatrix capella deneb electra
# Extra flags for Cargo
CARGO_INSTALL_EXTRA_FLAGS?=
@@ -79,6 +82,11 @@ build-aarch64:
build-aarch64-portable:
cross build --bin lighthouse --target aarch64-unknown-linux-gnu --features "portable,$(CROSS_FEATURES)" --profile "$(CROSS_PROFILE)" --locked
build-lcli-x86_64:
cross build --bin lcli --target x86_64-unknown-linux-gnu --features "portable" --profile "$(CROSS_PROFILE)" --locked
build-lcli-aarch64:
cross build --bin lcli --target aarch64-unknown-linux-gnu --features "portable" --profile "$(CROSS_PROFILE)" --locked
# Create a `.tar.gz` containing a binary for a specific target.
define tarball_release_binary
cp $(1)/lighthouse $(BIN_DIR)/lighthouse
@@ -106,12 +114,26 @@ build-release-tarballs:
# Runs the full workspace tests in **release**, without downloading any additional
# test vectors.
test-release:
cargo test --workspace --release --exclude ef_tests --exclude beacon_chain --exclude slasher
cargo test --workspace --release --features "$(TEST_FEATURES)" \
--exclude ef_tests --exclude beacon_chain --exclude slasher --exclude network
# Runs the full workspace tests in **release**, without downloading any additional
# test vectors, using nextest.
nextest-release:
cargo nextest run --workspace --release --features "$(TEST_FEATURES)" \
--exclude ef_tests --exclude beacon_chain --exclude slasher --exclude network
# Runs the full workspace tests in **debug**, without downloading any additional test
# vectors.
test-debug:
cargo test --workspace --exclude ef_tests --exclude beacon_chain
cargo test --workspace --features "$(TEST_FEATURES)" \
--exclude ef_tests --exclude beacon_chain --exclude network
# Runs the full workspace tests in **debug**, without downloading any additional test
# vectors, using nextest.
nextest-debug:
cargo nextest run --workspace --features "$(TEST_FEATURES)" \
--exclude ef_tests --exclude beacon_chain --exclude network
# Runs cargo-fmt (linter).
cargo-fmt:
@@ -119,35 +141,49 @@ cargo-fmt:
# Typechecks benchmark code
check-benches:
cargo check --workspace --benches
cargo check --workspace --benches --features "$(TEST_FEATURES)"
# Runs only the ef-test vectors.
run-ef-tests:
rm -rf $(EF_TESTS)/.accessed_file_log.txt
cargo test --release -p ef_tests --features "ef_tests,$(EF_TEST_FEATURES)"
cargo test --release -p ef_tests --features "ef_tests,$(EF_TEST_FEATURES),fake_crypto"
cargo test --release -p ef_tests --features "ef_tests,$(EF_TEST_FEATURES),milagro"
./$(EF_TESTS)/check_all_files_accessed.py $(EF_TESTS)/.accessed_file_log.txt $(EF_TESTS)/consensus-spec-tests
# Runs EF test vectors with nextest
nextest-run-ef-tests:
rm -rf $(EF_TESTS)/.accessed_file_log.txt
cargo nextest run --release -p ef_tests --features "ef_tests,$(EF_TEST_FEATURES)"
cargo nextest run --release -p ef_tests --features "ef_tests,$(EF_TEST_FEATURES),fake_crypto"
./$(EF_TESTS)/check_all_files_accessed.py $(EF_TESTS)/.accessed_file_log.txt $(EF_TESTS)/consensus-spec-tests
# Run the tests in the `beacon_chain` crate for all known forks.
test-beacon-chain: $(patsubst %,test-beacon-chain-%,$(FORKS))
test-beacon-chain-%:
env FORK_NAME=$* cargo test --release --features fork_from_env,slasher/lmdb -p beacon_chain
env FORK_NAME=$* cargo nextest run --release --features "fork_from_env,slasher/lmdb,$(TEST_FEATURES)" -p beacon_chain
# Run the tests in the `operation_pool` crate for all known forks.
test-op-pool: $(patsubst %,test-op-pool-%,$(FORKS))
test-op-pool-%:
env FORK_NAME=$* cargo test --release \
--features 'beacon_chain/fork_from_env'\
env FORK_NAME=$* cargo nextest run --release \
--features "beacon_chain/fork_from_env,$(TEST_FEATURES)"\
-p operation_pool
# Run the tests in the `network` crate for all known forks.
test-network: $(patsubst %,test-network-%,$(FORKS))
test-network-%:
env FORK_NAME=$* cargo nextest run --release \
--features "fork_from_env,$(TEST_FEATURES)" \
-p network
# Run the tests in the `slasher` crate for all supported database backends.
test-slasher:
cargo test --release -p slasher --features lmdb
cargo test --release -p slasher --no-default-features --features mdbx
cargo test --release -p slasher --features lmdb,mdbx # both backends enabled
cargo nextest run --release -p slasher --features "lmdb,$(TEST_FEATURES)"
cargo nextest run --release -p slasher --no-default-features --features "mdbx,$(TEST_FEATURES)"
cargo nextest run --release -p slasher --features "lmdb,mdbx,$(TEST_FEATURES)" # both backends enabled
# Runs only the tests/state_transition_vectors tests.
run-state-transition-tests:
@@ -156,6 +192,9 @@ run-state-transition-tests:
# Downloads and runs the EF test vectors.
test-ef: make-ef-tests run-ef-tests
# Downloads and runs the EF test vectors with nextest.
nextest-ef: make-ef-tests nextest-run-ef-tests
# Runs tests checking interop between Lighthouse and execution clients.
test-exec-engine:
make -C $(EXECUTION_ENGINE_INTEGRATION) test
@@ -164,21 +203,37 @@ test-exec-engine:
# test vectors.
test: test-release
# Updates the CLI help text pages in the Lighthouse book, building with Docker.
cli:
docker run --rm --user=root \
-v ${PWD}:/home/runner/actions-runner/lighthouse sigmaprime/github-runner \
bash -c 'cd lighthouse && make && ./scripts/cli.sh'
# Updates the CLI help text pages in the Lighthouse book, building using local
# `cargo`.
cli-local:
make && ./scripts/cli.sh
# Check for markdown files
mdlint:
./scripts/mdlint.sh
# Runs the entire test suite, downloading test vectors if required.
test-full: cargo-fmt test-release test-debug test-ef test-exec-engine
# Lints the code for bad style and potentially unsafe arithmetic using Clippy.
# Clippy lints are opt-in per-crate for now. By default, everything is allowed except for performance and correctness lints.
lint:
cargo clippy --workspace --tests $(EXTRA_CLIPPY_OPTS) -- \
cargo clippy --workspace --tests $(EXTRA_CLIPPY_OPTS) --features "$(TEST_FEATURES)" -- \
-D clippy::fn_to_numeric_cast_any \
-D clippy::manual_let_else \
-D warnings \
-A clippy::derive_partial_eq_without_eq \
-A clippy::from-over-into \
-A clippy::upper-case-acronyms \
-A clippy::vec-init-then-push \
-A clippy::question-mark \
-A clippy::uninlined-format-args
-A clippy::uninlined-format-args \
-A clippy::enum_variant_names
# Lints the code using Clippy and automatically fix some simple compiler warnings.
lint-fix:
@@ -201,12 +256,16 @@ make-ef-tests:
# Verifies that crates compile with fuzzing features enabled
arbitrary-fuzz:
cargo check -p state_processing --features arbitrary-fuzz
cargo check -p slashing_protection --features arbitrary-fuzz
cargo check -p state_processing --features arbitrary-fuzz,$(TEST_FEATURES)
cargo check -p slashing_protection --features arbitrary-fuzz,$(TEST_FEATURES)
# Runs cargo audit (Audit Cargo.lock files for crates with security vulnerabilities reported to the RustSec Advisory Database)
audit:
audit: install-audit audit-CI
install-audit:
cargo install --force cargo-audit
audit-CI:
cargo audit
# Runs `cargo vendor` to make sure dependencies can be vendored for packaging, reproducibility and archival purpose.
@@ -215,7 +274,7 @@ vendor:
# Runs `cargo udeps` to check for unused dependencies
udeps:
cargo +$(PINNED_NIGHTLY) udeps --tests --all-targets --release
cargo +$(PINNED_NIGHTLY) udeps --tests --all-targets --release --features "$(TEST_FEATURES)"
# Performs a `cargo` clean and cleans the `ef_tests` directory.
clean:

View File

@@ -41,7 +41,7 @@ as the canonical staking deposit contract address.
The [Lighthouse Book](https://lighthouse-book.sigmaprime.io) contains information for users and
developers.
The Lighthouse team maintains a blog at [lighthouse-blog.sigmaprime.io][blog] which contains periodical
The Lighthouse team maintains a blog at [lighthouse-blog.sigmaprime.io][blog] which contains periodic
progress updates, roadmap insights and interesting findings.
## Branches

View File

@@ -27,9 +27,6 @@ safe_arith = { workspace = true }
slot_clock = { workspace = true }
filesystem = { workspace = true }
sensitive_url = { workspace = true }
serde = { workspace = true }
serde_json = { workspace = true }
slog = { workspace = true }
[dev-dependencies]
tempfile = { workspace = true }

View File

@@ -29,6 +29,6 @@ Simply run `./account_manager generate` to generate a new random private key,
which will be automatically saved to the correct directory.
If you prefer to use our "deterministic" keys for testing purposes, simply
run `./accounts_manager generate_deterministic -i <index>`, where `index` is
run `./account_manager generate_deterministic -i <index>`, where `index` is
the validator index for the key. This will reliably produce the same key each time
and save it to the directory.
and save it to the directory.

View File

@@ -2,8 +2,11 @@ mod common;
pub mod validator;
pub mod wallet;
use clap::App;
use clap::Arg;
use clap::ArgAction;
use clap::ArgMatches;
use clap::Command;
use clap_utils::FLAG_HEADER;
use environment::Environment;
use types::EthSpec;
@@ -13,25 +16,36 @@ pub const VALIDATOR_DIR_FLAG: &str = "validator-dir";
pub const VALIDATOR_DIR_FLAG_ALIAS: &str = "validators-dir";
pub const WALLETS_DIR_FLAG: &str = "wallets-dir";
pub fn cli_app<'a, 'b>() -> App<'a, 'b> {
App::new(CMD)
.visible_aliases(&["a", "am", "account", CMD])
pub fn cli_app() -> Command {
Command::new(CMD)
.visible_aliases(["a", "am", "account"])
.about("Utilities for generating and managing Ethereum 2.0 accounts.")
.display_order(0)
.arg(
Arg::new("help")
.long("help")
.short('h')
.help("Prints help information")
.action(ArgAction::HelpLong)
.display_order(0)
.help_heading(FLAG_HEADER),
)
.subcommand(wallet::cli_app())
.subcommand(validator::cli_app())
}
/// Run the account manager, returning an error if the operation did not succeed.
pub fn run<T: EthSpec>(matches: &ArgMatches<'_>, env: Environment<T>) -> Result<(), String> {
pub fn run<E: EthSpec>(matches: &ArgMatches, env: Environment<E>) -> Result<(), String> {
match matches.subcommand() {
(wallet::CMD, Some(matches)) => wallet::cli_run(matches)?,
(validator::CMD, Some(matches)) => validator::cli_run(matches, env)?,
(unknown, _) => {
Some((wallet::CMD, matches)) => wallet::cli_run(matches)?,
Some((validator::CMD, matches)) => validator::cli_run(matches, env)?,
Some((unknown, _)) => {
return Err(format!(
"{} is not a valid {} command. See --help.",
unknown, CMD
));
}
_ => return Err("No subcommand provided, see --help for options".to_string()),
}
Ok(())

View File

@@ -4,7 +4,8 @@ use crate::{SECRETS_DIR_FLAG, WALLETS_DIR_FLAG};
use account_utils::{
random_password, read_password_from_user, strip_off_newlines, validator_definitions, PlainText,
};
use clap::{App, Arg, ArgMatches};
use clap::{Arg, ArgAction, ArgMatches, Command};
use clap_utils::FLAG_HEADER;
use directory::{
ensure_dir_exists, parse_path_or_default_with_flag, DEFAULT_SECRET_DIR, DEFAULT_WALLET_DIR,
};
@@ -26,36 +27,39 @@ pub const COUNT_FLAG: &str = "count";
pub const AT_MOST_FLAG: &str = "at-most";
pub const WALLET_PASSWORD_PROMPT: &str = "Enter your wallet's password:";
pub fn cli_app<'a, 'b>() -> App<'a, 'b> {
App::new(CMD)
pub fn cli_app() -> Command {
Command::new(CMD)
.about(
"Creates new validators from an existing EIP-2386 wallet using the EIP-2333 HD key \
derivation scheme.",
)
.arg(
Arg::with_name(WALLET_NAME_FLAG)
Arg::new(WALLET_NAME_FLAG)
.long(WALLET_NAME_FLAG)
.value_name("WALLET_NAME")
.help("Use the wallet identified by this name")
.takes_value(true),
.action(ArgAction::Set)
.display_order(0)
)
.arg(
Arg::with_name(WALLET_PASSWORD_FLAG)
Arg::new(WALLET_PASSWORD_FLAG)
.long(WALLET_PASSWORD_FLAG)
.value_name("WALLET_PASSWORD_PATH")
.help("A path to a file containing the password which will unlock the wallet.")
.takes_value(true),
.action(ArgAction::Set)
.display_order(0)
)
.arg(
Arg::with_name(WALLETS_DIR_FLAG)
Arg::new(WALLETS_DIR_FLAG)
.long(WALLETS_DIR_FLAG)
.value_name(WALLETS_DIR_FLAG)
.help("A path containing Eth2 EIP-2386 wallets. Defaults to ~/.lighthouse/{network}/wallets")
.takes_value(true)
.conflicts_with("datadir"),
.action(ArgAction::Set)
.conflicts_with("datadir")
.display_order(0)
)
.arg(
Arg::with_name(SECRETS_DIR_FLAG)
Arg::new(SECRETS_DIR_FLAG)
.long(SECRETS_DIR_FLAG)
.value_name("SECRETS_DIR")
.help(
@@ -63,37 +67,43 @@ pub fn cli_app<'a, 'b>() -> App<'a, 'b> {
Defaults to ~/.lighthouse/{network}/secrets",
)
.conflicts_with("datadir")
.takes_value(true),
.action(ArgAction::Set)
.display_order(0)
)
.arg(
Arg::with_name(DEPOSIT_GWEI_FLAG)
Arg::new(DEPOSIT_GWEI_FLAG)
.long(DEPOSIT_GWEI_FLAG)
.value_name("DEPOSIT_GWEI")
.help(
"The GWEI value of the deposit amount. Defaults to the minimum amount \
required for an active validator (MAX_EFFECTIVE_BALANCE)",
)
.takes_value(true),
.action(ArgAction::Set)
.display_order(0)
)
.arg(
Arg::with_name(STORE_WITHDRAW_FLAG)
Arg::new(STORE_WITHDRAW_FLAG)
.long(STORE_WITHDRAW_FLAG)
.help(
"If present, the withdrawal keystore will be stored alongside the voting \
keypair. It is generally recommended to *not* store the withdrawal key and \
instead generate them from the wallet seed when required.",
),
)
.action(ArgAction::SetTrue)
.help_heading(FLAG_HEADER)
.display_order(0)
)
.arg(
Arg::with_name(COUNT_FLAG)
Arg::new(COUNT_FLAG)
.long(COUNT_FLAG)
.value_name("VALIDATOR_COUNT")
.help("The number of validators to create, regardless of how many already exist")
.conflicts_with("at-most")
.takes_value(true),
.action(ArgAction::Set)
.display_order(0)
)
.arg(
Arg::with_name(AT_MOST_FLAG)
Arg::new(AT_MOST_FLAG)
.long(AT_MOST_FLAG)
.value_name("AT_MOST_VALIDATORS")
.help(
@@ -101,34 +111,38 @@ pub fn cli_app<'a, 'b>() -> App<'a, 'b> {
reach the given count. Never deletes an existing validator.",
)
.conflicts_with("count")
.takes_value(true),
.action(ArgAction::Set)
.display_order(0)
)
.arg(
Arg::with_name(STDIN_INPUTS_FLAG)
.takes_value(false)
.hidden(cfg!(windows))
Arg::new(STDIN_INPUTS_FLAG)
.action(ArgAction::SetTrue)
.help_heading(FLAG_HEADER)
.hide(cfg!(windows))
.long(STDIN_INPUTS_FLAG)
.help("If present, read all user inputs from stdin instead of tty."),
.help("If present, read all user inputs from stdin instead of tty.")
.display_order(0)
.action(ArgAction::SetTrue)
)
}
pub fn cli_run<T: EthSpec>(
pub fn cli_run<E: EthSpec>(
matches: &ArgMatches,
env: Environment<T>,
env: Environment<E>,
validator_dir: PathBuf,
) -> Result<(), String> {
let spec = env.core_context().eth2_config.spec;
let name: Option<String> = clap_utils::parse_optional(matches, WALLET_NAME_FLAG)?;
let stdin_inputs = cfg!(windows) || matches.is_present(STDIN_INPUTS_FLAG);
let stdin_inputs = cfg!(windows) || matches.get_flag(STDIN_INPUTS_FLAG);
let wallet_base_dir = if matches.value_of("datadir").is_some() {
let wallet_base_dir = if matches.get_one::<String>("datadir").is_some() {
let path: PathBuf = clap_utils::parse_required(matches, "datadir")?;
path.join(DEFAULT_WALLET_DIR)
} else {
parse_path_or_default_with_flag(matches, WALLETS_DIR_FLAG, DEFAULT_WALLET_DIR)?
};
let secrets_dir = if matches.value_of("datadir").is_some() {
let secrets_dir = if matches.get_one::<String>("datadir").is_some() {
let path: PathBuf = clap_utils::parse_required(matches, "datadir")?;
path.join(DEFAULT_SECRET_DIR)
} else {
@@ -145,7 +159,7 @@ pub fn cli_run<T: EthSpec>(
return Err(format!(
"No wallet directory at {:?}. Use the `lighthouse --network {} {} {} {}` command to create a wallet",
wallet_base_dir,
matches.value_of("network").unwrap_or("<NETWORK>"),
matches.get_one::<String>("network").unwrap_or(&String::from("<NETWORK>")),
crate::CMD,
crate::wallet::CMD,
crate::wallet::create::CMD
@@ -246,7 +260,7 @@ pub fn cli_run<T: EthSpec>(
.voting_keystore(keystores.voting, voting_password.as_bytes())
.withdrawal_keystore(keystores.withdrawal, withdrawal_password.as_bytes())
.create_eth1_tx_data(deposit_gwei, &spec)
.store_withdrawal_keystore(matches.is_present(STORE_WITHDRAW_FLAG))
.store_withdrawal_keystore(matches.get_flag(STORE_WITHDRAW_FLAG))
.build()
.map_err(|e| format!("Unable to build validator directory: {:?}", e))?;

View File

@@ -1,6 +1,7 @@
use crate::wallet::create::STDIN_INPUTS_FLAG;
use bls::{Keypair, PublicKey};
use clap::{App, Arg, ArgMatches};
use clap::{Arg, ArgAction, ArgMatches, Command};
use clap_utils::FLAG_HEADER;
use environment::Environment;
use eth2::{
types::{GenesisData, StateId, ValidatorData, ValidatorId, ValidatorStatus},
@@ -14,7 +15,7 @@ use slot_clock::{SlotClock, SystemTimeSlotClock};
use std::path::{Path, PathBuf};
use std::time::Duration;
use tokio::time::sleep;
use types::{ChainSpec, Epoch, EthSpec, Fork, VoluntaryExit};
use types::{ChainSpec, Epoch, EthSpec, VoluntaryExit};
pub const CMD: &str = "exit";
pub const KEYSTORE_FLAG: &str = "keystore";
@@ -28,48 +29,59 @@ pub const DEFAULT_BEACON_NODE: &str = "http://localhost:5052/";
pub const CONFIRMATION_PHRASE: &str = "Exit my validator";
pub const WEBSITE_URL: &str = "https://lighthouse-book.sigmaprime.io/voluntary-exit.html";
pub fn cli_app<'a, 'b>() -> App<'a, 'b> {
App::new("exit")
pub fn cli_app() -> Command {
Command::new("exit")
.about("Submits a VoluntaryExit to the beacon chain for a given validator keystore.")
.arg(
Arg::with_name(KEYSTORE_FLAG)
Arg::new(KEYSTORE_FLAG)
.long(KEYSTORE_FLAG)
.value_name("KEYSTORE_PATH")
.help("The path to the EIP-2335 voting keystore for the validator")
.takes_value(true)
.required(true),
.action(ArgAction::Set)
.required(true)
.display_order(0)
)
.arg(
Arg::with_name(PASSWORD_FILE_FLAG)
Arg::new(PASSWORD_FILE_FLAG)
.long(PASSWORD_FILE_FLAG)
.value_name("PASSWORD_FILE_PATH")
.help("The path to the password file which unlocks the validator voting keystore")
.takes_value(true),
.action(ArgAction::Set)
.display_order(0)
)
.arg(
Arg::with_name(BEACON_SERVER_FLAG)
Arg::new(BEACON_SERVER_FLAG)
.long(BEACON_SERVER_FLAG)
.value_name("NETWORK_ADDRESS")
.help("Address to a beacon node HTTP API")
.default_value(DEFAULT_BEACON_NODE)
.takes_value(true),
.action(ArgAction::Set)
.display_order(0)
)
.arg(
Arg::with_name(NO_WAIT)
Arg::new(NO_WAIT)
.long(NO_WAIT)
.help("Exits after publishing the voluntary exit without waiting for confirmation that the exit was included in the beacon chain")
.action(ArgAction::SetTrue)
.help_heading(FLAG_HEADER)
.display_order(0)
)
.arg(
Arg::with_name(NO_CONFIRMATION)
Arg::new(NO_CONFIRMATION)
.long(NO_CONFIRMATION)
.help("Exits without prompting for confirmation that you understand the implications of a voluntary exit. This should be used with caution")
.display_order(0)
.action(ArgAction::SetTrue)
.help_heading(FLAG_HEADER)
)
.arg(
Arg::with_name(STDIN_INPUTS_FLAG)
.takes_value(false)
.hidden(cfg!(windows))
Arg::new(STDIN_INPUTS_FLAG)
.action(ArgAction::SetTrue)
.help_heading(FLAG_HEADER)
.hide(cfg!(windows))
.long(STDIN_INPUTS_FLAG)
.help("If present, read all user inputs from stdin instead of tty."),
.help("If present, read all user inputs from stdin instead of tty.")
.display_order(0)
)
}
@@ -78,9 +90,9 @@ pub fn cli_run<E: EthSpec>(matches: &ArgMatches, env: Environment<E>) -> Result<
let password_file_path: Option<PathBuf> =
clap_utils::parse_optional(matches, PASSWORD_FILE_FLAG)?;
let stdin_inputs = cfg!(windows) || matches.is_present(STDIN_INPUTS_FLAG);
let no_wait = matches.is_present(NO_WAIT);
let no_confirmation = matches.is_present(NO_CONFIRMATION);
let stdin_inputs = cfg!(windows) || matches.get_flag(STDIN_INPUTS_FLAG);
let no_wait = matches.get_flag(NO_WAIT);
let no_confirmation = matches.get_flag(NO_CONFIRMATION);
let spec = env.eth2_config().spec.clone();
let server_url: String = clap_utils::parse_required(matches, BEACON_SERVER_FLAG)?;
@@ -146,7 +158,6 @@ async fn publish_voluntary_exit<E: EthSpec>(
.ok_or("Failed to get current epoch. Please check your system time")?;
let validator_index = get_validator_index_for_exit(client, &keypair.pk, epoch, spec).await?;
let fork = get_beacon_state_fork(client).await?;
let voluntary_exit = VoluntaryExit {
epoch,
validator_index,
@@ -173,12 +184,8 @@ async fn publish_voluntary_exit<E: EthSpec>(
if confirmation == CONFIRMATION_PHRASE {
// Sign and publish the voluntary exit to network
let signed_voluntary_exit = voluntary_exit.sign(
&keypair.sk,
&fork,
genesis_data.genesis_validators_root,
spec,
);
let signed_voluntary_exit =
voluntary_exit.sign(&keypair.sk, genesis_data.genesis_validators_root, spec);
client
.post_beacon_pool_voluntary_exits(&signed_voluntary_exit)
.await
@@ -316,16 +323,6 @@ async fn is_syncing(client: &BeaconNodeHttpClient) -> Result<bool, String> {
.is_syncing)
}
/// Get fork object for the current state by querying the beacon node client.
async fn get_beacon_state_fork(client: &BeaconNodeHttpClient) -> Result<Fork, String> {
Ok(client
.get_beacon_states_fork(StateId::Head)
.await
.map_err(|e| format!("Failed to get get fork: {:?}", e))?
.ok_or("Failed to get fork, state not found")?
.data)
}
/// Calculates the current epoch from the genesis time and current time.
fn get_current_epoch<E: EthSpec>(genesis_time: u64, spec: &ChainSpec) -> Option<Epoch> {
let slot_clock = SystemTimeSlotClock::new(

View File

@@ -9,7 +9,8 @@ use account_utils::{
},
ZeroizeString,
};
use clap::{App, Arg, ArgMatches};
use clap::{Arg, ArgAction, ArgMatches, Command};
use clap_utils::FLAG_HEADER;
use slashing_protection::{SlashingDatabase, SLASHING_PROTECTION_FILENAME};
use std::fs;
use std::path::PathBuf;
@@ -25,8 +26,8 @@ pub const PASSWORD_PROMPT: &str = "Enter the keystore password, or press enter t
pub const KEYSTORE_REUSE_WARNING: &str = "DO NOT USE THE ORIGINAL KEYSTORES TO VALIDATE WITH \
ANOTHER CLIENT, OR YOU WILL GET SLASHED.";
pub fn cli_app<'a, 'b>() -> App<'a, 'b> {
App::new(CMD)
pub fn cli_app() -> Command {
Command::new(CMD)
.about(
"Imports one or more EIP-2335 passwords into a Lighthouse VC directory, \
requesting passwords interactively. The directory flag provides a convenient \
@@ -34,16 +35,17 @@ pub fn cli_app<'a, 'b>() -> App<'a, 'b> {
Python utility.",
)
.arg(
Arg::with_name(KEYSTORE_FLAG)
Arg::new(KEYSTORE_FLAG)
.long(KEYSTORE_FLAG)
.value_name("KEYSTORE_PATH")
.help("Path to a single keystore to be imported.")
.conflicts_with(DIR_FLAG)
.required_unless(DIR_FLAG)
.takes_value(true),
.required_unless_present(DIR_FLAG)
.action(ArgAction::Set)
.display_order(0),
)
.arg(
Arg::with_name(DIR_FLAG)
Arg::new(DIR_FLAG)
.long(DIR_FLAG)
.value_name("KEYSTORES_DIRECTORY")
.help(
@@ -53,23 +55,29 @@ pub fn cli_app<'a, 'b>() -> App<'a, 'b> {
has the '.json' extension will be attempted to be imported.",
)
.conflicts_with(KEYSTORE_FLAG)
.required_unless(KEYSTORE_FLAG)
.takes_value(true),
.required_unless_present(KEYSTORE_FLAG)
.action(ArgAction::Set)
.display_order(0),
)
.arg(
Arg::with_name(STDIN_INPUTS_FLAG)
.takes_value(false)
.hidden(cfg!(windows))
Arg::new(STDIN_INPUTS_FLAG)
.action(ArgAction::SetTrue)
.help_heading(FLAG_HEADER)
.hide(cfg!(windows))
.long(STDIN_INPUTS_FLAG)
.help("If present, read all user inputs from stdin instead of tty."),
.help("If present, read all user inputs from stdin instead of tty.")
.display_order(0),
)
.arg(
Arg::with_name(REUSE_PASSWORD_FLAG)
Arg::new(REUSE_PASSWORD_FLAG)
.long(REUSE_PASSWORD_FLAG)
.help("If present, the same password will be used for all imported keystores."),
.action(ArgAction::SetTrue)
.help_heading(FLAG_HEADER)
.help("If present, the same password will be used for all imported keystores.")
.display_order(0),
)
.arg(
Arg::with_name(PASSWORD_FLAG)
Arg::new(PASSWORD_FLAG)
.long(PASSWORD_FLAG)
.value_name("KEYSTORE_PASSWORD_PATH")
.requires(REUSE_PASSWORD_FLAG)
@@ -79,15 +87,16 @@ pub fn cli_app<'a, 'b>() -> App<'a, 'b> {
The password will be copied to the `validator_definitions.yml` file, so after \
import we strongly recommend you delete the file at KEYSTORE_PASSWORD_PATH.",
)
.takes_value(true),
.action(ArgAction::Set)
.display_order(0),
)
}
pub fn cli_run(matches: &ArgMatches, validator_dir: PathBuf) -> Result<(), String> {
let keystore: Option<PathBuf> = clap_utils::parse_optional(matches, KEYSTORE_FLAG)?;
let keystores_dir: Option<PathBuf> = clap_utils::parse_optional(matches, DIR_FLAG)?;
let stdin_inputs = cfg!(windows) || matches.is_present(STDIN_INPUTS_FLAG);
let reuse_password = matches.is_present(REUSE_PASSWORD_FLAG);
let stdin_inputs = cfg!(windows) || matches.get_flag(STDIN_INPUTS_FLAG);
let reuse_password = matches.get_flag(REUSE_PASSWORD_FLAG);
let keystore_password_path: Option<PathBuf> =
clap_utils::parse_optional(matches, PASSWORD_FLAG)?;
@@ -284,6 +293,8 @@ pub fn cli_run(matches: &ArgMatches, validator_dir: PathBuf) -> Result<(), Strin
suggested_fee_recipient,
None,
None,
None,
None,
)
.map_err(|e| format!("Unable to create new validator definition: {:?}", e))?;

View File

@@ -1,11 +1,11 @@
use account_utils::validator_definitions::ValidatorDefinitions;
use clap::App;
use clap::Command;
use std::path::PathBuf;
pub const CMD: &str = "list";
pub fn cli_app<'a, 'b>() -> App<'a, 'b> {
App::new(CMD).about("Lists the public keys of all validators.")
pub fn cli_app() -> Command {
Command::new(CMD).about("Lists the public keys of all validators.")
}
pub fn cli_run(validator_dir: PathBuf) -> Result<(), String> {

View File

@@ -7,7 +7,8 @@ pub mod recover;
pub mod slashing_protection;
use crate::{VALIDATOR_DIR_FLAG, VALIDATOR_DIR_FLAG_ALIAS};
use clap::{App, Arg, ArgMatches};
use clap::{Arg, ArgAction, ArgMatches, Command};
use clap_utils::FLAG_HEADER;
use directory::{parse_path_or_default_with_flag, DEFAULT_VALIDATOR_DIR};
use environment::Environment;
use std::path::PathBuf;
@@ -15,11 +16,21 @@ use types::EthSpec;
pub const CMD: &str = "validator";
pub fn cli_app<'a, 'b>() -> App<'a, 'b> {
App::new(CMD)
pub fn cli_app() -> Command {
Command::new(CMD)
.display_order(0)
.about("Provides commands for managing Eth2 validators.")
.arg(
Arg::with_name(VALIDATOR_DIR_FLAG)
Arg::new("help")
.long("help")
.short('h')
.help("Prints help information")
.action(ArgAction::HelpLong)
.display_order(0)
.help_heading(FLAG_HEADER),
)
.arg(
Arg::new(VALIDATOR_DIR_FLAG)
.long(VALIDATOR_DIR_FLAG)
.alias(VALIDATOR_DIR_FLAG_ALIAS)
.value_name("VALIDATOR_DIRECTORY")
@@ -27,7 +38,7 @@ pub fn cli_app<'a, 'b>() -> App<'a, 'b> {
"The path to search for validator directories. \
Defaults to ~/.lighthouse/{network}/validators",
)
.takes_value(true)
.action(ArgAction::Set)
.conflicts_with("datadir"),
)
.subcommand(create::cli_app())
@@ -39,8 +50,8 @@ pub fn cli_app<'a, 'b>() -> App<'a, 'b> {
.subcommand(exit::cli_app())
}
pub fn cli_run<T: EthSpec>(matches: &ArgMatches, env: Environment<T>) -> Result<(), String> {
let validator_base_dir = if matches.value_of("datadir").is_some() {
pub fn cli_run<E: EthSpec>(matches: &ArgMatches, env: Environment<E>) -> Result<(), String> {
let validator_base_dir = if matches.get_one::<String>("datadir").is_some() {
let path: PathBuf = clap_utils::parse_required(matches, "datadir")?;
path.join(DEFAULT_VALIDATOR_DIR)
} else {
@@ -49,18 +60,19 @@ pub fn cli_run<T: EthSpec>(matches: &ArgMatches, env: Environment<T>) -> Result<
eprintln!("validator-dir path: {:?}", validator_base_dir);
match matches.subcommand() {
(create::CMD, Some(matches)) => create::cli_run::<T>(matches, env, validator_base_dir),
(modify::CMD, Some(matches)) => modify::cli_run(matches, validator_base_dir),
(import::CMD, Some(matches)) => import::cli_run(matches, validator_base_dir),
(list::CMD, Some(_)) => list::cli_run(validator_base_dir),
(recover::CMD, Some(matches)) => recover::cli_run(matches, validator_base_dir),
(slashing_protection::CMD, Some(matches)) => {
Some((create::CMD, matches)) => create::cli_run::<E>(matches, env, validator_base_dir),
Some((modify::CMD, matches)) => modify::cli_run(matches, validator_base_dir),
Some((import::CMD, matches)) => import::cli_run(matches, validator_base_dir),
Some((list::CMD, _)) => list::cli_run(validator_base_dir),
Some((recover::CMD, matches)) => recover::cli_run(matches, validator_base_dir),
Some((slashing_protection::CMD, matches)) => {
slashing_protection::cli_run(matches, env, validator_base_dir)
}
(exit::CMD, Some(matches)) => exit::cli_run(matches, env),
(unknown, _) => Err(format!(
Some((exit::CMD, matches)) => exit::cli_run(matches, env),
Some((unknown, _)) => Err(format!(
"{} does not have a {} command. See --help",
CMD, unknown
)),
_ => Err(format!("No command provided for {}. See --help", CMD)),
}
}

View File

@@ -1,6 +1,7 @@
use account_utils::validator_definitions::ValidatorDefinitions;
use bls::PublicKey;
use clap::{App, Arg, ArgMatches};
use clap::{Arg, ArgAction, ArgMatches, Command};
use clap_utils::FLAG_HEADER;
use std::{collections::HashSet, path::PathBuf};
pub const CMD: &str = "modify";
@@ -10,43 +11,50 @@ pub const DISABLE: &str = "disable";
pub const PUBKEY_FLAG: &str = "pubkey";
pub const ALL: &str = "all";
pub fn cli_app<'a, 'b>() -> App<'a, 'b> {
App::new(CMD)
pub fn cli_app() -> Command {
Command::new(CMD)
.about("Modify validator status in validator_definitions.yml.")
.display_order(0)
.subcommand(
App::new(ENABLE)
Command::new(ENABLE)
.about("Enable validator(s) in validator_definitions.yml.")
.arg(
Arg::with_name(PUBKEY_FLAG)
Arg::new(PUBKEY_FLAG)
.long(PUBKEY_FLAG)
.value_name("PUBKEY")
.help("Validator pubkey to enable")
.takes_value(true),
.action(ArgAction::Set)
.display_order(0),
)
.arg(
Arg::with_name(ALL)
Arg::new(ALL)
.long(ALL)
.help("Enable all validators in the validator directory")
.takes_value(false)
.conflicts_with(PUBKEY_FLAG),
.action(ArgAction::SetTrue)
.help_heading(FLAG_HEADER)
.conflicts_with(PUBKEY_FLAG)
.display_order(0),
),
)
.subcommand(
App::new(DISABLE)
Command::new(DISABLE)
.about("Disable validator(s) in validator_definitions.yml.")
.arg(
Arg::with_name(PUBKEY_FLAG)
Arg::new(PUBKEY_FLAG)
.long(PUBKEY_FLAG)
.value_name("PUBKEY")
.help("Validator pubkey to disable")
.takes_value(true),
.action(ArgAction::Set)
.display_order(0),
)
.arg(
Arg::with_name(ALL)
Arg::new(ALL)
.long(ALL)
.help("Disable all validators in the validator directory")
.takes_value(false)
.conflicts_with(PUBKEY_FLAG),
.action(ArgAction::SetTrue)
.help_heading(FLAG_HEADER)
.conflicts_with(PUBKEY_FLAG)
.display_order(0),
),
)
}
@@ -55,14 +63,15 @@ pub fn cli_run(matches: &ArgMatches, validator_dir: PathBuf) -> Result<(), Strin
// `true` implies we are setting `validator_definition.enabled = true` and
// vice versa.
let (enabled, sub_matches) = match matches.subcommand() {
(ENABLE, Some(sub_matches)) => (true, sub_matches),
(DISABLE, Some(sub_matches)) => (false, sub_matches),
(unknown, _) => {
Some((ENABLE, sub_matches)) => (true, sub_matches),
Some((DISABLE, sub_matches)) => (false, sub_matches),
Some((unknown, _)) => {
return Err(format!(
"{} does not have a {} command. See --help",
CMD, unknown
))
}
_ => return Err(format!("No command provided for {}. See --help", CMD)),
};
let mut defs = ValidatorDefinitions::open(&validator_dir).map_err(|e| {
format!(
@@ -70,7 +79,7 @@ pub fn cli_run(matches: &ArgMatches, validator_dir: PathBuf) -> Result<(), Strin
validator_dir, e
)
})?;
let pubkeys_to_modify = if sub_matches.is_present(ALL) {
let pubkeys_to_modify = if sub_matches.get_flag(ALL) {
defs.as_slice()
.iter()
.map(|def| def.voting_public_key.clone())

View File

@@ -4,7 +4,8 @@ use crate::wallet::create::STDIN_INPUTS_FLAG;
use crate::SECRETS_DIR_FLAG;
use account_utils::eth2_keystore::{keypair_from_secret, Keystore, KeystoreBuilder};
use account_utils::{random_password, read_mnemonic_from_cli};
use clap::{App, Arg, ArgMatches};
use clap::{Arg, ArgAction, ArgMatches, Command};
use clap_utils::FLAG_HEADER;
use directory::ensure_dir_exists;
use directory::{parse_path_or_default_with_flag, DEFAULT_SECRET_DIR};
use eth2_wallet::bip39::Seed;
@@ -15,70 +16,79 @@ pub const CMD: &str = "recover";
pub const FIRST_INDEX_FLAG: &str = "first-index";
pub const MNEMONIC_FLAG: &str = "mnemonic-path";
pub fn cli_app<'a, 'b>() -> App<'a, 'b> {
App::new(CMD)
pub fn cli_app() -> Command {
Command::new(CMD)
.about(
"Recovers validator private keys given a BIP-39 mnemonic phrase. \
If you did not specify a `--first-index` or count `--count`, by default this will \
only recover the keys associated with the validator at index 0 for an HD wallet \
in accordance with the EIP-2333 spec.")
.arg(
Arg::with_name(FIRST_INDEX_FLAG)
Arg::new(FIRST_INDEX_FLAG)
.long(FIRST_INDEX_FLAG)
.value_name("FIRST_INDEX")
.help("The first of consecutive key indexes you wish to recover.")
.takes_value(true)
.action(ArgAction::Set)
.required(false)
.default_value("0"),
.default_value("0")
.display_order(0)
)
.arg(
Arg::with_name(COUNT_FLAG)
Arg::new(COUNT_FLAG)
.long(COUNT_FLAG)
.value_name("COUNT")
.help("The number of validator keys you wish to recover. Counted consecutively from the provided `--first_index`.")
.takes_value(true)
.action(ArgAction::Set)
.required(false)
.default_value("1"),
.default_value("1")
.display_order(0)
)
.arg(
Arg::with_name(MNEMONIC_FLAG)
Arg::new(MNEMONIC_FLAG)
.long(MNEMONIC_FLAG)
.value_name("MNEMONIC_PATH")
.help(
"If present, the mnemonic will be read in from this file.",
)
.takes_value(true)
.action(ArgAction::Set)
.display_order(0)
)
.arg(
Arg::with_name(SECRETS_DIR_FLAG)
Arg::new(SECRETS_DIR_FLAG)
.long(SECRETS_DIR_FLAG)
.value_name("SECRETS_DIR")
.help(
"The path where the validator keystore passwords will be stored. \
Defaults to ~/.lighthouse/{network}/secrets",
)
.takes_value(true),
.action(ArgAction::Set)
.display_order(0)
)
.arg(
Arg::with_name(STORE_WITHDRAW_FLAG)
Arg::new(STORE_WITHDRAW_FLAG)
.long(STORE_WITHDRAW_FLAG)
.help(
"If present, the withdrawal keystore will be stored alongside the voting \
keypair. It is generally recommended to *not* store the withdrawal key and \
instead generate them from the wallet seed when required.",
),
)
.action(ArgAction::SetTrue)
.help_heading(FLAG_HEADER)
.display_order(0)
)
.arg(
Arg::with_name(STDIN_INPUTS_FLAG)
.takes_value(false)
.hidden(cfg!(windows))
Arg::new(STDIN_INPUTS_FLAG)
.action(ArgAction::SetTrue)
.help_heading(FLAG_HEADER)
.hide(cfg!(windows))
.long(STDIN_INPUTS_FLAG)
.help("If present, read all user inputs from stdin instead of tty."),
.help("If present, read all user inputs from stdin instead of tty.")
.display_order(0)
)
}
pub fn cli_run(matches: &ArgMatches, validator_dir: PathBuf) -> Result<(), String> {
let secrets_dir = if matches.value_of("datadir").is_some() {
let secrets_dir = if matches.get_one::<String>("datadir").is_some() {
let path: PathBuf = clap_utils::parse_required(matches, "datadir")?;
path.join(DEFAULT_SECRET_DIR)
} else {
@@ -87,7 +97,7 @@ pub fn cli_run(matches: &ArgMatches, validator_dir: PathBuf) -> Result<(), Strin
let first_index: u32 = clap_utils::parse_required(matches, FIRST_INDEX_FLAG)?;
let count: u32 = clap_utils::parse_required(matches, COUNT_FLAG)?;
let mnemonic_path: Option<PathBuf> = clap_utils::parse_optional(matches, MNEMONIC_FLAG)?;
let stdin_inputs = cfg!(windows) || matches.is_present(STDIN_INPUTS_FLAG);
let stdin_inputs = cfg!(windows) || matches.get_flag(STDIN_INPUTS_FLAG);
eprintln!("secrets-dir path: {:?}", secrets_dir);
@@ -131,7 +141,7 @@ pub fn cli_run(matches: &ArgMatches, validator_dir: PathBuf) -> Result<(), Strin
.password_dir(secrets_dir.clone())
.voting_keystore(keystores.voting, voting_password.as_bytes())
.withdrawal_keystore(keystores.withdrawal, withdrawal_password.as_bytes())
.store_withdrawal_keystore(matches.is_present(STORE_WITHDRAW_FLAG))
.store_withdrawal_keystore(matches.get_flag(STORE_WITHDRAW_FLAG))
.build()
.map_err(|e| format!("Unable to build validator directory: {:?}", e))?;

View File

@@ -1,4 +1,4 @@
use clap::{App, Arg, ArgMatches};
use clap::{Arg, ArgAction, ArgMatches, Command};
use environment::Environment;
use slashing_protection::{
interchange::Interchange, InterchangeError, InterchangeImportOutcome, SlashingDatabase,
@@ -16,68 +16,50 @@ pub const EXPORT_CMD: &str = "export";
pub const IMPORT_FILE_ARG: &str = "IMPORT-FILE";
pub const EXPORT_FILE_ARG: &str = "EXPORT-FILE";
pub const MINIFY_FLAG: &str = "minify";
pub const PUBKEYS_FLAG: &str = "pubkeys";
pub fn cli_app<'a, 'b>() -> App<'a, 'b> {
App::new(CMD)
pub fn cli_app() -> Command {
Command::new(CMD)
.about("Import or export slashing protection data to or from another client")
.display_order(0)
.subcommand(
App::new(IMPORT_CMD)
Command::new(IMPORT_CMD)
.about("Import an interchange file")
.arg(
Arg::with_name(IMPORT_FILE_ARG)
.takes_value(true)
Arg::new(IMPORT_FILE_ARG)
.action(ArgAction::Set)
.value_name("FILE")
.display_order(0)
.help("The slashing protection interchange file to import (.json)"),
)
.arg(
Arg::with_name(MINIFY_FLAG)
.long(MINIFY_FLAG)
.takes_value(true)
.possible_values(&["false", "true"])
.help(
"Deprecated: Lighthouse no longer requires minification on import \
because it always minifies",
),
),
)
.subcommand(
App::new(EXPORT_CMD)
Command::new(EXPORT_CMD)
.about("Export an interchange file")
.arg(
Arg::with_name(EXPORT_FILE_ARG)
.takes_value(true)
Arg::new(EXPORT_FILE_ARG)
.action(ArgAction::Set)
.value_name("FILE")
.help("The filename to export the interchange file to"),
.help("The filename to export the interchange file to")
.display_order(0)
)
.arg(
Arg::with_name(PUBKEYS_FLAG)
Arg::new(PUBKEYS_FLAG)
.long(PUBKEYS_FLAG)
.takes_value(true)
.action(ArgAction::Set)
.value_name("PUBKEYS")
.help(
"List of public keys to export history for. Keys should be 0x-prefixed, \
comma-separated. All known keys will be exported if omitted",
),
)
.display_order(0)
)
.arg(
Arg::with_name(MINIFY_FLAG)
.long(MINIFY_FLAG)
.takes_value(true)
.default_value("false")
.possible_values(&["false", "true"])
.help(
"Minify the output file. This will make it smaller and faster to \
import, but not faster to generate.",
),
),
)
}
pub fn cli_run<T: EthSpec>(
matches: &ArgMatches<'_>,
env: Environment<T>,
pub fn cli_run<E: EthSpec>(
matches: &ArgMatches,
env: Environment<E>,
validator_base_dir: PathBuf,
) -> Result<(), String> {
let slashing_protection_db_path = validator_base_dir.join(SLASHING_PROTECTION_FILENAME);
@@ -86,13 +68,12 @@ pub fn cli_run<T: EthSpec>(
.ok_or("Unable to get testnet configuration from the environment")?;
let genesis_validators_root = eth2_network_config
.genesis_validators_root::<T>()?
.genesis_validators_root::<E>()?
.ok_or_else(|| "Unable to get genesis state, has genesis occurred?".to_string())?;
match matches.subcommand() {
(IMPORT_CMD, Some(matches)) => {
Some((IMPORT_CMD, matches)) => {
let import_filename: PathBuf = clap_utils::parse_required(matches, IMPORT_FILE_ARG)?;
let minify: Option<bool> = clap_utils::parse_optional(matches, MINIFY_FLAG)?;
let import_file = File::open(&import_filename).map_err(|e| {
format!(
"Unable to open import file at {}: {:?}",
@@ -102,23 +83,10 @@ pub fn cli_run<T: EthSpec>(
})?;
eprint!("Loading JSON file into memory & deserializing");
let mut interchange = Interchange::from_json_reader(&import_file)
let interchange = Interchange::from_json_reader(&import_file)
.map_err(|e| format!("Error parsing file for import: {:?}", e))?;
eprintln!(" [done].");
if let Some(minify) = minify {
eprintln!(
"WARNING: --minify flag is deprecated and will be removed in a future release"
);
if minify {
eprint!("Minifying input file for faster loading");
interchange = interchange
.minify()
.map_err(|e| format!("Minification failed: {:?}", e))?;
eprintln!(" [done].");
}
}
let slashing_protection_database =
SlashingDatabase::open_or_create(&slashing_protection_db_path).map_err(|e| {
format!(
@@ -204,9 +172,8 @@ pub fn cli_run<T: EthSpec>(
Ok(())
}
(EXPORT_CMD, Some(matches)) => {
Some((EXPORT_CMD, matches)) => {
let export_filename: PathBuf = clap_utils::parse_required(matches, EXPORT_FILE_ARG)?;
let minify: bool = clap_utils::parse_required(matches, MINIFY_FLAG)?;
let selected_pubkeys = if let Some(pubkeys) =
clap_utils::parse_optional::<String>(matches, PUBKEYS_FLAG)?
@@ -237,17 +204,10 @@ pub fn cli_run<T: EthSpec>(
)
})?;
let mut interchange = slashing_protection_database
let interchange = slashing_protection_database
.export_interchange_info(genesis_validators_root, selected_pubkeys.as_deref())
.map_err(|e| format!("Error during export: {:?}", e))?;
if minify {
eprintln!("Minifying output file");
interchange = interchange
.minify()
.map_err(|e| format!("Unable to minify output: {:?}", e))?;
}
let output_file = File::create(export_filename)
.map_err(|e| format!("Error creating output file: {:?}", e))?;
@@ -259,7 +219,7 @@ pub fn cli_run<T: EthSpec>(
Ok(())
}
("", _) => Err("No subcommand provided, see --help for options".to_string()),
(command, _) => Err(format!("No such subcommand `{}`", command)),
Some((command, _)) => Err(format!("No such subcommand `{}`", command)),
_ => Err("No subcommand provided, see --help for options".to_string()),
}
}

View File

@@ -3,7 +3,7 @@ use crate::WALLETS_DIR_FLAG;
use account_utils::{
is_password_sufficiently_complex, random_password, read_password_from_user, strip_off_newlines,
};
use clap::{App, Arg, ArgMatches};
use clap::{Arg, ArgAction, ArgMatches, Command};
use eth2_wallet::{
bip39::{Language, Mnemonic, MnemonicType},
PlainText,
@@ -33,21 +33,22 @@ pub const NEW_WALLET_PASSWORD_PROMPT: &str =
"Enter a password for your new wallet that is at least 12 characters long:";
pub const RETYPE_PASSWORD_PROMPT: &str = "Please re-enter your wallet's new password:";
pub fn cli_app<'a, 'b>() -> App<'a, 'b> {
App::new(CMD)
pub fn cli_app() -> Command {
Command::new(CMD)
.about("Creates a new HD (hierarchical-deterministic) EIP-2386 wallet.")
.arg(
Arg::with_name(NAME_FLAG)
Arg::new(NAME_FLAG)
.long(NAME_FLAG)
.value_name("WALLET_NAME")
.help(
"The wallet will be created with this name. It is not allowed to \
create two wallets with the same name for the same --base-dir.",
)
.takes_value(true),
.action(ArgAction::Set)
.display_order(0)
)
.arg(
Arg::with_name(PASSWORD_FLAG)
Arg::new(PASSWORD_FLAG)
.long(PASSWORD_FLAG)
.value_name("WALLET_PASSWORD_PATH")
.help(
@@ -56,49 +57,65 @@ pub fn cli_app<'a, 'b>() -> App<'a, 'b> {
saved at that path. To avoid confusion, if the file does not already \
exist it must include a '.pass' suffix.",
)
.takes_value(true),
.action(ArgAction::Set)
.display_order(0)
)
.arg(
Arg::with_name(TYPE_FLAG)
Arg::new(TYPE_FLAG)
.long(TYPE_FLAG)
.value_name("WALLET_TYPE")
.help(
"The type of wallet to create. Only HD (hierarchical-deterministic) \
wallets are supported presently..",
)
.takes_value(true)
.possible_values(&[HD_TYPE])
.default_value(HD_TYPE),
.action(ArgAction::Set)
.value_parser([HD_TYPE])
.default_value(HD_TYPE)
.display_order(0)
)
.arg(
Arg::with_name(MNEMONIC_FLAG)
Arg::new(MNEMONIC_FLAG)
.long(MNEMONIC_FLAG)
.value_name("MNEMONIC_PATH")
.help(
"If present, the mnemonic will be saved to this file. DO NOT SHARE THE MNEMONIC.",
)
.takes_value(true)
.action(ArgAction::Set)
.display_order(0)
)
.arg(
Arg::with_name(STDIN_INPUTS_FLAG)
.takes_value(false)
.hidden(cfg!(windows))
Arg::new(STDIN_INPUTS_FLAG)
.action(ArgAction::SetTrue)
.hide(cfg!(windows))
.long(STDIN_INPUTS_FLAG)
.help("If present, read all user inputs from stdin instead of tty."),
.help("If present, read all user inputs from stdin instead of tty.")
.display_order(0)
)
.arg(
Arg::with_name(MNEMONIC_LENGTH_FLAG)
Arg::new(MNEMONIC_LENGTH_FLAG)
.long(MNEMONIC_LENGTH_FLAG)
.value_name("MNEMONIC_LENGTH")
.help("The number of words to use for the mnemonic phrase.")
.takes_value(true)
.validator(|len| {
match len.parse::<usize>().ok().and_then(|words| MnemonicType::for_word_count(words).ok()) {
Some(_) => Ok(()),
None => Err(format!("Mnemonic length must be one of {}", MNEMONIC_TYPES.iter().map(|t| t.word_count().to_string()).collect::<Vec<_>>().join(", "))),
}
.action(ArgAction::Set)
.value_parser(|len: &str| {
match len
.parse::<usize>()
.ok()
.and_then(|words| MnemonicType::for_word_count(words).ok())
{
Some(_) => Ok(len.to_string()),
None => Err(format!(
"Mnemonic length must be one of {}",
MNEMONIC_TYPES
.iter()
.map(|t| t.word_count().to_string())
.collect::<Vec<_>>()
.join(", ")
)),
}
})
.default_value("24"),
.default_value("24")
.display_order(0)
)
}
@@ -153,7 +170,7 @@ pub fn create_wallet_from_mnemonic(
let name: Option<String> = clap_utils::parse_optional(matches, NAME_FLAG)?;
let wallet_password_path: Option<PathBuf> = clap_utils::parse_optional(matches, PASSWORD_FLAG)?;
let type_field: String = clap_utils::parse_required(matches, TYPE_FLAG)?;
let stdin_inputs = cfg!(windows) || matches.is_present(STDIN_INPUTS_FLAG);
let stdin_inputs = cfg!(windows) || matches.get_flag(STDIN_INPUTS_FLAG);
let wallet_type = match type_field.as_ref() {
HD_TYPE => WalletType::Hd,
unknown => return Err(format!("--{} {} is not supported", TYPE_FLAG, unknown)),

View File

@@ -1,12 +1,12 @@
use crate::WALLETS_DIR_FLAG;
use clap::App;
use clap::Command;
use eth2_wallet_manager::WalletManager;
use std::path::PathBuf;
pub const CMD: &str = "list";
pub fn cli_app<'a, 'b>() -> App<'a, 'b> {
App::new(CMD).about("Lists the names of all wallets.")
pub fn cli_app() -> Command {
Command::new(CMD).about("Lists the names of all wallets.")
}
pub fn cli_run(wallet_base_dir: PathBuf) -> Result<(), String> {

View File

@@ -3,21 +3,32 @@ pub mod list;
pub mod recover;
use crate::WALLETS_DIR_FLAG;
use clap::{App, Arg, ArgMatches};
use clap::{Arg, ArgAction, ArgMatches, Command};
use clap_utils::FLAG_HEADER;
use directory::{ensure_dir_exists, parse_path_or_default_with_flag, DEFAULT_WALLET_DIR};
use std::path::PathBuf;
pub const CMD: &str = "wallet";
pub fn cli_app<'a, 'b>() -> App<'a, 'b> {
App::new(CMD)
pub fn cli_app() -> Command {
Command::new(CMD)
.about("Manage wallets, from which validator keys can be derived.")
.display_order(0)
.arg(
Arg::with_name(WALLETS_DIR_FLAG)
Arg::new("help")
.long("help")
.short('h')
.help("Prints help information")
.action(ArgAction::HelpLong)
.display_order(0)
.help_heading(FLAG_HEADER)
)
.arg(
Arg::new(WALLETS_DIR_FLAG)
.long(WALLETS_DIR_FLAG)
.value_name("WALLETS_DIRECTORY")
.help("A path containing Eth2 EIP-2386 wallets. Defaults to ~/.lighthouse/{network}/wallets")
.takes_value(true)
.action(ArgAction::Set)
.conflicts_with("datadir"),
)
.subcommand(create::cli_app())
@@ -26,7 +37,7 @@ pub fn cli_app<'a, 'b>() -> App<'a, 'b> {
}
pub fn cli_run(matches: &ArgMatches) -> Result<(), String> {
let wallet_base_dir = if matches.value_of("datadir").is_some() {
let wallet_base_dir = if matches.get_one::<String>("datadir").is_some() {
let path: PathBuf = clap_utils::parse_required(matches, "datadir")?;
path.join(DEFAULT_WALLET_DIR)
} else {
@@ -37,12 +48,13 @@ pub fn cli_run(matches: &ArgMatches) -> Result<(), String> {
eprintln!("wallet-dir path: {:?}", wallet_base_dir);
match matches.subcommand() {
(create::CMD, Some(matches)) => create::cli_run(matches, wallet_base_dir),
(list::CMD, Some(_)) => list::cli_run(wallet_base_dir),
(recover::CMD, Some(matches)) => recover::cli_run(matches, wallet_base_dir),
(unknown, _) => Err(format!(
Some((create::CMD, matches)) => create::cli_run(matches, wallet_base_dir),
Some((list::CMD, _)) => list::cli_run(wallet_base_dir),
Some((recover::CMD, matches)) => recover::cli_run(matches, wallet_base_dir),
Some((unknown, _)) => Err(format!(
"{} does not have a {} command. See --help",
CMD, unknown
)),
_ => Err("No subcommand provided, see --help for options".to_string()),
}
}

View File

@@ -1,27 +1,28 @@
use crate::wallet::create::{create_wallet_from_mnemonic, STDIN_INPUTS_FLAG};
use crate::wallet::create::{HD_TYPE, NAME_FLAG, PASSWORD_FLAG, TYPE_FLAG};
use account_utils::read_mnemonic_from_cli;
use clap::{App, Arg, ArgMatches};
use clap::{Arg, ArgAction, ArgMatches, Command};
use std::path::PathBuf;
pub const CMD: &str = "recover";
pub const MNEMONIC_FLAG: &str = "mnemonic-path";
pub fn cli_app<'a, 'b>() -> App<'a, 'b> {
App::new(CMD)
pub fn cli_app() -> Command {
Command::new(CMD)
.about("Recovers an EIP-2386 wallet from a given a BIP-39 mnemonic phrase.")
.arg(
Arg::with_name(NAME_FLAG)
Arg::new(NAME_FLAG)
.long(NAME_FLAG)
.value_name("WALLET_NAME")
.help(
"The wallet will be created with this name. It is not allowed to \
create two wallets with the same name for the same --base-dir.",
)
.takes_value(true),
.action(ArgAction::Set)
.display_order(0),
)
.arg(
Arg::with_name(PASSWORD_FLAG)
Arg::new(PASSWORD_FLAG)
.long(PASSWORD_FLAG)
.value_name("PASSWORD_FILE_PATH")
.help(
@@ -31,39 +32,43 @@ pub fn cli_app<'a, 'b>() -> App<'a, 'b> {
saved at that path. To avoid confusion, if the file does not already \
exist it must include a '.pass' suffix.",
)
.takes_value(true),
.action(ArgAction::Set)
.display_order(0),
)
.arg(
Arg::with_name(MNEMONIC_FLAG)
Arg::new(MNEMONIC_FLAG)
.long(MNEMONIC_FLAG)
.value_name("MNEMONIC_PATH")
.help("If present, the mnemonic will be read in from this file.")
.takes_value(true),
.action(ArgAction::Set)
.display_order(0),
)
.arg(
Arg::with_name(TYPE_FLAG)
Arg::new(TYPE_FLAG)
.long(TYPE_FLAG)
.value_name("WALLET_TYPE")
.help(
"The type of wallet to create. Only HD (hierarchical-deterministic) \
wallets are supported presently..",
)
.takes_value(true)
.possible_values(&[HD_TYPE])
.default_value(HD_TYPE),
.action(ArgAction::Set)
.value_parser([HD_TYPE])
.default_value(HD_TYPE)
.display_order(0),
)
.arg(
Arg::with_name(STDIN_INPUTS_FLAG)
.takes_value(false)
.hidden(cfg!(windows))
Arg::new(STDIN_INPUTS_FLAG)
.action(ArgAction::SetTrue)
.hide(cfg!(windows))
.long(STDIN_INPUTS_FLAG)
.help("If present, read all user inputs from stdin instead of tty."),
.help("If present, read all user inputs from stdin instead of tty.")
.display_order(0),
)
}
pub fn cli_run(matches: &ArgMatches, wallet_base_dir: PathBuf) -> Result<(), String> {
let mnemonic_path: Option<PathBuf> = clap_utils::parse_optional(matches, MNEMONIC_FLAG)?;
let stdin_inputs = cfg!(windows) || matches.is_present(STDIN_INPUTS_FLAG);
let stdin_inputs = cfg!(windows) || matches.get_flag(STDIN_INPUTS_FLAG);
eprintln!();
eprintln!("WARNING: KEY RECOVERY CAN LEAD TO DUPLICATING VALIDATORS KEYS, WHICH CAN LEAD TO SLASHING.");

View File

@@ -1,6 +1,6 @@
[package]
name = "beacon_node"
version = "4.5.0"
version = "5.2.0"
authors = [
"Paul Hauner <paul@paulhauner.com>",
"Age Manning <Age@AgeManning.com",
@@ -29,17 +29,14 @@ clap = { workspace = true }
slog = { workspace = true }
dirs = { workspace = true }
directory = { workspace = true }
futures = { workspace = true }
environment = { workspace = true }
task_executor = { workspace = true }
genesis = { workspace = true }
eth2_network_config = { workspace = true }
execution_layer = { workspace = true }
lighthouse_network = { workspace = true }
serde = { workspace = true }
serde_json = { workspace = true }
clap_utils = { workspace = true }
hyper = { workspace = true }
lighthouse_version = { workspace = true }
hex = { workspace = true }
slasher = { workspace = true }
monitoring_api = { workspace = true }

View File

@@ -10,6 +10,8 @@ default = ["participation_metrics"]
write_ssz_files = [] # Writes debugging .ssz files to /tmp during block processing.
participation_metrics = [] # Exposes validator participation metrics to Prometheus.
fork_from_env = [] # Initialise the harness chain spec from the FORK_NAME env variable
portable = ["bls/supranational-portable"]
test_backfill = []
[dev-dependencies]
maplit = { workspace = true }
@@ -17,54 +19,59 @@ environment = { workspace = true }
serde_json = { workspace = true }
[dependencies]
merkle_proof = { workspace = true }
store = { workspace = true }
parking_lot = { workspace = true }
lazy_static = { workspace = true }
smallvec = { workspace = true }
lighthouse_metrics = { workspace = true }
operation_pool = { workspace = true }
rayon = { workspace = true }
serde = { workspace = true }
ethereum_serde_utils = { workspace = true }
slog = { workspace = true }
sloggers = { workspace = true }
slot_clock = { workspace = true }
ethereum_hashing = { workspace = true }
ethereum_ssz = { workspace = true }
ssz_types = { workspace = true }
ethereum_ssz_derive = { workspace = true }
state_processing = { workspace = true }
tree_hash_derive = { workspace = true }
tree_hash = { workspace = true }
types = { workspace = true }
tokio = { workspace = true }
tokio-stream = { workspace = true }
eth1 = { workspace = true }
futures = { workspace = true }
genesis = { workspace = true }
int_to_bytes = { workspace = true }
rand = { workspace = true }
proto_array = { workspace = true }
lru = { workspace = true }
tempfile = { workspace = true }
bitvec = { workspace = true }
bls = { workspace = true }
safe_arith = { workspace = true }
fork_choice = { workspace = true }
task_executor = { workspace = true }
derivative = { workspace = true }
itertools = { workspace = true }
slasher = { workspace = true }
eth1 = { workspace = true }
eth2 = { workspace = true }
strum = { workspace = true }
logging = { workspace = true }
eth2_network_config = { workspace = true }
ethereum_hashing = { workspace = true }
ethereum_serde_utils = { workspace = true }
ethereum_ssz = { workspace = true }
ethereum_ssz_derive = { workspace = true }
execution_layer = { workspace = true }
sensitive_url = { workspace = true }
superstruct = { workspace = true }
fork_choice = { workspace = true }
futures = { workspace = true }
genesis = { workspace = true }
hex = { workspace = true }
exit-future = { workspace = true }
int_to_bytes = { workspace = true }
itertools = { workspace = true }
kzg = { workspace = true }
lazy_static = { workspace = true }
lighthouse_metrics = { workspace = true }
lighthouse_version = { workspace = true }
logging = { workspace = true }
lru = { workspace = true }
merkle_proof = { workspace = true }
oneshot_broadcast = { path = "../../common/oneshot_broadcast/" }
operation_pool = { workspace = true }
parking_lot = { workspace = true }
proto_array = { workspace = true }
rand = { workspace = true }
rayon = { workspace = true }
safe_arith = { workspace = true }
sensitive_url = { workspace = true }
serde = { workspace = true }
serde_json = { workspace = true }
slasher = { workspace = true }
slog = { workspace = true }
slog-async = { workspace = true }
slog-term = { workspace = true }
sloggers = { workspace = true }
slot_clock = { workspace = true }
smallvec = { workspace = true }
ssz_types = { workspace = true }
state_processing = { workspace = true }
store = { workspace = true }
strum = { workspace = true }
superstruct = { workspace = true }
task_executor = { workspace = true }
tempfile = { workspace = true }
tokio = { workspace = true }
tokio-stream = { workspace = true }
tree_hash = { workspace = true }
tree_hash_derive = { workspace = true }
types = { workspace = true }
[[test]]
name = "beacon_chain_tests"

View File

@@ -1,13 +1,28 @@
use crate::{BeaconChain, BeaconChainError, BeaconChainTypes};
use eth2::lighthouse::attestation_rewards::{IdealAttestationRewards, TotalAttestationRewards};
use eth2::lighthouse::StandardAttestationRewards;
use participation_cache::ParticipationCache;
use eth2::types::ValidatorId;
use safe_arith::SafeArith;
use serde_utils::quoted_u64::Quoted;
use slog::debug;
use state_processing::common::base::{self, SqrtTotalActiveBalance};
use state_processing::per_epoch_processing::altair::{
process_inactivity_updates_slow, process_justification_and_finalization,
};
use state_processing::per_epoch_processing::base::rewards_and_penalties::{
get_attestation_component_delta, get_attestation_deltas_all, get_attestation_deltas_subset,
get_inactivity_penalty_delta, get_inclusion_delay_delta,
};
use state_processing::per_epoch_processing::base::validator_statuses::InclusionInfo;
use state_processing::per_epoch_processing::base::{
process_justification_and_finalization as process_justification_and_finalization_base,
TotalBalances, ValidatorStatus, ValidatorStatuses,
};
use state_processing::{
common::altair::BaseRewardPerIncrement,
per_epoch_processing::altair::{participation_cache, rewards_and_penalties::get_flag_weight},
common::update_progressive_balances_cache::initialize_progressive_balances_cache,
epoch_cache::initialize_epoch_cache,
per_epoch_processing::altair::rewards_and_penalties::get_flag_weight,
};
use std::collections::HashMap;
use store::consts::altair::{
@@ -15,19 +30,7 @@ use store::consts::altair::{
TIMELY_TARGET_FLAG_INDEX,
};
use types::consts::altair::WEIGHT_DENOMINATOR;
use types::{BeaconState, Epoch, EthSpec};
use eth2::types::ValidatorId;
use state_processing::common::base::get_base_reward_from_effective_balance;
use state_processing::per_epoch_processing::base::rewards_and_penalties::{
get_attestation_component_delta, get_attestation_deltas_all, get_attestation_deltas_subset,
get_inactivity_penalty_delta, get_inclusion_delay_delta,
};
use state_processing::per_epoch_processing::base::validator_statuses::InclusionInfo;
use state_processing::per_epoch_processing::base::{
TotalBalances, ValidatorStatus, ValidatorStatuses,
};
use types::{BeaconState, Epoch, EthSpec, RelativeEpoch};
impl<T: BeaconChainTypes> BeaconChain<T> {
pub fn compute_attestation_rewards(
@@ -50,9 +53,11 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
match state {
BeaconState::Base(_) => self.compute_attestation_rewards_base(state, validators),
BeaconState::Altair(_) | BeaconState::Merge(_) | BeaconState::Capella(_) => {
self.compute_attestation_rewards_altair(state, validators)
}
BeaconState::Altair(_)
| BeaconState::Bellatrix(_)
| BeaconState::Capella(_)
| BeaconState::Deneb(_)
| BeaconState::Electra(_) => self.compute_attestation_rewards_altair(state, validators),
}
}
@@ -65,6 +70,13 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
let mut validator_statuses = ValidatorStatuses::new(&state, spec)?;
validator_statuses.process_attestations(&state)?;
process_justification_and_finalization_base(
&state,
&validator_statuses.total_balances,
spec,
)?
.apply_changes_to_state(&mut state);
let ideal_rewards =
self.compute_ideal_rewards_base(&state, &validator_statuses.total_balances)?;
@@ -121,8 +133,16 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
) -> Result<StandardAttestationRewards, BeaconChainError> {
let spec = &self.spec;
// Build required caches.
initialize_epoch_cache(&mut state, spec)?;
initialize_progressive_balances_cache(&mut state, spec)?;
state.build_exit_cache(spec)?;
state.build_committee_cache(RelativeEpoch::Previous, spec)?;
state.build_committee_cache(RelativeEpoch::Current, spec)?;
// Calculate ideal_rewards
let participation_cache = ParticipationCache::new(&state, spec)?;
process_justification_and_finalization(&state)?.apply_changes_to_state(&mut state);
process_inactivity_updates_slow(&mut state, spec)?;
let previous_epoch = state.previous_epoch();
@@ -132,18 +152,14 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
let weight = get_flag_weight(flag_index)
.map_err(|_| BeaconChainError::AttestationRewardsError)?;
let unslashed_participating_indices = participation_cache
.get_unslashed_participating_indices(flag_index, previous_epoch)?;
let unslashed_participating_balance =
unslashed_participating_indices
.total_balance()
.map_err(|_| BeaconChainError::AttestationRewardsError)?;
let unslashed_participating_balance = state
.progressive_balances_cache()
.previous_epoch_flag_attesting_balance(flag_index)?;
let unslashed_participating_increments =
unslashed_participating_balance.safe_div(spec.effective_balance_increment)?;
let total_active_balance = participation_cache.current_epoch_total_active_balance();
let total_active_balance = state.get_total_active_balance()?;
let active_increments =
total_active_balance.safe_div(spec.effective_balance_increment)?;
@@ -179,29 +195,49 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
let mut total_rewards: Vec<TotalAttestationRewards> = Vec::new();
let validators = if validators.is_empty() {
participation_cache.eligible_validator_indices().to_vec()
Self::all_eligible_validator_indices(&state, previous_epoch)?
} else {
Self::validators_ids_to_indices(&mut state, validators)?
};
for validator_index in &validators {
let eligible = state.is_eligible_validator(previous_epoch, *validator_index)?;
for &validator_index in &validators {
// Return 0s for unknown/inactive validator indices.
let Ok(validator) = state.get_validator(validator_index) else {
debug!(
self.log,
"No rewards for inactive/unknown validator";
"index" => validator_index,
"epoch" => previous_epoch
);
total_rewards.push(TotalAttestationRewards {
validator_index: validator_index as u64,
head: 0,
target: 0,
source: 0,
inclusion_delay: None,
inactivity: 0,
});
continue;
};
let previous_epoch_participation_flags = state
.previous_epoch_participation()?
.get(validator_index)
.ok_or(BeaconChainError::AttestationRewardsError)?;
let eligible = state.is_eligible_validator(previous_epoch, validator)?;
let mut head_reward = 0i64;
let mut target_reward = 0i64;
let mut source_reward = 0i64;
let mut inactivity_penalty = 0i64;
if eligible {
let effective_balance = state.get_effective_balance(*validator_index)?;
let effective_balance = validator.effective_balance;
for flag_index in 0..PARTICIPATION_FLAG_WEIGHTS.len() {
let (ideal_reward, penalty) = ideal_rewards_hashmap
.get(&(flag_index, effective_balance))
.ok_or(BeaconChainError::AttestationRewardsError)?;
let voted_correctly = participation_cache
.get_unslashed_participating_indices(flag_index, previous_epoch)
.map_err(|_| BeaconChainError::AttestationRewardsError)?
.contains(*validator_index)
.map_err(|_| BeaconChainError::AttestationRewardsError)?;
let voted_correctly = !validator.slashed
&& previous_epoch_participation_flags.has_flag(flag_index)?;
if voted_correctly {
if flag_index == TIMELY_HEAD_FLAG_INDEX {
head_reward += *ideal_reward as i64;
@@ -214,19 +250,26 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
head_reward = 0;
} else if flag_index == TIMELY_TARGET_FLAG_INDEX {
target_reward = *penalty;
let penalty_numerator = effective_balance
.safe_mul(state.get_inactivity_score(validator_index)?)?;
let penalty_denominator = spec.inactivity_score_bias.safe_mul(
spec.inactivity_penalty_quotient_for_fork(state.fork_name_unchecked()),
)?;
inactivity_penalty =
-(penalty_numerator.safe_div(penalty_denominator)? as i64);
} else if flag_index == TIMELY_SOURCE_FLAG_INDEX {
source_reward = *penalty;
}
}
}
total_rewards.push(TotalAttestationRewards {
validator_index: *validator_index as u64,
validator_index: validator_index as u64,
head: head_reward,
target: target_reward,
source: source_reward,
inclusion_delay: None,
// TODO: altair calculation logic needs to be updated to include inactivity penalty
inactivity: 0,
inactivity: inactivity_penalty,
});
}
@@ -249,7 +292,6 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
target: 0,
source: 0,
inclusion_delay: None,
// TODO: altair calculation logic needs to be updated to include inactivity penalty
inactivity: 0,
});
match *flag_index {
@@ -279,6 +321,24 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
Ok(max_steps)
}
fn all_eligible_validator_indices(
state: &BeaconState<T::EthSpec>,
previous_epoch: Epoch,
) -> Result<Vec<usize>, BeaconChainError> {
state
.validators()
.iter()
.enumerate()
.filter_map(|(i, validator)| {
state
.is_eligible_validator(previous_epoch, validator)
.map(|eligible| eligible.then_some(i))
.map_err(BeaconChainError::BeaconStateError)
.transpose()
})
.collect()
}
fn validators_ids_to_indices(
state: &mut BeaconState<T::EthSpec>,
validators: Vec<ValidatorId>,
@@ -317,15 +377,12 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
};
let mut ideal_attestation_rewards_list = Vec::new();
let sqrt_total_active_balance = SqrtTotalActiveBalance::new(total_balances.current_epoch());
for effective_balance_step in 1..=self.max_effective_balance_increment_steps()? {
let effective_balance =
effective_balance_step.safe_mul(spec.effective_balance_increment)?;
let base_reward = get_base_reward_from_effective_balance::<T::EthSpec>(
effective_balance,
total_balances.current_epoch(),
spec,
)?;
let base_reward =
base::get_base_reward(effective_balance, sqrt_total_active_balance, spec)?;
// compute ideal head rewards
let head = get_attestation_component_delta(

View File

@@ -0,0 +1,107 @@
use crate::{BeaconChain, BeaconChainTypes};
use slog::{debug, error};
use slot_clock::SlotClock;
use std::sync::Arc;
use task_executor::TaskExecutor;
use tokio::time::sleep;
use types::{EthSpec, Slot};
/// Don't run the attestation simulator if the head slot is this many epochs
/// behind the wall-clock slot.
const SYNCING_TOLERANCE_EPOCHS: u64 = 2;
/// Spawns a routine which produces an unaggregated attestation at every slot.
///
/// This routine will run once per slot
pub fn start_attestation_simulator_service<T: BeaconChainTypes>(
executor: TaskExecutor,
chain: Arc<BeaconChain<T>>,
) {
executor.clone().spawn(
async move { attestation_simulator_service(executor, chain).await },
"attestation_simulator_service",
);
}
/// Loop indefinitely, calling `BeaconChain::produce_unaggregated_attestation` every 4s into each slot.
async fn attestation_simulator_service<T: BeaconChainTypes>(
executor: TaskExecutor,
chain: Arc<BeaconChain<T>>,
) {
let slot_duration = chain.slot_clock.slot_duration();
let additional_delay = slot_duration / 3;
loop {
match chain.slot_clock.duration_to_next_slot() {
Some(duration) => {
sleep(duration + additional_delay).await;
debug!(
chain.log,
"Simulating unagg. attestation production";
);
// Run the task in the executor
let inner_chain = chain.clone();
executor.spawn(
async move {
if let Ok(current_slot) = inner_chain.slot() {
produce_unaggregated_attestation(inner_chain, current_slot);
}
},
"attestation_simulator_service",
);
}
None => {
error!(chain.log, "Failed to read slot clock");
// If we can't read the slot clock, just wait another slot.
sleep(slot_duration).await;
}
};
}
}
pub fn produce_unaggregated_attestation<T: BeaconChainTypes>(
chain: Arc<BeaconChain<T>>,
current_slot: Slot,
) {
// Don't run the attestation simulator when the head slot is far behind the
// wall-clock slot.
//
// This helps prevent the simulator from becoming a burden by computing
// committees from old states.
let syncing_tolerance_slots = SYNCING_TOLERANCE_EPOCHS * T::EthSpec::slots_per_epoch();
if chain.best_slot() + syncing_tolerance_slots < current_slot {
return;
}
// Since attestations for different committees are practically identical (apart from the committee index field)
// Committee 0 is guaranteed to exist. That means there's no need to load the committee.
let beacon_committee_index = 0;
// Store the unaggregated attestation in the validator monitor for later processing
match chain.produce_unaggregated_attestation(current_slot, beacon_committee_index) {
Ok(unaggregated_attestation) => {
let data = unaggregated_attestation.data();
debug!(
chain.log,
"Produce unagg. attestation";
"attestation_source" => data.source.root.to_string(),
"attestation_target" => data.target.root.to_string(),
);
chain
.validator_monitor
.write()
.set_unaggregated_attestation(unaggregated_attestation);
}
Err(e) => {
debug!(
chain.log,
"Failed to simulate attestation";
"error" => ?e
);
}
}
}

View File

@@ -35,17 +35,23 @@
mod batch;
use crate::{
beacon_chain::VALIDATOR_PUBKEY_CACHE_LOCK_TIMEOUT, metrics,
observed_aggregates::ObserveOutcome, observed_attesters::Error as ObservedAttestersError,
beacon_chain::VALIDATOR_PUBKEY_CACHE_LOCK_TIMEOUT,
metrics,
observed_aggregates::{ObserveOutcome, ObservedAttestationKey},
observed_attesters::Error as ObservedAttestersError,
BeaconChain, BeaconChainError, BeaconChainTypes,
};
use bls::verify_signature_sets;
use itertools::Itertools;
use proto_array::Block as ProtoBlock;
use slog::debug;
use slot_clock::SlotClock;
use state_processing::{
common::get_indexed_attestation,
per_block_processing::errors::AttestationValidationError,
common::{
attesting_indices_base,
attesting_indices_electra::{self, get_committee_indices},
},
per_block_processing::errors::{AttestationValidationError, BlockOperationError},
signature_sets::{
indexed_attestation_signature_set_from_pubkeys,
signed_aggregate_selection_proof_signature_set, signed_aggregate_signature_set,
@@ -55,8 +61,9 @@ use std::borrow::Cow;
use strum::AsRefStr;
use tree_hash::TreeHash;
use types::{
Attestation, BeaconCommittee, ChainSpec, CommitteeIndex, Epoch, EthSpec, Hash256,
IndexedAttestation, SelectionProof, SignedAggregateAndProof, Slot, SubnetId,
Attestation, AttestationRef, BeaconCommittee, BeaconStateError::NoCommitteeFound, ChainSpec,
CommitteeIndex, Epoch, EthSpec, Hash256, IndexedAttestation, SelectionProof,
SignedAggregateAndProof, Slot, SubnetId,
};
pub use batch::{batch_verify_aggregated_attestations, batch_verify_unaggregated_attestations};
@@ -137,6 +144,12 @@ pub enum Error {
///
/// The peer has sent an invalid message.
ValidatorIndexTooHigh(usize),
/// The validator index is not set to zero after Electra.
///
/// ## Peer scoring
///
/// The peer has sent an invalid message.
CommitteeIndexNonZero(usize),
/// The `attestation.data.beacon_block_root` block is unknown.
///
/// ## Peer scoring
@@ -185,6 +198,12 @@ pub enum Error {
///
/// The peer has sent an invalid message.
NotExactlyOneAggregationBitSet(usize),
/// The attestation doesn't have only one aggregation bit set.
///
/// ## Peer scoring
///
/// The peer has sent an invalid message.
NotExactlyOneCommitteeBitSet(usize),
/// We have already observed an attestation for the `validator_index` and refuse to process
/// another.
///
@@ -248,7 +267,7 @@ pub enum Error {
impl From<BeaconChainError> for Error {
fn from(e: BeaconChainError) -> Self {
Error::BeaconChainError(e)
Self::BeaconChainError(e)
}
}
@@ -263,10 +282,11 @@ enum CheckAttestationSignature {
/// `IndexedAttestation` can be derived.
///
/// These attestations have *not* undergone signature verification.
/// The `observed_attestation_key_root` is the hashed value of an `ObservedAttestationKey`.
struct IndexedAggregatedAttestation<'a, T: BeaconChainTypes> {
signed_aggregate: &'a SignedAggregateAndProof<T::EthSpec>,
indexed_attestation: IndexedAttestation<T::EthSpec>,
attestation_data_root: Hash256,
observed_attestation_key_root: Hash256,
}
/// Wraps a `Attestation` that has been verified up until the point that an `IndexedAttestation` can
@@ -274,7 +294,7 @@ struct IndexedAggregatedAttestation<'a, T: BeaconChainTypes> {
///
/// These attestations have *not* undergone signature verification.
struct IndexedUnaggregatedAttestation<'a, T: BeaconChainTypes> {
attestation: &'a Attestation<T::EthSpec>,
attestation: AttestationRef<'a, T::EthSpec>,
indexed_attestation: IndexedAttestation<T::EthSpec>,
subnet_id: SubnetId,
validator_index: u64,
@@ -295,7 +315,7 @@ impl<'a, T: BeaconChainTypes> VerifiedAggregatedAttestation<'a, T> {
/// Wraps an `Attestation` that has been fully verified for propagation on the gossip network.
pub struct VerifiedUnaggregatedAttestation<'a, T: BeaconChainTypes> {
attestation: &'a Attestation<T::EthSpec>,
attestation: AttestationRef<'a, T::EthSpec>,
indexed_attestation: IndexedAttestation<T::EthSpec>,
subnet_id: SubnetId,
}
@@ -322,20 +342,20 @@ impl<'a, T: BeaconChainTypes> Clone for IndexedUnaggregatedAttestation<'a, T> {
/// A helper trait implemented on wrapper types that can be progressed to a state where they can be
/// verified for application to fork choice.
pub trait VerifiedAttestation<T: BeaconChainTypes>: Sized {
fn attestation(&self) -> &Attestation<T::EthSpec>;
fn attestation(&self) -> AttestationRef<T::EthSpec>;
fn indexed_attestation(&self) -> &IndexedAttestation<T::EthSpec>;
// Inefficient default implementation. This is overridden for gossip verified attestations.
fn into_attestation_and_indices(self) -> (Attestation<T::EthSpec>, Vec<u64>) {
let attestation = self.attestation().clone();
let attesting_indices = self.indexed_attestation().attesting_indices.clone().into();
let attestation = self.attestation().clone_as_attestation();
let attesting_indices = self.indexed_attestation().attesting_indices_to_vec();
(attestation, attesting_indices)
}
}
impl<'a, T: BeaconChainTypes> VerifiedAttestation<T> for VerifiedAggregatedAttestation<'a, T> {
fn attestation(&self) -> &Attestation<T::EthSpec> {
fn attestation(&self) -> AttestationRef<T::EthSpec> {
self.attestation()
}
@@ -345,7 +365,7 @@ impl<'a, T: BeaconChainTypes> VerifiedAttestation<T> for VerifiedAggregatedAttes
}
impl<'a, T: BeaconChainTypes> VerifiedAttestation<T> for VerifiedUnaggregatedAttestation<'a, T> {
fn attestation(&self) -> &Attestation<T::EthSpec> {
fn attestation(&self) -> AttestationRef<T::EthSpec> {
self.attestation
}
@@ -357,7 +377,7 @@ impl<'a, T: BeaconChainTypes> VerifiedAttestation<T> for VerifiedUnaggregatedAtt
/// Information about invalid attestations which might still be slashable despite being invalid.
pub enum AttestationSlashInfo<'a, T: BeaconChainTypes, TErr> {
/// The attestation is invalid, but its signature wasn't checked.
SignatureNotChecked(&'a Attestation<T::EthSpec>, TErr),
SignatureNotChecked(AttestationRef<'a, T::EthSpec>, TErr),
/// As for `SignatureNotChecked`, but we know the `IndexedAttestation`.
SignatureNotCheckedIndexed(IndexedAttestation<T::EthSpec>, TErr),
/// The attestation's signature is invalid, so it will never be slashable.
@@ -381,6 +401,11 @@ fn process_slash_info<T: BeaconChainTypes>(
if let Some(slasher) = chain.slasher.as_ref() {
let (indexed_attestation, check_signature, err) = match slash_info {
SignatureNotChecked(attestation, err) => {
if let Error::UnknownHeadBlock { .. } = err {
if attestation.data().beacon_block_root == attestation.data().target.root {
return err;
}
}
match obtain_indexed_attestation_and_committees_per_slot(chain, attestation) {
Ok((indexed, _)) => (indexed, true, err),
Err(e) => {
@@ -446,7 +471,7 @@ impl<'a, T: BeaconChainTypes> IndexedAggregatedAttestation<'a, T> {
signed_aggregate: &SignedAggregateAndProof<T::EthSpec>,
chain: &BeaconChain<T>,
) -> Result<Hash256, Error> {
let attestation = &signed_aggregate.message.aggregate;
let attestation = signed_aggregate.message().aggregate();
// Ensure attestation is within the last ATTESTATION_PROPAGATION_SLOT_RANGE slots (within a
// MAXIMUM_GOSSIP_CLOCK_DISPARITY allowance).
@@ -455,30 +480,39 @@ impl<'a, T: BeaconChainTypes> IndexedAggregatedAttestation<'a, T> {
verify_propagation_slot_range(&chain.slot_clock, attestation, &chain.spec)?;
// Check the attestation's epoch matches its target.
if attestation.data.slot.epoch(T::EthSpec::slots_per_epoch())
!= attestation.data.target.epoch
if attestation.data().slot.epoch(T::EthSpec::slots_per_epoch())
!= attestation.data().target.epoch
{
return Err(Error::InvalidTargetEpoch {
slot: attestation.data.slot,
epoch: attestation.data.target.epoch,
slot: attestation.data().slot,
epoch: attestation.data().target.epoch,
});
}
// Ensure the valid aggregated attestation has not already been seen locally.
let attestation_data = &attestation.data;
let attestation_data_root = attestation_data.tree_hash_root();
let observed_attestation_key_root = ObservedAttestationKey {
committee_index: attestation
.committee_index()
.ok_or(Error::NotExactlyOneCommitteeBitSet(0))?,
attestation_data: attestation.data().clone(),
}
.tree_hash_root();
// [New in Electra:EIP7549]
verify_committee_index(attestation)?;
if chain
.observed_attestations
.write()
.is_known_subset(attestation, attestation_data_root)
.is_known_subset(attestation, observed_attestation_key_root)
.map_err(|e| Error::BeaconChainError(e.into()))?
{
metrics::inc_counter(&metrics::AGGREGATED_ATTESTATION_SUBSETS);
return Err(Error::AttestationSupersetKnown(attestation_data_root));
return Err(Error::AttestationSupersetKnown(
observed_attestation_key_root,
));
}
let aggregator_index = signed_aggregate.message.aggregator_index;
let aggregator_index = signed_aggregate.message().aggregator_index();
// Ensure there has been no other observed aggregate for the given `aggregator_index`.
//
@@ -486,7 +520,7 @@ impl<'a, T: BeaconChainTypes> IndexedAggregatedAttestation<'a, T> {
match chain
.observed_aggregators
.read()
.validator_has_been_observed(attestation.data.target.epoch, aggregator_index as usize)
.validator_has_been_observed(attestation.data().target.epoch, aggregator_index as usize)
{
Ok(true) => Err(Error::AggregatorAlreadyKnown(aggregator_index)),
Ok(false) => Ok(()),
@@ -518,10 +552,10 @@ impl<'a, T: BeaconChainTypes> IndexedAggregatedAttestation<'a, T> {
verify_attestation_target_root::<T::EthSpec>(&head_block, attestation)?;
// Ensure that the attestation has participants.
if attestation.aggregation_bits.is_zero() {
if attestation.is_aggregation_bits_zero() {
Err(Error::EmptyAggregationBitfield)
} else {
Ok(attestation_data_root)
Ok(observed_attestation_key_root)
}
}
@@ -531,23 +565,47 @@ impl<'a, T: BeaconChainTypes> IndexedAggregatedAttestation<'a, T> {
chain: &BeaconChain<T>,
) -> Result<Self, AttestationSlashInfo<'a, T, Error>> {
use AttestationSlashInfo::*;
let attestation = &signed_aggregate.message.aggregate;
let aggregator_index = signed_aggregate.message.aggregator_index;
let attestation_data_root = match Self::verify_early_checks(signed_aggregate, chain) {
let observed_attestation_key_root = match Self::verify_early_checks(signed_aggregate, chain)
{
Ok(root) => root,
Err(e) => return Err(SignatureNotChecked(&signed_aggregate.message.aggregate, e)),
Err(e) => {
return Err(SignatureNotChecked(
signed_aggregate.message().aggregate(),
e,
))
}
};
let indexed_attestation =
match map_attestation_committee(chain, attestation, |(committee, _)| {
// Note: this clones the signature which is known to be a relatively slow operation.
//
// Future optimizations should remove this clone.
let selection_proof =
SelectionProof::from(signed_aggregate.message.selection_proof.clone());
// Committees must be sorted by ascending index order 0..committees_per_slot
let get_indexed_attestation_with_committee =
|(committees, _): (Vec<BeaconCommittee>, CommitteesPerSlot)| {
let (index, aggregator_index, selection_proof, data) = match signed_aggregate {
SignedAggregateAndProof::Base(signed_aggregate) => (
signed_aggregate.message.aggregate.data.index,
signed_aggregate.message.aggregator_index,
// Note: this clones the signature which is known to be a relatively slow operation.
// Future optimizations should remove this clone.
signed_aggregate.message.selection_proof.clone(),
signed_aggregate.message.aggregate.data.clone(),
),
SignedAggregateAndProof::Electra(signed_aggregate) => (
signed_aggregate
.message
.aggregate
.committee_index()
.ok_or(Error::NotExactlyOneCommitteeBitSet(0))?,
signed_aggregate.message.aggregator_index,
signed_aggregate.message.selection_proof.clone(),
signed_aggregate.message.aggregate.data.clone(),
),
};
let slot = data.slot;
if !selection_proof
let committee = committees
.get(index as usize)
.ok_or(Error::NoCommitteeForSlotAndIndex { slot, index })?;
if !SelectionProof::from(selection_proof)
.is_aggregator(committee.committee.len(), &chain.spec)
.map_err(|e| Error::BeaconChainError(e.into()))?
{
@@ -559,17 +617,44 @@ impl<'a, T: BeaconChainTypes> IndexedAggregatedAttestation<'a, T> {
return Err(Error::AggregatorNotInCommittee { aggregator_index });
}
get_indexed_attestation(committee.committee, attestation)
.map_err(|e| BeaconChainError::from(e).into())
}) {
Ok(indexed_attestation) => indexed_attestation,
Err(e) => return Err(SignatureNotChecked(&signed_aggregate.message.aggregate, e)),
// p2p aggregates have a single committee, we can assert that aggregation_bits is always
// less then MaxValidatorsPerCommittee
match signed_aggregate {
SignedAggregateAndProof::Base(signed_aggregate) => {
attesting_indices_base::get_indexed_attestation(
committee.committee,
&signed_aggregate.message.aggregate,
)
.map_err(|e| BeaconChainError::from(e).into())
}
SignedAggregateAndProof::Electra(signed_aggregate) => {
attesting_indices_electra::get_indexed_attestation(
&committees,
&signed_aggregate.message.aggregate,
)
.map_err(|e| BeaconChainError::from(e).into())
}
}
};
let attestation = signed_aggregate.message().aggregate();
let indexed_attestation = match map_attestation_committees(
chain,
attestation,
get_indexed_attestation_with_committee,
) {
Ok(indexed_attestation) => indexed_attestation,
Err(e) => {
return Err(SignatureNotChecked(
signed_aggregate.message().aggregate(),
e,
))
}
};
Ok(IndexedAggregatedAttestation {
signed_aggregate,
indexed_attestation,
attestation_data_root,
observed_attestation_key_root,
})
}
}
@@ -578,11 +663,11 @@ impl<'a, T: BeaconChainTypes> VerifiedAggregatedAttestation<'a, T> {
/// Run the checks that happen after the indexed attestation and signature have been checked.
fn verify_late_checks(
signed_aggregate: &SignedAggregateAndProof<T::EthSpec>,
attestation_data_root: Hash256,
observed_attestation_key_root: Hash256,
chain: &BeaconChain<T>,
) -> Result<(), Error> {
let attestation = &signed_aggregate.message.aggregate;
let aggregator_index = signed_aggregate.message.aggregator_index;
let attestation = signed_aggregate.message().aggregate();
let aggregator_index = signed_aggregate.message().aggregator_index();
// Observe the valid attestation so we do not re-process it.
//
@@ -591,11 +676,13 @@ impl<'a, T: BeaconChainTypes> VerifiedAggregatedAttestation<'a, T> {
if let ObserveOutcome::Subset = chain
.observed_attestations
.write()
.observe_item(attestation, Some(attestation_data_root))
.observe_item(attestation, Some(observed_attestation_key_root))
.map_err(|e| Error::BeaconChainError(e.into()))?
{
metrics::inc_counter(&metrics::AGGREGATED_ATTESTATION_SUBSETS);
return Err(Error::AttestationSupersetKnown(attestation_data_root));
return Err(Error::AttestationSupersetKnown(
observed_attestation_key_root,
));
}
// Observe the aggregator so we don't process another aggregate from them.
@@ -605,12 +692,12 @@ impl<'a, T: BeaconChainTypes> VerifiedAggregatedAttestation<'a, T> {
if chain
.observed_aggregators
.write()
.observe_validator(attestation.data.target.epoch, aggregator_index as usize)
.observe_validator(attestation.data().target.epoch, aggregator_index as usize)
.map_err(BeaconChainError::from)?
{
return Err(Error::PriorAttestationKnown {
validator_index: aggregator_index,
epoch: attestation.data.target.epoch,
epoch: attestation.data().target.epoch,
});
}
@@ -655,7 +742,7 @@ impl<'a, T: BeaconChainTypes> VerifiedAggregatedAttestation<'a, T> {
let IndexedAggregatedAttestation {
signed_aggregate,
indexed_attestation,
attestation_data_root,
observed_attestation_key_root,
} = signed_aggregate;
match check_signature {
@@ -679,7 +766,9 @@ impl<'a, T: BeaconChainTypes> VerifiedAggregatedAttestation<'a, T> {
CheckAttestationSignature::No => (),
};
if let Err(e) = Self::verify_late_checks(signed_aggregate, attestation_data_root, chain) {
if let Err(e) =
Self::verify_late_checks(signed_aggregate, observed_attestation_key_root, chain)
{
return Err(SignatureValid(indexed_attestation, e));
}
@@ -690,8 +779,8 @@ impl<'a, T: BeaconChainTypes> VerifiedAggregatedAttestation<'a, T> {
}
/// Returns the underlying `attestation` for the `signed_aggregate`.
pub fn attestation(&self) -> &Attestation<T::EthSpec> {
&self.signed_aggregate.message.aggregate
pub fn attestation(&self) -> AttestationRef<'a, T::EthSpec> {
self.signed_aggregate.message().aggregate()
}
/// Returns the underlying `signed_aggregate`.
@@ -703,16 +792,16 @@ impl<'a, T: BeaconChainTypes> VerifiedAggregatedAttestation<'a, T> {
impl<'a, T: BeaconChainTypes> IndexedUnaggregatedAttestation<'a, T> {
/// Run the checks that happen before an indexed attestation is constructed.
pub fn verify_early_checks(
attestation: &Attestation<T::EthSpec>,
attestation: AttestationRef<T::EthSpec>,
chain: &BeaconChain<T>,
) -> Result<(), Error> {
let attestation_epoch = attestation.data.slot.epoch(T::EthSpec::slots_per_epoch());
let attestation_epoch = attestation.data().slot.epoch(T::EthSpec::slots_per_epoch());
// Check the attestation's epoch matches its target.
if attestation_epoch != attestation.data.target.epoch {
if attestation_epoch != attestation.data().target.epoch {
return Err(Error::InvalidTargetEpoch {
slot: attestation.data.slot,
epoch: attestation.data.target.epoch,
slot: attestation.data().slot,
epoch: attestation.data().target.epoch,
});
}
@@ -724,11 +813,14 @@ impl<'a, T: BeaconChainTypes> IndexedUnaggregatedAttestation<'a, T> {
// Check to ensure that the attestation is "unaggregated". I.e., it has exactly one
// aggregation bit set.
let num_aggregation_bits = attestation.aggregation_bits.num_set_bits();
let num_aggregation_bits = attestation.num_set_aggregation_bits();
if num_aggregation_bits != 1 {
return Err(Error::NotExactlyOneAggregationBitSet(num_aggregation_bits));
}
// [New in Electra:EIP7549]
verify_committee_index(attestation)?;
// Attestations must be for a known block. If the block is unknown, we simply drop the
// attestation and do not delay consideration for later.
//
@@ -744,14 +836,14 @@ impl<'a, T: BeaconChainTypes> IndexedUnaggregatedAttestation<'a, T> {
/// Run the checks that apply to the indexed attestation before the signature is checked.
pub fn verify_middle_checks(
attestation: &Attestation<T::EthSpec>,
attestation: AttestationRef<T::EthSpec>,
indexed_attestation: &IndexedAttestation<T::EthSpec>,
committees_per_slot: u64,
subnet_id: Option<SubnetId>,
chain: &BeaconChain<T>,
) -> Result<(u64, SubnetId), Error> {
let expected_subnet_id = SubnetId::compute_subnet_for_attestation_data::<T::EthSpec>(
&indexed_attestation.data,
let expected_subnet_id = SubnetId::compute_subnet_for_attestation::<T::EthSpec>(
attestation,
committees_per_slot,
&chain.spec,
)
@@ -768,8 +860,7 @@ impl<'a, T: BeaconChainTypes> IndexedUnaggregatedAttestation<'a, T> {
};
let validator_index = *indexed_attestation
.attesting_indices
.first()
.attesting_indices_first()
.ok_or(Error::NotExactlyOneAggregationBitSet(0))?;
/*
@@ -779,12 +870,12 @@ impl<'a, T: BeaconChainTypes> IndexedUnaggregatedAttestation<'a, T> {
if chain
.observed_gossip_attesters
.read()
.validator_has_been_observed(attestation.data.target.epoch, validator_index as usize)
.validator_has_been_observed(attestation.data().target.epoch, validator_index as usize)
.map_err(BeaconChainError::from)?
{
return Err(Error::PriorAttestationKnown {
validator_index,
epoch: attestation.data.target.epoch,
epoch: attestation.data().target.epoch,
});
}
@@ -801,7 +892,7 @@ impl<'a, T: BeaconChainTypes> IndexedUnaggregatedAttestation<'a, T> {
subnet_id: Option<SubnetId>,
chain: &BeaconChain<T>,
) -> Result<Self, Error> {
Self::verify_slashable(attestation, subnet_id, chain)
Self::verify_slashable(attestation.to_ref(), subnet_id, chain)
.map(|verified_unaggregated| {
if let Some(slasher) = chain.slasher.as_ref() {
slasher.accept_attestation(verified_unaggregated.indexed_attestation.clone());
@@ -813,7 +904,7 @@ impl<'a, T: BeaconChainTypes> IndexedUnaggregatedAttestation<'a, T> {
/// Verify the attestation, producing extra information about whether it might be slashable.
pub fn verify_slashable(
attestation: &'a Attestation<T::EthSpec>,
attestation: AttestationRef<'a, T::EthSpec>,
subnet_id: Option<SubnetId>,
chain: &BeaconChain<T>,
) -> Result<Self, AttestationSlashInfo<'a, T, Error>> {
@@ -862,7 +953,7 @@ impl<'a, T: BeaconChainTypes> IndexedUnaggregatedAttestation<'a, T> {
impl<'a, T: BeaconChainTypes> VerifiedUnaggregatedAttestation<'a, T> {
/// Run the checks that apply after the signature has been checked.
fn verify_late_checks(
attestation: &Attestation<T::EthSpec>,
attestation: AttestationRef<T::EthSpec>,
validator_index: u64,
chain: &BeaconChain<T>,
) -> Result<(), Error> {
@@ -875,12 +966,12 @@ impl<'a, T: BeaconChainTypes> VerifiedUnaggregatedAttestation<'a, T> {
if chain
.observed_gossip_attesters
.write()
.observe_validator(attestation.data.target.epoch, validator_index as usize)
.observe_validator(attestation.data().target.epoch, validator_index as usize)
.map_err(BeaconChainError::from)?
{
return Err(Error::PriorAttestationKnown {
validator_index,
epoch: attestation.data.target.epoch,
epoch: attestation.data().target.epoch,
});
}
Ok(())
@@ -956,7 +1047,7 @@ impl<'a, T: BeaconChainTypes> VerifiedUnaggregatedAttestation<'a, T> {
}
/// Returns the wrapped `attestation`.
pub fn attestation(&self) -> &Attestation<T::EthSpec> {
pub fn attestation(&self) -> AttestationRef<T::EthSpec> {
self.attestation
}
@@ -986,34 +1077,34 @@ impl<'a, T: BeaconChainTypes> VerifiedUnaggregatedAttestation<'a, T> {
/// already finalized.
fn verify_head_block_is_known<T: BeaconChainTypes>(
chain: &BeaconChain<T>,
attestation: &Attestation<T::EthSpec>,
attestation: AttestationRef<T::EthSpec>,
max_skip_slots: Option<u64>,
) -> Result<ProtoBlock, Error> {
let block_opt = chain
.canonical_head
.fork_choice_read_lock()
.get_block(&attestation.data.beacon_block_root)
.get_block(&attestation.data().beacon_block_root)
.or_else(|| {
chain
.early_attester_cache
.get_proto_block(attestation.data.beacon_block_root)
.get_proto_block(attestation.data().beacon_block_root)
});
if let Some(block) = block_opt {
// Reject any block that exceeds our limit on skipped slots.
if let Some(max_skip_slots) = max_skip_slots {
if attestation.data.slot > block.slot + max_skip_slots {
if attestation.data().slot > block.slot + max_skip_slots {
return Err(Error::TooManySkippedSlots {
head_block_slot: block.slot,
attestation_slot: attestation.data.slot,
attestation_slot: attestation.data().slot,
});
}
}
Ok(block)
} else if chain.is_pre_finalization_block(attestation.data.beacon_block_root)? {
} else if chain.is_pre_finalization_block(attestation.data().beacon_block_root)? {
Err(Error::HeadBlockFinalized {
beacon_block_root: attestation.data.beacon_block_root,
beacon_block_root: attestation.data().beacon_block_root,
})
} else {
// The block is either:
@@ -1023,7 +1114,7 @@ fn verify_head_block_is_known<T: BeaconChainTypes>(
// 2) A post-finalization block that we don't know about yet. We'll queue
// the attestation until the block becomes available (or we time out).
Err(Error::UnknownHeadBlock {
beacon_block_root: attestation.data.beacon_block_root,
beacon_block_root: attestation.data().beacon_block_root,
})
}
}
@@ -1034,10 +1125,10 @@ fn verify_head_block_is_known<T: BeaconChainTypes>(
/// Accounts for `MAXIMUM_GOSSIP_CLOCK_DISPARITY`.
pub fn verify_propagation_slot_range<S: SlotClock, E: EthSpec>(
slot_clock: &S,
attestation: &Attestation<E>,
attestation: AttestationRef<E>,
spec: &ChainSpec,
) -> Result<(), Error> {
let attestation_slot = attestation.data.slot;
let attestation_slot = attestation.data().slot;
let latest_permissible_slot = slot_clock
.now_with_future_tolerance(spec.maximum_gossip_clock_disparity())
.ok_or(BeaconChainError::UnableToReadSlot)?;
@@ -1049,10 +1140,22 @@ pub fn verify_propagation_slot_range<S: SlotClock, E: EthSpec>(
}
// Taking advantage of saturating subtraction on `Slot`.
let earliest_permissible_slot = slot_clock
let one_epoch_prior = slot_clock
.now_with_past_tolerance(spec.maximum_gossip_clock_disparity())
.ok_or(BeaconChainError::UnableToReadSlot)?
- E::slots_per_epoch();
let current_fork =
spec.fork_name_at_slot::<E>(slot_clock.now().ok_or(BeaconChainError::UnableToReadSlot)?);
let earliest_permissible_slot = if !current_fork.deneb_enabled() {
one_epoch_prior
// EIP-7045
} else {
one_epoch_prior
.epoch(E::slots_per_epoch())
.start_slot(E::slots_per_epoch())
};
if attestation_slot < earliest_permissible_slot {
return Err(Error::PastSlot {
attestation_slot,
@@ -1078,18 +1181,17 @@ pub fn verify_attestation_signature<T: BeaconChainTypes>(
let fork = chain
.spec
.fork_at_epoch(indexed_attestation.data.target.epoch);
.fork_at_epoch(indexed_attestation.data().target.epoch);
let signature_set = indexed_attestation_signature_set_from_pubkeys(
|validator_index| pubkey_cache.get(validator_index).map(Cow::Borrowed),
&indexed_attestation.signature,
indexed_attestation.signature(),
indexed_attestation,
&fork,
chain.genesis_validators_root,
&chain.spec,
)
.map_err(BeaconChainError::SignatureSetError)?;
metrics::stop_timer(signature_setup_timer);
let _signature_verification_timer =
@@ -1104,13 +1206,13 @@ pub fn verify_attestation_signature<T: BeaconChainTypes>(
/// Verifies that the `attestation.data.target.root` is indeed the target root of the block at
/// `attestation.data.beacon_block_root`.
pub fn verify_attestation_target_root<T: EthSpec>(
pub fn verify_attestation_target_root<E: EthSpec>(
head_block: &ProtoBlock,
attestation: &Attestation<T>,
attestation: AttestationRef<E>,
) -> Result<(), Error> {
// Check the attestation target root.
let head_block_epoch = head_block.slot.epoch(T::slots_per_epoch());
let attestation_epoch = attestation.data.slot.epoch(T::slots_per_epoch());
let head_block_epoch = head_block.slot.epoch(E::slots_per_epoch());
let attestation_epoch = attestation.data().slot.epoch(E::slots_per_epoch());
if head_block_epoch > attestation_epoch {
// The epoch references an invalid head block from a future epoch.
//
@@ -1123,7 +1225,7 @@ pub fn verify_attestation_target_root<T: EthSpec>(
// Reference:
// https://github.com/ethereum/eth2.0-specs/pull/2001#issuecomment-699246659
return Err(Error::InvalidTargetRoot {
attestation: attestation.data.target.root,
attestation: attestation.data().target.root,
// It is not clear what root we should expect in this case, since the attestation is
// fundamentally invalid.
expected: None,
@@ -1142,9 +1244,9 @@ pub fn verify_attestation_target_root<T: EthSpec>(
};
// Reject any attestation with an invalid target root.
if target_root != attestation.data.target.root {
if target_root != attestation.data().target.root {
return Err(Error::InvalidTargetRoot {
attestation: attestation.data.target.root,
attestation: attestation.data().target.root,
expected: Some(target_root),
});
}
@@ -1175,14 +1277,14 @@ pub fn verify_signed_aggregate_signatures<T: BeaconChainTypes>(
.try_read_for(VALIDATOR_PUBKEY_CACHE_LOCK_TIMEOUT)
.ok_or(BeaconChainError::ValidatorPubkeyCacheLockTimeout)?;
let aggregator_index = signed_aggregate.message.aggregator_index;
let aggregator_index = signed_aggregate.message().aggregator_index();
if aggregator_index >= pubkey_cache.len() as u64 {
return Err(Error::AggregatorPubkeyUnknown(aggregator_index));
}
let fork = chain
.spec
.fork_at_epoch(indexed_attestation.data.target.epoch);
.fork_at_epoch(indexed_attestation.data().target.epoch);
let signature_sets = vec![
signed_aggregate_selection_proof_signature_set(
@@ -1203,7 +1305,7 @@ pub fn verify_signed_aggregate_signatures<T: BeaconChainTypes>(
.map_err(BeaconChainError::SignatureSetError)?,
indexed_attestation_signature_set_from_pubkeys(
|validator_index| pubkey_cache.get(validator_index).map(Cow::Borrowed),
&indexed_attestation.signature,
indexed_attestation.signature(),
indexed_attestation,
&fork,
chain.genesis_validators_root,
@@ -1215,6 +1317,28 @@ pub fn verify_signed_aggregate_signatures<T: BeaconChainTypes>(
Ok(verify_signature_sets(signature_sets.iter()))
}
/// Verify that the `attestation` committee index is properly set for the attestation's fork.
/// This function will only apply verification post-Electra.
pub fn verify_committee_index<E: EthSpec>(attestation: AttestationRef<E>) -> Result<(), Error> {
if let Ok(committee_bits) = attestation.committee_bits() {
// Check to ensure that the attestation is for a single committee.
let num_committee_bits = get_committee_indices::<E>(committee_bits);
if num_committee_bits.len() != 1 {
return Err(Error::NotExactlyOneCommitteeBitSet(
num_committee_bits.len(),
));
}
// Ensure the attestation index is set to zero post Electra.
if attestation.data().index != 0 {
return Err(Error::CommitteeIndexNonZero(
attestation.data().index as usize,
));
}
}
Ok(())
}
/// Assists in readability.
type CommitteesPerSlot = u64;
@@ -1222,35 +1346,71 @@ type CommitteesPerSlot = u64;
/// public keys cached in the `chain`.
pub fn obtain_indexed_attestation_and_committees_per_slot<T: BeaconChainTypes>(
chain: &BeaconChain<T>,
attestation: &Attestation<T::EthSpec>,
attestation: AttestationRef<T::EthSpec>,
) -> Result<(IndexedAttestation<T::EthSpec>, CommitteesPerSlot), Error> {
map_attestation_committee(chain, attestation, |(committee, committees_per_slot)| {
get_indexed_attestation(committee.committee, attestation)
.map(|attestation| (attestation, committees_per_slot))
.map_err(Error::Invalid)
map_attestation_committees(chain, attestation, |(committees, committees_per_slot)| {
match attestation {
AttestationRef::Base(att) => {
let committee = committees
.iter()
.filter(|&committee| committee.index == att.data.index)
.at_most_one()
.map_err(|_| Error::NoCommitteeForSlotAndIndex {
slot: att.data.slot,
index: att.data.index,
})?;
if let Some(committee) = committee {
attesting_indices_base::get_indexed_attestation(committee.committee, att)
.map(|attestation| (attestation, committees_per_slot))
.map_err(Error::Invalid)
} else {
Err(Error::NoCommitteeForSlotAndIndex {
slot: att.data.slot,
index: att.data.index,
})
}
}
AttestationRef::Electra(att) => {
attesting_indices_electra::get_indexed_attestation(&committees, att)
.map(|attestation| (attestation, committees_per_slot))
.map_err(|e| {
if let BlockOperationError::BeaconStateError(NoCommitteeFound(index)) = e {
Error::NoCommitteeForSlotAndIndex {
slot: att.data.slot,
index,
}
} else {
Error::Invalid(e)
}
})
}
}
})
}
/// Runs the `map_fn` with the committee and committee count per slot for the given `attestation`.
///
/// This function exists in this odd "map" pattern because efficiently obtaining the committee for
/// an attestation can be complex. It might involve reading straight from the
/// This function exists in this odd "map" pattern because efficiently obtaining the committees for
/// an attestation's slot can be complex. It might involve reading straight from the
/// `beacon_chain.shuffling_cache` or it might involve reading it from a state from the DB. Due to
/// the complexities of `RwLock`s on the shuffling cache, a simple `Cow` isn't suitable here.
///
/// If the committee for `attestation` isn't found in the `shuffling_cache`, we will read a state
/// If the committees for an `attestation`'s slot aren't found in the `shuffling_cache`, we will read a state
/// from disk and then update the `shuffling_cache`.
fn map_attestation_committee<T, F, R>(
///
/// Committees are sorted by ascending index order 0..committees_per_slot
fn map_attestation_committees<T, F, R>(
chain: &BeaconChain<T>,
attestation: &Attestation<T::EthSpec>,
attestation: AttestationRef<T::EthSpec>,
map_fn: F,
) -> Result<R, Error>
where
T: BeaconChainTypes,
F: Fn((BeaconCommittee, CommitteesPerSlot)) -> Result<R, Error>,
F: Fn((Vec<BeaconCommittee>, CommitteesPerSlot)) -> Result<R, Error>,
{
let attestation_epoch = attestation.data.slot.epoch(T::EthSpec::slots_per_epoch());
let target = &attestation.data.target;
let attestation_epoch = attestation.data().slot.epoch(T::EthSpec::slots_per_epoch());
let target = &attestation.data().target;
// Attestation target must be for a known block.
//
@@ -1273,12 +1433,12 @@ where
let committees_per_slot = committee_cache.committees_per_slot();
Ok(committee_cache
.get_beacon_committee(attestation.data.slot, attestation.data.index)
.map(|committee| map_fn((committee, committees_per_slot)))
.unwrap_or_else(|| {
.get_beacon_committees_at_slot(attestation.data().slot)
.map(|committees| map_fn((committees, committees_per_slot)))
.unwrap_or_else(|_| {
Err(Error::NoCommitteeForSlotAndIndex {
slot: attestation.data.slot,
index: attestation.data.index,
slot: attestation.data().slot,
index: attestation.committee_index().unwrap_or(0),
})
}))
})

View File

@@ -66,14 +66,13 @@ where
.ok_or(BeaconChainError::ValidatorPubkeyCacheLockTimeout)?;
let mut signature_sets = Vec::with_capacity(num_indexed * 3);
// Iterate, flattening to get only the `Ok` values.
for indexed in indexing_results.iter().flatten() {
let signed_aggregate = &indexed.signed_aggregate;
let indexed_attestation = &indexed.indexed_attestation;
let fork = chain
.spec
.fork_at_epoch(indexed_attestation.data.target.epoch);
.fork_at_epoch(indexed_attestation.data().target.epoch);
signature_sets.push(
signed_aggregate_selection_proof_signature_set(
@@ -98,7 +97,7 @@ where
signature_sets.push(
indexed_attestation_signature_set_from_pubkeys(
|validator_index| pubkey_cache.get(validator_index).map(Cow::Borrowed),
&indexed_attestation.signature,
indexed_attestation.signature(),
indexed_attestation,
&fork,
chain.genesis_validators_root,
@@ -182,11 +181,11 @@ where
let indexed_attestation = &partially_verified.indexed_attestation;
let fork = chain
.spec
.fork_at_epoch(indexed_attestation.data.target.epoch);
.fork_at_epoch(indexed_attestation.data().target.epoch);
let signature_set = indexed_attestation_signature_set_from_pubkeys(
|validator_index| pubkey_cache.get(validator_index).map(Cow::Borrowed),
&indexed_attestation.signature,
indexed_attestation.signature(),
indexed_attestation,
&fork,
chain.genesis_validators_root,

View File

@@ -15,6 +15,7 @@ use state_processing::state_advance::{partial_state_advance, Error as StateAdvan
use std::collections::HashMap;
use std::ops::Range;
use types::{
attestation::Error as AttestationError,
beacon_state::{
compute_committee_index_in_epoch, compute_committee_range_in_epoch, epoch_committee_count,
},
@@ -59,6 +60,7 @@ pub enum Error {
InverseRange {
range: Range<usize>,
},
AttestationError(AttestationError),
}
impl From<BeaconStateError> for Error {
@@ -84,7 +86,7 @@ pub struct CommitteeLengths {
impl CommitteeLengths {
/// Instantiate `Self` using `state.current_epoch()`.
pub fn new<T: EthSpec>(state: &BeaconState<T>, spec: &ChainSpec) -> Result<Self, Error> {
pub fn new<E: EthSpec>(state: &BeaconState<E>, spec: &ChainSpec) -> Result<Self, Error> {
let active_validator_indices_len = if let Ok(committee_cache) =
state.committee_cache(RelativeEpoch::Current)
{
@@ -102,21 +104,21 @@ impl CommitteeLengths {
}
/// Get the count of committees per each slot of `self.epoch`.
pub fn get_committee_count_per_slot<T: EthSpec>(
pub fn get_committee_count_per_slot<E: EthSpec>(
&self,
spec: &ChainSpec,
) -> Result<usize, Error> {
T::get_committee_count_per_slot(self.active_validator_indices_len, spec).map_err(Into::into)
E::get_committee_count_per_slot(self.active_validator_indices_len, spec).map_err(Into::into)
}
/// Get the length of the committee at the given `slot` and `committee_index`.
pub fn get_committee_length<T: EthSpec>(
pub fn get_committee_length<E: EthSpec>(
&self,
slot: Slot,
committee_index: CommitteeIndex,
spec: &ChainSpec,
) -> Result<CommitteeLength, Error> {
let slots_per_epoch = T::slots_per_epoch();
let slots_per_epoch = E::slots_per_epoch();
let request_epoch = slot.epoch(slots_per_epoch);
// Sanity check.
@@ -128,7 +130,7 @@ impl CommitteeLengths {
}
let slots_per_epoch = slots_per_epoch as usize;
let committees_per_slot = self.get_committee_count_per_slot::<T>(spec)?;
let committees_per_slot = self.get_committee_count_per_slot::<E>(spec)?;
let index_in_epoch = compute_committee_index_in_epoch(
slot,
slots_per_epoch,
@@ -162,7 +164,7 @@ pub struct AttesterCacheValue {
impl AttesterCacheValue {
/// Instantiate `Self` using `state.current_epoch()`.
pub fn new<T: EthSpec>(state: &BeaconState<T>, spec: &ChainSpec) -> Result<Self, Error> {
pub fn new<E: EthSpec>(state: &BeaconState<E>, spec: &ChainSpec) -> Result<Self, Error> {
let current_justified_checkpoint = state.current_justified_checkpoint();
let committee_lengths = CommitteeLengths::new(state, spec)?;
Ok(Self {
@@ -172,14 +174,14 @@ impl AttesterCacheValue {
}
/// Get the justified checkpoint and committee length for some `slot` and `committee_index`.
fn get<T: EthSpec>(
fn get<E: EthSpec>(
&self,
slot: Slot,
committee_index: CommitteeIndex,
spec: &ChainSpec,
) -> Result<(JustifiedCheckpoint, CommitteeLength), Error> {
self.committee_lengths
.get_committee_length::<T>(slot, committee_index, spec)
.get_committee_length::<E>(slot, committee_index, spec)
.map(|committee_length| (self.current_justified_checkpoint, committee_length))
}
}
@@ -216,12 +218,12 @@ impl AttesterCacheKey {
/// ## Errors
///
/// May error if `epoch` is out of the range of `state.block_roots`.
pub fn new<T: EthSpec>(
pub fn new<E: EthSpec>(
epoch: Epoch,
state: &BeaconState<T>,
state: &BeaconState<E>,
latest_block_root: Hash256,
) -> Result<Self, Error> {
let slots_per_epoch = T::slots_per_epoch();
let slots_per_epoch = E::slots_per_epoch();
let decision_slot = epoch.start_slot(slots_per_epoch).saturating_sub(1_u64);
let decision_root = if decision_slot.epoch(slots_per_epoch) == epoch {
@@ -255,7 +257,7 @@ pub struct AttesterCache {
impl AttesterCache {
/// Get the justified checkpoint and committee length for the `slot` and `committee_index` in
/// the state identified by the cache `key`.
pub fn get<T: EthSpec>(
pub fn get<E: EthSpec>(
&self,
key: &AttesterCacheKey,
slot: Slot,
@@ -265,14 +267,14 @@ impl AttesterCache {
self.cache
.read()
.get(key)
.map(|cache_item| cache_item.get::<T>(slot, committee_index, spec))
.map(|cache_item| cache_item.get::<E>(slot, committee_index, spec))
.transpose()
}
/// Cache the `state.current_epoch()` values if they are not already present in the state.
pub fn maybe_cache_state<T: EthSpec>(
pub fn maybe_cache_state<E: EthSpec>(
&self,
state: &BeaconState<T>,
state: &BeaconState<E>,
latest_block_root: Hash256,
spec: &ChainSpec,
) -> Result<(), Error> {

View File

@@ -4,9 +4,8 @@ use operation_pool::RewardCache;
use safe_arith::SafeArith;
use slog::error;
use state_processing::{
common::{
altair, get_attestation_participation_flag_indices, get_attesting_indices_from_state,
},
common::{get_attestation_participation_flag_indices, get_attesting_indices_from_state},
epoch_cache::initialize_epoch_cache,
per_block_processing::{
altair::sync_committee::compute_sync_aggregate_rewards, get_slashable_indices,
},
@@ -32,7 +31,19 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
state.build_committee_cache(RelativeEpoch::Previous, &self.spec)?;
state.build_committee_cache(RelativeEpoch::Current, &self.spec)?;
initialize_epoch_cache(state, &self.spec)?;
self.compute_beacon_block_reward_with_cache(block, block_root, state)
}
// This should only be called after a committee cache has been built
// for both the previous and current epoch
fn compute_beacon_block_reward_with_cache<Payload: AbstractExecPayload<T::EthSpec>>(
&self,
block: BeaconBlockRef<'_, T::EthSpec, Payload>,
block_root: Hash256,
state: &BeaconState<T::EthSpec>,
) -> Result<StandardBlockReward, BeaconChainError> {
let proposer_index = block.proposer_index();
let sync_aggregate_reward =
@@ -64,19 +75,19 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
self.compute_beacon_block_attestation_reward_base(block, block_root, state)
.map_err(|e| {
error!(
self.log,
"Error calculating base block attestation reward";
"error" => ?e
self.log,
"Error calculating base block attestation reward";
"error" => ?e
);
BeaconChainError::BlockRewardAttestationError
})?
} else {
self.compute_beacon_block_attestation_reward_altair(block, state)
self.compute_beacon_block_attestation_reward_altair_deneb(block, state)
.map_err(|e| {
error!(
self.log,
"Error calculating altair block attestation reward";
"error" => ?e
self.log,
"Error calculating altair block attestation reward";
"error" => ?e
);
BeaconChainError::BlockRewardAttestationError
})?
@@ -173,15 +184,13 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
Ok(block_attestation_reward)
}
fn compute_beacon_block_attestation_reward_altair<Payload: AbstractExecPayload<T::EthSpec>>(
fn compute_beacon_block_attestation_reward_altair_deneb<
Payload: AbstractExecPayload<T::EthSpec>,
>(
&self,
block: BeaconBlockRef<'_, T::EthSpec, Payload>,
state: &mut BeaconState<T::EthSpec>,
state: &BeaconState<T::EthSpec>,
) -> Result<BeaconBlockSubRewardValue, BeaconChainError> {
let total_active_balance = state.get_total_active_balance()?;
let base_reward_per_increment =
altair::BaseRewardPerIncrement::new(total_active_balance, &self.spec)?;
let mut total_proposer_reward = 0;
let proposer_reward_denominator = WEIGHT_DENOMINATOR
@@ -189,9 +198,13 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
.safe_mul(WEIGHT_DENOMINATOR)?
.safe_div(PROPOSER_WEIGHT)?;
let mut current_epoch_participation = state.current_epoch_participation()?.clone();
let mut previous_epoch_participation = state.previous_epoch_participation()?.clone();
for attestation in block.body().attestations() {
let data = &attestation.data;
let data = attestation.data();
let inclusion_delay = state.slot().safe_sub(data.slot)?.as_u64();
// [Modified in Deneb:EIP7045]
let participation_flag_indices = get_attestation_participation_flag_indices(
state,
data,
@@ -200,13 +213,16 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
)?;
let attesting_indices = get_attesting_indices_from_state(state, attestation)?;
let mut proposer_reward_numerator = 0;
for index in attesting_indices {
let index = index as usize;
for (flag_index, &weight) in PARTICIPATION_FLAG_WEIGHTS.iter().enumerate() {
let epoch_participation =
state.get_epoch_participation_mut(data.target.epoch)?;
let epoch_participation = if data.target.epoch == state.current_epoch() {
&mut current_epoch_participation
} else {
&mut previous_epoch_participation
};
let validator_participation = epoch_participation
.get_mut(index)
.ok_or(BeaconStateError::ParticipationOutOfBounds(index))?;
@@ -215,15 +231,8 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
&& !validator_participation.has_flag(flag_index)?
{
validator_participation.add_flag(flag_index)?;
proposer_reward_numerator.safe_add_assign(
altair::get_base_reward(
state,
index,
base_reward_per_increment,
&self.spec,
)?
.safe_mul(weight)?,
)?;
proposer_reward_numerator
.safe_add_assign(state.get_base_reward(index)?.safe_mul(weight)?)?;
}
}
}

View File

@@ -1,10 +1,9 @@
use crate::{BeaconChain, BeaconChainError, BeaconChainTypes};
use crate::{metrics, BeaconChain, BeaconChainError, BeaconChainTypes, BlockProcessStatus};
use execution_layer::{ExecutionLayer, ExecutionPayloadBodyV1};
use slog::{crit, debug, Logger};
use slog::{crit, debug, error, Logger};
use std::collections::HashMap;
use std::sync::Arc;
use store::DatabaseBlock;
use task_executor::TaskExecutor;
use store::{DatabaseBlock, ExecutionPayloadDeneb};
use tokio::sync::{
mpsc::{self, UnboundedSender},
RwLock,
@@ -15,11 +14,12 @@ use types::{
SignedBlindedBeaconBlock, Slot,
};
use types::{
ExecutionPayload, ExecutionPayloadCapella, ExecutionPayloadHeader, ExecutionPayloadMerge,
ExecutionPayload, ExecutionPayloadBellatrix, ExecutionPayloadCapella, ExecutionPayloadElectra,
ExecutionPayloadHeader,
};
#[derive(PartialEq)]
pub enum CheckEarlyAttesterCache {
pub enum CheckCaches {
Yes,
No,
}
@@ -95,8 +95,10 @@ fn reconstruct_default_header_block<E: EthSpec>(
.map_err(BeaconChainError::InconsistentFork)?;
let payload: ExecutionPayload<E> = match fork {
ForkName::Merge => ExecutionPayloadMerge::default().into(),
ForkName::Bellatrix => ExecutionPayloadBellatrix::default().into(),
ForkName::Capella => ExecutionPayloadCapella::default().into(),
ForkName::Deneb => ExecutionPayloadDeneb::default().into(),
ForkName::Electra => ExecutionPayloadElectra::default().into(),
ForkName::Base | ForkName::Altair => {
return Err(Error::PayloadReconstruction(format!(
"Block with fork variant {} has execution payload",
@@ -384,66 +386,81 @@ impl<E: EthSpec> EngineRequest<E> {
pub struct BeaconBlockStreamer<T: BeaconChainTypes> {
execution_layer: ExecutionLayer<T::EthSpec>,
check_early_attester_cache: CheckEarlyAttesterCache,
check_caches: CheckCaches,
beacon_chain: Arc<BeaconChain<T>>,
}
impl<T: BeaconChainTypes> BeaconBlockStreamer<T> {
pub fn new(
beacon_chain: &Arc<BeaconChain<T>>,
check_early_attester_cache: CheckEarlyAttesterCache,
) -> Result<Self, BeaconChainError> {
check_caches: CheckCaches,
) -> Result<Arc<Self>, BeaconChainError> {
let execution_layer = beacon_chain
.execution_layer
.as_ref()
.ok_or(BeaconChainError::ExecutionLayerMissing)?
.clone();
Ok(Self {
Ok(Arc::new(Self {
execution_layer,
check_early_attester_cache,
check_caches,
beacon_chain: beacon_chain.clone(),
})
}))
}
fn check_early_attester_cache(
&self,
root: Hash256,
) -> Option<Arc<SignedBeaconBlock<T::EthSpec>>> {
if self.check_early_attester_cache == CheckEarlyAttesterCache::Yes {
self.beacon_chain.early_attester_cache.get_block(root)
fn check_caches(&self, root: Hash256) -> Option<Arc<SignedBeaconBlock<T::EthSpec>>> {
if self.check_caches == CheckCaches::Yes {
match self.beacon_chain.get_block_process_status(&root) {
BlockProcessStatus::Unknown => None,
BlockProcessStatus::NotValidated(block)
| BlockProcessStatus::ExecutionValidated(block) => {
metrics::inc_counter(&metrics::BEACON_REQRESP_PRE_IMPORT_CACHE_HITS);
Some(block)
}
}
} else {
None
}
}
fn load_payloads(&self, block_roots: Vec<Hash256>) -> Vec<(Hash256, LoadResult<T::EthSpec>)> {
let mut db_blocks = Vec::new();
for root in block_roots {
if let Some(cached_block) = self
.check_early_attester_cache(root)
.map(LoadedBeaconBlock::Full)
{
db_blocks.push((root, Ok(Some(cached_block))));
continue;
}
match self.beacon_chain.store.try_get_full_block(&root) {
Err(e) => db_blocks.push((root, Err(e.into()))),
Ok(opt_block) => db_blocks.push((
root,
Ok(opt_block.map(|db_block| match db_block {
DatabaseBlock::Full(block) => LoadedBeaconBlock::Full(Arc::new(block)),
DatabaseBlock::Blinded(block) => {
LoadedBeaconBlock::Blinded(Box::new(block))
async fn load_payloads(
self: &Arc<Self>,
block_roots: Vec<Hash256>,
) -> Result<Vec<(Hash256, LoadResult<T::EthSpec>)>, BeaconChainError> {
let streamer = self.clone();
// Loading from the DB is slow -> spawn a blocking task
self.beacon_chain
.spawn_blocking_handle(
move || {
let mut db_blocks = Vec::new();
for root in block_roots {
if let Some(cached_block) =
streamer.check_caches(root).map(LoadedBeaconBlock::Full)
{
db_blocks.push((root, Ok(Some(cached_block))));
continue;
}
})),
)),
}
}
db_blocks
match streamer.beacon_chain.store.try_get_full_block(&root) {
Err(e) => db_blocks.push((root, Err(e.into()))),
Ok(opt_block) => db_blocks.push((
root,
Ok(opt_block.map(|db_block| match db_block {
DatabaseBlock::Full(block) => {
LoadedBeaconBlock::Full(Arc::new(block))
}
DatabaseBlock::Blinded(block) => {
LoadedBeaconBlock::Blinded(Box::new(block))
}
})),
)),
}
}
db_blocks
},
"load_beacon_blocks",
)
.await
}
/// Pre-process the loaded blocks into execution engine requests.
@@ -544,7 +561,7 @@ impl<T: BeaconChainTypes> BeaconBlockStreamer<T> {
// used when the execution engine doesn't support the payload bodies methods
async fn stream_blocks_fallback(
&self,
self: Arc<Self>,
block_roots: Vec<Hash256>,
sender: UnboundedSender<(Hash256, Arc<BlockResult<T::EthSpec>>)>,
) {
@@ -553,7 +570,7 @@ impl<T: BeaconChainTypes> BeaconBlockStreamer<T> {
"Using slower fallback method of eth_getBlockByHash()"
);
for root in block_roots {
let cached_block = self.check_early_attester_cache(root);
let cached_block = self.check_caches(root);
let block_result = if cached_block.is_some() {
Ok(cached_block)
} else {
@@ -570,7 +587,7 @@ impl<T: BeaconChainTypes> BeaconBlockStreamer<T> {
}
async fn stream_blocks(
&self,
self: Arc<Self>,
block_roots: Vec<Hash256>,
sender: UnboundedSender<(Hash256, Arc<BlockResult<T::EthSpec>>)>,
) {
@@ -579,7 +596,17 @@ impl<T: BeaconChainTypes> BeaconBlockStreamer<T> {
let mut n_sent = 0usize;
let mut engine_requests = 0usize;
let payloads = self.load_payloads(block_roots);
let payloads = match self.load_payloads(block_roots).await {
Ok(payloads) => payloads,
Err(e) => {
error!(
self.beacon_chain.log,
"BeaconBlockStreamer: Failed to load payloads";
"error" => ?e
);
return;
}
};
let requests = self.get_requests(payloads).await;
for (root, request) in requests {
@@ -619,7 +646,7 @@ impl<T: BeaconChainTypes> BeaconBlockStreamer<T> {
}
pub async fn stream(
self,
self: Arc<Self>,
block_roots: Vec<Hash256>,
sender: UnboundedSender<(Hash256, Arc<BlockResult<T::EthSpec>>)>,
) {
@@ -645,9 +672,8 @@ impl<T: BeaconChainTypes> BeaconBlockStreamer<T> {
}
pub fn launch_stream(
self,
self: Arc<Self>,
block_roots: Vec<Hash256>,
executor: &TaskExecutor,
) -> impl Stream<Item = (Hash256, Arc<BlockResult<T::EthSpec>>)> {
let (block_tx, block_rx) = mpsc::unbounded_channel();
debug!(
@@ -655,6 +681,7 @@ impl<T: BeaconChainTypes> BeaconBlockStreamer<T> {
"Launching a BeaconBlockStreamer";
"blocks" => block_roots.len(),
);
let executor = self.beacon_chain.task_executor.clone();
executor.spawn(self.stream(block_roots, block_tx), "get_blocks_sender");
UnboundedReceiverStream::new(block_rx)
}
@@ -681,7 +708,7 @@ impl From<Error> for BeaconChainError {
#[cfg(test)]
mod tests {
use crate::beacon_block_streamer::{BeaconBlockStreamer, CheckEarlyAttesterCache};
use crate::beacon_block_streamer::{BeaconBlockStreamer, CheckCaches};
use crate::test_utils::{test_spec, BeaconChainHarness, EphemeralHarnessType};
use execution_layer::test_utils::{Block, DEFAULT_ENGINE_CAPABILITIES};
use execution_layer::EngineCapabilities;
@@ -714,19 +741,23 @@ mod tests {
}
#[tokio::test]
async fn check_all_blocks_from_altair_to_capella() {
async fn check_all_blocks_from_altair_to_electra() {
let slots_per_epoch = MinimalEthSpec::slots_per_epoch() as usize;
let num_epochs = 8;
let num_epochs = 10;
let bellatrix_fork_epoch = 2usize;
let capella_fork_epoch = 4usize;
let deneb_fork_epoch = 6usize;
let electra_fork_epoch = 8usize;
let num_blocks_produced = num_epochs * slots_per_epoch;
let mut spec = test_spec::<MinimalEthSpec>();
spec.altair_fork_epoch = Some(Epoch::new(0));
spec.bellatrix_fork_epoch = Some(Epoch::new(bellatrix_fork_epoch as u64));
spec.capella_fork_epoch = Some(Epoch::new(capella_fork_epoch as u64));
spec.deneb_fork_epoch = Some(Epoch::new(deneb_fork_epoch as u64));
spec.electra_fork_epoch = Some(Epoch::new(electra_fork_epoch as u64));
let harness = get_harness(VALIDATOR_COUNT, spec);
let harness = get_harness(VALIDATOR_COUNT, spec.clone());
// go to bellatrix fork
harness
.extend_slots(bellatrix_fork_epoch * slots_per_epoch)
@@ -801,7 +832,7 @@ mod tests {
let start = epoch * slots_per_epoch;
let mut epoch_roots = vec![Hash256::zero(); slots_per_epoch];
epoch_roots[..].clone_from_slice(&block_roots[start..(start + slots_per_epoch)]);
let streamer = BeaconBlockStreamer::new(&harness.chain, CheckEarlyAttesterCache::No)
let streamer = BeaconBlockStreamer::new(&harness.chain, CheckCaches::No)
.expect("should create streamer");
let (block_tx, mut block_rx) = mpsc::unbounded_channel();
streamer.stream(epoch_roots.clone(), block_tx).await;
@@ -833,17 +864,21 @@ mod tests {
}
#[tokio::test]
async fn check_fallback_altair_to_capella() {
async fn check_fallback_altair_to_electra() {
let slots_per_epoch = MinimalEthSpec::slots_per_epoch() as usize;
let num_epochs = 8;
let num_epochs = 10;
let bellatrix_fork_epoch = 2usize;
let capella_fork_epoch = 4usize;
let deneb_fork_epoch = 6usize;
let electra_fork_epoch = 8usize;
let num_blocks_produced = num_epochs * slots_per_epoch;
let mut spec = test_spec::<MinimalEthSpec>();
spec.altair_fork_epoch = Some(Epoch::new(0));
spec.bellatrix_fork_epoch = Some(Epoch::new(bellatrix_fork_epoch as u64));
spec.capella_fork_epoch = Some(Epoch::new(capella_fork_epoch as u64));
spec.deneb_fork_epoch = Some(Epoch::new(deneb_fork_epoch as u64));
spec.electra_fork_epoch = Some(Epoch::new(electra_fork_epoch as u64));
let harness = get_harness(VALIDATOR_COUNT, spec);
@@ -940,7 +975,7 @@ mod tests {
let start = epoch * slots_per_epoch;
let mut epoch_roots = vec![Hash256::zero(); slots_per_epoch];
epoch_roots[..].clone_from_slice(&block_roots[start..(start + slots_per_epoch)]);
let streamer = BeaconBlockStreamer::new(&harness.chain, CheckEarlyAttesterCache::No)
let streamer = BeaconBlockStreamer::new(&harness.chain, CheckCaches::No)
.expect("should create streamer");
let (block_tx, mut block_rx) = mpsc::unbounded_channel();
streamer.stream(epoch_roots.clone(), block_tx).await;

File diff suppressed because it is too large Load Diff

View File

@@ -389,35 +389,35 @@ pub struct PersistedForkChoiceStore {
pub equivocating_indices: BTreeSet<u64>,
}
impl Into<PersistedForkChoiceStore> for PersistedForkChoiceStoreV11 {
fn into(self) -> PersistedForkChoiceStore {
impl From<PersistedForkChoiceStoreV11> for PersistedForkChoiceStore {
fn from(from: PersistedForkChoiceStoreV11) -> PersistedForkChoiceStore {
PersistedForkChoiceStore {
balances_cache: self.balances_cache,
time: self.time,
finalized_checkpoint: self.finalized_checkpoint,
justified_checkpoint: self.justified_checkpoint,
justified_balances: self.justified_balances,
unrealized_justified_checkpoint: self.unrealized_justified_checkpoint,
unrealized_finalized_checkpoint: self.unrealized_finalized_checkpoint,
proposer_boost_root: self.proposer_boost_root,
equivocating_indices: self.equivocating_indices,
balances_cache: from.balances_cache,
time: from.time,
finalized_checkpoint: from.finalized_checkpoint,
justified_checkpoint: from.justified_checkpoint,
justified_balances: from.justified_balances,
unrealized_justified_checkpoint: from.unrealized_justified_checkpoint,
unrealized_finalized_checkpoint: from.unrealized_finalized_checkpoint,
proposer_boost_root: from.proposer_boost_root,
equivocating_indices: from.equivocating_indices,
}
}
}
impl Into<PersistedForkChoiceStoreV11> for PersistedForkChoiceStore {
fn into(self) -> PersistedForkChoiceStoreV11 {
impl From<PersistedForkChoiceStore> for PersistedForkChoiceStoreV11 {
fn from(from: PersistedForkChoiceStore) -> PersistedForkChoiceStoreV11 {
PersistedForkChoiceStoreV11 {
balances_cache: self.balances_cache,
time: self.time,
finalized_checkpoint: self.finalized_checkpoint,
justified_checkpoint: self.justified_checkpoint,
justified_balances: self.justified_balances,
balances_cache: from.balances_cache,
time: from.time,
finalized_checkpoint: from.finalized_checkpoint,
justified_checkpoint: from.justified_checkpoint,
justified_balances: from.justified_balances,
best_justified_checkpoint: JUNK_BEST_JUSTIFIED_CHECKPOINT,
unrealized_justified_checkpoint: self.unrealized_justified_checkpoint,
unrealized_finalized_checkpoint: self.unrealized_finalized_checkpoint,
proposer_boost_root: self.proposer_boost_root,
equivocating_indices: self.equivocating_indices,
unrealized_justified_checkpoint: from.unrealized_justified_checkpoint,
unrealized_finalized_checkpoint: from.unrealized_finalized_checkpoint,
proposer_boost_root: from.proposer_boost_root,
equivocating_indices: from.equivocating_indices,
}
}
}

View File

@@ -14,18 +14,19 @@ use lru::LruCache;
use smallvec::SmallVec;
use state_processing::state_advance::partial_state_advance;
use std::cmp::Ordering;
use std::num::NonZeroUsize;
use types::non_zero_usize::new_non_zero_usize;
use types::{
BeaconState, BeaconStateError, ChainSpec, CloneConfig, Epoch, EthSpec, Fork, Hash256, Slot,
Unsigned,
BeaconState, BeaconStateError, ChainSpec, Epoch, EthSpec, Fork, Hash256, Slot, Unsigned,
};
/// The number of sets of proposer indices that should be cached.
const CACHE_SIZE: usize = 16;
const CACHE_SIZE: NonZeroUsize = new_non_zero_usize(16);
/// This value is fairly unimportant, it's used to avoid heap allocations. The result of it being
/// incorrect is non-substantial from a consensus perspective (and probably also from a
/// performance perspective).
const TYPICAL_SLOTS_PER_EPOCH: usize = 32;
pub const TYPICAL_SLOTS_PER_EPOCH: usize = 32;
/// For some given slot, this contains the proposer index (`index`) and the `fork` that should be
/// used to verify their signature.
@@ -66,19 +67,19 @@ impl Default for BeaconProposerCache {
impl BeaconProposerCache {
/// If it is cached, returns the proposer for the block at `slot` where the block has the
/// ancestor block root of `shuffling_decision_block` at `end_slot(slot.epoch() - 1)`.
pub fn get_slot<T: EthSpec>(
pub fn get_slot<E: EthSpec>(
&mut self,
shuffling_decision_block: Hash256,
slot: Slot,
) -> Option<Proposer> {
let epoch = slot.epoch(T::slots_per_epoch());
let epoch = slot.epoch(E::slots_per_epoch());
let key = (epoch, shuffling_decision_block);
if let Some(cache) = self.cache.get(&key) {
// This `if` statement is likely unnecessary, but it feels like good practice.
if epoch == cache.epoch {
cache
.proposers
.get(slot.as_usize() % T::SlotsPerEpoch::to_usize())
.get(slot.as_usize() % E::SlotsPerEpoch::to_usize())
.map(|&index| Proposer {
index,
fork: cache.fork,
@@ -96,7 +97,7 @@ impl BeaconProposerCache {
/// The nth slot in the returned `SmallVec` will be equal to the nth slot in the given `epoch`.
/// E.g., if `epoch == 1` then `smallvec[0]` refers to slot 32 (assuming `SLOTS_PER_EPOCH ==
/// 32`).
pub fn get_epoch<T: EthSpec>(
pub fn get_epoch<E: EthSpec>(
&mut self,
shuffling_decision_block: Hash256,
epoch: Epoch,
@@ -143,10 +144,7 @@ pub fn compute_proposer_duties_from_head<T: BeaconChainTypes>(
let (mut state, head_state_root, head_block_root) = {
let head = chain.canonical_head.cached_head();
// Take a copy of the head state.
let head_state = head
.snapshot
.beacon_state
.clone_with(CloneConfig::committee_caches_only());
let head_state = head.snapshot.beacon_state.clone();
let head_state_root = head.head_state_root();
let head_block_root = head.head_block_root();
(head_state, head_state_root, head_block_root)

View File

@@ -1,8 +1,8 @@
use serde::Serialize;
use std::sync::Arc;
use types::{
beacon_state::CloneConfig, AbstractExecPayload, BeaconState, EthSpec, FullPayload, Hash256,
SignedBeaconBlock,
AbstractExecPayload, BeaconState, EthSpec, FullPayload, Hash256, SignedBeaconBlock,
SignedBlindedBeaconBlock,
};
/// Represents some block and its associated state. Generally, this will be used for tracking the
@@ -14,6 +14,19 @@ pub struct BeaconSnapshot<E: EthSpec, Payload: AbstractExecPayload<E> = FullPayl
pub beacon_state: BeaconState<E>,
}
/// This snapshot is to be used for verifying a child of `self.beacon_block`.
#[derive(Debug)]
pub struct PreProcessingSnapshot<T: EthSpec> {
/// This state is equivalent to the `self.beacon_block.state_root()` state that has been
/// advanced forward one slot using `per_slot_processing`. This state is "primed and ready" for
/// the application of another block.
pub pre_state: BeaconState<T>,
/// This value is only set to `Some` if the `pre_state` was *not* advanced forward.
pub beacon_state_root: Option<Hash256>,
pub beacon_block: SignedBlindedBeaconBlock<T>,
pub beacon_block_root: Hash256,
}
impl<E: EthSpec, Payload: AbstractExecPayload<E>> BeaconSnapshot<E, Payload> {
/// Create a new checkpoint.
pub fn new(
@@ -48,12 +61,4 @@ impl<E: EthSpec, Payload: AbstractExecPayload<E>> BeaconSnapshot<E, Payload> {
self.beacon_block_root = beacon_block_root;
self.beacon_state = beacon_state;
}
pub fn clone_with(&self, clone_config: CloneConfig) -> Self {
Self {
beacon_block: self.beacon_block.clone(),
beacon_block_root: self.beacon_block_root,
beacon_state: self.beacon_state.clone_with(clone_config),
}
}
}

View File

@@ -11,7 +11,7 @@ use types::*;
/// The time before the Bellatrix fork when we will start issuing warnings about preparation.
pub const SECONDS_IN_A_WEEK: u64 = 604800;
pub const MERGE_READINESS_PREPARATION_SECONDS: u64 = SECONDS_IN_A_WEEK * 2;
pub const BELLATRIX_READINESS_PREPARATION_SECONDS: u64 = SECONDS_IN_A_WEEK * 2;
#[derive(Default, Debug, Serialize, Deserialize)]
pub struct MergeConfig {
@@ -81,7 +81,7 @@ impl MergeConfig {
#[derive(Debug, Serialize, Deserialize)]
#[serde(rename_all = "snake_case")]
#[serde(tag = "type")]
pub enum MergeReadiness {
pub enum BellatrixReadiness {
/// The node is ready, as far as we can tell.
Ready {
config: MergeConfig,
@@ -94,29 +94,29 @@ pub enum MergeReadiness {
NoExecutionEndpoint,
}
impl fmt::Display for MergeReadiness {
impl fmt::Display for BellatrixReadiness {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
match self {
MergeReadiness::Ready {
BellatrixReadiness::Ready {
config: params,
current_difficulty,
} => {
write!(
f,
"This node appears ready for the merge. \
"This node appears ready for Bellatrix \
Params: {}, current_difficulty: {:?}",
params, current_difficulty
)
}
MergeReadiness::NotSynced => write!(
BellatrixReadiness::NotSynced => write!(
f,
"The execution endpoint is connected and configured, \
however it is not yet synced"
),
MergeReadiness::NoExecutionEndpoint => write!(
BellatrixReadiness::NoExecutionEndpoint => write!(
f,
"The --execution-endpoint flag is not specified, this is a \
requirement for the merge"
requirement for Bellatrix"
),
}
}
@@ -143,12 +143,12 @@ pub enum GenesisExecutionPayloadStatus {
impl<T: BeaconChainTypes> BeaconChain<T> {
/// Returns `true` if user has an EL configured, or if the Bellatrix fork has occurred or will
/// occur within `MERGE_READINESS_PREPARATION_SECONDS`.
/// occur within `BELLATRIX_READINESS_PREPARATION_SECONDS`.
pub fn is_time_to_prepare_for_bellatrix(&self, current_slot: Slot) -> bool {
if let Some(bellatrix_epoch) = self.spec.bellatrix_fork_epoch {
let bellatrix_slot = bellatrix_epoch.start_slot(T::EthSpec::slots_per_epoch());
let merge_readiness_preparation_slots =
MERGE_READINESS_PREPARATION_SECONDS / self.spec.seconds_per_slot;
let bellatrix_readiness_preparation_slots =
BELLATRIX_READINESS_PREPARATION_SECONDS / self.spec.seconds_per_slot;
if self.execution_layer.is_some() {
// The user has already configured an execution layer, start checking for readiness
@@ -156,7 +156,7 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
true
} else {
// Return `true` if Bellatrix has happened or is within the preparation time.
current_slot + merge_readiness_preparation_slots > bellatrix_slot
current_slot + bellatrix_readiness_preparation_slots > bellatrix_slot
}
} else {
// The Bellatrix fork epoch has not been defined yet, no need to prepare.
@@ -164,22 +164,22 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
}
}
/// Attempts to connect to the EL and confirm that it is ready for the merge.
pub async fn check_merge_readiness(&self, current_slot: Slot) -> MergeReadiness {
/// Attempts to connect to the EL and confirm that it is ready for Bellatrix.
pub async fn check_bellatrix_readiness(&self, current_slot: Slot) -> BellatrixReadiness {
if let Some(el) = self.execution_layer.as_ref() {
if !el.is_synced_for_notifier(current_slot).await {
// The EL is not synced.
return MergeReadiness::NotSynced;
return BellatrixReadiness::NotSynced;
}
let params = MergeConfig::from_chainspec(&self.spec);
let current_difficulty = el.get_current_difficulty().await.ok();
MergeReadiness::Ready {
BellatrixReadiness::Ready {
config: params,
current_difficulty,
}
} else {
// There is no EL configured.
MergeReadiness::NoExecutionEndpoint
BellatrixReadiness::NoExecutionEndpoint
}
}
@@ -244,8 +244,8 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
});
}
if let Some(&expected) = expected_withdrawals_root {
if let Some(&got) = got_withdrawals_root {
if let Some(expected) = expected_withdrawals_root {
if let Some(got) = got_withdrawals_root {
if got != expected {
return Ok(GenesisExecutionPayloadStatus::WithdrawalsRootMismatch {
got,

View File

@@ -0,0 +1,622 @@
use derivative::Derivative;
use slot_clock::SlotClock;
use std::sync::Arc;
use crate::beacon_chain::{BeaconChain, BeaconChainTypes};
use crate::block_verification::{
cheap_state_advance_to_obtain_committees, get_validator_pubkey_cache, process_block_slash_info,
BlockSlashInfo,
};
use crate::kzg_utils::{validate_blob, validate_blobs};
use crate::{metrics, BeaconChainError};
use kzg::{Error as KzgError, Kzg, KzgCommitment};
use slog::debug;
use ssz_derive::{Decode, Encode};
use ssz_types::VariableList;
use std::time::Duration;
use tree_hash::TreeHash;
use types::blob_sidecar::BlobIdentifier;
use types::{BeaconStateError, BlobSidecar, EthSpec, Hash256, SignedBeaconBlockHeader, Slot};
/// An error occurred while validating a gossip blob.
#[derive(Debug)]
pub enum GossipBlobError<E: EthSpec> {
/// The blob sidecar is from a slot that is later than the current slot (with respect to the
/// gossip clock disparity).
///
/// ## Peer scoring
///
/// Assuming the local clock is correct, the peer has sent an invalid message.
FutureSlot {
message_slot: Slot,
latest_permissible_slot: Slot,
},
/// There was an error whilst processing the blob. It is not known if it is
/// valid or invalid.
///
/// ## Peer scoring
///
/// We were unable to process this blob due to an internal error. It's
/// unclear if the blob is valid.
BeaconChainError(BeaconChainError),
/// The `BlobSidecar` was gossiped over an incorrect subnet.
///
/// ## Peer scoring
///
/// The blob is invalid or the peer is faulty.
InvalidSubnet { expected: u64, received: u64 },
/// The sidecar corresponds to a slot older than the finalized head slot.
///
/// ## Peer scoring
///
/// It's unclear if this blob is valid, but this blob is for a finalized slot and is
/// therefore useless to us.
PastFinalizedSlot {
blob_slot: Slot,
finalized_slot: Slot,
},
/// The proposer index specified in the sidecar does not match the locally computed
/// proposer index.
///
/// ## Peer scoring
///
/// The blob is invalid and the peer is faulty.
ProposerIndexMismatch { sidecar: usize, local: usize },
/// The proposal signature in invalid.
///
/// ## Peer scoring
///
/// The blob is invalid and the peer is faulty.
ProposalSignatureInvalid,
/// The proposal_index corresponding to blob.beacon_block_root is not known.
///
/// ## Peer scoring
///
/// The blob is invalid and the peer is faulty.
UnknownValidator(u64),
/// The provided blob is not from a later slot than its parent.
///
/// ## Peer scoring
///
/// The blob is invalid and the peer is faulty.
BlobIsNotLaterThanParent { blob_slot: Slot, parent_slot: Slot },
/// The provided blob's parent block is unknown.
///
/// ## Peer scoring
///
/// We cannot process the blob without validating its parent, the peer isn't necessarily faulty.
BlobParentUnknown(Arc<BlobSidecar<E>>),
/// Invalid kzg commitment inclusion proof
/// ## Peer scoring
///
/// The blob sidecar is invalid and the peer is faulty
InvalidInclusionProof,
/// A blob has already been seen for the given `(sidecar.block_root, sidecar.index)` tuple
/// over gossip or no gossip sources.
///
/// ## Peer scoring
///
/// The peer isn't faulty, but we do not forward it over gossip.
RepeatBlob {
proposer: u64,
slot: Slot,
index: u64,
},
/// `Kzg` struct hasn't been initialized. This is an internal error.
///
/// ## Peer scoring
///
/// The peer isn't faulty, This is an internal error.
KzgNotInitialized,
/// The kzg verification failed.
///
/// ## Peer scoring
///
/// The blob sidecar is invalid and the peer is faulty.
KzgError(kzg::Error),
/// The pubkey cache timed out.
///
/// ## Peer scoring
///
/// The blob sidecar may be valid, this is an internal error.
PubkeyCacheTimeout,
/// The block conflicts with finalization, no need to propagate.
///
/// ## Peer scoring
///
/// It's unclear if this block is valid, but it conflicts with finality and shouldn't be
/// imported.
NotFinalizedDescendant { block_parent_root: Hash256 },
}
impl<E: EthSpec> std::fmt::Display for GossipBlobError<E> {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self {
GossipBlobError::BlobParentUnknown(blob_sidecar) => {
write!(
f,
"BlobParentUnknown(parent_root:{})",
blob_sidecar.block_parent_root()
)
}
other => write!(f, "{:?}", other),
}
}
}
impl<E: EthSpec> From<BeaconChainError> for GossipBlobError<E> {
fn from(e: BeaconChainError) -> Self {
GossipBlobError::BeaconChainError(e)
}
}
impl<E: EthSpec> From<BeaconStateError> for GossipBlobError<E> {
fn from(e: BeaconStateError) -> Self {
GossipBlobError::BeaconChainError(BeaconChainError::BeaconStateError(e))
}
}
pub type GossipVerifiedBlobList<T> = VariableList<
GossipVerifiedBlob<T>,
<<T as BeaconChainTypes>::EthSpec as EthSpec>::MaxBlobsPerBlock,
>;
/// A wrapper around a `BlobSidecar` that indicates it has been approved for re-gossiping on
/// the p2p network.
#[derive(Debug)]
pub struct GossipVerifiedBlob<T: BeaconChainTypes> {
block_root: Hash256,
blob: KzgVerifiedBlob<T::EthSpec>,
}
impl<T: BeaconChainTypes> GossipVerifiedBlob<T> {
pub fn new(
blob: Arc<BlobSidecar<T::EthSpec>>,
subnet_id: u64,
chain: &BeaconChain<T>,
) -> Result<Self, GossipBlobError<T::EthSpec>> {
let header = blob.signed_block_header.clone();
// We only process slashing info if the gossip verification failed
// since we do not process the blob any further in that case.
validate_blob_sidecar_for_gossip(blob, subnet_id, chain).map_err(|e| {
process_block_slash_info::<_, GossipBlobError<T::EthSpec>>(
chain,
BlockSlashInfo::from_early_error_blob(header, e),
)
})
}
/// Construct a `GossipVerifiedBlob` that is assumed to be valid.
///
/// This should ONLY be used for testing.
pub fn __assumed_valid(blob: Arc<BlobSidecar<T::EthSpec>>) -> Self {
Self {
block_root: blob.block_root(),
blob: KzgVerifiedBlob {
blob,
seen_timestamp: Duration::from_secs(0),
},
}
}
pub fn id(&self) -> BlobIdentifier {
BlobIdentifier {
block_root: self.block_root,
index: self.blob.blob_index(),
}
}
pub fn block_root(&self) -> Hash256 {
self.block_root
}
pub fn slot(&self) -> Slot {
self.blob.blob.slot()
}
pub fn index(&self) -> u64 {
self.blob.blob.index
}
pub fn kzg_commitment(&self) -> KzgCommitment {
self.blob.blob.kzg_commitment
}
pub fn signed_block_header(&self) -> SignedBeaconBlockHeader {
self.blob.blob.signed_block_header.clone()
}
pub fn block_proposer_index(&self) -> u64 {
self.blob.blob.block_proposer_index()
}
pub fn into_inner(self) -> KzgVerifiedBlob<T::EthSpec> {
self.blob
}
pub fn as_blob(&self) -> &BlobSidecar<T::EthSpec> {
self.blob.as_blob()
}
/// This is cheap as we're calling clone on an Arc
pub fn clone_blob(&self) -> Arc<BlobSidecar<T::EthSpec>> {
self.blob.clone_blob()
}
}
/// Wrapper over a `BlobSidecar` for which we have completed kzg verification.
/// i.e. `verify_blob_kzg_proof(blob, commitment, proof) == true`.
#[derive(Debug, Derivative, Clone, Encode, Decode)]
#[derivative(PartialEq, Eq)]
#[ssz(struct_behaviour = "transparent")]
pub struct KzgVerifiedBlob<E: EthSpec> {
blob: Arc<BlobSidecar<E>>,
#[ssz(skip_serializing, skip_deserializing)]
seen_timestamp: Duration,
}
impl<E: EthSpec> PartialOrd for KzgVerifiedBlob<E> {
fn partial_cmp(&self, other: &Self) -> Option<std::cmp::Ordering> {
Some(self.cmp(other))
}
}
impl<E: EthSpec> Ord for KzgVerifiedBlob<E> {
fn cmp(&self, other: &Self) -> std::cmp::Ordering {
self.blob.cmp(&other.blob)
}
}
impl<E: EthSpec> KzgVerifiedBlob<E> {
pub fn new(
blob: Arc<BlobSidecar<E>>,
kzg: &Kzg,
seen_timestamp: Duration,
) -> Result<Self, KzgError> {
verify_kzg_for_blob(blob, kzg, seen_timestamp)
}
pub fn to_blob(self) -> Arc<BlobSidecar<E>> {
self.blob
}
pub fn as_blob(&self) -> &BlobSidecar<E> {
&self.blob
}
pub fn get_commitment(&self) -> &KzgCommitment {
&self.blob.kzg_commitment
}
/// This is cheap as we're calling clone on an Arc
pub fn clone_blob(&self) -> Arc<BlobSidecar<E>> {
self.blob.clone()
}
pub fn blob_index(&self) -> u64 {
self.blob.index
}
pub fn seen_timestamp(&self) -> Duration {
self.seen_timestamp
}
/// Construct a `KzgVerifiedBlob` that is assumed to be valid.
///
/// This should ONLY be used for testing.
#[cfg(test)]
pub fn __assumed_valid(blob: Arc<BlobSidecar<E>>) -> Self {
Self {
blob,
seen_timestamp: Duration::from_secs(0),
}
}
}
/// Complete kzg verification for a `BlobSidecar`.
///
/// Returns an error if the kzg verification check fails.
pub fn verify_kzg_for_blob<E: EthSpec>(
blob: Arc<BlobSidecar<E>>,
kzg: &Kzg,
seen_timestamp: Duration,
) -> Result<KzgVerifiedBlob<E>, KzgError> {
validate_blob::<E>(kzg, &blob.blob, blob.kzg_commitment, blob.kzg_proof)?;
Ok(KzgVerifiedBlob {
blob,
seen_timestamp,
})
}
pub struct KzgVerifiedBlobList<E: EthSpec> {
verified_blobs: Vec<KzgVerifiedBlob<E>>,
}
impl<E: EthSpec> KzgVerifiedBlobList<E> {
pub fn new<I: IntoIterator<Item = Arc<BlobSidecar<E>>>>(
blob_list: I,
kzg: &Kzg,
seen_timestamp: Duration,
) -> Result<Self, KzgError> {
let blobs = blob_list.into_iter().collect::<Vec<_>>();
verify_kzg_for_blob_list(blobs.iter(), kzg)?;
Ok(Self {
verified_blobs: blobs
.into_iter()
.map(|blob| KzgVerifiedBlob {
blob,
seen_timestamp,
})
.collect(),
})
}
}
impl<E: EthSpec> IntoIterator for KzgVerifiedBlobList<E> {
type Item = KzgVerifiedBlob<E>;
type IntoIter = std::vec::IntoIter<Self::Item>;
fn into_iter(self) -> Self::IntoIter {
self.verified_blobs.into_iter()
}
}
/// Complete kzg verification for a list of `BlobSidecar`s.
/// Returns an error if any of the `BlobSidecar`s fails kzg verification.
///
/// Note: This function should be preferred over calling `verify_kzg_for_blob`
/// in a loop since this function kzg verifies a list of blobs more efficiently.
pub fn verify_kzg_for_blob_list<'a, E: EthSpec, I>(
blob_iter: I,
kzg: &'a Kzg,
) -> Result<(), KzgError>
where
I: Iterator<Item = &'a Arc<BlobSidecar<E>>>,
{
let (blobs, (commitments, proofs)): (Vec<_>, (Vec<_>, Vec<_>)) = blob_iter
.map(|blob| (&blob.blob, (blob.kzg_commitment, blob.kzg_proof)))
.unzip();
validate_blobs::<E>(kzg, commitments.as_slice(), blobs, proofs.as_slice())
}
pub fn validate_blob_sidecar_for_gossip<T: BeaconChainTypes>(
blob_sidecar: Arc<BlobSidecar<T::EthSpec>>,
subnet: u64,
chain: &BeaconChain<T>,
) -> Result<GossipVerifiedBlob<T>, GossipBlobError<T::EthSpec>> {
let blob_slot = blob_sidecar.slot();
let blob_index = blob_sidecar.index;
let block_parent_root = blob_sidecar.block_parent_root();
let blob_proposer_index = blob_sidecar.block_proposer_index();
let block_root = blob_sidecar.block_root();
let blob_epoch = blob_slot.epoch(T::EthSpec::slots_per_epoch());
let signed_block_header = &blob_sidecar.signed_block_header;
let seen_timestamp = chain.slot_clock.now_duration().unwrap_or_default();
// This condition is not possible if we have received the blob from the network
// since we only subscribe to `MaxBlobsPerBlock` subnets over gossip network.
// We include this check only for completeness.
// Getting this error would imply something very wrong with our networking decoding logic.
if blob_index >= T::EthSpec::max_blobs_per_block() as u64 {
return Err(GossipBlobError::InvalidSubnet {
expected: subnet,
received: blob_index,
});
}
// Verify that the blob_sidecar was received on the correct subnet.
if blob_index != subnet {
return Err(GossipBlobError::InvalidSubnet {
expected: blob_index,
received: subnet,
});
}
// Verify that the sidecar is not from a future slot.
let latest_permissible_slot = chain
.slot_clock
.now_with_future_tolerance(chain.spec.maximum_gossip_clock_disparity())
.ok_or(BeaconChainError::UnableToReadSlot)?;
if blob_slot > latest_permissible_slot {
return Err(GossipBlobError::FutureSlot {
message_slot: blob_slot,
latest_permissible_slot,
});
}
// Verify that the sidecar slot is greater than the latest finalized slot
let latest_finalized_slot = chain
.head()
.finalized_checkpoint()
.epoch
.start_slot(T::EthSpec::slots_per_epoch());
if blob_slot <= latest_finalized_slot {
return Err(GossipBlobError::PastFinalizedSlot {
blob_slot,
finalized_slot: latest_finalized_slot,
});
}
// Verify that this is the first blob sidecar received for the tuple:
// (block_header.slot, block_header.proposer_index, blob_sidecar.index)
if chain
.observed_blob_sidecars
.read()
.proposer_is_known(&blob_sidecar)
.map_err(|e| GossipBlobError::BeaconChainError(e.into()))?
{
return Err(GossipBlobError::RepeatBlob {
proposer: blob_proposer_index,
slot: blob_slot,
index: blob_index,
});
}
// Verify the inclusion proof in the sidecar
let _timer = metrics::start_timer(&metrics::BLOB_SIDECAR_INCLUSION_PROOF_VERIFICATION);
if !blob_sidecar.verify_blob_sidecar_inclusion_proof() {
return Err(GossipBlobError::InvalidInclusionProof);
}
drop(_timer);
let fork_choice = chain.canonical_head.fork_choice_read_lock();
// We have already verified that the blob is past finalization, so we can
// just check fork choice for the block's parent.
let Some(parent_block) = fork_choice.get_block(&block_parent_root) else {
return Err(GossipBlobError::BlobParentUnknown(blob_sidecar));
};
// Do not process a blob that does not descend from the finalized root.
// We just loaded the parent_block, so we can be sure that it exists in fork choice.
if !fork_choice.is_finalized_checkpoint_or_descendant(block_parent_root) {
return Err(GossipBlobError::NotFinalizedDescendant { block_parent_root });
}
drop(fork_choice);
if parent_block.slot >= blob_slot {
return Err(GossipBlobError::BlobIsNotLaterThanParent {
blob_slot,
parent_slot: parent_block.slot,
});
}
let proposer_shuffling_root =
if parent_block.slot.epoch(T::EthSpec::slots_per_epoch()) == blob_epoch {
parent_block
.next_epoch_shuffling_id
.shuffling_decision_block
} else {
parent_block.root
};
let proposer_opt = chain
.beacon_proposer_cache
.lock()
.get_slot::<T::EthSpec>(proposer_shuffling_root, blob_slot);
let (proposer_index, fork) = if let Some(proposer) = proposer_opt {
(proposer.index, proposer.fork)
} else {
debug!(
chain.log,
"Proposer shuffling cache miss for blob verification";
"block_root" => %block_root,
"index" => %blob_index,
);
let (parent_state_root, mut parent_state) = chain
.store
.get_advanced_hot_state(block_parent_root, blob_slot, parent_block.state_root)
.map_err(|e| GossipBlobError::BeaconChainError(e.into()))?
.ok_or_else(|| {
BeaconChainError::DBInconsistent(format!(
"Missing state for parent block {block_parent_root:?}",
))
})?;
let state = cheap_state_advance_to_obtain_committees::<_, GossipBlobError<T::EthSpec>>(
&mut parent_state,
Some(parent_state_root),
blob_slot,
&chain.spec,
)?;
let proposers = state.get_beacon_proposer_indices(&chain.spec)?;
let proposer_index = *proposers
.get(blob_slot.as_usize() % T::EthSpec::slots_per_epoch() as usize)
.ok_or_else(|| BeaconChainError::NoProposerForSlot(blob_slot))?;
// Prime the proposer shuffling cache with the newly-learned value.
chain.beacon_proposer_cache.lock().insert(
blob_epoch,
proposer_shuffling_root,
proposers,
state.fork(),
)?;
(proposer_index, state.fork())
};
// Signature verify the signed block header.
let signature_is_valid = {
let pubkey_cache =
get_validator_pubkey_cache(chain).map_err(|_| GossipBlobError::PubkeyCacheTimeout)?;
let pubkey = pubkey_cache
.get(proposer_index)
.ok_or_else(|| GossipBlobError::UnknownValidator(proposer_index as u64))?;
signed_block_header.verify_signature::<T::EthSpec>(
pubkey,
&fork,
chain.genesis_validators_root,
&chain.spec,
)
};
if !signature_is_valid {
return Err(GossipBlobError::ProposalSignatureInvalid);
}
if proposer_index != blob_proposer_index as usize {
return Err(GossipBlobError::ProposerIndexMismatch {
sidecar: blob_proposer_index as usize,
local: proposer_index,
});
}
// Kzg verification for gossip blob sidecar
let kzg = chain
.kzg
.as_ref()
.ok_or(GossipBlobError::KzgNotInitialized)?;
let kzg_verified_blob = KzgVerifiedBlob::new(blob_sidecar.clone(), kzg, seen_timestamp)
.map_err(GossipBlobError::KzgError)?;
chain
.observed_slashable
.write()
.observe_slashable(
blob_sidecar.slot(),
blob_sidecar.block_proposer_index(),
block_root,
)
.map_err(|e| GossipBlobError::BeaconChainError(e.into()))?;
// Now the signature is valid, store the proposal so we don't accept another blob sidecar
// with the same `BlobIdentifier`.
// It's important to double-check that the proposer still hasn't been observed so we don't
// have a race-condition when verifying two blocks simultaneously.
//
// Note: If this BlobSidecar goes on to fail full verification, we do not evict it from the seen_cache
// as alternate blob_sidecars for the same identifier can still be retrieved
// over rpc. Evicting them from this cache would allow faster propagation over gossip. So we allow
// retrieval of potentially valid blocks over rpc, but try to punish the proposer for signing
// invalid messages. Issue for more background
// https://github.com/ethereum/consensus-specs/issues/3261
if chain
.observed_blob_sidecars
.write()
.observe_sidecar(&blob_sidecar)
.map_err(|e| GossipBlobError::BeaconChainError(e.into()))?
{
return Err(GossipBlobError::RepeatBlob {
proposer: proposer_index as u64,
slot: blob_slot,
index: blob_index,
});
}
Ok(GossipVerifiedBlob {
block_root,
blob: kzg_verified_blob,
})
}
/// Returns the canonical root of the given `blob`.
///
/// Use this function to ensure that we report the blob hashing time Prometheus metric.
pub fn get_blob_root<E: EthSpec>(blob: &BlobSidecar<E>) -> Hash256 {
let blob_root_timer = metrics::start_timer(&metrics::BLOCK_PROCESSING_BLOB_ROOT);
let blob_root = blob.tree_hash_root();
metrics::stop_timer(blob_root_timer);
blob_root
}

View File

@@ -27,10 +27,12 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
let split_attestations = block
.body()
.attestations()
.iter()
.map(|att| {
let attesting_indices = get_attesting_indices_from_state(state, att)?;
Ok(SplitAttestation::new(att.clone(), attesting_indices))
Ok(SplitAttestation::new(
att.clone_as_attestation(),
attesting_indices,
))
})
.collect::<Result<Vec<_>, BeaconChainError>>()?;
@@ -86,8 +88,7 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
block
.body()
.attestations()
.iter()
.map(|a| a.data.clone())
.map(|a| a.data().clone())
.collect()
} else {
vec![]

View File

@@ -18,15 +18,35 @@ type BlockRoot = Hash256;
#[derive(Clone, Default)]
pub struct Timestamps {
pub observed: Option<Duration>,
pub all_blobs_observed: Option<Duration>,
pub execution_time: Option<Duration>,
pub attestable: Option<Duration>,
pub imported: Option<Duration>,
pub set_as_head: Option<Duration>,
}
// Helps arrange delay data so it is more relevant to metrics.
#[derive(Default)]
#[derive(Debug, Default)]
pub struct BlockDelays {
/// Time after start of slot we saw the block.
pub observed: Option<Duration>,
/// The time after the start of the slot we saw all blobs.
pub all_blobs_observed: Option<Duration>,
/// The time it took to get verification from the EL for the block.
pub execution_time: Option<Duration>,
/// The delay from the start of the slot before the block became available
///
/// Equal to max(`observed + execution_time`, `all_blobs_observed`).
pub available: Option<Duration>,
/// Time after `available`.
pub attestable: Option<Duration>,
/// Time
/// ALSO time after `available`.
///
/// We need to use `available` again rather than `attestable` to handle the case where the block
/// does not get added to the early-attester cache.
pub imported: Option<Duration>,
/// Time after `imported`.
pub set_as_head: Option<Duration>,
}
@@ -35,14 +55,34 @@ impl BlockDelays {
let observed = times
.observed
.and_then(|observed_time| observed_time.checked_sub(slot_start_time));
let all_blobs_observed = times
.all_blobs_observed
.and_then(|all_blobs_observed| all_blobs_observed.checked_sub(slot_start_time));
let execution_time = times
.execution_time
.and_then(|execution_time| execution_time.checked_sub(times.observed?));
// Duration since UNIX epoch at which block became available.
let available_time = times.execution_time.map(|execution_time| {
std::cmp::max(execution_time, times.all_blobs_observed.unwrap_or_default())
});
// Duration from the start of the slot until the block became available.
let available_delay =
available_time.and_then(|available_time| available_time.checked_sub(slot_start_time));
let attestable = times
.attestable
.and_then(|attestable_time| attestable_time.checked_sub(slot_start_time));
let imported = times
.imported
.and_then(|imported_time| imported_time.checked_sub(times.observed?));
.and_then(|imported_time| imported_time.checked_sub(available_time?));
let set_as_head = times
.set_as_head
.and_then(|set_as_head_time| set_as_head_time.checked_sub(times.imported?));
BlockDelays {
observed,
all_blobs_observed,
execution_time,
available: available_delay,
attestable,
imported,
set_as_head,
}
@@ -51,7 +91,7 @@ impl BlockDelays {
// If the block was received via gossip, we can record the client type of the peer which sent us
// the block.
#[derive(Clone, Default)]
#[derive(Debug, Clone, Default, PartialEq)]
pub struct BlockPeerInfo {
pub id: Option<String>,
pub client: Option<String>,
@@ -80,6 +120,8 @@ pub struct BlockTimesCache {
/// Helper methods to read from and write to the cache.
impl BlockTimesCache {
/// Set the observation time for `block_root` to `timestamp` if `timestamp` is less than
/// any previous timestamp at which this block was observed.
pub fn set_time_observed(
&mut self,
block_root: BlockRoot,
@@ -92,11 +134,66 @@ impl BlockTimesCache {
.cache
.entry(block_root)
.or_insert_with(|| BlockTimesCacheValue::new(slot));
block_times.timestamps.observed = Some(timestamp);
block_times.peer_info = BlockPeerInfo {
id: peer_id,
client: peer_client,
};
match block_times.timestamps.observed {
Some(existing_observation_time) if existing_observation_time <= timestamp => {
// Existing timestamp is earlier, do nothing.
}
_ => {
// No existing timestamp, or new timestamp is earlier.
block_times.timestamps.observed = Some(timestamp);
block_times.peer_info = BlockPeerInfo {
id: peer_id,
client: peer_client,
};
}
}
}
pub fn set_time_blob_observed(
&mut self,
block_root: BlockRoot,
slot: Slot,
timestamp: Duration,
) {
let block_times = self
.cache
.entry(block_root)
.or_insert_with(|| BlockTimesCacheValue::new(slot));
if block_times
.timestamps
.all_blobs_observed
.map_or(true, |prev| timestamp > prev)
{
block_times.timestamps.all_blobs_observed = Some(timestamp);
}
}
pub fn set_execution_time(&mut self, block_root: BlockRoot, slot: Slot, timestamp: Duration) {
let block_times = self
.cache
.entry(block_root)
.or_insert_with(|| BlockTimesCacheValue::new(slot));
if block_times
.timestamps
.execution_time
.map_or(true, |prev| timestamp < prev)
{
block_times.timestamps.execution_time = Some(timestamp);
}
}
pub fn set_time_attestable(&mut self, block_root: BlockRoot, slot: Slot, timestamp: Duration) {
let block_times = self
.cache
.entry(block_root)
.or_insert_with(|| BlockTimesCacheValue::new(slot));
if block_times
.timestamps
.attestable
.map_or(true, |prev| timestamp < prev)
{
block_times.timestamps.attestable = Some(timestamp);
}
}
pub fn set_time_imported(&mut self, block_root: BlockRoot, slot: Slot, timestamp: Duration) {
@@ -141,3 +238,71 @@ impl BlockTimesCache {
.retain(|_, cache| cache.slot > current_slot.saturating_sub(64_u64));
}
}
#[cfg(test)]
mod test {
use super::*;
#[test]
fn observed_time_uses_minimum() {
let mut cache = BlockTimesCache::default();
let block_root = Hash256::zero();
let slot = Slot::new(100);
let slot_start_time = Duration::from_secs(0);
let ts1 = Duration::from_secs(5);
let ts2 = Duration::from_secs(6);
let ts3 = Duration::from_secs(4);
let peer_info2 = BlockPeerInfo {
id: Some("peer2".to_string()),
client: Some("lighthouse".to_string()),
};
let peer_info3 = BlockPeerInfo {
id: Some("peer3".to_string()),
client: Some("prysm".to_string()),
};
cache.set_time_observed(block_root, slot, ts1, None, None);
assert_eq!(
cache.get_block_delays(block_root, slot_start_time).observed,
Some(ts1)
);
assert_eq!(cache.get_peer_info(block_root), BlockPeerInfo::default());
// Second observation with higher timestamp should not override anything, even though it has
// superior peer info.
cache.set_time_observed(
block_root,
slot,
ts2,
peer_info2.id.clone(),
peer_info2.client.clone(),
);
assert_eq!(
cache.get_block_delays(block_root, slot_start_time).observed,
Some(ts1)
);
assert_eq!(cache.get_peer_info(block_root), BlockPeerInfo::default());
// Third observation with lower timestamp should override everything.
cache.set_time_observed(
block_root,
slot,
ts3,
peer_info3.id.clone(),
peer_info3.client.clone(),
);
assert_eq!(
cache.get_block_delays(block_root, slot_start_time).observed,
Some(ts3)
);
assert_eq!(cache.get_peer_info(block_root), peer_info3);
}
}

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,572 @@
use crate::blob_verification::{GossipBlobError, GossipVerifiedBlobList};
use crate::block_verification::BlockError;
use crate::data_availability_checker::AvailabilityCheckError;
pub use crate::data_availability_checker::{AvailableBlock, MaybeAvailableBlock};
use crate::eth1_finalization_cache::Eth1FinalizationData;
use crate::{get_block_root, GossipVerifiedBlock, PayloadVerificationOutcome};
use derivative::Derivative;
use ssz_types::VariableList;
use state_processing::ConsensusContext;
use std::fmt::{Debug, Formatter};
use std::sync::Arc;
use types::blob_sidecar::{BlobIdentifier, BlobSidecarError, FixedBlobSidecarList};
use types::{
BeaconBlockRef, BeaconState, BlindedPayload, BlobSidecarList, Epoch, EthSpec, Hash256,
SignedBeaconBlock, SignedBeaconBlockHeader, Slot,
};
/// A block that has been received over RPC. It has 2 internal variants:
///
/// 1. `BlockAndBlobs`: A fully available post deneb block with all the blobs available. This variant
/// is only constructed after making consistency checks between blocks and blobs.
/// Hence, it is fully self contained w.r.t verification. i.e. this block has all the required
/// data to get verified and imported into fork choice.
///
/// 2. `Block`: This can be a fully available pre-deneb block **or** a post-deneb block that may or may
/// not require blobs to be considered fully available.
///
/// Note: We make a distinction over blocks received over gossip because
/// in a post-deneb world, the blobs corresponding to a given block that are received
/// over rpc do not contain the proposer signature for dos resistance.
#[derive(Clone, Derivative)]
#[derivative(Hash(bound = "E: EthSpec"))]
pub struct RpcBlock<E: EthSpec> {
block_root: Hash256,
block: RpcBlockInner<E>,
}
impl<E: EthSpec> Debug for RpcBlock<E> {
fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
write!(f, "RpcBlock({:?})", self.block_root)
}
}
impl<E: EthSpec> RpcBlock<E> {
pub fn block_root(&self) -> Hash256 {
self.block_root
}
pub fn as_block(&self) -> &SignedBeaconBlock<E> {
match &self.block {
RpcBlockInner::Block(block) => block,
RpcBlockInner::BlockAndBlobs(block, _) => block,
}
}
pub fn block_cloned(&self) -> Arc<SignedBeaconBlock<E>> {
match &self.block {
RpcBlockInner::Block(block) => block.clone(),
RpcBlockInner::BlockAndBlobs(block, _) => block.clone(),
}
}
pub fn blobs(&self) -> Option<&BlobSidecarList<E>> {
match &self.block {
RpcBlockInner::Block(_) => None,
RpcBlockInner::BlockAndBlobs(_, blobs) => Some(blobs),
}
}
}
/// Note: This variant is intentionally private because we want to safely construct the
/// internal variants after applying consistency checks to ensure that the block and blobs
/// are consistent with respect to each other.
#[derive(Debug, Clone, Derivative)]
#[derivative(Hash(bound = "E: EthSpec"))]
enum RpcBlockInner<E: EthSpec> {
/// Single block lookup response. This should potentially hit the data availability cache.
Block(Arc<SignedBeaconBlock<E>>),
/// This variant is used with parent lookups and by-range responses. It should have all blobs
/// ordered, all block roots matching, and the correct number of blobs for this block.
BlockAndBlobs(Arc<SignedBeaconBlock<E>>, BlobSidecarList<E>),
}
impl<E: EthSpec> RpcBlock<E> {
/// Constructs a `Block` variant.
pub fn new_without_blobs(
block_root: Option<Hash256>,
block: Arc<SignedBeaconBlock<E>>,
) -> Self {
let block_root = block_root.unwrap_or_else(|| get_block_root(&block));
Self {
block_root,
block: RpcBlockInner::Block(block),
}
}
/// Constructs a new `BlockAndBlobs` variant after making consistency
/// checks between the provided blocks and blobs. This struct makes no
/// guarantees about whether blobs should be present, only that they are
/// consistent with the block. An empty list passed in for `blobs` is
/// viewed the same as `None` passed in.
pub fn new(
block_root: Option<Hash256>,
block: Arc<SignedBeaconBlock<E>>,
blobs: Option<BlobSidecarList<E>>,
) -> Result<Self, AvailabilityCheckError> {
let block_root = block_root.unwrap_or_else(|| get_block_root(&block));
// Treat empty blob lists as if they are missing.
let blobs = blobs.filter(|b| !b.is_empty());
if let (Some(blobs), Ok(block_commitments)) = (
blobs.as_ref(),
block.message().body().blob_kzg_commitments(),
) {
if blobs.len() != block_commitments.len() {
return Err(AvailabilityCheckError::MissingBlobs);
}
for (blob, &block_commitment) in blobs.iter().zip(block_commitments.iter()) {
let blob_commitment = blob.kzg_commitment;
if blob_commitment != block_commitment {
return Err(AvailabilityCheckError::KzgCommitmentMismatch {
block_commitment,
blob_commitment,
});
}
}
}
let inner = match blobs {
Some(blobs) => RpcBlockInner::BlockAndBlobs(block, blobs),
None => RpcBlockInner::Block(block),
};
Ok(Self {
block_root,
block: inner,
})
}
pub fn new_from_fixed(
block_root: Hash256,
block: Arc<SignedBeaconBlock<E>>,
blobs: FixedBlobSidecarList<E>,
) -> Result<Self, AvailabilityCheckError> {
let filtered = blobs
.into_iter()
.filter_map(|b| b.clone())
.collect::<Vec<_>>();
let blobs = if filtered.is_empty() {
None
} else {
Some(VariableList::from(filtered))
};
Self::new(Some(block_root), block, blobs)
}
pub fn deconstruct(
self,
) -> (
Hash256,
Arc<SignedBeaconBlock<E>>,
Option<BlobSidecarList<E>>,
) {
let block_root = self.block_root();
match self.block {
RpcBlockInner::Block(block) => (block_root, block, None),
RpcBlockInner::BlockAndBlobs(block, blobs) => (block_root, block, Some(blobs)),
}
}
pub fn n_blobs(&self) -> usize {
match &self.block {
RpcBlockInner::Block(_) => 0,
RpcBlockInner::BlockAndBlobs(_, blobs) => blobs.len(),
}
}
}
/// A block that has gone through all pre-deneb block processing checks including block processing
/// and execution by an EL client. This block hasn't necessarily completed data availability checks.
///
///
/// It contains 2 variants:
/// 1. `Available`: This block has been executed and also contains all data to consider it a
/// fully available block. i.e. for post-deneb, this implies that this contains all the
/// required blobs.
/// 2. `AvailabilityPending`: This block hasn't received all required blobs to consider it a
/// fully available block.
pub enum ExecutedBlock<E: EthSpec> {
Available(AvailableExecutedBlock<E>),
AvailabilityPending(AvailabilityPendingExecutedBlock<E>),
}
impl<E: EthSpec> ExecutedBlock<E> {
pub fn new(
block: MaybeAvailableBlock<E>,
import_data: BlockImportData<E>,
payload_verification_outcome: PayloadVerificationOutcome,
) -> Self {
match block {
MaybeAvailableBlock::Available(available_block) => {
Self::Available(AvailableExecutedBlock::new(
available_block,
import_data,
payload_verification_outcome,
))
}
MaybeAvailableBlock::AvailabilityPending {
block_root: _,
block: pending_block,
} => Self::AvailabilityPending(AvailabilityPendingExecutedBlock::new(
pending_block,
import_data,
payload_verification_outcome,
)),
}
}
pub fn as_block(&self) -> &SignedBeaconBlock<E> {
match self {
Self::Available(available) => available.block.block(),
Self::AvailabilityPending(pending) => &pending.block,
}
}
pub fn block_root(&self) -> Hash256 {
match self {
ExecutedBlock::AvailabilityPending(pending) => pending.import_data.block_root,
ExecutedBlock::Available(available) => available.import_data.block_root,
}
}
}
/// A block that has completed all pre-deneb block processing checks including verification
/// by an EL client **and** has all requisite blob data to be imported into fork choice.
#[derive(PartialEq)]
pub struct AvailableExecutedBlock<E: EthSpec> {
pub block: AvailableBlock<E>,
pub import_data: BlockImportData<E>,
pub payload_verification_outcome: PayloadVerificationOutcome,
}
impl<E: EthSpec> AvailableExecutedBlock<E> {
pub fn new(
block: AvailableBlock<E>,
import_data: BlockImportData<E>,
payload_verification_outcome: PayloadVerificationOutcome,
) -> Self {
Self {
block,
import_data,
payload_verification_outcome,
}
}
pub fn get_all_blob_ids(&self) -> Vec<BlobIdentifier> {
let num_blobs_expected = self
.block
.message()
.body()
.blob_kzg_commitments()
.map_or(0, |commitments| commitments.len());
let mut blob_ids = Vec::with_capacity(num_blobs_expected);
for i in 0..num_blobs_expected {
blob_ids.push(BlobIdentifier {
block_root: self.import_data.block_root,
index: i as u64,
});
}
blob_ids
}
}
/// A block that has completed all pre-deneb block processing checks, verification
/// by an EL client but does not have all requisite blob data to get imported into
/// fork choice.
pub struct AvailabilityPendingExecutedBlock<E: EthSpec> {
pub block: Arc<SignedBeaconBlock<E>>,
pub import_data: BlockImportData<E>,
pub payload_verification_outcome: PayloadVerificationOutcome,
}
impl<E: EthSpec> AvailabilityPendingExecutedBlock<E> {
pub fn new(
block: Arc<SignedBeaconBlock<E>>,
import_data: BlockImportData<E>,
payload_verification_outcome: PayloadVerificationOutcome,
) -> Self {
Self {
block,
import_data,
payload_verification_outcome,
}
}
pub fn as_block(&self) -> &SignedBeaconBlock<E> {
&self.block
}
pub fn num_blobs_expected(&self) -> usize {
self.block
.message()
.body()
.blob_kzg_commitments()
.map_or(0, |commitments| commitments.len())
}
}
#[derive(Debug, PartialEq)]
pub struct BlockImportData<E: EthSpec> {
pub block_root: Hash256,
pub state: BeaconState<E>,
pub parent_block: SignedBeaconBlock<E, BlindedPayload<E>>,
pub parent_eth1_finalization_data: Eth1FinalizationData,
pub confirmed_state_roots: Vec<Hash256>,
pub consensus_context: ConsensusContext<E>,
}
impl<E: EthSpec> BlockImportData<E> {
pub fn __new_for_test(
block_root: Hash256,
state: BeaconState<E>,
parent_block: SignedBeaconBlock<E, BlindedPayload<E>>,
) -> Self {
Self {
block_root,
state,
parent_block,
parent_eth1_finalization_data: Eth1FinalizationData {
eth1_data: <_>::default(),
eth1_deposit_index: 0,
},
confirmed_state_roots: vec![],
consensus_context: ConsensusContext::new(Slot::new(0)),
}
}
}
pub type GossipVerifiedBlockContents<E> =
(GossipVerifiedBlock<E>, Option<GossipVerifiedBlobList<E>>);
#[derive(Debug)]
pub enum BlockContentsError<E: EthSpec> {
BlockError(BlockError<E>),
BlobError(GossipBlobError<E>),
SidecarError(BlobSidecarError),
}
impl<E: EthSpec> From<BlockError<E>> for BlockContentsError<E> {
fn from(value: BlockError<E>) -> Self {
Self::BlockError(value)
}
}
impl<E: EthSpec> From<GossipBlobError<E>> for BlockContentsError<E> {
fn from(value: GossipBlobError<E>) -> Self {
Self::BlobError(value)
}
}
impl<E: EthSpec> std::fmt::Display for BlockContentsError<E> {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self {
BlockContentsError::BlockError(err) => {
write!(f, "BlockError({})", err)
}
BlockContentsError::BlobError(err) => {
write!(f, "BlobError({})", err)
}
BlockContentsError::SidecarError(err) => {
write!(f, "SidecarError({:?})", err)
}
}
}
}
/// Trait for common block operations.
pub trait AsBlock<E: EthSpec> {
fn slot(&self) -> Slot;
fn epoch(&self) -> Epoch;
fn parent_root(&self) -> Hash256;
fn state_root(&self) -> Hash256;
fn signed_block_header(&self) -> SignedBeaconBlockHeader;
fn message(&self) -> BeaconBlockRef<E>;
fn as_block(&self) -> &SignedBeaconBlock<E>;
fn block_cloned(&self) -> Arc<SignedBeaconBlock<E>>;
fn canonical_root(&self) -> Hash256;
fn into_rpc_block(self) -> RpcBlock<E>;
}
impl<E: EthSpec> AsBlock<E> for Arc<SignedBeaconBlock<E>> {
fn slot(&self) -> Slot {
SignedBeaconBlock::slot(self)
}
fn epoch(&self) -> Epoch {
SignedBeaconBlock::epoch(self)
}
fn parent_root(&self) -> Hash256 {
SignedBeaconBlock::parent_root(self)
}
fn state_root(&self) -> Hash256 {
SignedBeaconBlock::state_root(self)
}
fn signed_block_header(&self) -> SignedBeaconBlockHeader {
SignedBeaconBlock::signed_block_header(self)
}
fn message(&self) -> BeaconBlockRef<E> {
SignedBeaconBlock::message(self)
}
fn as_block(&self) -> &SignedBeaconBlock<E> {
self
}
fn block_cloned(&self) -> Arc<SignedBeaconBlock<E>> {
Arc::<SignedBeaconBlock<E>>::clone(self)
}
fn canonical_root(&self) -> Hash256 {
SignedBeaconBlock::canonical_root(self)
}
fn into_rpc_block(self) -> RpcBlock<E> {
RpcBlock::new_without_blobs(None, self)
}
}
impl<E: EthSpec> AsBlock<E> for MaybeAvailableBlock<E> {
fn slot(&self) -> Slot {
self.as_block().slot()
}
fn epoch(&self) -> Epoch {
self.as_block().epoch()
}
fn parent_root(&self) -> Hash256 {
self.as_block().parent_root()
}
fn state_root(&self) -> Hash256 {
self.as_block().state_root()
}
fn signed_block_header(&self) -> SignedBeaconBlockHeader {
self.as_block().signed_block_header()
}
fn message(&self) -> BeaconBlockRef<E> {
self.as_block().message()
}
fn as_block(&self) -> &SignedBeaconBlock<E> {
match &self {
MaybeAvailableBlock::Available(block) => block.as_block(),
MaybeAvailableBlock::AvailabilityPending {
block_root: _,
block,
} => block,
}
}
fn block_cloned(&self) -> Arc<SignedBeaconBlock<E>> {
match &self {
MaybeAvailableBlock::Available(block) => block.block_cloned(),
MaybeAvailableBlock::AvailabilityPending {
block_root: _,
block,
} => block.clone(),
}
}
fn canonical_root(&self) -> Hash256 {
self.as_block().canonical_root()
}
fn into_rpc_block(self) -> RpcBlock<E> {
match self {
MaybeAvailableBlock::Available(available_block) => available_block.into_rpc_block(),
MaybeAvailableBlock::AvailabilityPending { block_root, block } => {
RpcBlock::new_without_blobs(Some(block_root), block)
}
}
}
}
impl<E: EthSpec> AsBlock<E> for AvailableBlock<E> {
fn slot(&self) -> Slot {
self.block().slot()
}
fn epoch(&self) -> Epoch {
self.block().epoch()
}
fn parent_root(&self) -> Hash256 {
self.block().parent_root()
}
fn state_root(&self) -> Hash256 {
self.block().state_root()
}
fn signed_block_header(&self) -> SignedBeaconBlockHeader {
self.block().signed_block_header()
}
fn message(&self) -> BeaconBlockRef<E> {
self.block().message()
}
fn as_block(&self) -> &SignedBeaconBlock<E> {
self.block()
}
fn block_cloned(&self) -> Arc<SignedBeaconBlock<E>> {
AvailableBlock::block_cloned(self)
}
fn canonical_root(&self) -> Hash256 {
self.block().canonical_root()
}
fn into_rpc_block(self) -> RpcBlock<E> {
let (block_root, block, blobs_opt) = self.deconstruct();
// Circumvent the constructor here, because an Available block will have already had
// consistency checks performed.
let inner = match blobs_opt {
None => RpcBlockInner::Block(block),
Some(blobs) => RpcBlockInner::BlockAndBlobs(block, blobs),
};
RpcBlock {
block_root,
block: inner,
}
}
}
impl<E: EthSpec> AsBlock<E> for RpcBlock<E> {
fn slot(&self) -> Slot {
self.as_block().slot()
}
fn epoch(&self) -> Epoch {
self.as_block().epoch()
}
fn parent_root(&self) -> Hash256 {
self.as_block().parent_root()
}
fn state_root(&self) -> Hash256 {
self.as_block().state_root()
}
fn signed_block_header(&self) -> SignedBeaconBlockHeader {
self.as_block().signed_block_header()
}
fn message(&self) -> BeaconBlockRef<E> {
self.as_block().message()
}
fn as_block(&self) -> &SignedBeaconBlock<E> {
match &self.block {
RpcBlockInner::Block(block) => block,
RpcBlockInner::BlockAndBlobs(block, _) => block,
}
}
fn block_cloned(&self) -> Arc<SignedBeaconBlock<E>> {
match &self.block {
RpcBlockInner::Block(block) => block.clone(),
RpcBlockInner::BlockAndBlobs(block, _) => block.clone(),
}
}
fn canonical_root(&self) -> Hash256 {
self.as_block().canonical_root()
}
fn into_rpc_block(self) -> RpcBlock<E> {
self
}
}

View File

@@ -1,15 +1,20 @@
use crate::beacon_chain::{CanonicalHead, BEACON_CHAIN_DB_KEY, ETH1_CACHE_DB_KEY, OP_POOL_DB_KEY};
use crate::beacon_chain::{
CanonicalHead, LightClientProducerEvent, BEACON_CHAIN_DB_KEY, ETH1_CACHE_DB_KEY, OP_POOL_DB_KEY,
};
use crate::beacon_proposer_cache::BeaconProposerCache;
use crate::data_availability_checker::DataAvailabilityChecker;
use crate::eth1_chain::{CachingEth1Backend, SszEth1};
use crate::eth1_finalization_cache::Eth1FinalizationCache;
use crate::fork_choice_signal::ForkChoiceSignalTx;
use crate::fork_revert::{reset_fork_choice_to_finalization, revert_to_fork_boundary};
use crate::graffiti_calculator::{GraffitiCalculator, GraffitiOrigin};
use crate::head_tracker::HeadTracker;
use crate::light_client_server_cache::LightClientServerCache;
use crate::migrate::{BackgroundMigrator, MigratorConfig};
use crate::persisted_beacon_chain::PersistedBeaconChain;
use crate::shuffling_cache::{BlockShufflingIds, ShufflingCache};
use crate::snapshot_cache::{SnapshotCache, DEFAULT_SNAPSHOT_CACHE_SIZE};
use crate::timeout_rw_lock::TimeoutRwLock;
use crate::validator_monitor::ValidatorMonitor;
use crate::validator_monitor::{ValidatorMonitor, ValidatorMonitorConfig};
use crate::validator_pubkey_cache::ValidatorPubkeyCache;
use crate::ChainConfig;
use crate::{
@@ -20,43 +25,44 @@ use eth1::Config as Eth1Config;
use execution_layer::ExecutionLayer;
use fork_choice::{ForkChoice, ResetPayloadStatuses};
use futures::channel::mpsc::Sender;
use kzg::Kzg;
use operation_pool::{OperationPool, PersistedOperationPool};
use parking_lot::RwLock;
use parking_lot::{Mutex, RwLock};
use proto_array::{DisallowedReOrgOffsets, ReOrgThreshold};
use slasher::Slasher;
use slog::{crit, debug, error, info, Logger};
use slog::{crit, debug, error, info, o, Logger};
use slot_clock::{SlotClock, TestingSlotClock};
use state_processing::per_slot_processing;
use state_processing::{per_slot_processing, AllCaches};
use std::marker::PhantomData;
use std::sync::Arc;
use std::time::Duration;
use store::{Error as StoreError, HotColdDB, ItemStore, KeyValueStoreOp};
use task_executor::{ShutdownReason, TaskExecutor};
use types::{
BeaconBlock, BeaconState, ChainSpec, Checkpoint, Epoch, EthSpec, Graffiti, Hash256,
PublicKeyBytes, Signature, SignedBeaconBlock, Slot,
BeaconBlock, BeaconState, BlobSidecarList, ChainSpec, Checkpoint, Epoch, EthSpec, Hash256,
Signature, SignedBeaconBlock, Slot,
};
/// An empty struct used to "witness" all the `BeaconChainTypes` traits. It has no user-facing
/// functionality and only exists to satisfy the type system.
pub struct Witness<TSlotClock, TEth1Backend, TEthSpec, THotStore, TColdStore>(
PhantomData<(TSlotClock, TEth1Backend, TEthSpec, THotStore, TColdStore)>,
pub struct Witness<TSlotClock, TEth1Backend, E, THotStore, TColdStore>(
PhantomData<(TSlotClock, TEth1Backend, E, THotStore, TColdStore)>,
);
impl<TSlotClock, TEth1Backend, TEthSpec, THotStore, TColdStore> BeaconChainTypes
for Witness<TSlotClock, TEth1Backend, TEthSpec, THotStore, TColdStore>
impl<TSlotClock, TEth1Backend, E, THotStore, TColdStore> BeaconChainTypes
for Witness<TSlotClock, TEth1Backend, E, THotStore, TColdStore>
where
THotStore: ItemStore<TEthSpec> + 'static,
TColdStore: ItemStore<TEthSpec> + 'static,
THotStore: ItemStore<E> + 'static,
TColdStore: ItemStore<E> + 'static,
TSlotClock: SlotClock + 'static,
TEth1Backend: Eth1ChainBackend<TEthSpec> + 'static,
TEthSpec: EthSpec + 'static,
TEth1Backend: Eth1ChainBackend<E> + 'static,
E: EthSpec + 'static,
{
type HotStore = THotStore;
type ColdStore = TColdStore;
type SlotClock = TSlotClock;
type Eth1Chain = TEth1Backend;
type EthSpec = TEthSpec;
type EthSpec = E;
}
/// Builds a `BeaconChain` by either creating anew from genesis, or, resuming from an existing chain
@@ -84,34 +90,36 @@ pub struct BeaconChainBuilder<T: BeaconChainTypes> {
event_handler: Option<ServerSentEventHandler<T::EthSpec>>,
slot_clock: Option<T::SlotClock>,
shutdown_sender: Option<Sender<ShutdownReason>>,
light_client_server_tx: Option<Sender<LightClientProducerEvent<T::EthSpec>>>,
head_tracker: Option<HeadTracker>,
validator_pubkey_cache: Option<ValidatorPubkeyCache<T>>,
spec: ChainSpec,
chain_config: ChainConfig,
log: Option<Logger>,
graffiti: Graffiti,
beacon_graffiti: GraffitiOrigin,
slasher: Option<Arc<Slasher<T::EthSpec>>>,
validator_monitor: Option<ValidatorMonitor<T::EthSpec>>,
// Pending I/O batch that is constructed during building and should be executed atomically
// alongside `PersistedBeaconChain` storage when `BeaconChainBuilder::build` is called.
pending_io_batch: Vec<KeyValueStoreOp>,
kzg: Option<Arc<Kzg>>,
task_executor: Option<TaskExecutor>,
validator_monitor_config: Option<ValidatorMonitorConfig>,
}
impl<TSlotClock, TEth1Backend, TEthSpec, THotStore, TColdStore>
BeaconChainBuilder<Witness<TSlotClock, TEth1Backend, TEthSpec, THotStore, TColdStore>>
impl<TSlotClock, TEth1Backend, E, THotStore, TColdStore>
BeaconChainBuilder<Witness<TSlotClock, TEth1Backend, E, THotStore, TColdStore>>
where
THotStore: ItemStore<TEthSpec> + 'static,
TColdStore: ItemStore<TEthSpec> + 'static,
THotStore: ItemStore<E> + 'static,
TColdStore: ItemStore<E> + 'static,
TSlotClock: SlotClock + 'static,
TEth1Backend: Eth1ChainBackend<TEthSpec> + 'static,
TEthSpec: EthSpec + 'static,
TEth1Backend: Eth1ChainBackend<E> + 'static,
E: EthSpec + 'static,
{
/// Returns a new builder.
///
/// The `_eth_spec_instance` parameter is only supplied to make concrete the `TEthSpec` trait.
/// The `_eth_spec_instance` parameter is only supplied to make concrete the `E` trait.
/// This should generally be either the `MinimalEthSpec` or `MainnetEthSpec` types.
pub fn new(_eth_spec_instance: TEthSpec) -> Self {
pub fn new(_eth_spec_instance: E) -> Self {
Self {
store: None,
store_migrator_config: None,
@@ -125,20 +133,22 @@ where
event_handler: None,
slot_clock: None,
shutdown_sender: None,
light_client_server_tx: None,
head_tracker: None,
validator_pubkey_cache: None,
spec: TEthSpec::default_spec(),
spec: E::default_spec(),
chain_config: ChainConfig::default(),
log: None,
graffiti: Graffiti::default(),
beacon_graffiti: GraffitiOrigin::default(),
slasher: None,
validator_monitor: None,
pending_io_batch: vec![],
kzg: None,
task_executor: None,
validator_monitor_config: None,
}
}
/// Override the default spec (as defined by `TEthSpec`).
/// Override the default spec (as defined by `E`).
///
/// This method should generally be called immediately after `Self::new` to ensure components
/// are started with a consistent spec.
@@ -162,8 +172,8 @@ where
}
/// Sets the proposer re-org threshold.
pub fn proposer_re_org_threshold(mut self, threshold: Option<ReOrgThreshold>) -> Self {
self.chain_config.re_org_threshold = threshold;
pub fn proposer_re_org_head_threshold(mut self, threshold: Option<ReOrgThreshold>) -> Self {
self.chain_config.re_org_head_threshold = threshold;
self
}
@@ -188,7 +198,7 @@ where
/// Sets the store (database).
///
/// Should generally be called early in the build chain.
pub fn store(mut self, store: Arc<HotColdDB<TEthSpec, THotStore, TColdStore>>) -> Self {
pub fn store(mut self, store: Arc<HotColdDB<E, THotStore, TColdStore>>) -> Self {
self.store = Some(store);
self
}
@@ -200,7 +210,7 @@ where
}
/// Sets the slasher.
pub fn slasher(mut self, slasher: Arc<Slasher<TEthSpec>>) -> Self {
pub fn slasher(mut self, slasher: Arc<Slasher<E>>) -> Self {
self.slasher = Some(slasher);
self
}
@@ -294,7 +304,7 @@ where
self.op_pool = Some(
store
.get_item::<PersistedOperationPool<TEthSpec>>(&OP_POOL_DB_KEY)
.get_item::<PersistedOperationPool<E>>(&OP_POOL_DB_KEY)
.map_err(|e| format!("DB error whilst reading persisted op pool: {:?}", e))?
.map(PersistedOperationPool::into_operation_pool)
.transpose()
@@ -329,8 +339,8 @@ where
/// Return the `BeaconSnapshot` representing genesis as well as the mutated builder.
fn set_genesis_state(
mut self,
mut beacon_state: BeaconState<TEthSpec>,
) -> Result<(BeaconSnapshot<TEthSpec>, Self), String> {
mut beacon_state: BeaconState<E>,
) -> Result<(BeaconSnapshot<E>, Self), String> {
let store = self
.store
.clone()
@@ -377,7 +387,7 @@ where
}
/// Starts a new chain from a genesis state.
pub fn genesis_state(mut self, beacon_state: BeaconState<TEthSpec>) -> Result<Self, String> {
pub fn genesis_state(mut self, beacon_state: BeaconState<E>) -> Result<Self, String> {
let store = self.store.clone().ok_or("genesis_state requires a store")?;
let (genesis, updated_builder) = self.set_genesis_state(beacon_state)?;
@@ -392,6 +402,11 @@ where
.init_anchor_info(genesis.beacon_block.message(), retain_historic_states)
.map_err(|e| format!("Failed to initialize genesis anchor: {:?}", e))?,
);
self.pending_io_batch.push(
store
.init_blob_info(genesis.beacon_block.slot())
.map_err(|e| format!("Failed to initialize genesis blob info: {:?}", e))?,
);
let fc_store = BeaconForkChoiceStore::get_forkchoice_store(store, &genesis)
.map_err(|e| format!("Unable to initialize fork choice store: {e:?}"))?;
@@ -415,9 +430,10 @@ where
/// Start the chain from a weak subjectivity state.
pub fn weak_subjectivity_state(
mut self,
mut weak_subj_state: BeaconState<TEthSpec>,
weak_subj_block: SignedBeaconBlock<TEthSpec>,
genesis_state: BeaconState<TEthSpec>,
mut weak_subj_state: BeaconState<E>,
weak_subj_block: SignedBeaconBlock<E>,
weak_subj_blobs: Option<BlobSidecarList<E>>,
genesis_state: BeaconState<E>,
) -> Result<Self, String> {
let store = self
.store
@@ -429,7 +445,7 @@ where
.ok_or("weak_subjectivity_state requires a log")?;
// Ensure the state is advanced to an epoch boundary.
let slots_per_epoch = TEthSpec::slots_per_epoch();
let slots_per_epoch = E::slots_per_epoch();
if weak_subj_state.slot() % slots_per_epoch != 0 {
debug!(
log,
@@ -446,7 +462,7 @@ where
// Prime all caches before storing the state in the database and computing the tree hash
// root.
weak_subj_state
.build_caches(&self.spec)
.build_all_caches(&self.spec)
.map_err(|e| format!("Error building caches on checkpoint state: {e:?}"))?;
let weak_subj_state_root = weak_subj_state
.update_tree_hash_cache()
@@ -475,6 +491,29 @@ where
));
}
// Verify that blobs (if provided) match the block.
if let Some(blobs) = &weak_subj_blobs {
let commitments = weak_subj_block
.message()
.body()
.blob_kzg_commitments()
.map_err(|e| format!("Blobs provided but block does not reference them: {e:?}"))?;
if blobs.len() != commitments.len() {
return Err(format!(
"Wrong number of blobs, expected: {}, got: {}",
commitments.len(),
blobs.len()
));
}
if commitments
.iter()
.zip(blobs.iter())
.any(|(commitment, blob)| *commitment != blob.kzg_commitment)
{
return Err("Checkpoint blob does not match block commitment".into());
}
}
// Set the store's split point *before* storing genesis so that genesis is stored
// immediately in the freezer DB.
store.set_split(weak_subj_slot, weak_subj_state_root, weak_subj_block_root);
@@ -496,14 +535,26 @@ where
.do_atomically(block_root_batch)
.map_err(|e| format!("Error writing frozen block roots: {e:?}"))?;
// Write the state and block non-atomically, it doesn't matter if they're forgotten
// Write the state, block and blobs non-atomically, it doesn't matter if they're forgotten
// about on a crash restart.
store
.update_finalized_state(
weak_subj_state_root,
weak_subj_block_root,
weak_subj_state.clone(),
)
.map_err(|e| format!("Failed to set checkpoint state as finalized state: {:?}", e))?;
store
.put_state(&weak_subj_state_root, &weak_subj_state)
.map_err(|e| format!("Failed to store weak subjectivity state: {:?}", e))?;
.map_err(|e| format!("Failed to store weak subjectivity state: {e:?}"))?;
store
.put_block(&weak_subj_block_root, weak_subj_block.clone())
.map_err(|e| format!("Failed to store weak subjectivity block: {:?}", e))?;
.map_err(|e| format!("Failed to store weak subjectivity block: {e:?}"))?;
if let Some(blobs) = weak_subj_blobs {
store
.put_blobs(&weak_subj_block_root, blobs)
.map_err(|e| format!("Failed to store weak subjectivity blobs: {e:?}"))?;
}
// Stage the database's metadata fields for atomic storage when `build` is called.
// This prevents the database from restarting in an inconsistent state if the anchor
@@ -515,12 +566,17 @@ where
.init_anchor_info(weak_subj_block.message(), retain_historic_states)
.map_err(|e| format!("Failed to initialize anchor info: {:?}", e))?,
);
self.pending_io_batch.push(
store
.init_blob_info(weak_subj_block.slot())
.map_err(|e| format!("Failed to initialize blob info: {:?}", e))?,
);
// Store pruning checkpoint to prevent attempting to prune before the anchor state.
self.pending_io_batch
.push(store.pruning_checkpoint_store_op(Checkpoint {
root: weak_subj_block_root,
epoch: weak_subj_state.slot().epoch(TEthSpec::slots_per_epoch()),
epoch: weak_subj_state.slot().epoch(E::slots_per_epoch()),
}));
let snapshot = BeaconSnapshot {
@@ -554,7 +610,7 @@ where
}
/// Sets the `BeaconChain` execution layer.
pub fn execution_layer(mut self, execution_layer: Option<ExecutionLayer<TEthSpec>>) -> Self {
pub fn execution_layer(mut self, execution_layer: Option<ExecutionLayer<E>>) -> Self {
self.execution_layer = execution_layer;
self
}
@@ -562,7 +618,7 @@ where
/// Sets the `BeaconChain` event handler backend.
///
/// For example, provide `ServerSentEventHandler` as a `handler`.
pub fn event_handler(mut self, handler: Option<ServerSentEventHandler<TEthSpec>>) -> Self {
pub fn event_handler(mut self, handler: Option<ServerSentEventHandler<E>>) -> Self {
self.event_handler = handler;
self
}
@@ -588,15 +644,21 @@ where
self
}
/// Sets a `Sender` to allow the beacon chain to trigger light_client update production.
pub fn light_client_server_tx(mut self, sender: Sender<LightClientProducerEvent<E>>) -> Self {
self.light_client_server_tx = Some(sender);
self
}
/// Creates a new, empty operation pool.
fn empty_op_pool(mut self) -> Self {
self.op_pool = Some(OperationPool::new());
self
}
/// Sets the `graffiti` field.
pub fn graffiti(mut self, graffiti: Graffiti) -> Self {
self.graffiti = graffiti;
/// Sets the `beacon_graffiti` field.
pub fn beacon_graffiti(mut self, beacon_graffiti: GraffitiOrigin) -> Self {
self.beacon_graffiti = beacon_graffiti;
self
}
@@ -609,19 +671,13 @@ where
/// Register some validators for additional monitoring.
///
/// `validators` is a comma-separated string of 0x-formatted BLS pubkeys.
pub fn monitor_validators(
mut self,
auto_register: bool,
validators: Vec<PublicKeyBytes>,
individual_metrics_threshold: usize,
log: Logger,
) -> Self {
self.validator_monitor = Some(ValidatorMonitor::new(
validators,
auto_register,
individual_metrics_threshold,
log.clone(),
));
pub fn validator_monitor_config(mut self, config: ValidatorMonitorConfig) -> Self {
self.validator_monitor_config = Some(config);
self
}
pub fn kzg(mut self, kzg: Option<Arc<Kzg>>) -> Self {
self.kzg = kzg;
self
}
@@ -634,10 +690,8 @@ where
#[allow(clippy::type_complexity)] // I think there's nothing to be gained here from a type alias.
pub fn build(
mut self,
) -> Result<
BeaconChain<Witness<TSlotClock, TEth1Backend, TEthSpec, THotStore, TColdStore>>,
String,
> {
) -> Result<BeaconChain<Witness<TSlotClock, TEth1Backend, E, THotStore, TColdStore>>, String>
{
let log = self.log.ok_or("Cannot build without a logger")?;
let slot_clock = self
.slot_clock
@@ -652,10 +706,15 @@ where
let genesis_state_root = self
.genesis_state_root
.ok_or("Cannot build without a genesis state root")?;
let mut validator_monitor = self
.validator_monitor
.ok_or("Cannot build without a validator monitor")?;
let validator_monitor_config = self.validator_monitor_config.unwrap_or_default();
let head_tracker = Arc::new(self.head_tracker.unwrap_or_default());
let beacon_proposer_cache: Arc<Mutex<BeaconProposerCache>> = <_>::default();
let mut validator_monitor = ValidatorMonitor::new(
validator_monitor_config,
beacon_proposer_cache.clone(),
log.new(o!("service" => "val_mon")),
);
let current_slot = if slot_clock
.is_prior_to_genesis()
@@ -714,8 +773,6 @@ where
store.clone(),
Some(current_slot),
&self.spec,
self.chain_config.progressive_balances_mode,
&log,
)?;
}
@@ -761,8 +818,9 @@ where
if let Some(slot) = slot_clock.now() {
validator_monitor.process_valid_state(
slot.epoch(TEthSpec::slots_per_epoch()),
slot.epoch(E::slots_per_epoch()),
&head_snapshot.beacon_state,
&self.spec,
);
}
@@ -781,13 +839,14 @@ where
//
// This *must* be stored before constructing the `BeaconChain`, so that its `Drop` instance
// doesn't write a `PersistedBeaconChain` without the rest of the batch.
let head_tracker_reader = head_tracker.0.read();
self.pending_io_batch.push(BeaconChain::<
Witness<TSlotClock, TEth1Backend, TEthSpec, THotStore, TColdStore>,
Witness<TSlotClock, TEth1Backend, E, THotStore, TColdStore>,
>::persist_head_in_batch_standalone(
genesis_block_root, &head_tracker
genesis_block_root, &head_tracker_reader
));
self.pending_io_batch.push(BeaconChain::<
Witness<TSlotClock, TEth1Backend, TEthSpec, THotStore, TColdStore>,
Witness<TSlotClock, TEth1Backend, E, THotStore, TColdStore>,
>::persist_fork_choice_in_batch_standalone(
&fork_choice
));
@@ -795,10 +854,10 @@ where
.hot_db
.do_atomically(self.pending_io_batch)
.map_err(|e| format!("Error writing chain & metadata to disk: {:?}", e))?;
drop(head_tracker_reader);
let genesis_validators_root = head_snapshot.beacon_state.genesis_validators_root();
let genesis_time = head_snapshot.beacon_state.genesis_time();
let head_for_snapshot_cache = head_snapshot.clone();
let canonical_head = CanonicalHead::new(fork_choice, Arc::new(head_snapshot));
let shuffling_cache_size = self.chain_config.shuffling_cache_size;
@@ -806,16 +865,20 @@ where
let genesis_backfill_slot = if self.chain_config.genesis_backfill {
Slot::new(0)
} else {
let backfill_epoch_range = (self.spec.min_validator_withdrawability_delay
+ self.spec.churn_limit_quotient)
.as_u64()
/ 2;
let backfill_epoch_range = if cfg!(feature = "test_backfill") {
3
} else {
(self.spec.min_validator_withdrawability_delay + self.spec.churn_limit_quotient)
.as_u64()
/ 2
};
match slot_clock.now() {
Some(current_slot) => {
let genesis_backfill_epoch = current_slot
.epoch(TEthSpec::slots_per_epoch())
.epoch(E::slots_per_epoch())
.saturating_sub(backfill_epoch_range);
genesis_backfill_epoch.start_slot(TEthSpec::slots_per_epoch())
genesis_backfill_epoch.start_slot(E::slots_per_epoch())
}
None => {
// The slot clock cannot derive the current slot. We therefore assume we are
@@ -826,14 +889,14 @@ where
};
let beacon_chain = BeaconChain {
spec: self.spec,
spec: self.spec.clone(),
config: self.chain_config,
store,
store: store.clone(),
task_executor: self
.task_executor
.ok_or("Cannot build without task executor")?,
store_migrator,
slot_clock,
slot_clock: slot_clock.clone(),
op_pool: self.op_pool.ok_or("Cannot build without op pool")?,
// TODO: allow for persisting and loading the pool from disk.
naive_aggregation_pool: <_>::default(),
@@ -855,14 +918,14 @@ where
observed_sync_aggregators: <_>::default(),
// TODO: allow for persisting and loading the pool from disk.
observed_block_producers: <_>::default(),
observed_blob_sidecars: <_>::default(),
observed_slashable: <_>::default(),
observed_voluntary_exits: <_>::default(),
observed_proposer_slashings: <_>::default(),
observed_attester_slashings: <_>::default(),
observed_bls_to_execution_changes: <_>::default(),
latest_seen_finality_update: <_>::default(),
latest_seen_optimistic_update: <_>::default(),
eth1_chain: self.eth1_chain,
execution_layer: self.execution_layer,
execution_layer: self.execution_layer.clone(),
genesis_validators_root,
genesis_time,
canonical_head,
@@ -872,30 +935,39 @@ where
fork_choice_signal_rx,
event_handler: self.event_handler,
head_tracker,
snapshot_cache: TimeoutRwLock::new(SnapshotCache::new(
DEFAULT_SNAPSHOT_CACHE_SIZE,
head_for_snapshot_cache,
)),
shuffling_cache: TimeoutRwLock::new(ShufflingCache::new(
shuffling_cache_size,
head_shuffling_ids,
log.clone(),
)),
eth1_finalization_cache: TimeoutRwLock::new(Eth1FinalizationCache::new(log.clone())),
beacon_proposer_cache: <_>::default(),
beacon_proposer_cache,
block_times_cache: <_>::default(),
pre_finalization_block_cache: <_>::default(),
validator_pubkey_cache: TimeoutRwLock::new(validator_pubkey_cache),
attester_cache: <_>::default(),
early_attester_cache: <_>::default(),
reqresp_pre_import_cache: <_>::default(),
light_client_server_cache: LightClientServerCache::new(),
light_client_server_tx: self.light_client_server_tx,
shutdown_sender: self
.shutdown_sender
.ok_or("Cannot build without a shutdown sender.")?,
log: log.clone(),
graffiti: self.graffiti,
graffiti_calculator: GraffitiCalculator::new(
self.beacon_graffiti,
self.execution_layer,
slot_clock.slot_duration() * E::slots_per_epoch() as u32,
log.clone(),
),
slasher: self.slasher.clone(),
validator_monitor: RwLock::new(validator_monitor),
genesis_backfill_slot,
data_availability_checker: Arc::new(
DataAvailabilityChecker::new(slot_clock, self.kzg.clone(), store, &log, self.spec)
.map_err(|e| format!("Error initializing DataAvailabiltyChecker: {:?}", e))?,
),
kzg: self.kzg.clone(),
};
let head = beacon_chain.head_snapshot();
@@ -958,19 +1030,24 @@ where
);
}
// Prune blobs older than the blob data availability boundary in the background.
if let Some(data_availability_boundary) = beacon_chain.data_availability_boundary() {
beacon_chain
.store_migrator
.process_prune_blobs(data_availability_boundary);
}
Ok(beacon_chain)
}
}
impl<TSlotClock, TEthSpec, THotStore, TColdStore>
BeaconChainBuilder<
Witness<TSlotClock, CachingEth1Backend<TEthSpec>, TEthSpec, THotStore, TColdStore>,
>
impl<TSlotClock, E, THotStore, TColdStore>
BeaconChainBuilder<Witness<TSlotClock, CachingEth1Backend<E>, E, THotStore, TColdStore>>
where
THotStore: ItemStore<TEthSpec> + 'static,
TColdStore: ItemStore<TEthSpec> + 'static,
THotStore: ItemStore<E> + 'static,
TColdStore: ItemStore<E> + 'static,
TSlotClock: SlotClock + 'static,
TEthSpec: EthSpec + 'static,
E: EthSpec + 'static,
{
/// Do not use any eth1 backend. The client will not be able to produce beacon blocks.
pub fn no_eth1_backend(self) -> Self {
@@ -993,13 +1070,13 @@ where
}
}
impl<TEth1Backend, TEthSpec, THotStore, TColdStore>
BeaconChainBuilder<Witness<TestingSlotClock, TEth1Backend, TEthSpec, THotStore, TColdStore>>
impl<TEth1Backend, E, THotStore, TColdStore>
BeaconChainBuilder<Witness<TestingSlotClock, TEth1Backend, E, THotStore, TColdStore>>
where
THotStore: ItemStore<TEthSpec> + 'static,
TColdStore: ItemStore<TEthSpec> + 'static,
TEth1Backend: Eth1ChainBackend<TEthSpec> + 'static,
TEthSpec: EthSpec + 'static,
THotStore: ItemStore<E> + 'static,
TColdStore: ItemStore<E> + 'static,
TEth1Backend: Eth1ChainBackend<E> + 'static,
E: EthSpec + 'static,
{
/// Sets the `BeaconChain` slot clock to `TestingSlotClock`.
///
@@ -1019,10 +1096,10 @@ where
}
}
fn genesis_block<T: EthSpec>(
genesis_state: &mut BeaconState<T>,
fn genesis_block<E: EthSpec>(
genesis_state: &mut BeaconState<E>,
spec: &ChainSpec,
) -> Result<SignedBeaconBlock<T>, String> {
) -> Result<SignedBeaconBlock<E>, String> {
let mut genesis_block = BeaconBlock::empty(spec);
*genesis_block.state_root_mut() = genesis_state
.update_tree_hash_cache()
@@ -1055,7 +1132,7 @@ fn descriptive_db_error(item: &str, error: &StoreError) -> String {
#[cfg(test)]
mod test {
use super::*;
use crate::validator_monitor::DEFAULT_INDIVIDUAL_TRACKING_THRESHOLD;
use crate::test_utils::EphemeralHarnessType;
use ethereum_hashing::hash;
use genesis::{
generate_deterministic_keypairs, interop_genesis_state, DEFAULT_ETH1_BLOCK_HASH,
@@ -1069,6 +1146,7 @@ mod test {
use types::{EthSpec, MinimalEthSpec, Slot};
type TestEthSpec = MinimalEthSpec;
type Builder = BeaconChainBuilder<EphemeralHarnessType<TestEthSpec>>;
fn get_logger() -> Logger {
let builder = NullLoggerBuilder;
@@ -1101,7 +1179,7 @@ mod test {
let (shutdown_tx, _) = futures::channel::mpsc::channel(1);
let runtime = TestRuntime::default();
let chain = BeaconChainBuilder::new(MinimalEthSpec)
let chain = Builder::new(MinimalEthSpec)
.logger(log.clone())
.store(Arc::new(store))
.task_executor(runtime.task_executor.clone())
@@ -1112,12 +1190,6 @@ mod test {
.testing_slot_clock(Duration::from_secs(1))
.expect("should configure testing slot clock")
.shutdown_sender(shutdown_tx)
.monitor_validators(
true,
vec![],
DEFAULT_INDIVIDUAL_TRACKING_THRESHOLD,
log.clone(),
)
.build()
.expect("should build");

View File

@@ -35,10 +35,7 @@ use crate::beacon_chain::ATTESTATION_CACHE_LOCK_TIMEOUT;
use crate::persisted_fork_choice::PersistedForkChoice;
use crate::shuffling_cache::BlockShufflingIds;
use crate::{
beacon_chain::{
BeaconForkChoice, BeaconStore, OverrideForkchoiceUpdate,
BLOCK_PROCESSING_CACHE_LOCK_TIMEOUT, FORK_CHOICE_DB_KEY,
},
beacon_chain::{BeaconForkChoice, BeaconStore, OverrideForkchoiceUpdate, FORK_CHOICE_DB_KEY},
block_times_cache::BlockTimesCache,
events::ServerSentEventHandler,
metrics,
@@ -54,6 +51,7 @@ use itertools::process_results;
use parking_lot::{Mutex, RwLock, RwLockReadGuard, RwLockWriteGuard};
use slog::{crit, debug, error, warn, Logger};
use slot_clock::SlotClock;
use state_processing::AllCaches;
use std::sync::Arc;
use std::time::Duration;
use store::{iter::StateRootsIterator, KeyValueStoreOp, StoreItem};
@@ -466,9 +464,7 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
pub fn head_beacon_state_cloned(&self) -> BeaconState<T::EthSpec> {
// Don't clone whilst holding the read-lock, take an Arc-clone to reduce lock contention.
let snapshot: Arc<_> = self.head_snapshot();
snapshot
.beacon_state
.clone_with(CloneConfig::committee_caches_only())
snapshot.beacon_state.clone()
}
/// Execute the fork choice algorithm and enthrone the result as the canonical head.
@@ -652,48 +648,31 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
let new_cached_head = if new_view.head_block_root != old_view.head_block_root {
metrics::inc_counter(&metrics::FORK_CHOICE_CHANGED_HEAD);
// Try and obtain the snapshot for `beacon_block_root` from the snapshot cache, falling
// back to a database read if that fails.
let new_snapshot = self
.snapshot_cache
.try_read_for(BLOCK_PROCESSING_CACHE_LOCK_TIMEOUT)
.and_then(|snapshot_cache| {
snapshot_cache.get_cloned(
let mut new_snapshot = {
let beacon_block = self
.store
.get_full_block(&new_view.head_block_root)?
.ok_or(Error::MissingBeaconBlock(new_view.head_block_root))?;
let (_, beacon_state) = self
.store
.get_advanced_hot_state(
new_view.head_block_root,
CloneConfig::committee_caches_only(),
)
})
.map::<Result<_, Error>, _>(Ok)
.unwrap_or_else(|| {
let beacon_block = self
.store
.get_full_block(&new_view.head_block_root)?
.ok_or(Error::MissingBeaconBlock(new_view.head_block_root))?;
current_slot,
beacon_block.state_root(),
)?
.ok_or(Error::MissingBeaconState(beacon_block.state_root()))?;
let (_, beacon_state) = self
.store
.get_advanced_hot_state(
new_view.head_block_root,
current_slot,
beacon_block.state_root(),
)?
.ok_or(Error::MissingBeaconState(beacon_block.state_root()))?;
BeaconSnapshot {
beacon_block: Arc::new(beacon_block),
beacon_block_root: new_view.head_block_root,
beacon_state,
}
};
Ok(BeaconSnapshot {
beacon_block: Arc::new(beacon_block),
beacon_block_root: new_view.head_block_root,
beacon_state,
})
})
.and_then(|mut snapshot| {
// Regardless of where we got the state from, attempt to build the committee
// caches.
snapshot
.beacon_state
.build_all_committee_caches(&self.spec)
.map_err(Into::into)
.map(|()| snapshot)
})?;
// Regardless of where we got the state from, attempt to build all the
// caches except the tree hash cache.
new_snapshot.beacon_state.build_all_caches(&self.spec)?;
let new_cached_head = CachedHead {
snapshot: Arc::new(new_snapshot),
@@ -834,25 +813,6 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
.beacon_state
.attester_shuffling_decision_root(self.genesis_block_root, RelativeEpoch::Current);
// Update the snapshot cache with the latest head value.
//
// This *could* be done inside `recompute_head`, however updating the head on the snapshot
// cache is not critical so we avoid placing it on a critical path. Note that this function
// will not return an error if the update fails, it will just log an error.
self.snapshot_cache
.try_write_for(BLOCK_PROCESSING_CACHE_LOCK_TIMEOUT)
.map(|mut snapshot_cache| {
snapshot_cache.update_head(new_snapshot.beacon_block_root);
})
.unwrap_or_else(|| {
error!(
self.log,
"Failed to obtain cache write lock";
"lock" => "snapshot_cache",
"task" => "update head"
);
});
match BlockShufflingIds::try_from_head(
new_snapshot.beacon_block_root,
&new_snapshot.beacon_state,
@@ -984,25 +944,19 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
.start_slot(T::EthSpec::slots_per_epoch()),
);
self.snapshot_cache
.try_write_for(BLOCK_PROCESSING_CACHE_LOCK_TIMEOUT)
.map(|mut snapshot_cache| {
snapshot_cache.prune(new_view.finalized_checkpoint.epoch);
debug!(
self.log,
"Snapshot cache pruned";
"new_len" => snapshot_cache.len(),
"remaining_roots" => ?snapshot_cache.beacon_block_roots(),
);
})
.unwrap_or_else(|| {
error!(
self.log,
"Failed to obtain cache write lock";
"lock" => "snapshot_cache",
"task" => "prune"
);
});
self.observed_blob_sidecars.write().prune(
new_view
.finalized_checkpoint
.epoch
.start_slot(T::EthSpec::slots_per_epoch()),
);
self.observed_slashable.write().prune(
new_view
.finalized_checkpoint
.epoch
.start_slot(T::EthSpec::slots_per_epoch()),
);
self.attester_cache
.prune_below(new_view.finalized_checkpoint.epoch);
@@ -1051,6 +1005,12 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
self.head_tracker.clone(),
)?;
// Prune blobs in the background.
if let Some(data_availability_boundary) = self.data_availability_boundary() {
self.store_migrator
.process_prune_blobs(data_availability_boundary);
}
// Take a write-lock on the canonical head and signal for it to prune.
self.canonical_head.fork_choice_write_lock().prune()?;
@@ -1385,13 +1345,6 @@ fn observe_head_block_delays<E: EthSpec, S: SlotClock>(
// Do not store metrics if the block was > 4 slots old, this helps prevent noise during
// sync.
if !block_from_sync {
// Observe the total block delay. This is the delay between the time the slot started
// and when the block was set as head.
metrics::observe_duration(
&metrics::BEACON_BLOCK_HEAD_SLOT_START_DELAY_TIME,
block_delay_total,
);
// Observe the delay between when we imported the block and when we set the block as
// head.
let block_delays = block_times_cache.get_block_delays(
@@ -1401,34 +1354,120 @@ fn observe_head_block_delays<E: EthSpec, S: SlotClock>(
.unwrap_or_else(|| Duration::from_secs(0)),
);
metrics::observe_duration(
&metrics::BEACON_BLOCK_OBSERVED_SLOT_START_DELAY_TIME,
block_delays
.observed
.unwrap_or_else(|| Duration::from_secs(0)),
// Update all the metrics
// Convention here is to use "Time" to indicate the duration of the event and "Delay"
// to indicate the time since the start of the slot.
//
// Observe the total block delay. This is the delay between the time the slot started
// and when the block was set as head.
metrics::set_gauge(
&metrics::BEACON_BLOCK_DELAY_TOTAL,
block_delay_total.as_millis() as i64,
);
metrics::observe_duration(
&metrics::BEACON_BLOCK_HEAD_IMPORTED_DELAY_TIME,
// The time at which the beacon block was first observed to be processed
metrics::set_gauge(
&metrics::BEACON_BLOCK_DELAY_OBSERVED_SLOT_START,
block_delays
.observed
.unwrap_or_else(|| Duration::from_secs(0))
.as_millis() as i64,
);
// The time from the start of the slot when all blobs have been observed. Technically this
// is the time we last saw a blob related to this block/slot.
metrics::set_gauge(
&metrics::BEACON_BLOB_DELAY_ALL_OBSERVED_SLOT_START,
block_delays
.all_blobs_observed
.unwrap_or_else(|| Duration::from_secs(0))
.as_millis() as i64,
);
// The time it took to check the validity with the EL
metrics::set_gauge(
&metrics::BEACON_BLOCK_DELAY_EXECUTION_TIME,
block_delays
.execution_time
.unwrap_or_else(|| Duration::from_secs(0))
.as_millis() as i64,
);
// The time the block became available after the start of the slot. Available here means
// that all the blobs have arrived and the block has been verified by the execution layer.
metrics::set_gauge(
&metrics::BEACON_BLOCK_DELAY_AVAILABLE_SLOT_START,
block_delays
.available
.unwrap_or_else(|| Duration::from_secs(0))
.as_millis() as i64,
);
// The time the block became attestable after the start of the slot.
metrics::set_gauge(
&metrics::BEACON_BLOCK_DELAY_ATTESTABLE_SLOT_START,
block_delays
.attestable
.unwrap_or_else(|| Duration::from_secs(0))
.as_millis() as i64,
);
// The time the block was imported since becoming available.
metrics::set_gauge(
&metrics::BEACON_BLOCK_DELAY_IMPORTED_TIME,
block_delays
.imported
.unwrap_or_else(|| Duration::from_secs(0))
.as_millis() as i64,
);
// The time the block was imported and setting it as head
metrics::set_gauge(
&metrics::BEACON_BLOCK_DELAY_HEAD_IMPORTED_TIME,
block_delays
.set_as_head
.unwrap_or_else(|| Duration::from_secs(0)),
.unwrap_or_else(|| Duration::from_secs(0))
.as_millis() as i64,
);
// If the block was enshrined as head too late for attestations to be created for it,
// log a debug warning and increment a metric.
let format_delay = |delay: &Option<Duration>| {
delay.map_or("unknown".to_string(), |d| format!("{}", d.as_millis()))
};
if late_head {
metrics::inc_counter(&metrics::BEACON_BLOCK_HEAD_SLOT_START_DELAY_EXCEEDED_TOTAL);
metrics::inc_counter(&metrics::BEACON_BLOCK_DELAY_HEAD_SLOT_START_EXCEEDED_TOTAL);
debug!(
log,
"Delayed head block";
"block_root" => ?head_block_root,
"proposer_index" => head_block_proposer_index,
"slot" => head_block_slot,
"block_delay" => ?block_delay_total,
"observed_delay" => ?block_delays.observed,
"imported_delay" => ?block_delays.imported,
"set_as_head_delay" => ?block_delays.set_as_head,
"total_delay_ms" => block_delay_total.as_millis(),
"observed_delay_ms" => format_delay(&block_delays.observed),
"blob_delay_ms" => format_delay(&block_delays.all_blobs_observed),
"execution_time_ms" => format_delay(&block_delays.execution_time),
"available_delay_ms" => format_delay(&block_delays.available),
"attestable_delay_ms" => format_delay(&block_delays.attestable),
"imported_time_ms" => format_delay(&block_delays.imported),
"set_as_head_time_ms" => format_delay(&block_delays.set_as_head),
);
} else {
debug!(
log,
"On-time head block";
"block_root" => ?head_block_root,
"proposer_index" => head_block_proposer_index,
"slot" => head_block_slot,
"total_delay_ms" => block_delay_total.as_millis(),
"observed_delay_ms" => format_delay(&block_delays.observed),
"blob_delay_ms" => format_delay(&block_delays.all_blobs_observed),
"execution_time_ms" => format_delay(&block_delays.execution_time),
"available_delay_ms" => format_delay(&block_delays.available),
"attestable_delay_ms" => format_delay(&block_delays.attestable),
"imported_time_ms" => format_delay(&block_delays.imported),
"set_as_head_time_ms" => format_delay(&block_delays.set_as_head),
);
}
}

View File

@@ -1,5 +1,4 @@
//! Provides tools for checking if a node is ready for the Capella upgrade and following merge
//! transition.
//! Provides tools for checking if a node is ready for the Capella upgrade.
use crate::{BeaconChain, BeaconChainTypes};
use execution_layer::http::{
@@ -11,7 +10,7 @@ use std::time::Duration;
use types::*;
/// The time before the Capella fork when we will start issuing warnings about preparation.
use super::merge_readiness::SECONDS_IN_A_WEEK;
use super::bellatrix_readiness::SECONDS_IN_A_WEEK;
pub const CAPELLA_READINESS_PREPARATION_SECONDS: u64 = SECONDS_IN_A_WEEK * 2;
pub const ENGINE_CAPABILITIES_REFRESH_INTERVAL: u64 = 300;

View File

@@ -1,9 +1,10 @@
pub use proto_array::{DisallowedReOrgOffsets, ReOrgThreshold};
use serde::{Deserialize, Serialize};
use std::time::Duration;
use types::{Checkpoint, Epoch, ProgressiveBalancesMode};
use types::{Checkpoint, Epoch};
pub const DEFAULT_RE_ORG_THRESHOLD: ReOrgThreshold = ReOrgThreshold(20);
pub const DEFAULT_RE_ORG_HEAD_THRESHOLD: ReOrgThreshold = ReOrgThreshold(20);
pub const DEFAULT_RE_ORG_PARENT_THRESHOLD: ReOrgThreshold = ReOrgThreshold(160);
pub const DEFAULT_RE_ORG_MAX_EPOCHS_SINCE_FINALIZATION: Epoch = Epoch::new(2);
/// Default to 1/12th of the slot, which is 1 second on mainnet.
pub const DEFAULT_RE_ORG_CUTOFF_DENOMINATOR: u32 = 12;
@@ -31,8 +32,10 @@ pub struct ChainConfig {
pub enable_lock_timeouts: bool,
/// The max size of a message that can be sent over the network.
pub max_network_size: usize,
/// Maximum percentage of committee weight at which to attempt re-orging the canonical head.
pub re_org_threshold: Option<ReOrgThreshold>,
/// Maximum percentage of the head committee weight at which to attempt re-orging the canonical head.
pub re_org_head_threshold: Option<ReOrgThreshold>,
/// Minimum percentage of the parent committee weight at which to attempt re-orging the canonical head.
pub re_org_parent_threshold: Option<ReOrgThreshold>,
/// Maximum number of epochs since finalization for attempting a proposer re-org.
pub re_org_max_epochs_since_finalization: Epoch,
/// Maximum delay after the start of the slot at which to propose a reorging block.
@@ -79,10 +82,10 @@ pub struct ChainConfig {
///
/// This is useful for block builders and testing.
pub always_prepare_payload: bool,
/// Whether to use `ProgressiveBalancesCache` in unrealized FFG progression calculation.
pub progressive_balances_mode: ProgressiveBalancesMode,
/// Number of epochs between each migration of data from the hot database to the freezer.
pub epochs_per_migration: u64,
/// When set to true Light client server computes and caches state proofs for serving updates
pub enable_light_client_server: bool,
}
impl Default for ChainConfig {
@@ -93,7 +96,8 @@ impl Default for ChainConfig {
reconstruct_historic_states: false,
enable_lock_timeouts: true,
max_network_size: 10 * 1_048_576, // 10M
re_org_threshold: Some(DEFAULT_RE_ORG_THRESHOLD),
re_org_head_threshold: Some(DEFAULT_RE_ORG_HEAD_THRESHOLD),
re_org_parent_threshold: Some(DEFAULT_RE_ORG_PARENT_THRESHOLD),
re_org_max_epochs_since_finalization: DEFAULT_RE_ORG_MAX_EPOCHS_SINCE_FINALIZATION,
re_org_cutoff_millis: None,
re_org_disallowed_offsets: DisallowedReOrgOffsets::default(),
@@ -112,8 +116,8 @@ impl Default for ChainConfig {
shuffling_cache_size: crate::shuffling_cache::DEFAULT_CACHE_SIZE,
genesis_backfill: false,
always_prepare_payload: false,
progressive_balances_mode: ProgressiveBalancesMode::Checked,
epochs_per_migration: crate::migrate::DEFAULT_EPOCHS_PER_MIGRATION,
enable_light_client_server: false,
}
}
}

View File

@@ -0,0 +1,514 @@
use crate::blob_verification::{verify_kzg_for_blob_list, GossipVerifiedBlob, KzgVerifiedBlobList};
use crate::block_verification_types::{
AvailabilityPendingExecutedBlock, AvailableExecutedBlock, RpcBlock,
};
use crate::data_availability_checker::overflow_lru_cache::OverflowLRUCache;
use crate::{BeaconChain, BeaconChainTypes, BeaconStore};
use kzg::Kzg;
use slog::{debug, error, Logger};
use slot_clock::SlotClock;
use std::fmt;
use std::fmt::Debug;
use std::num::NonZeroUsize;
use std::sync::Arc;
use std::time::Duration;
use task_executor::TaskExecutor;
use types::blob_sidecar::{BlobIdentifier, BlobSidecar, FixedBlobSidecarList};
use types::{BlobSidecarList, ChainSpec, Epoch, EthSpec, Hash256, SignedBeaconBlock};
mod error;
mod overflow_lru_cache;
mod state_lru_cache;
pub use error::{Error as AvailabilityCheckError, ErrorCategory as AvailabilityCheckErrorCategory};
use types::non_zero_usize::new_non_zero_usize;
/// The LRU Cache stores `PendingComponents` which can store up to
/// `MAX_BLOBS_PER_BLOCK = 6` blobs each. A `BlobSidecar` is 0.131256 MB. So
/// the maximum size of a `PendingComponents` is ~ 0.787536 MB. Setting this
/// to 1024 means the maximum size of the cache is ~ 0.8 GB. But the cache
/// will target a size of less than 75% of capacity.
pub const OVERFLOW_LRU_CAPACITY: NonZeroUsize = new_non_zero_usize(1024);
/// Until tree-states is implemented, we can't store very many states in memory :(
pub const STATE_LRU_CAPACITY_NON_ZERO: NonZeroUsize = new_non_zero_usize(2);
pub const STATE_LRU_CAPACITY: usize = STATE_LRU_CAPACITY_NON_ZERO.get();
/// This includes a cache for any blocks or blobs that have been received over gossip or RPC
/// and are awaiting more components before they can be imported. Additionally the
/// `DataAvailabilityChecker` is responsible for KZG verification of block components as well as
/// checking whether a "availability check" is required at all.
pub struct DataAvailabilityChecker<T: BeaconChainTypes> {
availability_cache: Arc<OverflowLRUCache<T>>,
slot_clock: T::SlotClock,
kzg: Option<Arc<Kzg>>,
log: Logger,
spec: ChainSpec,
}
/// This type is returned after adding a block / blob to the `DataAvailabilityChecker`.
///
/// Indicates if the block is fully `Available` or if we need blobs or blocks
/// to "complete" the requirements for an `AvailableBlock`.
#[derive(PartialEq)]
pub enum Availability<E: EthSpec> {
MissingComponents(Hash256),
Available(Box<AvailableExecutedBlock<E>>),
}
impl<E: EthSpec> Debug for Availability<E> {
fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
match self {
Self::MissingComponents(block_root) => {
write!(f, "MissingComponents({})", block_root)
}
Self::Available(block) => write!(f, "Available({:?})", block.import_data.block_root),
}
}
}
impl<T: BeaconChainTypes> DataAvailabilityChecker<T> {
pub fn new(
slot_clock: T::SlotClock,
kzg: Option<Arc<Kzg>>,
store: BeaconStore<T>,
log: &Logger,
spec: ChainSpec,
) -> Result<Self, AvailabilityCheckError> {
let overflow_cache = OverflowLRUCache::new(OVERFLOW_LRU_CAPACITY, store, spec.clone())?;
Ok(Self {
availability_cache: Arc::new(overflow_cache),
slot_clock,
log: log.clone(),
kzg,
spec,
})
}
/// Checks if the block root is currenlty in the availability cache awaiting import because
/// of missing components.
pub fn get_execution_valid_block(
&self,
block_root: &Hash256,
) -> Option<Arc<SignedBeaconBlock<T::EthSpec>>> {
self.availability_cache
.get_execution_valid_block(block_root)
}
/// Return the set of imported blob indexes for `block_root`. Returns None if there is no block
/// component for `block_root`.
pub fn imported_blob_indexes(&self, block_root: &Hash256) -> Option<Vec<u64>> {
self.availability_cache
.peek_pending_components(block_root, |components| {
components.map(|components| {
components
.get_cached_blobs()
.iter()
.filter_map(|blob| blob.as_ref().map(|blob| blob.blob_index()))
.collect::<Vec<_>>()
})
})
}
/// Get a blob from the availability cache.
pub fn get_blob(
&self,
blob_id: &BlobIdentifier,
) -> Result<Option<Arc<BlobSidecar<T::EthSpec>>>, AvailabilityCheckError> {
self.availability_cache.peek_blob(blob_id)
}
/// Put a list of blobs received via RPC into the availability cache. This performs KZG
/// verification on the blobs in the list.
pub fn put_rpc_blobs(
&self,
block_root: Hash256,
blobs: FixedBlobSidecarList<T::EthSpec>,
) -> Result<Availability<T::EthSpec>, AvailabilityCheckError> {
let Some(kzg) = self.kzg.as_ref() else {
return Err(AvailabilityCheckError::KzgNotInitialized);
};
let seen_timestamp = self
.slot_clock
.now_duration()
.ok_or(AvailabilityCheckError::SlotClockError)?;
let verified_blobs =
KzgVerifiedBlobList::new(Vec::from(blobs).into_iter().flatten(), kzg, seen_timestamp)
.map_err(AvailabilityCheckError::Kzg)?;
self.availability_cache
.put_kzg_verified_blobs(block_root, verified_blobs)
}
/// Check if we've cached other blobs for this block. If it completes a set and we also
/// have a block cached, return the `Availability` variant triggering block import.
/// Otherwise cache the blob sidecar.
///
/// This should only accept gossip verified blobs, so we should not have to worry about dupes.
pub fn put_gossip_blob(
&self,
gossip_blob: GossipVerifiedBlob<T>,
) -> Result<Availability<T::EthSpec>, AvailabilityCheckError> {
self.availability_cache
.put_kzg_verified_blobs(gossip_blob.block_root(), vec![gossip_blob.into_inner()])
}
/// Check if we have all the blobs for a block. Returns `Availability` which has information
/// about whether all components have been received or more are required.
pub fn put_pending_executed_block(
&self,
executed_block: AvailabilityPendingExecutedBlock<T::EthSpec>,
) -> Result<Availability<T::EthSpec>, AvailabilityCheckError> {
self.availability_cache
.put_pending_executed_block(executed_block)
}
pub fn remove_pending_components(&self, block_root: Hash256) {
self.availability_cache
.remove_pending_components(block_root)
}
/// Verifies kzg commitments for an RpcBlock, returns a `MaybeAvailableBlock` that may
/// include the fully available block.
///
/// WARNING: This function assumes all required blobs are already present, it does NOT
/// check if there are any missing blobs.
pub fn verify_kzg_for_rpc_block(
&self,
block: RpcBlock<T::EthSpec>,
) -> Result<MaybeAvailableBlock<T::EthSpec>, AvailabilityCheckError> {
let (block_root, block, blobs) = block.deconstruct();
match blobs {
None => {
if self.blobs_required_for_block(&block) {
Ok(MaybeAvailableBlock::AvailabilityPending { block_root, block })
} else {
Ok(MaybeAvailableBlock::Available(AvailableBlock {
block_root,
block,
blobs: None,
blobs_available_timestamp: None,
}))
}
}
Some(blob_list) => {
let verified_blobs = if self.blobs_required_for_block(&block) {
let kzg = self
.kzg
.as_ref()
.ok_or(AvailabilityCheckError::KzgNotInitialized)?;
verify_kzg_for_blob_list(blob_list.iter(), kzg)
.map_err(AvailabilityCheckError::Kzg)?;
Some(blob_list)
} else {
None
};
Ok(MaybeAvailableBlock::Available(AvailableBlock {
block_root,
block,
blobs: verified_blobs,
blobs_available_timestamp: None,
}))
}
}
}
/// Checks if a vector of blocks are available. Returns a vector of `MaybeAvailableBlock`
/// This is more efficient than calling `verify_kzg_for_rpc_block` in a loop as it does
/// all kzg verification at once
///
/// WARNING: This function assumes all required blobs are already present, it does NOT
/// check if there are any missing blobs.
pub fn verify_kzg_for_rpc_blocks(
&self,
blocks: Vec<RpcBlock<T::EthSpec>>,
) -> Result<Vec<MaybeAvailableBlock<T::EthSpec>>, AvailabilityCheckError> {
let mut results = Vec::with_capacity(blocks.len());
let all_blobs: BlobSidecarList<T::EthSpec> = blocks
.iter()
.filter(|block| self.blobs_required_for_block(block.as_block()))
// this clone is cheap as it's cloning an Arc
.filter_map(|block| block.blobs().cloned())
.flatten()
.collect::<Vec<_>>()
.into();
// verify kzg for all blobs at once
if !all_blobs.is_empty() {
let kzg = self
.kzg
.as_ref()
.ok_or(AvailabilityCheckError::KzgNotInitialized)?;
verify_kzg_for_blob_list(all_blobs.iter(), kzg)?;
}
for block in blocks {
let (block_root, block, blobs) = block.deconstruct();
match blobs {
None => {
if self.blobs_required_for_block(&block) {
results.push(MaybeAvailableBlock::AvailabilityPending { block_root, block })
} else {
results.push(MaybeAvailableBlock::Available(AvailableBlock {
block_root,
block,
blobs: None,
blobs_available_timestamp: None,
}))
}
}
Some(blob_list) => {
let verified_blobs = if self.blobs_required_for_block(&block) {
Some(blob_list)
} else {
None
};
// already verified kzg for all blobs
results.push(MaybeAvailableBlock::Available(AvailableBlock {
block_root,
block,
blobs: verified_blobs,
blobs_available_timestamp: None,
}))
}
}
}
Ok(results)
}
/// Determines the blob requirements for a block. If the block is pre-deneb, no blobs are required.
/// If the block's epoch is from prior to the data availability boundary, no blobs are required.
fn blobs_required_for_block(&self, block: &SignedBeaconBlock<T::EthSpec>) -> bool {
block.num_expected_blobs() > 0 && self.da_check_required_for_epoch(block.epoch())
}
/// The epoch at which we require a data availability check in block processing.
/// `None` if the `Deneb` fork is disabled.
pub fn data_availability_boundary(&self) -> Option<Epoch> {
self.spec.deneb_fork_epoch.and_then(|fork_epoch| {
self.slot_clock
.now()
.map(|slot| slot.epoch(T::EthSpec::slots_per_epoch()))
.map(|current_epoch| {
std::cmp::max(
fork_epoch,
current_epoch
.saturating_sub(self.spec.min_epochs_for_blob_sidecars_requests),
)
})
})
}
/// Returns true if the given epoch lies within the da boundary and false otherwise.
pub fn da_check_required_for_epoch(&self, block_epoch: Epoch) -> bool {
self.data_availability_boundary()
.map_or(false, |da_epoch| block_epoch >= da_epoch)
}
pub fn da_check_required_for_current_epoch(&self) -> bool {
let Some(current_slot) = self.slot_clock.now_or_genesis() else {
error!(
self.log,
"Failed to read slot clock when checking for missing blob ids"
);
return false;
};
self.da_check_required_for_epoch(current_slot.epoch(T::EthSpec::slots_per_epoch()))
}
/// Returns `true` if the current epoch is greater than or equal to the `Deneb` epoch.
pub fn is_deneb(&self) -> bool {
self.slot_clock.now().map_or(false, |slot| {
self.spec.deneb_fork_epoch.map_or(false, |deneb_epoch| {
let now_epoch = slot.epoch(T::EthSpec::slots_per_epoch());
now_epoch >= deneb_epoch
})
})
}
/// Persist all in memory components to disk
pub fn persist_all(&self) -> Result<(), AvailabilityCheckError> {
self.availability_cache.write_all_to_disk()
}
/// Collects metrics from the data availability checker.
pub fn metrics(&self) -> DataAvailabilityCheckerMetrics {
DataAvailabilityCheckerMetrics {
num_store_entries: self.availability_cache.num_store_entries(),
state_cache_size: self.availability_cache.state_cache_size(),
block_cache_size: self.availability_cache.block_cache_size(),
}
}
}
/// Helper struct to group data availability checker metrics.
pub struct DataAvailabilityCheckerMetrics {
pub num_store_entries: usize,
pub state_cache_size: usize,
pub block_cache_size: usize,
}
pub fn start_availability_cache_maintenance_service<T: BeaconChainTypes>(
executor: TaskExecutor,
chain: Arc<BeaconChain<T>>,
) {
// this cache only needs to be maintained if deneb is configured
if chain.spec.deneb_fork_epoch.is_some() {
let overflow_cache = chain.data_availability_checker.availability_cache.clone();
executor.spawn(
async move { availability_cache_maintenance_service(chain, overflow_cache).await },
"availability_cache_service",
);
} else {
debug!(
chain.log,
"Deneb fork not configured, not starting availability cache maintenance service"
);
}
}
async fn availability_cache_maintenance_service<T: BeaconChainTypes>(
chain: Arc<BeaconChain<T>>,
overflow_cache: Arc<OverflowLRUCache<T>>,
) {
let epoch_duration = chain.slot_clock.slot_duration() * T::EthSpec::slots_per_epoch() as u32;
loop {
match chain
.slot_clock
.duration_to_next_epoch(T::EthSpec::slots_per_epoch())
{
Some(duration) => {
// this service should run 3/4 of the way through the epoch
let additional_delay = (epoch_duration * 3) / 4;
tokio::time::sleep(duration + additional_delay).await;
let Some(deneb_fork_epoch) = chain.spec.deneb_fork_epoch else {
// shutdown service if deneb fork epoch not set
break;
};
debug!(
chain.log,
"Availability cache maintenance service firing";
);
let Some(current_epoch) = chain
.slot_clock
.now()
.map(|slot| slot.epoch(T::EthSpec::slots_per_epoch()))
else {
continue;
};
if current_epoch < deneb_fork_epoch {
// we are not in deneb yet
continue;
}
let finalized_epoch = chain
.canonical_head
.fork_choice_read_lock()
.finalized_checkpoint()
.epoch;
// any data belonging to an epoch before this should be pruned
let cutoff_epoch = std::cmp::max(
finalized_epoch + 1,
std::cmp::max(
current_epoch
.saturating_sub(chain.spec.min_epochs_for_blob_sidecars_requests),
deneb_fork_epoch,
),
);
if let Err(e) = overflow_cache.do_maintenance(cutoff_epoch) {
error!(chain.log, "Failed to maintain availability cache"; "error" => ?e);
}
}
None => {
error!(chain.log, "Failed to read slot clock");
// If we can't read the slot clock, just wait another slot.
tokio::time::sleep(chain.slot_clock.slot_duration()).await;
}
};
}
}
/// A fully available block that is ready to be imported into fork choice.
#[derive(Clone, Debug, PartialEq)]
pub struct AvailableBlock<E: EthSpec> {
block_root: Hash256,
block: Arc<SignedBeaconBlock<E>>,
blobs: Option<BlobSidecarList<E>>,
/// Timestamp at which this block first became available (UNIX timestamp, time since 1970).
blobs_available_timestamp: Option<Duration>,
}
impl<E: EthSpec> AvailableBlock<E> {
pub fn __new_for_testing(
block_root: Hash256,
block: Arc<SignedBeaconBlock<E>>,
blobs: Option<BlobSidecarList<E>>,
) -> Self {
Self {
block_root,
block,
blobs,
blobs_available_timestamp: None,
}
}
pub fn block(&self) -> &SignedBeaconBlock<E> {
&self.block
}
pub fn block_cloned(&self) -> Arc<SignedBeaconBlock<E>> {
self.block.clone()
}
pub fn blobs(&self) -> Option<&BlobSidecarList<E>> {
self.blobs.as_ref()
}
pub fn blobs_available_timestamp(&self) -> Option<Duration> {
self.blobs_available_timestamp
}
pub fn deconstruct(
self,
) -> (
Hash256,
Arc<SignedBeaconBlock<E>>,
Option<BlobSidecarList<E>>,
) {
let AvailableBlock {
block_root,
block,
blobs,
blobs_available_timestamp: _,
} = self;
(block_root, block, blobs)
}
}
#[derive(Debug, Clone)]
pub enum MaybeAvailableBlock<E: EthSpec> {
/// This variant is fully available.
/// i.e. for pre-deneb blocks, it contains a (`SignedBeaconBlock`, `Blobs::None`) and for
/// post-4844 blocks, it contains a `SignedBeaconBlock` and a Blobs variant other than `Blobs::None`.
Available(AvailableBlock<E>),
/// This variant is not fully available and requires blobs to become fully available.
AvailabilityPending {
block_root: Hash256,
block: Arc<SignedBeaconBlock<E>>,
},
}
impl<E: EthSpec> MaybeAvailableBlock<E> {
pub fn block_cloned(&self) -> Arc<SignedBeaconBlock<E>> {
match self {
Self::Available(block) => block.block_cloned(),
Self::AvailabilityPending { block, .. } => block.clone(),
}
}
}

View File

@@ -0,0 +1,82 @@
use kzg::{Error as KzgError, KzgCommitment};
use types::{BeaconStateError, Hash256};
#[derive(Debug)]
pub enum Error {
Kzg(KzgError),
KzgNotInitialized,
KzgVerificationFailed,
KzgCommitmentMismatch {
blob_commitment: KzgCommitment,
block_commitment: KzgCommitment,
},
Unexpected,
SszTypes(ssz_types::Error),
MissingBlobs,
BlobIndexInvalid(u64),
StoreError(store::Error),
DecodeError(ssz::DecodeError),
ParentStateMissing(Hash256),
BlockReplayError(state_processing::BlockReplayError),
RebuildingStateCaches(BeaconStateError),
SlotClockError,
}
#[derive(PartialEq, Eq)]
pub enum ErrorCategory {
/// Internal Errors (not caused by peers)
Internal,
/// Errors caused by faulty / malicious peers
Malicious,
}
impl Error {
pub fn category(&self) -> ErrorCategory {
match self {
Error::KzgNotInitialized
| Error::SszTypes(_)
| Error::MissingBlobs
| Error::StoreError(_)
| Error::DecodeError(_)
| Error::Unexpected
| Error::ParentStateMissing(_)
| Error::BlockReplayError(_)
| Error::RebuildingStateCaches(_)
| Error::SlotClockError => ErrorCategory::Internal,
Error::Kzg(_)
| Error::BlobIndexInvalid(_)
| Error::KzgCommitmentMismatch { .. }
| Error::KzgVerificationFailed => ErrorCategory::Malicious,
}
}
}
impl From<ssz_types::Error> for Error {
fn from(value: ssz_types::Error) -> Self {
Self::SszTypes(value)
}
}
impl From<store::Error> for Error {
fn from(value: store::Error) -> Self {
Self::StoreError(value)
}
}
impl From<ssz::DecodeError> for Error {
fn from(value: ssz::DecodeError) -> Self {
Self::DecodeError(value)
}
}
impl From<state_processing::BlockReplayError> for Error {
fn from(value: state_processing::BlockReplayError) -> Self {
Self::BlockReplayError(value)
}
}
impl From<KzgError> for Error {
fn from(value: KzgError) -> Self {
Self::Kzg(value)
}
}

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,251 @@
use crate::block_verification_types::AsBlock;
use crate::{
block_verification_types::BlockImportData,
data_availability_checker::{AvailabilityCheckError, STATE_LRU_CAPACITY_NON_ZERO},
eth1_finalization_cache::Eth1FinalizationData,
AvailabilityPendingExecutedBlock, BeaconChainTypes, BeaconStore, PayloadVerificationOutcome,
};
use lru::LruCache;
use parking_lot::RwLock;
use ssz_derive::{Decode, Encode};
use state_processing::BlockReplayer;
use std::sync::Arc;
use store::OnDiskConsensusContext;
use types::beacon_block_body::KzgCommitments;
use types::{ssz_tagged_signed_beacon_block, ssz_tagged_signed_beacon_block_arc};
use types::{BeaconState, BlindedPayload, ChainSpec, Epoch, EthSpec, Hash256, SignedBeaconBlock};
/// This mirrors everything in the `AvailabilityPendingExecutedBlock`, except
/// that it is much smaller because it contains only a state root instead of
/// a full `BeaconState`.
#[derive(Encode, Decode, Clone)]
pub struct DietAvailabilityPendingExecutedBlock<E: EthSpec> {
#[ssz(with = "ssz_tagged_signed_beacon_block_arc")]
block: Arc<SignedBeaconBlock<E>>,
state_root: Hash256,
#[ssz(with = "ssz_tagged_signed_beacon_block")]
parent_block: SignedBeaconBlock<E, BlindedPayload<E>>,
parent_eth1_finalization_data: Eth1FinalizationData,
confirmed_state_roots: Vec<Hash256>,
consensus_context: OnDiskConsensusContext<E>,
payload_verification_outcome: PayloadVerificationOutcome,
}
/// just implementing the same methods as `AvailabilityPendingExecutedBlock`
impl<E: EthSpec> DietAvailabilityPendingExecutedBlock<E> {
pub fn as_block(&self) -> &SignedBeaconBlock<E> {
&self.block
}
pub fn block_cloned(&self) -> Arc<SignedBeaconBlock<E>> {
self.block.clone()
}
pub fn num_blobs_expected(&self) -> usize {
self.block
.message()
.body()
.blob_kzg_commitments()
.map_or(0, |commitments| commitments.len())
}
pub fn get_commitments(&self) -> KzgCommitments<E> {
self.as_block()
.message()
.body()
.blob_kzg_commitments()
.cloned()
.unwrap_or_default()
}
}
/// This LRU cache holds BeaconStates used for block import. If the cache overflows,
/// the least recently used state will be dropped. If the dropped state is needed
/// later on, it will be recovered from the parent state and replaying the block.
///
/// WARNING: This cache assumes the parent block of any `AvailabilityPendingExecutedBlock`
/// has already been imported into ForkChoice. If this is not the case, the cache
/// will fail to recover the state when the cache overflows because it can't load
/// the parent state!
pub struct StateLRUCache<T: BeaconChainTypes> {
states: RwLock<LruCache<Hash256, BeaconState<T::EthSpec>>>,
store: BeaconStore<T>,
spec: ChainSpec,
}
impl<T: BeaconChainTypes> StateLRUCache<T> {
pub fn new(store: BeaconStore<T>, spec: ChainSpec) -> Self {
Self {
states: RwLock::new(LruCache::new(STATE_LRU_CAPACITY_NON_ZERO)),
store,
spec,
}
}
/// This will store the state in the LRU cache and return a
/// `DietAvailabilityPendingExecutedBlock` which is much cheaper to
/// keep around in memory.
pub fn register_pending_executed_block(
&self,
executed_block: AvailabilityPendingExecutedBlock<T::EthSpec>,
) -> DietAvailabilityPendingExecutedBlock<T::EthSpec> {
let state = executed_block.import_data.state;
let state_root = executed_block.block.state_root();
self.states.write().put(state_root, state);
DietAvailabilityPendingExecutedBlock {
block: executed_block.block,
state_root,
parent_block: executed_block.import_data.parent_block,
parent_eth1_finalization_data: executed_block.import_data.parent_eth1_finalization_data,
confirmed_state_roots: executed_block.import_data.confirmed_state_roots,
consensus_context: OnDiskConsensusContext::from_consensus_context(
executed_block.import_data.consensus_context,
),
payload_verification_outcome: executed_block.payload_verification_outcome,
}
}
/// Recover the `AvailabilityPendingExecutedBlock` from the diet version.
/// This method will first check the cache and if the state is not found
/// it will reconstruct the state by loading the parent state from disk and
/// replaying the block.
pub fn recover_pending_executed_block(
&self,
diet_executed_block: DietAvailabilityPendingExecutedBlock<T::EthSpec>,
) -> Result<AvailabilityPendingExecutedBlock<T::EthSpec>, AvailabilityCheckError> {
let maybe_state = self.states.write().pop(&diet_executed_block.state_root);
if let Some(state) = maybe_state {
let block_root = diet_executed_block.block.canonical_root();
Ok(AvailabilityPendingExecutedBlock {
block: diet_executed_block.block,
import_data: BlockImportData {
block_root,
state,
parent_block: diet_executed_block.parent_block,
parent_eth1_finalization_data: diet_executed_block
.parent_eth1_finalization_data,
confirmed_state_roots: diet_executed_block.confirmed_state_roots,
consensus_context: diet_executed_block
.consensus_context
.into_consensus_context(),
},
payload_verification_outcome: diet_executed_block.payload_verification_outcome,
})
} else {
self.reconstruct_pending_executed_block(diet_executed_block)
}
}
/// Reconstruct the `AvailabilityPendingExecutedBlock` by loading the parent
/// state from disk and replaying the block. This function does NOT check the
/// LRU cache.
pub fn reconstruct_pending_executed_block(
&self,
diet_executed_block: DietAvailabilityPendingExecutedBlock<T::EthSpec>,
) -> Result<AvailabilityPendingExecutedBlock<T::EthSpec>, AvailabilityCheckError> {
let block_root = diet_executed_block.block.canonical_root();
let state = self.reconstruct_state(&diet_executed_block)?;
Ok(AvailabilityPendingExecutedBlock {
block: diet_executed_block.block,
import_data: BlockImportData {
block_root,
state,
parent_block: diet_executed_block.parent_block,
parent_eth1_finalization_data: diet_executed_block.parent_eth1_finalization_data,
confirmed_state_roots: diet_executed_block.confirmed_state_roots,
consensus_context: diet_executed_block
.consensus_context
.into_consensus_context(),
},
payload_verification_outcome: diet_executed_block.payload_verification_outcome,
})
}
/// Reconstruct the state by loading the parent state from disk and replaying
/// the block.
fn reconstruct_state(
&self,
diet_executed_block: &DietAvailabilityPendingExecutedBlock<T::EthSpec>,
) -> Result<BeaconState<T::EthSpec>, AvailabilityCheckError> {
let parent_block_root = diet_executed_block.parent_block.canonical_root();
let parent_block_state_root = diet_executed_block.parent_block.state_root();
let (parent_state_root, parent_state) = self
.store
.get_advanced_hot_state(
parent_block_root,
diet_executed_block.parent_block.slot(),
parent_block_state_root,
)
.map_err(AvailabilityCheckError::StoreError)?
.ok_or(AvailabilityCheckError::ParentStateMissing(
parent_block_state_root,
))?;
let state_roots = vec![
Ok((parent_state_root, diet_executed_block.parent_block.slot())),
Ok((
diet_executed_block.state_root,
diet_executed_block.block.slot(),
)),
];
let block_replayer: BlockReplayer<'_, T::EthSpec, AvailabilityCheckError, _> =
BlockReplayer::new(parent_state, &self.spec)
.no_signature_verification()
.state_root_iter(state_roots.into_iter())
.minimal_block_root_verification();
block_replayer
.apply_blocks(vec![diet_executed_block.block.clone_as_blinded()], None)
.map(|block_replayer| block_replayer.into_state())
.and_then(|mut state| {
state
.build_exit_cache(&self.spec)
.map_err(AvailabilityCheckError::RebuildingStateCaches)?;
state
.update_tree_hash_cache()
.map_err(AvailabilityCheckError::RebuildingStateCaches)?;
Ok(state)
})
}
/// returns the state cache for inspection
pub fn lru_cache(&self) -> &RwLock<LruCache<Hash256, BeaconState<T::EthSpec>>> {
&self.states
}
/// remove any states from the cache from before the given epoch
pub fn do_maintenance(&self, cutoff_epoch: Epoch) {
let mut write_lock = self.states.write();
while let Some((_, state)) = write_lock.peek_lru() {
if state.slot().epoch(T::EthSpec::slots_per_epoch()) < cutoff_epoch {
write_lock.pop_lru();
} else {
break;
}
}
}
}
/// This can only be used during testing. The intended way to
/// obtain a `DietAvailabilityPendingExecutedBlock` is to call
/// `register_pending_executed_block` on the `StateLRUCache`.
#[cfg(test)]
impl<E: EthSpec> From<AvailabilityPendingExecutedBlock<E>>
for DietAvailabilityPendingExecutedBlock<E>
{
fn from(value: AvailabilityPendingExecutedBlock<E>) -> Self {
Self {
block: value.block,
state_root: value.import_data.state.canonical_root(),
parent_block: value.import_data.parent_block,
parent_eth1_finalization_data: value.import_data.parent_eth1_finalization_data,
confirmed_state_roots: value.import_data.confirmed_state_roots,
consensus_context: OnDiskConsensusContext::from_consensus_context(
value.import_data.consensus_context,
),
payload_verification_outcome: value.payload_verification_outcome,
}
}
}

View File

@@ -0,0 +1,121 @@
//! Provides tools for checking if a node is ready for the Deneb upgrade.
use crate::{BeaconChain, BeaconChainTypes};
use execution_layer::http::{
ENGINE_FORKCHOICE_UPDATED_V3, ENGINE_GET_PAYLOAD_V3, ENGINE_NEW_PAYLOAD_V3,
};
use serde::{Deserialize, Serialize};
use std::fmt;
use std::time::Duration;
use types::*;
/// The time before the Deneb fork when we will start issuing warnings about preparation.
use super::bellatrix_readiness::SECONDS_IN_A_WEEK;
pub const DENEB_READINESS_PREPARATION_SECONDS: u64 = SECONDS_IN_A_WEEK * 2;
pub const ENGINE_CAPABILITIES_REFRESH_INTERVAL: u64 = 300;
#[derive(Debug, Serialize, Deserialize)]
#[serde(rename_all = "snake_case")]
#[serde(tag = "type")]
pub enum DenebReadiness {
/// The execution engine is deneb-enabled (as far as we can tell)
Ready,
/// We are connected to an execution engine which doesn't support the V3 engine api methods
V3MethodsNotSupported { error: String },
/// The transition configuration with the EL failed, there might be a problem with
/// connectivity, authentication or a difference in configuration.
ExchangeCapabilitiesFailed { error: String },
/// The user has not configured an execution endpoint
NoExecutionEndpoint,
}
impl fmt::Display for DenebReadiness {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
match self {
DenebReadiness::Ready => {
write!(f, "This node appears ready for Deneb.")
}
DenebReadiness::ExchangeCapabilitiesFailed { error } => write!(
f,
"Could not exchange capabilities with the \
execution endpoint: {}",
error
),
DenebReadiness::NoExecutionEndpoint => write!(
f,
"The --execution-endpoint flag is not specified, this is a \
requirement post-merge"
),
DenebReadiness::V3MethodsNotSupported { error } => write!(
f,
"Execution endpoint does not support Deneb methods: {}",
error
),
}
}
}
impl<T: BeaconChainTypes> BeaconChain<T> {
/// Returns `true` if deneb epoch is set and Deneb fork has occurred or will
/// occur within `DENEB_READINESS_PREPARATION_SECONDS`
pub fn is_time_to_prepare_for_deneb(&self, current_slot: Slot) -> bool {
if let Some(deneb_epoch) = self.spec.deneb_fork_epoch {
let deneb_slot = deneb_epoch.start_slot(T::EthSpec::slots_per_epoch());
let deneb_readiness_preparation_slots =
DENEB_READINESS_PREPARATION_SECONDS / self.spec.seconds_per_slot;
// Return `true` if Deneb has happened or is within the preparation time.
current_slot + deneb_readiness_preparation_slots > deneb_slot
} else {
// The Deneb fork epoch has not been defined yet, no need to prepare.
false
}
}
/// Attempts to connect to the EL and confirm that it is ready for capella.
pub async fn check_deneb_readiness(&self) -> DenebReadiness {
if let Some(el) = self.execution_layer.as_ref() {
match el
.get_engine_capabilities(Some(Duration::from_secs(
ENGINE_CAPABILITIES_REFRESH_INTERVAL,
)))
.await
{
Err(e) => {
// The EL was either unreachable or responded with an error
DenebReadiness::ExchangeCapabilitiesFailed {
error: format!("{:?}", e),
}
}
Ok(capabilities) => {
let mut missing_methods = String::from("Required Methods Unsupported:");
let mut all_good = true;
if !capabilities.get_payload_v3 {
missing_methods.push(' ');
missing_methods.push_str(ENGINE_GET_PAYLOAD_V3);
all_good = false;
}
if !capabilities.forkchoice_updated_v3 {
missing_methods.push(' ');
missing_methods.push_str(ENGINE_FORKCHOICE_UPDATED_V3);
all_good = false;
}
if !capabilities.new_payload_v3 {
missing_methods.push(' ');
missing_methods.push_str(ENGINE_NEW_PAYLOAD_V3);
all_good = false;
}
if all_good {
DenebReadiness::Ready
} else {
DenebReadiness::V3MethodsNotSupported {
error: missing_methods,
}
}
}
}
} else {
DenebReadiness::NoExecutionEndpoint
}
}
}

View File

@@ -1,3 +1,4 @@
use crate::data_availability_checker::AvailableBlock;
use crate::{
attester_cache::{CommitteeLengths, Error},
metrics,
@@ -20,6 +21,7 @@ pub struct CacheItem<E: EthSpec> {
* Values used to make the block available.
*/
block: Arc<SignedBeaconBlock<E>>,
blobs: Option<BlobSidecarList<E>>,
proto_block: ProtoBlock,
}
@@ -49,7 +51,7 @@ impl<E: EthSpec> EarlyAttesterCache<E> {
pub fn add_head_block(
&self,
beacon_block_root: Hash256,
block: Arc<SignedBeaconBlock<E>>,
block: AvailableBlock<E>,
proto_block: ProtoBlock,
state: &BeaconState<E>,
spec: &ChainSpec,
@@ -67,6 +69,7 @@ impl<E: EthSpec> EarlyAttesterCache<E> {
},
};
let (_, block, blobs) = block.deconstruct();
let item = CacheItem {
epoch,
committee_lengths,
@@ -74,6 +77,7 @@ impl<E: EthSpec> EarlyAttesterCache<E> {
source,
target,
block,
blobs,
proto_block,
};
@@ -94,9 +98,7 @@ impl<E: EthSpec> EarlyAttesterCache<E> {
spec: &ChainSpec,
) -> Result<Option<Attestation<E>>, Error> {
let lock = self.item.read();
let item = if let Some(item) = lock.as_ref() {
item
} else {
let Some(item) = lock.as_ref() else {
return Ok(None);
};
@@ -120,18 +122,16 @@ impl<E: EthSpec> EarlyAttesterCache<E> {
item.committee_lengths
.get_committee_length::<E>(request_slot, request_index, spec)?;
let attestation = Attestation {
aggregation_bits: BitList::with_capacity(committee_len)
.map_err(BeaconStateError::from)?,
data: AttestationData {
slot: request_slot,
index: request_index,
beacon_block_root: item.beacon_block_root,
source: item.source,
target: item.target,
},
signature: AggregateSignature::empty(),
};
let attestation = Attestation::empty_for_signing(
request_index,
committee_len,
request_slot,
item.beacon_block_root,
item.source,
item.target,
spec,
)
.map_err(Error::AttestationError)?;
metrics::inc_counter(&metrics::BEACON_EARLY_ATTESTER_CACHE_HITS);
@@ -155,6 +155,15 @@ impl<E: EthSpec> EarlyAttesterCache<E> {
.map(|item| item.block.clone())
}
/// Returns the blobs, if `block_root` matches the cached item.
pub fn get_blobs(&self, block_root: Hash256) -> Option<BlobSidecarList<E>> {
self.item
.read()
.as_ref()
.filter(|item| item.beacon_block_root == block_root)
.and_then(|item| item.blobs.clone())
}
/// Returns the proto-array block, if `block_root` matches the cached item.
pub fn get_proto_block(&self, block_root: Hash256) -> Option<ProtoBlock> {
self.item

View File

@@ -0,0 +1,123 @@
//! Provides tools for checking if a node is ready for the Electra upgrade and following merge
//! transition.
use crate::{BeaconChain, BeaconChainTypes};
use execution_layer::http::{
ENGINE_FORKCHOICE_UPDATED_V3, ENGINE_GET_PAYLOAD_V3, ENGINE_NEW_PAYLOAD_V3,
};
use serde::{Deserialize, Serialize};
use std::fmt;
use std::time::Duration;
use types::*;
/// The time before the Electra fork when we will start issuing warnings about preparation.
use super::bellatrix_readiness::SECONDS_IN_A_WEEK;
pub const ELECTRA_READINESS_PREPARATION_SECONDS: u64 = SECONDS_IN_A_WEEK * 2;
pub const ENGINE_CAPABILITIES_REFRESH_INTERVAL: u64 = 300;
#[derive(Debug, Serialize, Deserialize)]
#[serde(rename_all = "snake_case")]
#[serde(tag = "type")]
pub enum ElectraReadiness {
/// The execution engine is electra-enabled (as far as we can tell)
Ready,
/// We are connected to an execution engine which doesn't support the V3 engine api methods
V3MethodsNotSupported { error: String },
/// The transition configuration with the EL failed, there might be a problem with
/// connectivity, authentication or a difference in configuration.
ExchangeCapabilitiesFailed { error: String },
/// The user has not configured an execution endpoint
NoExecutionEndpoint,
}
impl fmt::Display for ElectraReadiness {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
match self {
ElectraReadiness::Ready => {
write!(f, "This node appears ready for Electra.")
}
ElectraReadiness::ExchangeCapabilitiesFailed { error } => write!(
f,
"Could not exchange capabilities with the \
execution endpoint: {}",
error
),
ElectraReadiness::NoExecutionEndpoint => write!(
f,
"The --execution-endpoint flag is not specified, this is a \
requirement post-merge"
),
ElectraReadiness::V3MethodsNotSupported { error } => write!(
f,
"Execution endpoint does not support Electra methods: {}",
error
),
}
}
}
impl<T: BeaconChainTypes> BeaconChain<T> {
/// Returns `true` if electra epoch is set and Electra fork has occurred or will
/// occur within `ELECTRA_READINESS_PREPARATION_SECONDS`
pub fn is_time_to_prepare_for_electra(&self, current_slot: Slot) -> bool {
if let Some(electra_epoch) = self.spec.electra_fork_epoch {
let electra_slot = electra_epoch.start_slot(T::EthSpec::slots_per_epoch());
let electra_readiness_preparation_slots =
ELECTRA_READINESS_PREPARATION_SECONDS / self.spec.seconds_per_slot;
// Return `true` if Electra has happened or is within the preparation time.
current_slot + electra_readiness_preparation_slots > electra_slot
} else {
// The Electra fork epoch has not been defined yet, no need to prepare.
false
}
}
/// Attempts to connect to the EL and confirm that it is ready for electra.
pub async fn check_electra_readiness(&self) -> ElectraReadiness {
if let Some(el) = self.execution_layer.as_ref() {
match el
.get_engine_capabilities(Some(Duration::from_secs(
ENGINE_CAPABILITIES_REFRESH_INTERVAL,
)))
.await
{
Err(e) => {
// The EL was either unreachable or responded with an error
ElectraReadiness::ExchangeCapabilitiesFailed {
error: format!("{:?}", e),
}
}
Ok(capabilities) => {
// TODO(electra): Update in the event we get V4s.
let mut missing_methods = String::from("Required Methods Unsupported:");
let mut all_good = true;
if !capabilities.get_payload_v3 {
missing_methods.push(' ');
missing_methods.push_str(ENGINE_GET_PAYLOAD_V3);
all_good = false;
}
if !capabilities.forkchoice_updated_v3 {
missing_methods.push(' ');
missing_methods.push_str(ENGINE_FORKCHOICE_UPDATED_V3);
all_good = false;
}
if !capabilities.new_payload_v3 {
missing_methods.push(' ');
missing_methods.push_str(ENGINE_NEW_PAYLOAD_V3);
all_good = false;
}
if all_good {
ElectraReadiness::Ready
} else {
ElectraReadiness::V3MethodsNotSupported {
error: missing_methods,
}
}
}
}
} else {
ElectraReadiness::NoExecutionEndpoint
}
}
}

View File

@@ -2,12 +2,14 @@ use crate::attester_cache::Error as AttesterCacheError;
use crate::beacon_block_streamer::Error as BlockStreamerError;
use crate::beacon_chain::ForkChoiceError;
use crate::beacon_fork_choice_store::Error as ForkChoiceStoreError;
use crate::data_availability_checker::AvailabilityCheckError;
use crate::eth1_chain::Error as Eth1ChainError;
use crate::historical_blocks::HistoricalBlockError;
use crate::migrate::PruningError;
use crate::naive_aggregation_pool::Error as NaiveAggregationError;
use crate::observed_aggregates::Error as ObservedAttestationsError;
use crate::observed_attesters::Error as ObservedAttestersError;
use crate::observed_blob_sidecars::Error as ObservedBlobSidecarsError;
use crate::observed_block_producers::Error as ObservedBlockProducersError;
use execution_layer::PayloadStatus;
use fork_choice::ExecutionStatus;
@@ -29,6 +31,7 @@ use state_processing::{
use std::time::Duration;
use task_executor::ShutdownReason;
use tokio::task::JoinError;
use types::milhouse::Error as MilhouseError;
use types::*;
macro_rules! easy_from_to {
@@ -53,6 +56,7 @@ pub enum BeaconChainError {
SlotClockDidNotStart,
NoStateForSlot(Slot),
BeaconStateError(BeaconStateError),
EpochCacheError(EpochCacheError),
DBInconsistent(String),
DBError(store::Error),
ForkChoiceError(ForkChoiceError),
@@ -102,6 +106,7 @@ pub enum BeaconChainError {
ObservedAttestationsError(ObservedAttestationsError),
ObservedAttestersError(ObservedAttestersError),
ObservedBlockProducersError(ObservedBlockProducersError),
ObservedBlobSidecarsError(ObservedBlobSidecarsError),
AttesterCacheError(AttesterCacheError),
PruningError(PruningError),
ArithError(ArithError),
@@ -217,6 +222,12 @@ pub enum BeaconChainError {
InconsistentFork(InconsistentFork),
ProposerHeadForkChoiceError(fork_choice::Error<proto_array::Error>),
UnableToPublish,
AvailabilityCheckError(AvailabilityCheckError),
LightClientError(LightClientError),
UnsupportedFork,
MilhouseError(MilhouseError),
AttestationError(AttestationError),
AttestationCommitteeIndexNotSet,
}
easy_from_to!(SlotProcessingError, BeaconChainError);
@@ -233,6 +244,7 @@ easy_from_to!(NaiveAggregationError, BeaconChainError);
easy_from_to!(ObservedAttestationsError, BeaconChainError);
easy_from_to!(ObservedAttestersError, BeaconChainError);
easy_from_to!(ObservedBlockProducersError, BeaconChainError);
easy_from_to!(ObservedBlobSidecarsError, BeaconChainError);
easy_from_to!(AttesterCacheError, BeaconChainError);
easy_from_to!(BlockSignatureVerifierError, BeaconChainError);
easy_from_to!(PruningError, BeaconChainError);
@@ -242,6 +254,11 @@ easy_from_to!(HistoricalBlockError, BeaconChainError);
easy_from_to!(StateAdvanceError, BeaconChainError);
easy_from_to!(BlockReplayError, BeaconChainError);
easy_from_to!(InconsistentFork, BeaconChainError);
easy_from_to!(AvailabilityCheckError, BeaconChainError);
easy_from_to!(EpochCacheError, BeaconChainError);
easy_from_to!(LightClientError, BeaconChainError);
easy_from_to!(MilhouseError, BeaconChainError);
easy_from_to!(AttestationError, BeaconChainError);
#[derive(Debug)]
pub enum BlockProductionError {
@@ -250,6 +267,7 @@ pub enum BlockProductionError {
UnableToProduceAtSlot(Slot),
SlotProcessingError(SlotProcessingError),
BlockProcessingError(BlockProcessingError),
EpochCacheError(EpochCacheError),
ForkChoiceError(ForkChoiceError),
Eth1ChainError(Eth1ChainError),
BeaconStateError(BeaconStateError),
@@ -267,14 +285,21 @@ pub enum BlockProductionError {
TerminalPoWBlockLookupFailed(execution_layer::Error),
GetPayloadFailed(execution_layer::Error),
FailedToReadFinalizedBlock(store::Error),
FailedToLoadState(store::Error),
MissingFinalizedBlock(Hash256),
BlockTooLarge(usize),
ShuttingDown,
MissingBlobs,
MissingSyncAggregate,
MissingExecutionPayload,
TokioJoin(tokio::task::JoinError),
MissingKzgCommitment(String),
TokioJoin(JoinError),
BeaconChain(BeaconChainError),
InvalidPayloadFork,
TrustedSetupNotInitialized,
InvalidBlockVariant(String),
KzgError(kzg::Error),
FailedToBuildBlobSidecars(String),
}
easy_from_to!(BlockProcessingError, BlockProductionError);
@@ -283,3 +308,4 @@ easy_from_to!(SlotProcessingError, BlockProductionError);
easy_from_to!(Eth1ChainError, BlockProductionError);
easy_from_to!(StateAdvanceError, BlockProductionError);
easy_from_to!(ForkChoiceError, BlockProductionError);
easy_from_to!(EpochCacheError, BlockProductionError);

View File

@@ -9,7 +9,6 @@ use ssz_derive::{Decode, Encode};
use state_processing::per_block_processing::get_new_eth1_data;
use std::cmp::Ordering;
use std::collections::HashMap;
use std::iter::DoubleEndedIterator;
use std::marker::PhantomData;
use std::time::{SystemTime, UNIX_EPOCH};
use store::{DBColumn, Error as StoreError, StoreItem};
@@ -67,7 +66,7 @@ impl From<safe_arith::ArithError> for Error {
/// - `genesis_time`: beacon chain genesis time.
/// - `current_slot`: current beacon chain slot.
/// - `spec`: current beacon chain specification.
fn get_sync_status<T: EthSpec>(
fn get_sync_status<E: EthSpec>(
latest_cached_block: Option<&Eth1Block>,
head_block: Option<&Eth1Block>,
genesis_time: u64,
@@ -85,7 +84,7 @@ fn get_sync_status<T: EthSpec>(
// that are *before* genesis, so that we can indicate to users that we're actually adequately
// cached for where they are in time.
let voting_target_timestamp = if let Some(current_slot) = current_slot {
let period = T::SlotsPerEth1VotingPeriod::to_u64();
let period = E::SlotsPerEth1VotingPeriod::to_u64();
let voting_period_start_slot = (current_slot / period) * period;
let period_start = slot_start_seconds(
@@ -98,7 +97,7 @@ fn get_sync_status<T: EthSpec>(
} else {
// The number of seconds in an eth1 voting period.
let voting_period_duration =
T::slots_per_eth1_voting_period() as u64 * spec.seconds_per_slot;
E::slots_per_eth1_voting_period() as u64 * spec.seconds_per_slot;
let now = SystemTime::now().duration_since(UNIX_EPOCH).ok()?.as_secs();
@@ -316,10 +315,10 @@ where
}
}
pub trait Eth1ChainBackend<T: EthSpec>: Sized + Send + Sync {
pub trait Eth1ChainBackend<E: EthSpec>: Sized + Send + Sync {
/// Returns the `Eth1Data` that should be included in a block being produced for the given
/// `state`.
fn eth1_data(&self, beacon_state: &BeaconState<T>, spec: &ChainSpec)
fn eth1_data(&self, beacon_state: &BeaconState<E>, spec: &ChainSpec)
-> Result<Eth1Data, Error>;
/// Returns all `Deposits` between `state.eth1_deposit_index` and
@@ -331,7 +330,7 @@ pub trait Eth1ChainBackend<T: EthSpec>: Sized + Send + Sync {
/// be more than `MAX_DEPOSIT_COUNT` or the churn may be too high.
fn queued_deposits(
&self,
beacon_state: &BeaconState<T>,
beacon_state: &BeaconState<E>,
eth1_data_vote: &Eth1Data,
spec: &ChainSpec,
) -> Result<Vec<Deposit>, Error>;
@@ -365,13 +364,13 @@ pub trait Eth1ChainBackend<T: EthSpec>: Sized + Send + Sync {
/// Never creates deposits, therefore the validator set is static.
///
/// This was used in the 2019 Canada interop workshops.
pub struct DummyEth1ChainBackend<T: EthSpec>(PhantomData<T>);
pub struct DummyEth1ChainBackend<E: EthSpec>(PhantomData<E>);
impl<T: EthSpec> Eth1ChainBackend<T> for DummyEth1ChainBackend<T> {
impl<E: EthSpec> Eth1ChainBackend<E> for DummyEth1ChainBackend<E> {
/// Produce some deterministic junk based upon the current epoch.
fn eth1_data(&self, state: &BeaconState<T>, _spec: &ChainSpec) -> Result<Eth1Data, Error> {
fn eth1_data(&self, state: &BeaconState<E>, _spec: &ChainSpec) -> Result<Eth1Data, Error> {
let current_epoch = state.current_epoch();
let slots_per_voting_period = T::slots_per_eth1_voting_period() as u64;
let slots_per_voting_period = E::slots_per_eth1_voting_period() as u64;
let current_voting_period: u64 = current_epoch.as_u64() / slots_per_voting_period;
let deposit_root = hash(&int_to_bytes32(current_voting_period));
@@ -387,7 +386,7 @@ impl<T: EthSpec> Eth1ChainBackend<T> for DummyEth1ChainBackend<T> {
/// The dummy back-end never produces deposits.
fn queued_deposits(
&self,
_: &BeaconState<T>,
_: &BeaconState<E>,
_: &Eth1Data,
_: &ChainSpec,
) -> Result<Vec<Deposit>, Error> {
@@ -420,7 +419,7 @@ impl<T: EthSpec> Eth1ChainBackend<T> for DummyEth1ChainBackend<T> {
}
}
impl<T: EthSpec> Default for DummyEth1ChainBackend<T> {
impl<E: EthSpec> Default for DummyEth1ChainBackend<E> {
fn default() -> Self {
Self(PhantomData)
}
@@ -432,13 +431,13 @@ impl<T: EthSpec> Default for DummyEth1ChainBackend<T> {
/// The `core` connects to some external eth1 client (e.g., Parity/Geth) and polls it for
/// information.
#[derive(Clone)]
pub struct CachingEth1Backend<T: EthSpec> {
pub struct CachingEth1Backend<E: EthSpec> {
pub core: HttpService,
log: Logger,
_phantom: PhantomData<T>,
_phantom: PhantomData<E>,
}
impl<T: EthSpec> CachingEth1Backend<T> {
impl<E: EthSpec> CachingEth1Backend<E> {
/// Instantiates `self` with empty caches.
///
/// Does not connect to the eth1 node or start any tasks to keep the cache updated.
@@ -466,9 +465,9 @@ impl<T: EthSpec> CachingEth1Backend<T> {
}
}
impl<T: EthSpec> Eth1ChainBackend<T> for CachingEth1Backend<T> {
fn eth1_data(&self, state: &BeaconState<T>, spec: &ChainSpec) -> Result<Eth1Data, Error> {
let period = T::SlotsPerEth1VotingPeriod::to_u64();
impl<E: EthSpec> Eth1ChainBackend<E> for CachingEth1Backend<E> {
fn eth1_data(&self, state: &BeaconState<E>, spec: &ChainSpec) -> Result<Eth1Data, Error> {
let period = E::SlotsPerEth1VotingPeriod::to_u64();
let voting_period_start_slot = (state.slot() / period) * period;
let voting_period_start_seconds = slot_start_seconds(
state.genesis_time(),
@@ -536,7 +535,7 @@ impl<T: EthSpec> Eth1ChainBackend<T> for CachingEth1Backend<T> {
fn queued_deposits(
&self,
state: &BeaconState<T>,
state: &BeaconState<E>,
eth1_data_vote: &Eth1Data,
_spec: &ChainSpec,
) -> Result<Vec<Deposit>, Error> {
@@ -552,7 +551,7 @@ impl<T: EthSpec> Eth1ChainBackend<T> for CachingEth1Backend<T> {
Ordering::Equal => Ok(vec![]),
Ordering::Less => {
let next = deposit_index;
let last = std::cmp::min(deposit_count, next + T::MaxDeposits::to_u64());
let last = std::cmp::min(deposit_count, next + E::MaxDeposits::to_u64());
self.core
.deposits()
@@ -627,8 +626,8 @@ where
/// Collect all valid votes that are cast during the current voting period.
/// Return hashmap with count of each vote cast.
fn collect_valid_votes<T: EthSpec>(
state: &BeaconState<T>,
fn collect_valid_votes<E: EthSpec>(
state: &BeaconState<E>,
votes_to_consider: &HashMap<Eth1Data, BlockNumber>,
) -> Eth1DataVoteCount {
let mut valid_votes = HashMap::new();
@@ -686,7 +685,7 @@ mod test {
fn get_eth1_data(i: u64) -> Eth1Data {
Eth1Data {
block_hash: Hash256::from_low_u64_be(i),
deposit_root: Hash256::from_low_u64_be(u64::max_value() - i),
deposit_root: Hash256::from_low_u64_be(u64::MAX - i),
deposit_count: i,
}
}
@@ -736,7 +735,7 @@ mod test {
mod eth1_chain_json_backend {
use super::*;
use eth1::DepositLog;
use types::{test_utils::generate_deterministic_keypair, EthSpec, MainnetEthSpec};
use types::{test_utils::generate_deterministic_keypair, MainnetEthSpec};
fn get_eth1_chain() -> Eth1Chain<CachingEth1Backend<E>, E> {
let eth1_config = Eth1Config {
@@ -967,7 +966,7 @@ mod test {
let spec = &E::default_spec();
let state: BeaconState<E> = BeaconState::new(0, get_eth1_data(0), spec);
let blocks = vec![];
let blocks = [];
assert_eq!(
get_votes_to_consider(
@@ -1021,6 +1020,7 @@ mod test {
mod collect_valid_votes {
use super::*;
use types::List;
fn get_eth1_data_vec(n: u64, block_number_offset: u64) -> Vec<(Eth1Data, BlockNumber)> {
(0..n)
@@ -1068,12 +1068,14 @@ mod test {
let votes_to_consider = get_eth1_data_vec(slots, 0);
*state.eth1_data_votes_mut() = votes_to_consider[0..slots as usize / 4]
.iter()
.map(|(eth1_data, _)| eth1_data)
.cloned()
.collect::<Vec<_>>()
.into();
*state.eth1_data_votes_mut() = List::new(
votes_to_consider[0..slots as usize / 4]
.iter()
.map(|(eth1_data, _)| eth1_data)
.cloned()
.collect::<Vec<_>>(),
)
.unwrap();
let votes =
collect_valid_votes(&state, &votes_to_consider.clone().into_iter().collect());
@@ -1097,12 +1099,14 @@ mod test {
.expect("should have some eth1 data")
.clone();
*state.eth1_data_votes_mut() = vec![duplicate_eth1_data.clone(); 4]
.iter()
.map(|(eth1_data, _)| eth1_data)
.cloned()
.collect::<Vec<_>>()
.into();
*state.eth1_data_votes_mut() = List::new(
vec![duplicate_eth1_data.clone(); 4]
.iter()
.map(|(eth1_data, _)| eth1_data)
.cloned()
.collect::<Vec<_>>(),
)
.unwrap();
let votes = collect_valid_votes(&state, &votes_to_consider.into_iter().collect());
assert_votes!(

View File

@@ -1,4 +1,5 @@
use slog::{debug, Logger};
use ssz_derive::{Decode, Encode};
use std::cmp;
use std::collections::BTreeMap;
use types::{Checkpoint, Epoch, Eth1Data, Hash256 as Root};
@@ -10,7 +11,7 @@ pub const DEFAULT_ETH1_CACHE_SIZE: usize = 5;
/// These fields are named the same as the corresponding fields in the `BeaconState`
/// as this structure stores these values from the `BeaconState` at a `Checkpoint`
#[derive(Clone)]
#[derive(Clone, Debug, PartialEq, Encode, Decode)]
pub struct Eth1FinalizationData {
pub eth1_data: Eth1Data,
pub eth1_deposit_index: u64,
@@ -66,7 +67,7 @@ impl CheckpointMap {
pub fn insert(&mut self, checkpoint: Checkpoint, eth1_finalization_data: Eth1FinalizationData) {
self.store
.entry(checkpoint.epoch)
.or_insert_with(Vec::new)
.or_default()
.push((checkpoint.root, eth1_finalization_data));
// faster to reduce size after the fact than do pre-checking to see

View File

@@ -6,21 +6,27 @@ use types::EthSpec;
const DEFAULT_CHANNEL_CAPACITY: usize = 16;
pub struct ServerSentEventHandler<T: EthSpec> {
attestation_tx: Sender<EventKind<T>>,
block_tx: Sender<EventKind<T>>,
finalized_tx: Sender<EventKind<T>>,
head_tx: Sender<EventKind<T>>,
exit_tx: Sender<EventKind<T>>,
chain_reorg_tx: Sender<EventKind<T>>,
contribution_tx: Sender<EventKind<T>>,
payload_attributes_tx: Sender<EventKind<T>>,
late_head: Sender<EventKind<T>>,
block_reward_tx: Sender<EventKind<T>>,
pub struct ServerSentEventHandler<E: EthSpec> {
attestation_tx: Sender<EventKind<E>>,
block_tx: Sender<EventKind<E>>,
blob_sidecar_tx: Sender<EventKind<E>>,
finalized_tx: Sender<EventKind<E>>,
head_tx: Sender<EventKind<E>>,
exit_tx: Sender<EventKind<E>>,
chain_reorg_tx: Sender<EventKind<E>>,
contribution_tx: Sender<EventKind<E>>,
payload_attributes_tx: Sender<EventKind<E>>,
late_head: Sender<EventKind<E>>,
light_client_finality_update_tx: Sender<EventKind<E>>,
light_client_optimistic_update_tx: Sender<EventKind<E>>,
block_reward_tx: Sender<EventKind<E>>,
proposer_slashing_tx: Sender<EventKind<E>>,
attester_slashing_tx: Sender<EventKind<E>>,
bls_to_execution_change_tx: Sender<EventKind<E>>,
log: Logger,
}
impl<T: EthSpec> ServerSentEventHandler<T> {
impl<E: EthSpec> ServerSentEventHandler<E> {
pub fn new(log: Logger, capacity_multiplier: usize) -> Self {
Self::new_with_capacity(
log,
@@ -31,6 +37,7 @@ impl<T: EthSpec> ServerSentEventHandler<T> {
pub fn new_with_capacity(log: Logger, capacity: usize) -> Self {
let (attestation_tx, _) = broadcast::channel(capacity);
let (block_tx, _) = broadcast::channel(capacity);
let (blob_sidecar_tx, _) = broadcast::channel(capacity);
let (finalized_tx, _) = broadcast::channel(capacity);
let (head_tx, _) = broadcast::channel(capacity);
let (exit_tx, _) = broadcast::channel(capacity);
@@ -38,11 +45,17 @@ impl<T: EthSpec> ServerSentEventHandler<T> {
let (contribution_tx, _) = broadcast::channel(capacity);
let (payload_attributes_tx, _) = broadcast::channel(capacity);
let (late_head, _) = broadcast::channel(capacity);
let (light_client_finality_update_tx, _) = broadcast::channel(capacity);
let (light_client_optimistic_update_tx, _) = broadcast::channel(capacity);
let (block_reward_tx, _) = broadcast::channel(capacity);
let (proposer_slashing_tx, _) = broadcast::channel(capacity);
let (attester_slashing_tx, _) = broadcast::channel(capacity);
let (bls_to_execution_change_tx, _) = broadcast::channel(capacity);
Self {
attestation_tx,
block_tx,
blob_sidecar_tx,
finalized_tx,
head_tx,
exit_tx,
@@ -50,12 +63,17 @@ impl<T: EthSpec> ServerSentEventHandler<T> {
contribution_tx,
payload_attributes_tx,
late_head,
light_client_finality_update_tx,
light_client_optimistic_update_tx,
block_reward_tx,
proposer_slashing_tx,
attester_slashing_tx,
bls_to_execution_change_tx,
log,
}
}
pub fn register(&self, kind: EventKind<T>) {
pub fn register(&self, kind: EventKind<E>) {
let log_count = |name, count| {
trace!(
self.log,
@@ -73,6 +91,10 @@ impl<T: EthSpec> ServerSentEventHandler<T> {
.block_tx
.send(kind)
.map(|count| log_count("block", count)),
EventKind::BlobSidecar(_) => self
.blob_sidecar_tx
.send(kind)
.map(|count| log_count("blob sidecar", count)),
EventKind::FinalizedCheckpoint(_) => self
.finalized_tx
.send(kind)
@@ -101,56 +123,100 @@ impl<T: EthSpec> ServerSentEventHandler<T> {
.late_head
.send(kind)
.map(|count| log_count("late head", count)),
EventKind::LightClientFinalityUpdate(_) => self
.light_client_finality_update_tx
.send(kind)
.map(|count| log_count("light client finality update", count)),
EventKind::LightClientOptimisticUpdate(_) => self
.light_client_optimistic_update_tx
.send(kind)
.map(|count| log_count("light client optimistic update", count)),
EventKind::BlockReward(_) => self
.block_reward_tx
.send(kind)
.map(|count| log_count("block reward", count)),
EventKind::ProposerSlashing(_) => self
.proposer_slashing_tx
.send(kind)
.map(|count| log_count("proposer slashing", count)),
EventKind::AttesterSlashing(_) => self
.attester_slashing_tx
.send(kind)
.map(|count| log_count("attester slashing", count)),
EventKind::BlsToExecutionChange(_) => self
.bls_to_execution_change_tx
.send(kind)
.map(|count| log_count("bls to execution change", count)),
};
if let Err(SendError(event)) = result {
trace!(self.log, "No receivers registered to listen for event"; "event" => ?event);
}
}
pub fn subscribe_attestation(&self) -> Receiver<EventKind<T>> {
pub fn subscribe_attestation(&self) -> Receiver<EventKind<E>> {
self.attestation_tx.subscribe()
}
pub fn subscribe_block(&self) -> Receiver<EventKind<T>> {
pub fn subscribe_block(&self) -> Receiver<EventKind<E>> {
self.block_tx.subscribe()
}
pub fn subscribe_finalized(&self) -> Receiver<EventKind<T>> {
pub fn subscribe_blob_sidecar(&self) -> Receiver<EventKind<E>> {
self.blob_sidecar_tx.subscribe()
}
pub fn subscribe_finalized(&self) -> Receiver<EventKind<E>> {
self.finalized_tx.subscribe()
}
pub fn subscribe_head(&self) -> Receiver<EventKind<T>> {
pub fn subscribe_head(&self) -> Receiver<EventKind<E>> {
self.head_tx.subscribe()
}
pub fn subscribe_exit(&self) -> Receiver<EventKind<T>> {
pub fn subscribe_exit(&self) -> Receiver<EventKind<E>> {
self.exit_tx.subscribe()
}
pub fn subscribe_reorgs(&self) -> Receiver<EventKind<T>> {
pub fn subscribe_reorgs(&self) -> Receiver<EventKind<E>> {
self.chain_reorg_tx.subscribe()
}
pub fn subscribe_contributions(&self) -> Receiver<EventKind<T>> {
pub fn subscribe_contributions(&self) -> Receiver<EventKind<E>> {
self.contribution_tx.subscribe()
}
pub fn subscribe_payload_attributes(&self) -> Receiver<EventKind<T>> {
pub fn subscribe_payload_attributes(&self) -> Receiver<EventKind<E>> {
self.payload_attributes_tx.subscribe()
}
pub fn subscribe_late_head(&self) -> Receiver<EventKind<T>> {
pub fn subscribe_late_head(&self) -> Receiver<EventKind<E>> {
self.late_head.subscribe()
}
pub fn subscribe_block_reward(&self) -> Receiver<EventKind<T>> {
pub fn subscribe_light_client_finality_update(&self) -> Receiver<EventKind<E>> {
self.light_client_finality_update_tx.subscribe()
}
pub fn subscribe_light_client_optimistic_update(&self) -> Receiver<EventKind<E>> {
self.light_client_optimistic_update_tx.subscribe()
}
pub fn subscribe_block_reward(&self) -> Receiver<EventKind<E>> {
self.block_reward_tx.subscribe()
}
pub fn subscribe_attester_slashing(&self) -> Receiver<EventKind<E>> {
self.attester_slashing_tx.subscribe()
}
pub fn subscribe_proposer_slashing(&self) -> Receiver<EventKind<E>> {
self.proposer_slashing_tx.subscribe()
}
pub fn subscribe_bls_to_execution_change(&self) -> Receiver<EventKind<E>> {
self.bls_to_execution_change_tx.subscribe()
}
pub fn has_attestation_subscribers(&self) -> bool {
self.attestation_tx.receiver_count() > 0
}
@@ -159,6 +225,10 @@ impl<T: EthSpec> ServerSentEventHandler<T> {
self.block_tx.receiver_count() > 0
}
pub fn has_blob_sidecar_subscribers(&self) -> bool {
self.blob_sidecar_tx.receiver_count() > 0
}
pub fn has_finalized_subscribers(&self) -> bool {
self.finalized_tx.receiver_count() > 0
}
@@ -190,4 +260,16 @@ impl<T: EthSpec> ServerSentEventHandler<T> {
pub fn has_block_reward_subscribers(&self) -> bool {
self.block_reward_tx.receiver_count() > 0
}
pub fn has_proposer_slashing_subscribers(&self) -> bool {
self.proposer_slashing_tx.receiver_count() > 0
}
pub fn has_attester_slashing_subscribers(&self) -> bool {
self.attester_slashing_tx.receiver_count() > 0
}
pub fn has_bls_to_execution_change_subscribers(&self) -> bool {
self.bls_to_execution_change_tx.receiver_count() > 0
}
}

View File

@@ -12,7 +12,10 @@ use crate::{
BeaconChain, BeaconChainError, BeaconChainTypes, BlockError, BlockProductionError,
ExecutionPayloadError,
};
use execution_layer::{BlockProposalContents, BuilderParams, PayloadAttributes, PayloadStatus};
use execution_layer::{
BlockProposalContents, BlockProposalContentsType, BuilderParams, NewPayloadRequest,
PayloadAttributes, PayloadStatus,
};
use fork_choice::{InvalidationOperation, PayloadVerificationStatus};
use proto_array::{Block as ProtoBlock, ExecutionStatus};
use slog::{debug, warn};
@@ -24,11 +27,11 @@ use state_processing::per_block_processing::{
use std::sync::Arc;
use tokio::task::JoinHandle;
use tree_hash::TreeHash;
use types::payload::BlockProductionVersion;
use types::*;
pub type PreparePayloadResult<E, Payload> =
Result<BlockProposalContents<E, Payload>, BlockProductionError>;
pub type PreparePayloadHandle<E, Payload> = JoinHandle<Option<PreparePayloadResult<E, Payload>>>;
pub type PreparePayloadResult<E> = Result<BlockProposalContentsType<E>, BlockProductionError>;
pub type PreparePayloadHandle<E> = JoinHandle<Option<PreparePayloadResult<E>>>;
#[derive(PartialEq)]
pub enum AllowOptimisticImport {
@@ -68,31 +71,24 @@ impl<T: BeaconChainTypes> PayloadNotifier<T> {
// the block as optimistically imported. This is particularly relevant in the case
// where we do not send the block to the EL at all.
let block_message = block.message();
let payload = block_message.execution_payload()?;
partially_verify_execution_payload::<_, FullPayload<_>>(
state,
block.slot(),
payload,
block_message.body(),
&chain.spec,
)
.map_err(BlockError::PerBlockProcessingError)?;
match notify_execution_layer {
NotifyExecutionLayer::No if chain.config.optimistic_finalized_sync => {
// Verify the block hash here in Lighthouse and immediately mark the block as
// optimistically imported. This saves a lot of roundtrips to the EL.
let execution_layer = chain
.execution_layer
.as_ref()
.ok_or(ExecutionPayloadError::NoExecutionConnection)?;
if let Err(e) =
execution_layer.verify_payload_block_hash(payload.execution_payload_ref())
{
// Create a NewPayloadRequest (no clones required) and check optimistic sync verifications
let new_payload_request: NewPayloadRequest<T::EthSpec> =
block_message.try_into()?;
if let Err(e) = new_payload_request.perform_optimistic_sync_verifications() {
warn!(
chain.log,
"Falling back to slow block hash verification";
"block_number" => payload.block_number(),
"block_number" => ?block_message.execution_payload().map(|payload| payload.block_number()),
"info" => "you can silence this warning with --disable-optimistic-finalized-sync",
"error" => ?e,
);
@@ -138,16 +134,13 @@ async fn notify_new_payload<'a, T: BeaconChainTypes>(
chain: &Arc<BeaconChain<T>>,
block: BeaconBlockRef<'a, T::EthSpec>,
) -> Result<PayloadVerificationStatus, BlockError<T::EthSpec>> {
let execution_payload = block.execution_payload()?;
let execution_layer = chain
.execution_layer
.as_ref()
.ok_or(ExecutionPayloadError::NoExecutionConnection)?;
let new_payload_response = execution_layer
.notify_new_payload(&execution_payload.into())
.await;
let execution_block_hash = block.execution_payload()?.block_hash();
let new_payload_response = execution_layer.notify_new_payload(block.try_into()?).await;
match new_payload_response {
Ok(status) => match status {
@@ -164,7 +157,7 @@ async fn notify_new_payload<'a, T: BeaconChainTypes>(
"Invalid execution payload";
"validation_error" => ?validation_error,
"latest_valid_hash" => ?latest_valid_hash,
"execution_block_hash" => ?execution_payload.block_hash(),
"execution_block_hash" => ?execution_block_hash,
"root" => ?block.tree_hash_root(),
"graffiti" => block.body().graffiti().as_utf8_lossy(),
"proposer_index" => block.proposer_index(),
@@ -210,7 +203,7 @@ async fn notify_new_payload<'a, T: BeaconChainTypes>(
chain.log,
"Invalid execution payload block hash";
"validation_error" => ?validation_error,
"execution_block_hash" => ?execution_payload.block_hash(),
"execution_block_hash" => ?execution_block_hash,
"root" => ?block.tree_hash_root(),
"graffiti" => block.body().graffiti().as_utf8_lossy(),
"proposer_index" => block.proposer_index(),
@@ -343,7 +336,7 @@ pub fn validate_execution_payload_for_gossip<T: BeaconChainTypes>(
block: BeaconBlockRef<'_, T::EthSpec>,
chain: &BeaconChain<T>,
) -> Result<(), BlockError<T::EthSpec>> {
// Only apply this validation if this is a merge beacon block.
// Only apply this validation if this is a Bellatrix beacon block.
if let Ok(execution_payload) = block.body().execution_payload() {
// This logic should match `is_execution_enabled`. We use only the execution block hash of
// the parent here in order to avoid loading the parent state during gossip verification.
@@ -392,22 +385,22 @@ pub fn validate_execution_payload_for_gossip<T: BeaconChainTypes>(
/// ## Errors
///
/// Will return an error when using a pre-merge fork `state`. Ensure to only run this function
/// after the merge fork.
/// after the Bellatrix fork.
///
/// ## Specification
///
/// Equivalent to the `get_execution_payload` function in the Validator Guide:
///
/// https://github.com/ethereum/consensus-specs/blob/v1.1.5/specs/merge/validator.md#block-proposal
pub fn get_execution_payload<
T: BeaconChainTypes,
Payload: AbstractExecPayload<T::EthSpec> + 'static,
>(
pub fn get_execution_payload<T: BeaconChainTypes>(
chain: Arc<BeaconChain<T>>,
state: &BeaconState<T::EthSpec>,
parent_block_root: Hash256,
proposer_index: u64,
builder_params: BuilderParams,
) -> Result<PreparePayloadHandle<T::EthSpec, Payload>, BlockProductionError> {
builder_boost_factor: Option<u64>,
block_production_version: BlockProductionVersion,
) -> Result<PreparePayloadHandle<T::EthSpec>, BlockProductionError> {
// Compute all required values from the `state` now to avoid needing to pass it into a spawned
// task.
let spec = &chain.spec;
@@ -419,11 +412,19 @@ pub fn get_execution_payload<
let latest_execution_payload_header_block_hash =
state.latest_execution_payload_header()?.block_hash();
let withdrawals = match state {
&BeaconState::Capella(_) => Some(get_expected_withdrawals(state, spec)?.into()),
&BeaconState::Merge(_) => None,
&BeaconState::Capella(_) | &BeaconState::Deneb(_) | &BeaconState::Electra(_) => {
Some(get_expected_withdrawals(state, spec)?.into())
}
&BeaconState::Bellatrix(_) => None,
// These shouldn't happen but they're here to make the pattern irrefutable
&BeaconState::Base(_) | &BeaconState::Altair(_) => None,
};
let parent_beacon_block_root = match state {
BeaconState::Deneb(_) | BeaconState::Electra(_) => Some(parent_block_root),
BeaconState::Bellatrix(_) | BeaconState::Capella(_) => None,
// These shouldn't happen but they're here to make the pattern irrefutable
BeaconState::Base(_) | BeaconState::Altair(_) => None,
};
// Spawn a task to obtain the execution payload from the EL via a series of async calls. The
// `join_handle` can be used to await the result of the function.
@@ -432,7 +433,7 @@ pub fn get_execution_payload<
.clone()
.spawn_handle(
async move {
prepare_execution_payload::<T, Payload>(
prepare_execution_payload::<T>(
&chain,
is_merge_transition_complete,
timestamp,
@@ -441,6 +442,9 @@ pub fn get_execution_payload<
latest_execution_payload_header_block_hash,
builder_params,
withdrawals,
parent_beacon_block_root,
builder_boost_factor,
block_production_version,
)
.await
},
@@ -453,12 +457,12 @@ pub fn get_execution_payload<
/// Prepares an execution payload for inclusion in a block.
///
/// Will return `Ok(None)` if the merge fork has occurred, but a terminal block has not been found.
/// Will return `Ok(None)` if the Bellatrix fork has occurred, but a terminal block has not been found.
///
/// ## Errors
///
/// Will return an error when using a pre-merge fork `state`. Ensure to only run this function
/// after the merge fork.
/// Will return an error when using a pre-Bellatrix fork `state`. Ensure to only run this function
/// after the Bellatrix fork.
///
/// ## Specification
///
@@ -466,7 +470,7 @@ pub fn get_execution_payload<
///
/// https://github.com/ethereum/consensus-specs/blob/v1.1.5/specs/merge/validator.md#block-proposal
#[allow(clippy::too_many_arguments)]
pub async fn prepare_execution_payload<T, Payload>(
pub async fn prepare_execution_payload<T>(
chain: &Arc<BeaconChain<T>>,
is_merge_transition_complete: bool,
timestamp: u64,
@@ -475,10 +479,12 @@ pub async fn prepare_execution_payload<T, Payload>(
latest_execution_payload_header_block_hash: ExecutionBlockHash,
builder_params: BuilderParams,
withdrawals: Option<Vec<Withdrawal>>,
) -> Result<BlockProposalContents<T::EthSpec, Payload>, BlockProductionError>
parent_beacon_block_root: Option<Hash256>,
builder_boost_factor: Option<u64>,
block_production_version: BlockProductionVersion,
) -> Result<BlockProposalContentsType<T::EthSpec>, BlockProductionError>
where
T: BeaconChainTypes,
Payload: AbstractExecPayload<T::EthSpec>,
{
let current_epoch = builder_params.slot.epoch(T::EthSpec::slots_per_epoch());
let spec = &chain.spec;
@@ -496,7 +502,12 @@ where
if is_terminal_block_hash_set && !is_activation_epoch_reached {
// Use the "empty" payload if there's a terminal block hash, but we haven't reached the
// terminal block epoch yet.
return BlockProposalContents::default_at_fork(fork).map_err(Into::into);
return Ok(BlockProposalContentsType::Full(
BlockProposalContents::Payload {
payload: FullPayload::default_at_fork(fork)?,
block_value: Uint256::zero(),
},
));
}
let terminal_pow_block_hash = execution_layer
@@ -509,7 +520,12 @@ where
} else {
// If the merge transition hasn't occurred yet and the EL hasn't found the terminal
// block, return an "empty" payload.
return BlockProposalContents::default_at_fork(fork).map_err(Into::into);
return Ok(BlockProposalContentsType::Full(
BlockProposalContents::Payload {
payload: FullPayload::default_at_fork(fork)?,
block_value: Uint256::zero(),
},
));
}
} else {
latest_execution_payload_header_block_hash
@@ -536,20 +552,24 @@ where
let suggested_fee_recipient = execution_layer
.get_suggested_fee_recipient(proposer_index)
.await;
let payload_attributes =
PayloadAttributes::new(timestamp, random, suggested_fee_recipient, withdrawals);
let payload_attributes = PayloadAttributes::new(
timestamp,
random,
suggested_fee_recipient,
withdrawals,
parent_beacon_block_root,
);
// Note: the suggested_fee_recipient is stored in the `execution_layer`, it will add this parameter.
//
// This future is not executed here, it's up to the caller to await it.
let block_contents = execution_layer
.get_payload::<Payload>(
.get_payload(
parent_hash,
&payload_attributes,
forkchoice_update_params,
builder_params,
fork,
&chain.spec,
builder_boost_factor,
block_production_version,
)
.await
.map_err(BlockProductionError::GetPayloadFailed)?;

View File

@@ -5,15 +5,12 @@ use slog::{info, warn, Logger};
use state_processing::state_advance::complete_state_advance;
use state_processing::{
per_block_processing, per_block_processing::BlockSignatureStrategy, ConsensusContext,
StateProcessingStrategy, VerifyBlockRoot,
VerifyBlockRoot,
};
use std::sync::Arc;
use std::time::Duration;
use store::{iter::ParentRootBlockIterator, HotColdDB, ItemStore};
use types::{
BeaconState, ChainSpec, EthSpec, ForkName, Hash256, ProgressiveBalancesMode, SignedBeaconBlock,
Slot,
};
use types::{BeaconState, ChainSpec, EthSpec, ForkName, Hash256, SignedBeaconBlock, Slot};
const CORRUPT_DB_MESSAGE: &str = "The database could be corrupt. Check its file permissions or \
consider deleting it by running with the --purge-db flag.";
@@ -103,8 +100,6 @@ pub fn reset_fork_choice_to_finalization<E: EthSpec, Hot: ItemStore<E>, Cold: It
store: Arc<HotColdDB<E, Hot, Cold>>,
current_slot: Option<Slot>,
spec: &ChainSpec,
progressive_balances_mode: ProgressiveBalancesMode,
log: &Logger,
) -> Result<ForkChoice<BeaconForkChoiceStore<E, Hot, Cold>, E>, String> {
// Fetch finalized block.
let finalized_checkpoint = head_state.finalized_checkpoint();
@@ -180,7 +175,6 @@ pub fn reset_fork_choice_to_finalization<E: EthSpec, Hot: ItemStore<E>, Cold: It
&mut state,
&block,
BlockSignatureStrategy::NoVerification,
StateProcessingStrategy::Accurate,
VerifyBlockRoot::True,
&mut ctxt,
spec,
@@ -202,9 +196,7 @@ pub fn reset_fork_choice_to_finalization<E: EthSpec, Hot: ItemStore<E>, Cold: It
Duration::from_secs(0),
&state,
payload_verification_status,
progressive_balances_mode,
spec,
log,
)
.map_err(|e| format!("Error applying replayed block to fork choice: {:?}", e))?;
}

View File

@@ -0,0 +1,377 @@
use crate::BeaconChain;
use crate::BeaconChainTypes;
use execution_layer::{http::ENGINE_GET_CLIENT_VERSION_V1, CommitPrefix, ExecutionLayer};
use serde::{Deserialize, Serialize};
use slog::{crit, debug, error, warn, Logger};
use slot_clock::SlotClock;
use std::{fmt::Debug, time::Duration};
use task_executor::TaskExecutor;
use types::{EthSpec, Graffiti, GRAFFITI_BYTES_LEN};
const ENGINE_VERSION_AGE_LIMIT_EPOCH_MULTIPLE: u32 = 6; // 6 epochs
const ENGINE_VERSION_CACHE_REFRESH_EPOCH_MULTIPLE: u32 = 2; // 2 epochs
const ENGINE_VERSION_CACHE_PRELOAD_STARTUP_DELAY: Duration = Duration::from_secs(60);
/// Represents the source and content of graffiti for block production, excluding
/// inputs from the validator client and execution engine. Graffiti is categorized
/// as either user-specified or calculated to facilitate decisions on graffiti
/// selection.
#[derive(Clone, Copy, Serialize, Deserialize)]
pub enum GraffitiOrigin {
UserSpecified(Graffiti),
Calculated(Graffiti),
}
impl GraffitiOrigin {
pub fn graffiti(&self) -> Graffiti {
match self {
GraffitiOrigin::UserSpecified(graffiti) => *graffiti,
GraffitiOrigin::Calculated(graffiti) => *graffiti,
}
}
}
impl Default for GraffitiOrigin {
fn default() -> Self {
let version_bytes = lighthouse_version::VERSION.as_bytes();
let trimmed_len = std::cmp::min(version_bytes.len(), GRAFFITI_BYTES_LEN);
let mut bytes = [0u8; GRAFFITI_BYTES_LEN];
bytes[..trimmed_len].copy_from_slice(&version_bytes[..trimmed_len]);
Self::Calculated(Graffiti::from(bytes))
}
}
impl Debug for GraffitiOrigin {
fn fmt(&self, f: &mut std::fmt::Formatter) -> std::fmt::Result {
self.graffiti().fmt(f)
}
}
pub struct GraffitiCalculator<T: BeaconChainTypes> {
pub beacon_graffiti: GraffitiOrigin,
execution_layer: Option<ExecutionLayer<T::EthSpec>>,
pub epoch_duration: Duration,
log: Logger,
}
impl<T: BeaconChainTypes> GraffitiCalculator<T> {
pub fn new(
beacon_graffiti: GraffitiOrigin,
execution_layer: Option<ExecutionLayer<T::EthSpec>>,
epoch_duration: Duration,
log: Logger,
) -> Self {
Self {
beacon_graffiti,
execution_layer,
epoch_duration,
log,
}
}
/// Returns the appropriate graffiti to use for block production, prioritizing
/// sources in the following order:
/// 1. Graffiti specified by the validator client.
/// 2. Graffiti specified by the user via beacon node CLI options.
/// 3. The EL & CL client version string, applicable when the EL supports version specification.
/// 4. The default lighthouse version string, used if the EL lacks version specification support.
pub async fn get_graffiti(&self, validator_graffiti: Option<Graffiti>) -> Graffiti {
if let Some(graffiti) = validator_graffiti {
return graffiti;
}
match self.beacon_graffiti {
GraffitiOrigin::UserSpecified(graffiti) => graffiti,
GraffitiOrigin::Calculated(default_graffiti) => {
let Some(execution_layer) = self.execution_layer.as_ref() else {
// Return default graffiti if there is no execution layer. This
// shouldn't occur if we're actually producing blocks.
crit!(self.log, "No execution layer available for graffiti calculation during block production!");
return default_graffiti;
};
// The engine version cache refresh service ensures this will almost always retrieve this data from the
// cache instead of making a request to the execution engine. A cache miss would only occur if lighthouse
// has recently started or the EL recently went offline.
let engine_versions = match execution_layer
.get_engine_version(Some(
self.epoch_duration * ENGINE_VERSION_AGE_LIMIT_EPOCH_MULTIPLE,
))
.await
{
Ok(engine_versions) => engine_versions,
Err(el_error) => {
warn!(self.log, "Failed to determine execution engine version for graffiti"; "error" => ?el_error);
return default_graffiti;
}
};
let Some(engine_version) = engine_versions.first() else {
// Got an empty array which indicates the EL doesn't support the method
debug!(
self.log,
"Using default lighthouse graffiti: EL does not support {} method",
ENGINE_GET_CLIENT_VERSION_V1;
);
return default_graffiti;
};
if engine_versions.len() != 1 {
// More than one version implies lighthouse is connected to
// an EL multiplexer. We don't support modifying the graffiti
// with these configurations.
warn!(
self.log,
"Execution Engine multiplexer detected, using default graffiti"
);
return default_graffiti;
}
let lighthouse_commit_prefix = CommitPrefix::try_from(lighthouse_version::COMMIT_PREFIX.to_string())
.unwrap_or_else(|error_message| {
// This really shouldn't happen but we want to definitly log if it does
crit!(self.log, "Failed to parse lighthouse commit prefix"; "error" => error_message);
CommitPrefix("00000000".to_string())
});
engine_version.calculate_graffiti(lighthouse_commit_prefix)
}
}
}
}
pub fn start_engine_version_cache_refresh_service<T: BeaconChainTypes>(
chain: &BeaconChain<T>,
executor: TaskExecutor,
) {
let Some(el_ref) = chain.execution_layer.as_ref() else {
debug!(
chain.log,
"No execution layer configured, not starting engine version cache refresh service"
);
return;
};
if matches!(
chain.graffiti_calculator.beacon_graffiti,
GraffitiOrigin::UserSpecified(_)
) {
debug!(
chain.log,
"Graffiti is user-specified, not starting engine version cache refresh service"
);
return;
}
let execution_layer = el_ref.clone();
let log = chain.log.clone();
let slot_clock = chain.slot_clock.clone();
let epoch_duration = chain.graffiti_calculator.epoch_duration;
executor.spawn(
async move {
engine_version_cache_refresh_service::<T>(
execution_layer,
slot_clock,
epoch_duration,
log,
)
.await
},
"engine_version_cache_refresh_service",
);
}
async fn engine_version_cache_refresh_service<T: BeaconChainTypes>(
execution_layer: ExecutionLayer<T::EthSpec>,
slot_clock: T::SlotClock,
epoch_duration: Duration,
log: Logger,
) {
// Preload the engine version cache after a brief delay to allow for EL initialization.
// This initial priming ensures cache readiness before the service's regular update cycle begins.
tokio::time::sleep(ENGINE_VERSION_CACHE_PRELOAD_STARTUP_DELAY).await;
if let Err(e) = execution_layer.get_engine_version(None).await {
debug!(log, "Failed to preload engine version cache"; "error" => format!("{:?}", e));
}
// this service should run 3/8 of the way through the epoch
let epoch_delay = (epoch_duration * 3) / 8;
// the duration of 1 epoch less than the total duration between firing of this service
let partial_firing_delay =
epoch_duration * ENGINE_VERSION_CACHE_REFRESH_EPOCH_MULTIPLE.saturating_sub(1);
loop {
match slot_clock.duration_to_next_epoch(T::EthSpec::slots_per_epoch()) {
Some(duration_to_next_epoch) => {
let firing_delay = partial_firing_delay + duration_to_next_epoch + epoch_delay;
tokio::time::sleep(firing_delay).await;
debug!(
log,
"Engine version cache refresh service firing";
);
match execution_layer.get_engine_version(None).await {
Err(e) => warn!(log, "Failed to populate engine version cache"; "error" => ?e),
Ok(versions) => {
if versions.is_empty() {
// Empty array indicates the EL doesn't support the method
debug!(
log,
"EL does not support {} method. Sleeping twice as long before retry",
ENGINE_GET_CLIENT_VERSION_V1
);
tokio::time::sleep(
epoch_duration * ENGINE_VERSION_CACHE_REFRESH_EPOCH_MULTIPLE,
)
.await;
}
}
}
}
None => {
error!(log, "Failed to read slot clock");
// If we can't read the slot clock, just wait another slot.
tokio::time::sleep(slot_clock.slot_duration()).await;
}
};
}
}
#[cfg(test)]
mod tests {
use crate::test_utils::{test_spec, BeaconChainHarness, EphemeralHarnessType};
use crate::ChainConfig;
use execution_layer::test_utils::{DEFAULT_CLIENT_VERSION, DEFAULT_ENGINE_CAPABILITIES};
use execution_layer::EngineCapabilities;
use lazy_static::lazy_static;
use slog::info;
use std::time::Duration;
use types::{ChainSpec, Graffiti, Keypair, MinimalEthSpec, GRAFFITI_BYTES_LEN};
const VALIDATOR_COUNT: usize = 48;
lazy_static! {
/// A cached set of keys.
static ref KEYPAIRS: Vec<Keypair> = types::test_utils::generate_deterministic_keypairs(VALIDATOR_COUNT);
}
fn get_harness(
validator_count: usize,
spec: ChainSpec,
chain_config: Option<ChainConfig>,
) -> BeaconChainHarness<EphemeralHarnessType<MinimalEthSpec>> {
let harness = BeaconChainHarness::builder(MinimalEthSpec)
.spec(spec)
.chain_config(chain_config.unwrap_or_default())
.keypairs(KEYPAIRS[0..validator_count].to_vec())
.logger(logging::test_logger())
.fresh_ephemeral_store()
.mock_execution_layer()
.build();
harness.advance_slot();
harness
}
#[tokio::test]
async fn check_graffiti_without_el_version_support() {
let spec = test_spec::<MinimalEthSpec>();
let harness = get_harness(VALIDATOR_COUNT, spec, None);
// modify execution engine so it doesn't support engine_getClientVersionV1 method
let mock_execution_layer = harness.mock_execution_layer.as_ref().unwrap();
mock_execution_layer
.server
.set_engine_capabilities(EngineCapabilities {
get_client_version_v1: false,
..DEFAULT_ENGINE_CAPABILITIES
});
// refresh capabilities cache
harness
.chain
.execution_layer
.as_ref()
.unwrap()
.get_engine_capabilities(Some(Duration::ZERO))
.await
.unwrap();
let version_bytes = std::cmp::min(
lighthouse_version::VERSION.as_bytes().len(),
GRAFFITI_BYTES_LEN,
);
// grab the slice of the graffiti that corresponds to the lighthouse version
let graffiti_slice =
&harness.chain.graffiti_calculator.get_graffiti(None).await.0[..version_bytes];
// convert graffiti bytes slice to ascii for easy debugging if this test should fail
let graffiti_str =
std::str::from_utf8(graffiti_slice).expect("bytes should convert nicely to ascii");
info!(harness.chain.log, "results"; "lighthouse_version" => lighthouse_version::VERSION, "graffiti_str" => graffiti_str);
println!("lighthouse_version: '{}'", lighthouse_version::VERSION);
println!("graffiti_str: '{}'", graffiti_str);
assert!(lighthouse_version::VERSION.starts_with(graffiti_str));
}
#[tokio::test]
async fn check_graffiti_with_el_version_support() {
let spec = test_spec::<MinimalEthSpec>();
let harness = get_harness(VALIDATOR_COUNT, spec, None);
let found_graffiti_bytes = harness.chain.graffiti_calculator.get_graffiti(None).await.0;
let mock_commit = DEFAULT_CLIENT_VERSION.commit.clone();
let expected_graffiti_string = format!(
"{}{}{}{}",
DEFAULT_CLIENT_VERSION.code,
mock_commit
.strip_prefix("0x")
.unwrap_or(&mock_commit)
.get(0..4)
.expect("should get first 2 bytes in hex"),
"LH",
lighthouse_version::COMMIT_PREFIX
.get(0..4)
.expect("should get first 2 bytes in hex")
);
let expected_graffiti_prefix_bytes = expected_graffiti_string.as_bytes();
let expected_graffiti_prefix_len =
std::cmp::min(expected_graffiti_prefix_bytes.len(), GRAFFITI_BYTES_LEN);
let found_graffiti_string =
std::str::from_utf8(&found_graffiti_bytes[..expected_graffiti_prefix_len])
.expect("bytes should convert nicely to ascii");
info!(harness.chain.log, "results"; "expected_graffiti_string" => &expected_graffiti_string, "found_graffiti_string" => &found_graffiti_string);
println!("expected_graffiti_string: '{}'", expected_graffiti_string);
println!("found_graffiti_string: '{}'", found_graffiti_string);
assert_eq!(expected_graffiti_string, found_graffiti_string);
let mut expected_graffiti_bytes = [0u8; GRAFFITI_BYTES_LEN];
expected_graffiti_bytes[..expected_graffiti_prefix_len]
.copy_from_slice(expected_graffiti_string.as_bytes());
assert_eq!(found_graffiti_bytes, expected_graffiti_bytes);
}
#[tokio::test]
async fn check_graffiti_with_validator_specified_value() {
let spec = test_spec::<MinimalEthSpec>();
let harness = get_harness(VALIDATOR_COUNT, spec, None);
let graffiti_str = "nice graffiti bro";
let mut graffiti_bytes = [0u8; GRAFFITI_BYTES_LEN];
graffiti_bytes[..graffiti_str.as_bytes().len()].copy_from_slice(graffiti_str.as_bytes());
let found_graffiti = harness
.chain
.graffiti_calculator
.get_graffiti(Some(Graffiti::from(graffiti_bytes)))
.await;
assert_eq!(
found_graffiti.to_string(),
"0x6e6963652067726166666974692062726f000000000000000000000000000000"
);
}
}

View File

@@ -1,4 +1,4 @@
use parking_lot::RwLock;
use parking_lot::{RwLock, RwLockReadGuard};
use ssz_derive::{Decode, Encode};
use std::collections::HashMap;
use types::{Hash256, Slot};
@@ -16,6 +16,8 @@ pub enum Error {
#[derive(Default, Debug)]
pub struct HeadTracker(pub RwLock<HashMap<Hash256, Slot>>);
pub type HeadTrackerReader<'a> = RwLockReadGuard<'a, HashMap<Hash256, Slot>>;
impl HeadTracker {
/// Register a block with `Self`, so it may or may not be included in a `Self::heads` call.
///
@@ -44,6 +46,11 @@ impl HeadTracker {
/// Returns a `SszHeadTracker`, which contains all necessary information to restore the state
/// of `Self` at some later point.
///
/// Should ONLY be used for tests, due to the potential for database races.
///
/// See <https://github.com/sigp/lighthouse/issues/4773>
#[cfg(test)]
pub fn to_ssz_container(&self) -> SszHeadTracker {
SszHeadTracker::from_map(&self.0.read())
}

View File

@@ -1,3 +1,4 @@
use crate::data_availability_checker::AvailableBlock;
use crate::{errors::BeaconChainError as Error, metrics, BeaconChain, BeaconChainTypes};
use itertools::Itertools;
use slog::debug;
@@ -7,10 +8,9 @@ use state_processing::{
};
use std::borrow::Cow;
use std::iter;
use std::sync::Arc;
use std::time::Duration;
use store::{chunked_vector::BlockRoots, AnchorInfo, ChunkWriter, KeyValueStore};
use types::{Hash256, SignedBlindedBeaconBlock, Slot};
use store::{chunked_vector::BlockRoots, AnchorInfo, BlobInfo, ChunkWriter, KeyValueStore};
use types::{Hash256, Slot};
/// Use a longer timeout on the pubkey cache.
///
@@ -59,27 +59,30 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
/// Return the number of blocks successfully imported.
pub fn import_historical_block_batch(
&self,
blocks: Vec<Arc<SignedBlindedBeaconBlock<T::EthSpec>>>,
mut blocks: Vec<AvailableBlock<T::EthSpec>>,
) -> Result<usize, Error> {
let anchor_info = self
.store
.get_anchor_info()
.ok_or(HistoricalBlockError::NoAnchorInfo)?;
let blob_info = self.store.get_blob_info();
// Take all blocks with slots less than the oldest block slot.
let num_relevant =
blocks.partition_point(|block| block.slot() < anchor_info.oldest_block_slot);
let blocks_to_import = &blocks
.get(..num_relevant)
.ok_or(HistoricalBlockError::IndexOutOfBounds)?;
let num_relevant = blocks.partition_point(|available_block| {
available_block.block().slot() < anchor_info.oldest_block_slot
});
if blocks_to_import.len() != blocks.len() {
let total_blocks = blocks.len();
blocks.truncate(num_relevant);
let blocks_to_import = blocks;
if blocks_to_import.len() != total_blocks {
debug!(
self.log,
"Ignoring some historic blocks";
"oldest_block_slot" => anchor_info.oldest_block_slot,
"total_blocks" => blocks.len(),
"ignored" => blocks.len().saturating_sub(blocks_to_import.len()),
"total_blocks" => total_blocks,
"ignored" => total_blocks.saturating_sub(blocks_to_import.len()),
);
}
@@ -87,17 +90,24 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
return Ok(0);
}
let n_blobs_lists_to_import = blocks_to_import
.iter()
.filter(|available_block| available_block.blobs().is_some())
.count();
let mut expected_block_root = anchor_info.oldest_block_parent;
let mut prev_block_slot = anchor_info.oldest_block_slot;
let mut chunk_writer =
ChunkWriter::<BlockRoots, _, _>::new(&self.store.cold_db, prev_block_slot.as_usize())?;
let mut new_oldest_blob_slot = blob_info.oldest_blob_slot;
let mut cold_batch = Vec::with_capacity(blocks.len());
let mut hot_batch = Vec::with_capacity(blocks.len());
let mut blob_batch = Vec::with_capacity(n_blobs_lists_to_import);
let mut cold_batch = Vec::with_capacity(blocks_to_import.len());
let mut hot_batch = Vec::with_capacity(blocks_to_import.len());
let mut signed_blocks = Vec::with_capacity(blocks_to_import.len());
for block in blocks_to_import.iter().rev() {
// Check chain integrity.
let block_root = block.canonical_root();
for available_block in blocks_to_import.into_iter().rev() {
let (block_root, block, maybe_blobs) = available_block.deconstruct();
if block_root != expected_block_root {
return Err(HistoricalBlockError::MismatchedBlockRoot {
@@ -107,9 +117,16 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
.into());
}
let blinded_block = block.clone_as_blinded();
// Store block in the hot database without payload.
self.store
.blinded_block_as_kv_store_ops(&block_root, block, &mut hot_batch);
.blinded_block_as_kv_store_ops(&block_root, &blinded_block, &mut hot_batch);
// Store the blobs too
if let Some(blobs) = maybe_blobs {
new_oldest_blob_slot = Some(block.slot());
self.store
.blobs_as_kv_store_ops(&block_root, blobs, &mut blob_batch);
}
// Store block roots, including at all skip slots in the freezer DB.
for slot in (block.slot().as_usize()..prev_block_slot.as_usize()).rev() {
@@ -118,22 +135,24 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
prev_block_slot = block.slot();
expected_block_root = block.message().parent_root();
signed_blocks.push(block);
// If we've reached genesis, add the genesis block root to the batch and set the
// anchor slot to 0 to indicate completion.
// If we've reached genesis, add the genesis block root to the batch for all slots
// between 0 and the first block slot, and set the anchor slot to 0 to indicate
// completion.
if expected_block_root == self.genesis_block_root {
let genesis_slot = self.spec.genesis_slot;
chunk_writer.set(
genesis_slot.as_usize(),
self.genesis_block_root,
&mut cold_batch,
)?;
for slot in genesis_slot.as_usize()..prev_block_slot.as_usize() {
chunk_writer.set(slot, self.genesis_block_root, &mut cold_batch)?;
}
prev_block_slot = genesis_slot;
expected_block_root = Hash256::zero();
break;
}
}
chunk_writer.write(&mut cold_batch)?;
// these were pushed in reverse order so we reverse again
signed_blocks.reverse();
// Verify signatures in one batch, holding the pubkey cache lock for the shortest duration
// possible. For each block fetch the parent root from its successor. Slicing from index 1
@@ -144,15 +163,16 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
.validator_pubkey_cache
.try_read_for(PUBKEY_CACHE_LOCK_TIMEOUT)
.ok_or(HistoricalBlockError::ValidatorPubkeyCacheTimeout)?;
let block_roots = blocks_to_import
let block_roots = signed_blocks
.get(1..)
.ok_or(HistoricalBlockError::IndexOutOfBounds)?
.iter()
.map(|block| block.parent_root())
.chain(iter::once(anchor_info.oldest_block_parent));
let signature_set = blocks_to_import
let signature_set = signed_blocks
.iter()
.zip_eq(block_roots)
.filter(|&(_block, block_root)| (block_root != self.genesis_block_root))
.map(|(block, block_root)| {
block_proposal_signature_set_from_parts(
block,
@@ -180,9 +200,26 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
// Write the I/O batches to disk, writing the blocks themselves first, as it's better
// for the hot DB to contain extra blocks than for the cold DB to point to blocks that
// do not exist.
self.store.blobs_db.do_atomically(blob_batch)?;
self.store.hot_db.do_atomically(hot_batch)?;
self.store.cold_db.do_atomically(cold_batch)?;
let mut anchor_and_blob_batch = Vec::with_capacity(2);
// Update the blob info.
if new_oldest_blob_slot != blob_info.oldest_blob_slot {
if let Some(oldest_blob_slot) = new_oldest_blob_slot {
let new_blob_info = BlobInfo {
oldest_blob_slot: Some(oldest_blob_slot),
..blob_info.clone()
};
anchor_and_blob_batch.push(
self.store
.compare_and_set_blob_info(blob_info, new_blob_info)?,
);
}
}
// Update the anchor.
let new_anchor = AnchorInfo {
oldest_block_slot: prev_block_slot,
@@ -190,8 +227,11 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
..anchor_info
};
let backfill_complete = new_anchor.block_backfill_complete(self.genesis_backfill_slot);
self.store
.compare_and_set_anchor_info_with_write(Some(anchor_info), Some(new_anchor))?;
anchor_and_blob_batch.push(
self.store
.compare_and_set_anchor_info(Some(anchor_info), Some(new_anchor))?,
);
self.store.hot_db.do_atomically(anchor_and_blob_batch)?;
// If backfill has completed and the chain is configured to reconstruct historic states,
// send a message to the background migrator instructing it to begin reconstruction.
@@ -203,6 +243,6 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
self.store_migrator.process_reconstruction();
}
Ok(blocks_to_import.len())
Ok(num_relevant)
}
}

View File

@@ -0,0 +1,78 @@
use kzg::{Blob as KzgBlob, Error as KzgError, Kzg};
use types::{Blob, EthSpec, Hash256, KzgCommitment, KzgProof};
/// Converts a blob ssz List object to an array to be used with the kzg
/// crypto library.
fn ssz_blob_to_crypto_blob<E: EthSpec>(blob: &Blob<E>) -> Result<KzgBlob, KzgError> {
KzgBlob::from_bytes(blob.as_ref()).map_err(Into::into)
}
/// Validate a single blob-commitment-proof triplet from a `BlobSidecar`.
pub fn validate_blob<E: EthSpec>(
kzg: &Kzg,
blob: &Blob<E>,
kzg_commitment: KzgCommitment,
kzg_proof: KzgProof,
) -> Result<(), KzgError> {
let _timer = crate::metrics::start_timer(&crate::metrics::KZG_VERIFICATION_SINGLE_TIMES);
let kzg_blob = ssz_blob_to_crypto_blob::<E>(blob)?;
kzg.verify_blob_kzg_proof(&kzg_blob, kzg_commitment, kzg_proof)
}
/// Validate a batch of blob-commitment-proof triplets from multiple `BlobSidecars`.
pub fn validate_blobs<E: EthSpec>(
kzg: &Kzg,
expected_kzg_commitments: &[KzgCommitment],
blobs: Vec<&Blob<E>>,
kzg_proofs: &[KzgProof],
) -> Result<(), KzgError> {
let _timer = crate::metrics::start_timer(&crate::metrics::KZG_VERIFICATION_BATCH_TIMES);
let blobs = blobs
.into_iter()
.map(|blob| ssz_blob_to_crypto_blob::<E>(blob))
.collect::<Result<Vec<_>, KzgError>>()?;
kzg.verify_blob_kzg_proof_batch(&blobs, expected_kzg_commitments, kzg_proofs)
}
/// Compute the kzg proof given an ssz blob and its kzg commitment.
pub fn compute_blob_kzg_proof<E: EthSpec>(
kzg: &Kzg,
blob: &Blob<E>,
kzg_commitment: KzgCommitment,
) -> Result<KzgProof, KzgError> {
let kzg_blob = ssz_blob_to_crypto_blob::<E>(blob)?;
kzg.compute_blob_kzg_proof(&kzg_blob, kzg_commitment)
}
/// Compute the kzg commitment for a given blob.
pub fn blob_to_kzg_commitment<E: EthSpec>(
kzg: &Kzg,
blob: &Blob<E>,
) -> Result<KzgCommitment, KzgError> {
let kzg_blob = ssz_blob_to_crypto_blob::<E>(blob)?;
kzg.blob_to_kzg_commitment(&kzg_blob)
}
/// Compute the kzg proof for a given blob and an evaluation point z.
pub fn compute_kzg_proof<E: EthSpec>(
kzg: &Kzg,
blob: &Blob<E>,
z: Hash256,
) -> Result<(KzgProof, Hash256), KzgError> {
let z = z.0.into();
let kzg_blob = ssz_blob_to_crypto_blob::<E>(blob)?;
kzg.compute_kzg_proof(&kzg_blob, &z)
.map(|(proof, z)| (proof, Hash256::from_slice(&z.to_vec())))
}
/// Verify a `kzg_proof` for a `kzg_commitment` that evaluating a polynomial at `z` results in `y`
pub fn verify_kzg_proof<E: EthSpec>(
kzg: &Kzg,
kzg_commitment: KzgCommitment,
kzg_proof: KzgProof,
z: Hash256,
y: Hash256,
) -> Result<bool, KzgError> {
kzg.verify_kzg_proof(kzg_commitment, &z.0.into(), &y.0.into(), kzg_proof)
}

View File

@@ -1,4 +1,5 @@
pub mod attestation_rewards;
pub mod attestation_simulator;
pub mod attestation_verification;
mod attester_cache;
pub mod beacon_block_reward;
@@ -7,14 +8,20 @@ mod beacon_chain;
mod beacon_fork_choice_store;
pub mod beacon_proposer_cache;
mod beacon_snapshot;
pub mod bellatrix_readiness;
pub mod blob_verification;
pub mod block_reward;
mod block_times_cache;
mod block_verification;
pub mod block_verification_types;
pub mod builder;
pub mod canonical_head;
pub mod capella_readiness;
pub mod chain_config;
pub mod data_availability_checker;
pub mod deneb_readiness;
mod early_attester_cache;
pub mod electra_readiness;
mod errors;
pub mod eth1_chain;
mod eth1_finalization_cache;
@@ -22,18 +29,22 @@ pub mod events;
pub mod execution_payload;
pub mod fork_choice_signal;
pub mod fork_revert;
pub mod graffiti_calculator;
mod head_tracker;
pub mod historical_blocks;
pub mod kzg_utils;
pub mod light_client_finality_update_verification;
pub mod light_client_optimistic_update_verification;
pub mod merge_readiness;
mod light_client_server_cache;
pub mod metrics;
pub mod migrate;
mod naive_aggregation_pool;
mod observed_aggregates;
pub mod observed_aggregates;
mod observed_attesters;
mod observed_blob_sidecars;
pub mod observed_block_producers;
pub mod observed_operations;
mod observed_slashable;
pub mod otb_verification_service;
mod persisted_beacon_chain;
mod persisted_fork_choice;
@@ -41,7 +52,6 @@ mod pre_finalization_cache;
pub mod proposer_prep_service;
pub mod schema_change;
pub mod shuffling_cache;
mod snapshot_cache;
pub mod state_advance_timer;
pub mod sync_committee_rewards;
pub mod sync_committee_verification;
@@ -51,9 +61,11 @@ pub mod validator_monitor;
pub mod validator_pubkey_cache;
pub use self::beacon_chain::{
AttestationProcessingOutcome, BeaconChain, BeaconChainTypes, BeaconStore, ChainSegmentResult,
ForkChoiceError, OverrideForkchoiceUpdate, ProduceBlockVerification, StateSkipConfig,
WhenSlotSkipped, INVALID_FINALIZED_MERGE_TRANSITION_BLOCK_SHUTDOWN_REASON,
AttestationProcessingOutcome, AvailabilityProcessingStatus, BeaconBlockResponse,
BeaconBlockResponseWrapper, BeaconChain, BeaconChainTypes, BeaconStore, BlockProcessStatus,
ChainSegmentResult, ForkChoiceError, LightClientProducerEvent, OverrideForkchoiceUpdate,
ProduceBlockVerification, StateSkipConfig, WhenSlotSkipped,
INVALID_FINALIZED_MERGE_TRANSITION_BLOCK_SHUTDOWN_REASON,
INVALID_JUSTIFIED_PAYLOAD_SHUTDOWN_REASON,
};
pub use self::beacon_snapshot::BeaconSnapshot;
@@ -63,15 +75,19 @@ pub use self::historical_blocks::HistoricalBlockError;
pub use attestation_verification::Error as AttestationError;
pub use beacon_fork_choice_store::{BeaconForkChoiceStore, Error as ForkChoiceStoreError};
pub use block_verification::{
get_block_root, BlockError, ExecutionPayloadError, GossipVerifiedBlock,
IntoExecutionPendingBlock, IntoGossipVerifiedBlock,
get_block_root, BlockError, ExecutionPayloadError, ExecutionPendingBlock, GossipVerifiedBlock,
IntoExecutionPendingBlock, IntoGossipVerifiedBlockContents, PayloadVerificationOutcome,
PayloadVerificationStatus,
};
pub use block_verification_types::AvailabilityPendingExecutedBlock;
pub use block_verification_types::ExecutedBlock;
pub use canonical_head::{CachedHead, CanonicalHead, CanonicalHeadRwLock};
pub use eth1_chain::{Eth1Chain, Eth1ChainBackend};
pub use events::ServerSentEventHandler;
pub use execution_layer::EngineState;
pub use execution_payload::NotifyExecutionLayer;
pub use fork_choice::{ExecutionStatus, ForkchoiceUpdateParameters};
pub use kzg::{Kzg, TrustedSetup};
pub use metrics::scrape_for_metrics;
pub use migrate::MigratorConfig;
pub use parking_lot;

View File

@@ -1,11 +1,9 @@
use crate::{BeaconChain, BeaconChainError, BeaconChainTypes};
use crate::{BeaconChain, BeaconChainTypes};
use derivative::Derivative;
use slot_clock::SlotClock;
use std::time::Duration;
use strum::AsRefStr;
use types::{
light_client_update::Error as LightClientUpdateError, LightClientFinalityUpdate, Slot,
};
use types::LightClientFinalityUpdate;
/// Returned when a light client finality update was not successfully verified. It might not have been verified for
/// two reasons:
@@ -16,8 +14,6 @@ use types::{
/// (the `BeaconChainError` variant)
#[derive(Debug, AsRefStr)]
pub enum Error {
/// Light client finality update message with a lower or equal finalized_header slot already forwarded.
FinalityUpdateAlreadySeen,
/// The light client finality message was received is prior to one-third of slot duration passage. (with
/// respect to the gossip clock disparity and slot clock duration).
///
@@ -26,29 +22,11 @@ pub enum Error {
/// Assuming the local clock is correct, the peer has sent an invalid message.
TooEarly,
/// Light client finality update message does not match the locally constructed one.
///
/// ## Peer Scoring
///
InvalidLightClientFinalityUpdate,
/// Signature slot start time is none.
SigSlotStartIsNone,
/// Failed to construct a LightClientFinalityUpdate from state.
FailedConstructingUpdate,
/// Beacon chain error occured.
BeaconChainError(BeaconChainError),
LightClientUpdateError(LightClientUpdateError),
}
impl From<BeaconChainError> for Error {
fn from(e: BeaconChainError) -> Self {
Error::BeaconChainError(e)
}
}
impl From<LightClientUpdateError> for Error {
fn from(e: LightClientUpdateError) -> Self {
Error::LightClientUpdateError(e)
}
}
/// Wraps a `LightClientFinalityUpdate` that has been verified for propagation on the gossip network.
@@ -63,71 +41,34 @@ impl<T: BeaconChainTypes> VerifiedLightClientFinalityUpdate<T> {
/// Returns `Ok(Self)` if the `light_client_finality_update` is valid to be (re)published on the gossip
/// network.
pub fn verify(
light_client_finality_update: LightClientFinalityUpdate<T::EthSpec>,
rcv_finality_update: LightClientFinalityUpdate<T::EthSpec>,
chain: &BeaconChain<T>,
seen_timestamp: Duration,
) -> Result<Self, Error> {
let gossiped_finality_slot = light_client_finality_update.finalized_header.slot;
let one_third_slot_duration = Duration::new(chain.spec.seconds_per_slot / 3, 0);
let signature_slot = light_client_finality_update.signature_slot;
let start_time = chain.slot_clock.start_of(signature_slot);
let mut latest_seen_finality_update = chain.latest_seen_finality_update.lock();
let head = chain.canonical_head.cached_head();
let head_block = &head.snapshot.beacon_block;
let attested_block_root = head_block.message().parent_root();
let attested_block = chain
.get_blinded_block(&attested_block_root)?
.ok_or(Error::FailedConstructingUpdate)?;
let mut attested_state = chain
.get_state(&attested_block.state_root(), Some(attested_block.slot()))?
.ok_or(Error::FailedConstructingUpdate)?;
let finalized_block_root = attested_state.finalized_checkpoint().root;
let finalized_block = chain
.get_blinded_block(&finalized_block_root)?
.ok_or(Error::FailedConstructingUpdate)?;
let latest_seen_finality_update_slot = match latest_seen_finality_update.as_ref() {
Some(update) => update.finalized_header.slot,
None => Slot::new(0),
};
// verify that no other finality_update with a lower or equal
// finalized_header.slot was already forwarded on the network
if gossiped_finality_slot <= latest_seen_finality_update_slot {
return Err(Error::FinalityUpdateAlreadySeen);
}
// verify that enough time has passed for the block to have been propagated
match start_time {
Some(time) => {
if seen_timestamp + chain.spec.maximum_gossip_clock_disparity()
< time + one_third_slot_duration
{
return Err(Error::TooEarly);
}
}
None => return Err(Error::SigSlotStartIsNone),
let start_time = chain
.slot_clock
.start_of(*rcv_finality_update.signature_slot())
.ok_or(Error::SigSlotStartIsNone)?;
let one_third_slot_duration = Duration::new(chain.spec.seconds_per_slot / 3, 0);
if seen_timestamp + chain.spec.maximum_gossip_clock_disparity()
< start_time + one_third_slot_duration
{
return Err(Error::TooEarly);
}
let head_state = &head.snapshot.beacon_state;
let finality_update = LightClientFinalityUpdate::new(
&chain.spec,
head_state,
head_block,
&mut attested_state,
&finalized_block,
)?;
let latest_finality_update = chain
.light_client_server_cache
.get_latest_finality_update()
.ok_or(Error::FailedConstructingUpdate)?;
// verify that the gossiped finality update is the same as the locally constructed one.
if finality_update != light_client_finality_update {
if latest_finality_update != rcv_finality_update {
return Err(Error::InvalidLightClientFinalityUpdate);
}
*latest_seen_finality_update = Some(light_client_finality_update.clone());
Ok(Self {
light_client_finality_update,
light_client_finality_update: rcv_finality_update,
seen_timestamp,
})
}

View File

@@ -1,12 +1,10 @@
use crate::{BeaconChain, BeaconChainError, BeaconChainTypes};
use crate::{BeaconChain, BeaconChainTypes};
use derivative::Derivative;
use eth2::types::Hash256;
use slot_clock::SlotClock;
use std::time::Duration;
use strum::AsRefStr;
use types::{
light_client_update::Error as LightClientUpdateError, LightClientOptimisticUpdate, Slot,
};
use types::LightClientOptimisticUpdate;
/// Returned when a light client optimistic update was not successfully verified. It might not have been verified for
/// two reasons:
@@ -17,8 +15,6 @@ use types::{
/// (the `BeaconChainError` variant)
#[derive(Debug, AsRefStr)]
pub enum Error {
/// Light client optimistic update message with a lower or equal optimistic_header slot already forwarded.
OptimisticUpdateAlreadySeen,
/// The light client optimistic message was received is prior to one-third of slot duration passage. (with
/// respect to the gossip clock disparity and slot clock duration).
///
@@ -27,9 +23,6 @@ pub enum Error {
/// Assuming the local clock is correct, the peer has sent an invalid message.
TooEarly,
/// Light client optimistic update message does not match the locally constructed one.
///
/// ## Peer Scoring
///
InvalidLightClientOptimisticUpdate,
/// Signature slot start time is none.
SigSlotStartIsNone,
@@ -37,21 +30,6 @@ pub enum Error {
FailedConstructingUpdate,
/// Unknown block with parent root.
UnknownBlockParentRoot(Hash256),
/// Beacon chain error occured.
BeaconChainError(BeaconChainError),
LightClientUpdateError(LightClientUpdateError),
}
impl From<BeaconChainError> for Error {
fn from(e: BeaconChainError) -> Self {
Error::BeaconChainError(e)
}
}
impl From<LightClientUpdateError> for Error {
fn from(e: LightClientUpdateError) -> Self {
Error::LightClientUpdateError(e)
}
}
/// Wraps a `LightClientOptimisticUpdate` that has been verified for propagation on the gossip network.
@@ -67,72 +45,46 @@ impl<T: BeaconChainTypes> VerifiedLightClientOptimisticUpdate<T> {
/// Returns `Ok(Self)` if the `light_client_optimistic_update` is valid to be (re)published on the gossip
/// network.
pub fn verify(
light_client_optimistic_update: LightClientOptimisticUpdate<T::EthSpec>,
rcv_optimistic_update: LightClientOptimisticUpdate<T::EthSpec>,
chain: &BeaconChain<T>,
seen_timestamp: Duration,
) -> Result<Self, Error> {
let gossiped_optimistic_slot = light_client_optimistic_update.attested_header.slot;
// verify that enough time has passed for the block to have been propagated
let start_time = chain
.slot_clock
.start_of(*rcv_optimistic_update.signature_slot())
.ok_or(Error::SigSlotStartIsNone)?;
let one_third_slot_duration = Duration::new(chain.spec.seconds_per_slot / 3, 0);
let signature_slot = light_client_optimistic_update.signature_slot;
let start_time = chain.slot_clock.start_of(signature_slot);
let mut latest_seen_optimistic_update = chain.latest_seen_optimistic_update.lock();
if seen_timestamp + chain.spec.maximum_gossip_clock_disparity()
< start_time + one_third_slot_duration
{
return Err(Error::TooEarly);
}
let head = chain.canonical_head.cached_head();
let head_block = &head.snapshot.beacon_block;
let attested_block_root = head_block.message().parent_root();
let attested_block = chain
.get_blinded_block(&attested_block_root)?
.ok_or(Error::FailedConstructingUpdate)?;
let attested_state = chain
.get_state(&attested_block.state_root(), Some(attested_block.slot()))?
.ok_or(Error::FailedConstructingUpdate)?;
let latest_seen_optimistic_update_slot = match latest_seen_optimistic_update.as_ref() {
Some(update) => update.attested_header.slot,
None => Slot::new(0),
};
// verify that no other optimistic_update with a lower or equal
// optimistic_header.slot was already forwarded on the network
if gossiped_optimistic_slot <= latest_seen_optimistic_update_slot {
return Err(Error::OptimisticUpdateAlreadySeen);
}
// verify that enough time has passed for the block to have been propagated
match start_time {
Some(time) => {
if seen_timestamp + chain.spec.maximum_gossip_clock_disparity()
< time + one_third_slot_duration
{
return Err(Error::TooEarly);
}
}
None => return Err(Error::SigSlotStartIsNone),
}
// check if we can process the optimistic update immediately
// otherwise queue
let canonical_root = light_client_optimistic_update
.attested_header
.canonical_root();
let canonical_root = rcv_optimistic_update.get_canonical_root();
if canonical_root != head_block.message().parent_root() {
return Err(Error::UnknownBlockParentRoot(canonical_root));
}
let optimistic_update =
LightClientOptimisticUpdate::new(&chain.spec, head_block, &attested_state)?;
let latest_optimistic_update = chain
.light_client_server_cache
.get_latest_optimistic_update()
.ok_or(Error::FailedConstructingUpdate)?;
// verify that the gossiped optimistic update is the same as the locally constructed one.
if optimistic_update != light_client_optimistic_update {
if latest_optimistic_update != rcv_optimistic_update {
return Err(Error::InvalidLightClientOptimisticUpdate);
}
*latest_seen_optimistic_update = Some(light_client_optimistic_update.clone());
let parent_root = rcv_optimistic_update.get_parent_root();
Ok(Self {
light_client_optimistic_update,
parent_root: canonical_root,
light_client_optimistic_update: rcv_optimistic_update,
parent_root,
seen_timestamp,
})
}

View File

@@ -0,0 +1,251 @@
use crate::errors::BeaconChainError;
use crate::{metrics, BeaconChainTypes, BeaconStore};
use parking_lot::{Mutex, RwLock};
use slog::{debug, Logger};
use ssz_types::FixedVector;
use std::num::NonZeroUsize;
use types::light_client_update::{FinalizedRootProofLen, FINALIZED_ROOT_INDEX};
use types::non_zero_usize::new_non_zero_usize;
use types::{
BeaconBlockRef, BeaconState, ChainSpec, EthSpec, ForkName, Hash256, LightClientFinalityUpdate,
LightClientOptimisticUpdate, Slot, SyncAggregate,
};
/// A prev block cache miss requires to re-generate the state of the post-parent block. Items in the
/// prev block cache are very small 32 * (6 + 1) = 224 bytes. 32 is an arbitrary number that
/// represents unlikely re-orgs, while keeping the cache very small.
const PREV_BLOCK_CACHE_SIZE: NonZeroUsize = new_non_zero_usize(32);
/// This cache computes light client messages ahead of time, required to satisfy p2p and API
/// requests. These messages include proofs on historical states, so on-demand computation is
/// expensive.
///
pub struct LightClientServerCache<T: BeaconChainTypes> {
/// Tracks a single global latest finality update out of all imported blocks.
///
/// TODO: Active discussion with @etan-status if this cache should be fork aware to return
/// latest canonical (update with highest signature slot, where its attested header is part of
/// the head chain) instead of global latest (update with highest signature slot, out of all
/// branches).
latest_finality_update: RwLock<Option<LightClientFinalityUpdate<T::EthSpec>>>,
/// Tracks a single global latest optimistic update out of all imported blocks.
latest_optimistic_update: RwLock<Option<LightClientOptimisticUpdate<T::EthSpec>>>,
/// Caches state proofs by block root
prev_block_cache: Mutex<lru::LruCache<Hash256, LightClientCachedData>>,
}
impl<T: BeaconChainTypes> LightClientServerCache<T> {
pub fn new() -> Self {
Self {
latest_finality_update: None.into(),
latest_optimistic_update: None.into(),
prev_block_cache: lru::LruCache::new(PREV_BLOCK_CACHE_SIZE).into(),
}
}
/// Compute and cache state proofs for latter production of light-client messages. Does not
/// trigger block replay.
pub fn cache_state_data(
&self,
spec: &ChainSpec,
block: BeaconBlockRef<T::EthSpec>,
block_root: Hash256,
block_post_state: &mut BeaconState<T::EthSpec>,
) -> Result<(), BeaconChainError> {
let _timer = metrics::start_timer(&metrics::LIGHT_CLIENT_SERVER_CACHE_STATE_DATA_TIMES);
// Only post-altair
if spec.fork_name_at_slot::<T::EthSpec>(block.slot()) == ForkName::Base {
return Ok(());
}
// Persist in memory cache for a descendent block
let cached_data = LightClientCachedData::from_state(block_post_state)?;
self.prev_block_cache.lock().put(block_root, cached_data);
Ok(())
}
/// Given a block with a SyncAggregte computes better or more recent light client updates. The
/// results are cached either on disk or memory to be served via p2p and rest API
pub fn recompute_and_cache_updates(
&self,
store: BeaconStore<T>,
block_parent_root: &Hash256,
block_slot: Slot,
sync_aggregate: &SyncAggregate<T::EthSpec>,
log: &Logger,
chain_spec: &ChainSpec,
) -> Result<(), BeaconChainError> {
let _timer =
metrics::start_timer(&metrics::LIGHT_CLIENT_SERVER_CACHE_RECOMPUTE_UPDATES_TIMES);
let signature_slot = block_slot;
let attested_block_root = block_parent_root;
let attested_block =
store
.get_full_block(attested_block_root)?
.ok_or(BeaconChainError::DBInconsistent(format!(
"Block not available {:?}",
attested_block_root
)))?;
let cached_parts = self.get_or_compute_prev_block_cache(
store.clone(),
attested_block_root,
&attested_block.state_root(),
attested_block.slot(),
)?;
let attested_slot = attested_block.slot();
// Spec: Full nodes SHOULD provide the LightClientOptimisticUpdate with the highest
// attested_header.beacon.slot (if multiple, highest signature_slot) as selected by fork choice
let is_latest_optimistic = match &self.latest_optimistic_update.read().clone() {
Some(latest_optimistic_update) => {
is_latest_optimistic_update(latest_optimistic_update, attested_slot, signature_slot)
}
None => true,
};
if is_latest_optimistic {
// can create an optimistic update, that is more recent
*self.latest_optimistic_update.write() = Some(LightClientOptimisticUpdate::new(
&attested_block,
sync_aggregate.clone(),
signature_slot,
chain_spec,
)?);
};
// Spec: Full nodes SHOULD provide the LightClientFinalityUpdate with the highest
// attested_header.beacon.slot (if multiple, highest signature_slot) as selected by fork choice
let is_latest_finality = match &self.latest_finality_update.read().clone() {
Some(latest_finality_update) => {
is_latest_finality_update(latest_finality_update, attested_slot, signature_slot)
}
None => true,
};
if is_latest_finality & !cached_parts.finalized_block_root.is_zero() {
// Immediately after checkpoint sync the finalized block may not be available yet.
if let Some(finalized_block) =
store.get_full_block(&cached_parts.finalized_block_root)?
{
*self.latest_finality_update.write() = Some(LightClientFinalityUpdate::new(
&attested_block,
&finalized_block,
cached_parts.finality_branch.clone(),
sync_aggregate.clone(),
signature_slot,
chain_spec,
)?);
} else {
debug!(
log,
"Finalized block not available in store for light_client server";
"finalized_block_root" => format!("{}", cached_parts.finalized_block_root),
);
}
}
Ok(())
}
/// Retrieves prev block cached data from cache. If not present re-computes by retrieving the
/// parent state, and inserts an entry to the cache.
///
/// In separate function since FnOnce of get_or_insert can not be fallible.
fn get_or_compute_prev_block_cache(
&self,
store: BeaconStore<T>,
block_root: &Hash256,
block_state_root: &Hash256,
block_slot: Slot,
) -> Result<LightClientCachedData, BeaconChainError> {
// Attempt to get the value from the cache first.
if let Some(cached_parts) = self.prev_block_cache.lock().get(block_root) {
return Ok(cached_parts.clone());
}
metrics::inc_counter(&metrics::LIGHT_CLIENT_SERVER_CACHE_PREV_BLOCK_CACHE_MISS);
// Compute the value, handling potential errors.
let mut state = store
.get_state(block_state_root, Some(block_slot))?
.ok_or_else(|| {
BeaconChainError::DBInconsistent(format!("Missing state {:?}", block_state_root))
})?;
let new_value = LightClientCachedData::from_state(&mut state)?;
// Insert value and return owned
self.prev_block_cache
.lock()
.put(*block_root, new_value.clone());
Ok(new_value)
}
pub fn get_latest_finality_update(&self) -> Option<LightClientFinalityUpdate<T::EthSpec>> {
self.latest_finality_update.read().clone()
}
pub fn get_latest_optimistic_update(&self) -> Option<LightClientOptimisticUpdate<T::EthSpec>> {
self.latest_optimistic_update.read().clone()
}
}
impl<T: BeaconChainTypes> Default for LightClientServerCache<T> {
fn default() -> Self {
Self::new()
}
}
type FinalityBranch = FixedVector<Hash256, FinalizedRootProofLen>;
#[derive(Clone)]
struct LightClientCachedData {
finality_branch: FinalityBranch,
finalized_block_root: Hash256,
}
impl LightClientCachedData {
fn from_state<E: EthSpec>(state: &mut BeaconState<E>) -> Result<Self, BeaconChainError> {
Ok(Self {
finality_branch: state.compute_merkle_proof(FINALIZED_ROOT_INDEX)?.into(),
finalized_block_root: state.finalized_checkpoint().root,
})
}
}
// Implements spec prioritization rules:
// > Full nodes SHOULD provide the LightClientFinalityUpdate with the highest attested_header.beacon.slot (if multiple, highest signature_slot)
//
// ref: https://github.com/ethereum/consensus-specs/blob/113c58f9bf9c08867f6f5f633c4d98e0364d612a/specs/altair/light-client/full-node.md#create_light_client_finality_update
fn is_latest_finality_update<E: EthSpec>(
prev: &LightClientFinalityUpdate<E>,
attested_slot: Slot,
signature_slot: Slot,
) -> bool {
let prev_slot = prev.get_attested_header_slot();
if attested_slot > prev_slot {
true
} else {
attested_slot == prev_slot && signature_slot > *prev.signature_slot()
}
}
// Implements spec prioritization rules:
// > Full nodes SHOULD provide the LightClientOptimisticUpdate with the highest attested_header.beacon.slot (if multiple, highest signature_slot)
//
// ref: https://github.com/ethereum/consensus-specs/blob/113c58f9bf9c08867f6f5f633c4d98e0364d612a/specs/altair/light-client/full-node.md#create_light_client_optimistic_update
fn is_latest_optimistic_update<E: EthSpec>(
prev: &LightClientOptimisticUpdate<E>,
attested_slot: Slot,
signature_slot: Slot,
) -> bool {
let prev_slot = prev.get_slot();
if attested_slot > prev_slot {
true
} else {
attested_slot == prev_slot && signature_slot > *prev.signature_slot()
}
}

View File

@@ -4,11 +4,21 @@ use crate::{BeaconChain, BeaconChainError, BeaconChainTypes};
use lazy_static::lazy_static;
pub use lighthouse_metrics::*;
use slot_clock::SlotClock;
use std::time::Duration;
use types::{BeaconState, Epoch, EthSpec, Hash256, Slot};
/// The maximum time to wait for the snapshot cache lock during a metrics scrape.
const SNAPSHOT_CACHE_TIMEOUT: Duration = Duration::from_millis(100);
// Attestation simulator metrics
pub const VALIDATOR_MONITOR_ATTESTATION_SIMULATOR_HEAD_ATTESTER_HIT_TOTAL: &str =
"validator_monitor_attestation_simulator_head_attester_hit_total";
pub const VALIDATOR_MONITOR_ATTESTATION_SIMULATOR_HEAD_ATTESTER_MISS_TOTAL: &str =
"validator_monitor_attestation_simulator_head_attester_miss_total";
pub const VALIDATOR_MONITOR_ATTESTATION_SIMULATOR_TARGET_ATTESTER_HIT_TOTAL: &str =
"validator_monitor_attestation_simulator_target_attester_hit_total";
pub const VALIDATOR_MONITOR_ATTESTATION_SIMULATOR_TARGET_ATTESTER_MISS_TOTAL: &str =
"validator_monitor_attestation_simulator_target_attester_miss_total";
pub const VALIDATOR_MONITOR_ATTESTATION_SIMULATOR_SOURCE_ATTESTER_HIT_TOTAL: &str =
"validator_monitor_attestation_simulator_source_attester_hit_total";
pub const VALIDATOR_MONITOR_ATTESTATION_SIMULATOR_SOURCE_ATTESTER_MISS_TOTAL: &str =
"validator_monitor_attestation_simulator_source_attester_miss_total";
lazy_static! {
/*
@@ -22,24 +32,25 @@ lazy_static! {
"beacon_block_processing_successes_total",
"Count of blocks processed without error"
);
// Keeping the existing "snapshot_cache" metric name as it would break existing dashboards
pub static ref BLOCK_PROCESSING_SNAPSHOT_CACHE_SIZE: Result<IntGauge> = try_create_int_gauge(
"beacon_block_processing_snapshot_cache_size",
"Count snapshots in the snapshot cache"
);
pub static ref BLOCK_PROCESSING_SNAPSHOT_CACHE_MISSES: Result<IntCounter> = try_create_int_counter(
"beacon_block_processing_snapshot_cache_misses",
"Count of snapshot cache misses"
);
pub static ref BLOCK_PROCESSING_SNAPSHOT_CACHE_CLONES: Result<IntCounter> = try_create_int_counter(
"beacon_block_processing_snapshot_cache_clones",
"Count of snapshot cache clones"
);
pub static ref BLOCK_PROCESSING_TIMES: Result<Histogram> =
try_create_histogram("beacon_block_processing_seconds", "Full runtime of block processing");
pub static ref BLOCK_PROCESSING_BLOCK_ROOT: Result<Histogram> = try_create_histogram(
"beacon_block_processing_block_root_seconds",
"Time spent calculating the block root when processing a block."
);
pub static ref BLOCK_HEADER_PROCESSING_BLOCK_ROOT: Result<Histogram> = try_create_histogram(
"beacon_block_header_processing_block_root_seconds",
"Time spent calculating the block root for a beacon block header."
);
pub static ref BLOCK_PROCESSING_BLOB_ROOT: Result<Histogram> = try_create_histogram(
"beacon_block_processing_blob_root_seconds",
"Time spent calculating the blob root when processing a block."
);
pub static ref BLOCK_PROCESSING_DB_READ: Result<Histogram> = try_create_histogram(
"beacon_block_processing_db_read_seconds",
"Time spent loading block and state from DB for block processing"
@@ -282,6 +293,19 @@ lazy_static! {
"Count of times the early attester cache returns an attestation"
);
pub static ref BEACON_REQRESP_PRE_IMPORT_CACHE_SIZE: Result<IntGauge> = try_create_int_gauge(
"beacon_reqresp_pre_import_cache_size",
"Current count of items of the reqresp pre import cache"
);
pub static ref BEACON_REQRESP_PRE_IMPORT_CACHE_HITS: Result<IntCounter> = try_create_int_counter(
"beacon_reqresp_pre_import_cache_hits",
"Count of times the reqresp pre import cache returns an item"
);
}
// Second lazy-static block is used to account for macro recursion limit.
lazy_static! {
/*
* Attestation Production
*/
@@ -301,10 +325,7 @@ lazy_static! {
"attestation_production_cache_prime_seconds",
"Time spent loading a new state from the disk due to a cache miss"
);
}
// Second lazy-static block is used to account for macro recursion limit.
lazy_static! {
/*
* Fork Choice
*/
@@ -380,6 +401,8 @@ lazy_static! {
try_create_histogram("beacon_persist_eth1_cache", "Time taken to persist the eth1 caches");
pub static ref PERSIST_FORK_CHOICE: Result<Histogram> =
try_create_histogram("beacon_persist_fork_choice", "Time taken to persist the fork choice struct");
pub static ref PERSIST_DATA_AVAILABILITY_CHECKER: Result<Histogram> =
try_create_histogram("beacon_persist_data_availability_checker", "Time taken to persist the data availability checker");
/*
* Eth1
@@ -813,37 +836,55 @@ lazy_static! {
"Number of attester slashings seen",
&["src", "validator"]
);
}
// Prevent recursion limit
lazy_static! {
/*
* Block Delay Metrics
*/
pub static ref BEACON_BLOCK_OBSERVED_SLOT_START_DELAY_TIME: Result<Histogram> = try_create_histogram_with_buckets(
"beacon_block_observed_slot_start_delay_time",
"Duration between the start of the block's slot and the time the block was observed.",
// [0.1, 0.2, 0.5, 1, 2, 5, 10, 20, 50]
decimal_buckets(-1,2)
);
pub static ref BEACON_BLOCK_IMPORTED_OBSERVED_DELAY_TIME: Result<Histogram> = try_create_histogram_with_buckets(
"beacon_block_imported_observed_delay_time",
"Duration between the time the block was observed and the time when it was imported.",
// [0.01, 0.02, 0.05, 0.1, 0.2, 0.5, 1, 2, 5]
decimal_buckets(-2,0)
);
pub static ref BEACON_BLOCK_HEAD_IMPORTED_DELAY_TIME: Result<Histogram> = try_create_histogram_with_buckets(
"beacon_block_head_imported_delay_time",
"Duration between the time the block was imported and the time when it was set as head.",
// [0.01, 0.02, 0.05, 0.1, 0.2, 0.5, 1, 2, 5]
decimal_buckets(-2,-1)
);
pub static ref BEACON_BLOCK_HEAD_SLOT_START_DELAY_TIME: Result<Histogram> = try_create_histogram_with_buckets(
"beacon_block_head_slot_start_delay_time",
pub static ref BEACON_BLOCK_DELAY_TOTAL: Result<IntGauge> = try_create_int_gauge(
"beacon_block_delay_total",
"Duration between the start of the block's slot and the time when it was set as head.",
// [0.1, 0.2, 0.5, 1, 2, 5, 10, 20, 50]
decimal_buckets(-1,2)
);
pub static ref BEACON_BLOCK_HEAD_SLOT_START_DELAY_EXCEEDED_TOTAL: Result<IntCounter> = try_create_int_counter(
"beacon_block_head_slot_start_delay_exceeded_total",
"Triggered when the duration between the start of the block's slot and the current time \
pub static ref BEACON_BLOCK_DELAY_OBSERVED_SLOT_START: Result<IntGauge> = try_create_int_gauge(
"beacon_block_delay_observed_slot_start",
"Duration between the start of the block's slot and the time the block was observed.",
);
pub static ref BEACON_BLOB_DELAY_ALL_OBSERVED_SLOT_START: Result<IntGauge> = try_create_int_gauge(
"beacon_blob_delay_all_observed_slot_start",
"Duration between the start of the block's slot and the time the block was observed.",
);
pub static ref BEACON_BLOCK_DELAY_EXECUTION_TIME: Result<IntGauge> = try_create_int_gauge(
"beacon_block_delay_execution_time",
"The duration in verifying the block with the execution layer.",
);
pub static ref BEACON_BLOCK_DELAY_AVAILABLE_SLOT_START: Result<IntGauge> = try_create_int_gauge(
"beacon_block_delay_available_slot_start",
"Duration between the time that block became available and the start of the slot.",
);
pub static ref BEACON_BLOCK_DELAY_ATTESTABLE_SLOT_START: Result<IntGauge> = try_create_int_gauge(
"beacon_block_delay_attestable_slot_start",
"Duration between the time that block became attestable and the start of the slot.",
);
pub static ref BEACON_BLOCK_DELAY_IMPORTED_TIME: Result<IntGauge> = try_create_int_gauge(
"beacon_block_delay_imported_time",
"Duration between the time the block became available and the time when it was imported.",
);
pub static ref BEACON_BLOCK_DELAY_HEAD_IMPORTED_TIME: Result<IntGauge> = try_create_int_gauge(
"beacon_block_delay_head_imported_time",
"Duration between the time that block was imported and the time when it was set as head.",
);
pub static ref BEACON_BLOCK_DELAY_HEAD_SLOT_START_EXCEEDED_TOTAL: Result<IntCounter> = try_create_int_counter(
"beacon_block_delay_head_slot_start_exceeded_total",
"A counter that is triggered when the duration between the start of the block's slot and the current time \
will result in failed attestations.",
);
@@ -980,6 +1021,30 @@ lazy_static! {
"beacon_pre_finalization_block_lookup_count",
"Number of block roots subject to single block lookups"
);
/*
* Blob sidecar Verification
*/
pub static ref BLOBS_SIDECAR_PROCESSING_REQUESTS: Result<IntCounter> = try_create_int_counter(
"beacon_blobs_sidecar_processing_requests_total",
"Count of all blob sidecars submitted for processing"
);
pub static ref BLOBS_SIDECAR_PROCESSING_SUCCESSES: Result<IntCounter> = try_create_int_counter(
"beacon_blobs_sidecar_processing_successes_total",
"Number of blob sidecars verified for gossip"
);
pub static ref BLOBS_SIDECAR_GOSSIP_VERIFICATION_TIMES: Result<Histogram> = try_create_histogram(
"beacon_blobs_sidecar_gossip_verification_seconds",
"Full runtime of blob sidecars gossip verification"
);
pub static ref BLOB_SIDECAR_INCLUSION_PROOF_VERIFICATION: Result<Histogram> = try_create_histogram(
"blob_sidecar_inclusion_proof_verification_seconds",
"Time taken to verify blob sidecar inclusion proof"
);
pub static ref BLOB_SIDECAR_INCLUSION_PROOF_COMPUTATION: Result<Histogram> = try_create_histogram(
"blob_sidecar_inclusion_proof_computation_seconds",
"Time taken to compute blob sidecar inclusion proof"
);
}
// Fifth lazy-static block is used to account for macro recursion limit.
@@ -1009,6 +1074,108 @@ lazy_static! {
"beacon_aggregated_attestation_subsets_total",
"Count of new aggregated attestations that are subsets of already known aggregates"
);
/*
* Attestation simulator metrics
*/
pub static ref VALIDATOR_MONITOR_ATTESTATION_SIMULATOR_HEAD_ATTESTER_HIT: Result<IntCounter> =
try_create_int_counter(
VALIDATOR_MONITOR_ATTESTATION_SIMULATOR_HEAD_ATTESTER_HIT_TOTAL,
"Incremented if a validator is flagged as a previous slot head attester \
during per slot processing",
);
pub static ref VALIDATOR_MONITOR_ATTESTATION_SIMULATOR_HEAD_ATTESTER_MISS: Result<IntCounter> =
try_create_int_counter(
VALIDATOR_MONITOR_ATTESTATION_SIMULATOR_HEAD_ATTESTER_MISS_TOTAL,
"Incremented if a validator is not flagged as a previous slot head attester \
during per slot processing",
);
pub static ref VALIDATOR_MONITOR_ATTESTATION_SIMULATOR_TARGET_ATTESTER_HIT: Result<IntCounter> =
try_create_int_counter(
VALIDATOR_MONITOR_ATTESTATION_SIMULATOR_TARGET_ATTESTER_HIT_TOTAL,
"Incremented if a validator is flagged as a previous slot target attester \
during per slot processing",
);
pub static ref VALIDATOR_MONITOR_ATTESTATION_SIMULATOR_TARGET_ATTESTER_MISS: Result<IntCounter> =
try_create_int_counter(
VALIDATOR_MONITOR_ATTESTATION_SIMULATOR_TARGET_ATTESTER_MISS_TOTAL,
"Incremented if a validator is not flagged as a previous slot target attester \
during per slot processing",
);
pub static ref VALIDATOR_MONITOR_ATTESTATION_SIMULATOR_SOURCE_ATTESTER_HIT: Result<IntCounter> =
try_create_int_counter(
VALIDATOR_MONITOR_ATTESTATION_SIMULATOR_SOURCE_ATTESTER_HIT_TOTAL,
"Incremented if a validator is flagged as a previous slot source attester \
during per slot processing",
);
pub static ref VALIDATOR_MONITOR_ATTESTATION_SIMULATOR_SOURCE_ATTESTER_MISS: Result<IntCounter> =
try_create_int_counter(
VALIDATOR_MONITOR_ATTESTATION_SIMULATOR_SOURCE_ATTESTER_MISS_TOTAL,
"Incremented if a validator is not flagged as a previous slot source attester \
during per slot processing",
);
/*
* Missed block metrics
*/
pub static ref VALIDATOR_MONITOR_MISSED_BLOCKS_TOTAL: Result<IntCounterVec> = try_create_int_counter_vec(
"validator_monitor_missed_blocks_total",
"Number of non-finalized blocks missed",
&["validator"]
);
/*
* Kzg related metrics
*/
pub static ref KZG_VERIFICATION_SINGLE_TIMES: Result<Histogram> =
try_create_histogram("kzg_verification_single_seconds", "Runtime of single kzg verification");
pub static ref KZG_VERIFICATION_BATCH_TIMES: Result<Histogram> =
try_create_histogram("kzg_verification_batch_seconds", "Runtime of batched kzg verification");
pub static ref BLOCK_PRODUCTION_BLOBS_VERIFICATION_TIMES: Result<Histogram> = try_create_histogram(
"beacon_block_production_blobs_verification_seconds",
"Time taken to verify blobs against commitments and creating BlobSidecar objects in block production"
);
/*
* Availability related metrics
*/
pub static ref BLOCK_AVAILABILITY_DELAY: Result<IntGauge> = try_create_int_gauge(
"block_availability_delay",
"Duration between start of the slot and the time at which all components of the block are available.",
);
/*
* Data Availability cache metrics
*/
pub static ref DATA_AVAILABILITY_OVERFLOW_MEMORY_BLOCK_CACHE_SIZE: Result<IntGauge> =
try_create_int_gauge(
"data_availability_overflow_memory_block_cache_size",
"Number of entries in the data availability overflow block memory cache."
);
pub static ref DATA_AVAILABILITY_OVERFLOW_MEMORY_STATE_CACHE_SIZE: Result<IntGauge> =
try_create_int_gauge(
"data_availability_overflow_memory_state_cache_size",
"Number of entries in the data availability overflow state memory cache."
);
pub static ref DATA_AVAILABILITY_OVERFLOW_STORE_CACHE_SIZE: Result<IntGauge> =
try_create_int_gauge(
"data_availability_overflow_store_cache_size",
"Number of entries in the data availability overflow store cache."
);
/*
* light_client server metrics
*/
pub static ref LIGHT_CLIENT_SERVER_CACHE_STATE_DATA_TIMES: Result<Histogram> = try_create_histogram(
"beacon_light_client_server_cache_state_data_seconds",
"Time taken to produce and cache state data",
);
pub static ref LIGHT_CLIENT_SERVER_CACHE_RECOMPUTE_UPDATES_TIMES: Result<Histogram> = try_create_histogram(
"beacon_light_client_server_cache_recompute_updates_seconds",
"Time taken to recompute and cache updates",
);
pub static ref LIGHT_CLIENT_SERVER_CACHE_PREV_BLOCK_CACHE_MISS: Result<IntCounter> = try_create_int_counter(
"beacon_light_client_server_cache_prev_block_cache_miss",
"Count of prev block cache misses",
);
}
/// Scrape the `beacon_chain` for metrics that are not constantly updated (e.g., the present slot,
@@ -1025,16 +1192,31 @@ pub fn scrape_for_metrics<T: BeaconChainTypes>(beacon_chain: &BeaconChain<T>) {
}
let attestation_stats = beacon_chain.op_pool.attestation_stats();
let chain_metrics = beacon_chain.metrics();
if let Some(snapshot_cache) = beacon_chain
.snapshot_cache
.try_write_for(SNAPSHOT_CACHE_TIMEOUT)
{
set_gauge(
&BLOCK_PROCESSING_SNAPSHOT_CACHE_SIZE,
snapshot_cache.len() as i64,
)
}
set_gauge_by_usize(
&BLOCK_PROCESSING_SNAPSHOT_CACHE_SIZE,
beacon_chain.store.state_cache_len(),
);
set_gauge_by_usize(
&BEACON_REQRESP_PRE_IMPORT_CACHE_SIZE,
chain_metrics.reqresp_pre_import_cache_len,
);
let da_checker_metrics = beacon_chain.data_availability_checker.metrics();
set_gauge_by_usize(
&DATA_AVAILABILITY_OVERFLOW_MEMORY_BLOCK_CACHE_SIZE,
da_checker_metrics.block_cache_size,
);
set_gauge_by_usize(
&DATA_AVAILABILITY_OVERFLOW_MEMORY_STATE_CACHE_SIZE,
da_checker_metrics.state_cache_size,
);
set_gauge_by_usize(
&DATA_AVAILABILITY_OVERFLOW_STORE_CACHE_SIZE,
da_checker_metrics.num_store_entries,
);
if let Some((size, num_lookups)) = beacon_chain.pre_finalization_block_cache.metrics() {
set_gauge_by_usize(&PRE_FINALIZATION_BLOCK_CACHE_SIZE, size);
@@ -1077,7 +1259,7 @@ pub fn scrape_for_metrics<T: BeaconChainTypes>(beacon_chain: &BeaconChain<T>) {
}
/// Scrape the given `state` assuming it's the head state, updating the `DEFAULT_REGISTRY`.
fn scrape_head_state<T: EthSpec>(state: &BeaconState<T>, state_root: Hash256) {
fn scrape_head_state<E: EthSpec>(state: &BeaconState<E>, state_root: Hash256) {
set_gauge_by_slot(&HEAD_STATE_SLOT, state.slot());
set_gauge_by_slot(&HEAD_STATE_SLOT_INTEROP, state.slot());
set_gauge_by_hash(&HEAD_STATE_ROOT, state_root);

View File

@@ -117,6 +117,7 @@ pub enum PruningError {
pub enum Notification {
Finalization(FinalizationNotification),
Reconstruction,
PruneBlobs(Epoch),
}
pub struct FinalizationNotification {
@@ -191,6 +192,14 @@ impl<E: EthSpec, Hot: ItemStore<E>, Cold: ItemStore<E>> BackgroundMigrator<E, Ho
}
}
pub fn process_prune_blobs(&self, data_availability_boundary: Epoch) {
if let Some(Notification::PruneBlobs(data_availability_boundary)) =
self.send_background_notification(Notification::PruneBlobs(data_availability_boundary))
{
Self::run_prune_blobs(self.db.clone(), data_availability_boundary, &self.log);
}
}
pub fn run_reconstruction(db: Arc<HotColdDB<E, Hot, Cold>>, log: &Logger) {
if let Err(e) = db.reconstruct_historic_states() {
error!(
@@ -201,6 +210,20 @@ impl<E: EthSpec, Hot: ItemStore<E>, Cold: ItemStore<E>> BackgroundMigrator<E, Ho
}
}
pub fn run_prune_blobs(
db: Arc<HotColdDB<E, Hot, Cold>>,
data_availability_boundary: Epoch,
log: &Logger,
) {
if let Err(e) = db.try_prune_blobs(false, data_availability_boundary) {
error!(
log,
"Blob pruning failed";
"error" => ?e,
);
}
}
/// If configured to run in the background, send `notif` to the background thread.
///
/// Return `None` if the message was sent to the background thread, `Some(notif)` otherwise.
@@ -367,29 +390,44 @@ impl<E: EthSpec, Hot: ItemStore<E>, Cold: ItemStore<E>> BackgroundMigrator<E, Ho
let (tx, rx) = mpsc::channel();
let thread = thread::spawn(move || {
while let Ok(notif) = rx.recv() {
// Read the rest of the messages in the channel, preferring any reconstruction
// notification, or the finalization notification with the greatest finalized epoch.
let notif =
rx.try_iter()
.fold(notif, |best, other: Notification| match (&best, &other) {
(Notification::Reconstruction, _)
| (_, Notification::Reconstruction) => Notification::Reconstruction,
(
Notification::Finalization(fin1),
Notification::Finalization(fin2),
) => {
if fin2.finalized_checkpoint.epoch > fin1.finalized_checkpoint.epoch
{
other
} else {
best
}
}
});
let mut reconstruction_notif = None;
let mut finalization_notif = None;
let mut prune_blobs_notif = None;
match notif {
Notification::Reconstruction => Self::run_reconstruction(db.clone(), &log),
Notification::Finalization(fin) => Self::run_migration(db.clone(), fin, &log),
Notification::Reconstruction => reconstruction_notif = Some(notif),
Notification::Finalization(fin) => finalization_notif = Some(fin),
Notification::PruneBlobs(dab) => prune_blobs_notif = Some(dab),
}
// Read the rest of the messages in the channel, taking the best of each type.
for notif in rx.try_iter() {
match notif {
Notification::Reconstruction => reconstruction_notif = Some(notif),
Notification::Finalization(fin) => {
if let Some(current) = finalization_notif.as_mut() {
if fin.finalized_checkpoint.epoch
> current.finalized_checkpoint.epoch
{
*current = fin;
}
} else {
finalization_notif = Some(fin);
}
}
Notification::PruneBlobs(dab) => {
prune_blobs_notif = std::cmp::max(prune_blobs_notif, Some(dab));
}
}
}
// If reconstruction is on-going, ignore finalization migration and blob pruning.
if reconstruction_notif.is_some() {
Self::run_reconstruction(db.clone(), &log);
} else {
if let Some(fin) = finalization_notif {
Self::run_migration(db.clone(), fin, &log);
}
if let Some(dab) = prune_blobs_notif {
Self::run_prune_blobs(db.clone(), dab, &log);
}
}
}
});
@@ -630,13 +668,14 @@ impl<E: EthSpec, Hot: ItemStore<E>, Cold: ItemStore<E>> BackgroundMigrator<E, Ho
head_tracker_lock.remove(&head_hash);
}
let batch: Vec<StoreOp<E>> = abandoned_blocks
let mut batch: Vec<StoreOp<E>> = abandoned_blocks
.into_iter()
.map(Into::into)
.flat_map(|block_root: Hash256| {
[
StoreOp::DeleteBlock(block_root),
StoreOp::DeleteExecutionPayload(block_root),
StoreOp::DeleteBlobs(block_root),
]
})
.chain(
@@ -646,8 +685,6 @@ impl<E: EthSpec, Hot: ItemStore<E>, Cold: ItemStore<E>> BackgroundMigrator<E, Ho
)
.collect();
let mut kv_batch = store.convert_to_kv_batch(batch)?;
// Persist the head in case the process is killed or crashes here. This prevents
// the head tracker reverting after our mutation above.
let persisted_head = PersistedBeaconChain {
@@ -656,12 +693,21 @@ impl<E: EthSpec, Hot: ItemStore<E>, Cold: ItemStore<E>> BackgroundMigrator<E, Ho
ssz_head_tracker: SszHeadTracker::from_map(&head_tracker_lock),
};
drop(head_tracker_lock);
kv_batch.push(persisted_head.as_kv_store_op(BEACON_CHAIN_DB_KEY));
batch.push(StoreOp::KeyValueOp(
persisted_head.as_kv_store_op(BEACON_CHAIN_DB_KEY),
));
// Persist the new finalized checkpoint as the pruning checkpoint.
kv_batch.push(store.pruning_checkpoint_store_op(new_finalized_checkpoint));
batch.push(StoreOp::KeyValueOp(
store.pruning_checkpoint_store_op(new_finalized_checkpoint),
));
store.do_atomically_with_block_and_blobs_cache(batch)?;
// Do a quick separate pass to delete obsoleted hot states, usually pre-states from the state
// advance which are not canonical due to blocks being applied on top.
store.prune_old_hot_states()?;
store.hot_db.do_atomically(kv_batch)?;
debug!(log, "Database pruning complete");
Ok(PruningOutcome::Successful {

View File

@@ -1,14 +1,124 @@
use crate::metrics;
use crate::observed_aggregates::AsReference;
use itertools::Itertools;
use smallvec::SmallVec;
use std::collections::HashMap;
use tree_hash::TreeHash;
use tree_hash::{MerkleHasher, TreeHash, TreeHashType};
use types::consts::altair::SYNC_COMMITTEE_SUBNET_COUNT;
use types::slot_data::SlotData;
use types::sync_committee_contribution::SyncContributionData;
use types::{Attestation, AttestationData, EthSpec, Hash256, Slot, SyncCommitteeContribution};
use types::{
Attestation, AttestationData, AttestationRef, CommitteeIndex, EthSpec, Hash256, Slot,
SyncCommitteeContribution,
};
type AttestationDataRoot = Hash256;
type AttestationKeyRoot = Hash256;
type SyncDataRoot = Hash256;
/// Post-Electra, we need a new key for Attestations that includes the committee index
#[derive(Debug, Clone, PartialEq)]
pub struct AttestationKey {
data_root: Hash256,
committee_index: Option<CommitteeIndex>,
slot: Slot,
}
// A custom implementation of `TreeHash` such that:
// AttestationKey(data, None).tree_hash_root() == data.tree_hash_root()
// AttestationKey(data, Some(index)).tree_hash_root() == (data, index).tree_hash_root()
// This is necessary because pre-Electra, the validator will ask for the tree_hash_root()
// of the `AttestationData`
impl TreeHash for AttestationKey {
fn tree_hash_type() -> TreeHashType {
TreeHashType::Container
}
fn tree_hash_packed_encoding(&self) -> SmallVec<[u8; 32]> {
unreachable!("AttestationKey should never be packed.")
}
fn tree_hash_packing_factor() -> usize {
unreachable!("AttestationKey should never be packed.")
}
fn tree_hash_root(&self) -> Hash256 {
match self.committee_index {
None => self.data_root, // Return just the data root if no committee index is present
Some(index) => {
// Combine the hash of the data with the hash of the index
let mut hasher = MerkleHasher::with_leaves(2);
hasher
.write(self.data_root.as_bytes())
.expect("should write data hash");
hasher
.write(&index.to_le_bytes())
.expect("should write index");
hasher.finish().expect("should give tree hash")
}
}
}
}
impl AttestationKey {
pub fn from_attestation_ref<E: EthSpec>(attestation: AttestationRef<E>) -> Result<Self, Error> {
let slot = attestation.data().slot;
match attestation {
AttestationRef::Base(att) => Ok(Self {
data_root: att.data.tree_hash_root(),
committee_index: None,
slot,
}),
AttestationRef::Electra(att) => {
let committee_index = att
.committee_bits
.iter()
.enumerate()
.filter_map(|(i, bit)| if bit { Some(i) } else { None })
.at_most_one()
.map_err(|_| Error::MoreThanOneCommitteeBitSet)?
.ok_or(Error::NoCommitteeBitSet)?;
Ok(Self {
data_root: att.data.tree_hash_root(),
committee_index: Some(committee_index as u64),
slot,
})
}
}
}
pub fn new_base(data: &AttestationData) -> Self {
let slot = data.slot;
Self {
data_root: data.tree_hash_root(),
committee_index: None,
slot,
}
}
pub fn new_electra(slot: Slot, data_root: Hash256, committee_index: CommitteeIndex) -> Self {
Self {
data_root,
committee_index: Some(committee_index),
slot,
}
}
pub fn new_base_from_slot_and_root(slot: Slot, data_root: Hash256) -> Self {
Self {
data_root,
committee_index: None,
slot,
}
}
}
impl SlotData for AttestationKey {
fn get_slot(&self) -> Slot {
self.slot
}
}
/// The number of slots that will be stored in the pool.
///
/// For example, if `SLOTS_RETAINED == 3` and the pool is pruned at slot `6`, then all items
@@ -46,6 +156,10 @@ pub enum Error {
/// The given `aggregation_bits` field had more than one signature. The number of
/// signatures found is included.
MoreThanOneAggregationBitSet(usize),
/// The electra attestation has more than one committee bit set
MoreThanOneCommitteeBitSet,
/// The electra attestation has NO committee bit set
NoCommitteeBitSet,
/// We have reached the maximum number of unique items that can be stored in a
/// slot. This is a DoS protection function.
ReachedMaxItemsPerSlot(usize),
@@ -59,12 +173,15 @@ pub enum Error {
/// Implemented for items in the `NaiveAggregationPool`. Requires that items implement `SlotData`,
/// which means they have an associated slot. This handles aggregation of items that are inserted.
pub trait AggregateMap {
pub trait AggregateMap
where
for<'a> <Self::Value as AsReference>::Reference<'a>: SlotData,
{
/// `Key` should be a hash of `Data`.
type Key;
/// The item stored in the map
type Value: Clone + SlotData;
type Value: Clone + SlotData + AsReference;
/// The unique fields of `Value`, hashed to create `Key`.
type Data: SlotData;
@@ -73,7 +190,10 @@ pub trait AggregateMap {
fn new(initial_capacity: usize) -> Self;
/// Insert a `Value` into `Self`, returning a result.
fn insert(&mut self, value: &Self::Value) -> Result<InsertOutcome, Error>;
fn insert(
&mut self,
value: <Self::Value as AsReference>::Reference<'_>,
) -> Result<InsertOutcome, Error>;
/// Get a `Value` from `Self` based on `Data`.
fn get(&self, data: &Self::Data) -> Option<Self::Value>;
@@ -81,9 +201,6 @@ pub trait AggregateMap {
/// Get a reference to the inner `HashMap`.
fn get_map(&self) -> &HashMap<Self::Key, Self::Value>;
/// Get a `Value` from `Self` based on `Key`, which is a hash of `Data`.
fn get_by_root(&self, root: &Self::Key) -> Option<&Self::Value>;
/// The number of items store in `Self`.
fn len(&self) -> usize;
@@ -103,13 +220,13 @@ pub trait AggregateMap {
/// A collection of `Attestation` objects, keyed by their `attestation.data`. Enforces that all
/// `attestation` are from the same slot.
pub struct AggregatedAttestationMap<E: EthSpec> {
map: HashMap<AttestationDataRoot, Attestation<E>>,
map: HashMap<AttestationKeyRoot, Attestation<E>>,
}
impl<E: EthSpec> AggregateMap for AggregatedAttestationMap<E> {
type Key = AttestationDataRoot;
type Key = AttestationKeyRoot;
type Value = Attestation<E>;
type Data = AttestationData;
type Data = AttestationKey;
/// Create an empty collection with the given `initial_capacity`.
fn new(initial_capacity: usize) -> Self {
@@ -121,48 +238,45 @@ impl<E: EthSpec> AggregateMap for AggregatedAttestationMap<E> {
/// Insert an attestation into `self`, aggregating it into the pool.
///
/// The given attestation (`a`) must only have one signature.
fn insert(&mut self, a: &Self::Value) -> Result<InsertOutcome, Error> {
fn insert(&mut self, a: AttestationRef<E>) -> Result<InsertOutcome, Error> {
let _timer = metrics::start_timer(&metrics::ATTESTATION_PROCESSING_AGG_POOL_CORE_INSERT);
let set_bits = a
.aggregation_bits
let aggregation_bit = *a
.set_aggregation_bits()
.iter()
.enumerate()
.filter(|(_i, bit)| *bit)
.map(|(i, _bit)| i)
.collect::<Vec<_>>();
let committee_index = set_bits
.first()
.copied()
.at_most_one()
.map_err(|iter| Error::MoreThanOneAggregationBitSet(iter.count()))?
.ok_or(Error::NoAggregationBitsSet)?;
if set_bits.len() > 1 {
return Err(Error::MoreThanOneAggregationBitSet(set_bits.len()));
}
let attestation_key = AttestationKey::from_attestation_ref(a)?;
let attestation_key_root = attestation_key.tree_hash_root();
let attestation_data_root = a.data.tree_hash_root();
if let Some(existing_attestation) = self.map.get_mut(&attestation_data_root) {
if let Some(existing_attestation) = self.map.get_mut(&attestation_key_root) {
if existing_attestation
.aggregation_bits
.get(committee_index)
.get_aggregation_bit(aggregation_bit)
.map_err(|_| Error::InconsistentBitfieldLengths)?
{
Ok(InsertOutcome::SignatureAlreadyKnown { committee_index })
Ok(InsertOutcome::SignatureAlreadyKnown {
committee_index: aggregation_bit,
})
} else {
let _timer =
metrics::start_timer(&metrics::ATTESTATION_PROCESSING_AGG_POOL_AGGREGATION);
existing_attestation.aggregate(a);
Ok(InsertOutcome::SignatureAggregated { committee_index })
Ok(InsertOutcome::SignatureAggregated {
committee_index: aggregation_bit,
})
}
} else {
if self.map.len() >= MAX_ATTESTATIONS_PER_SLOT {
return Err(Error::ReachedMaxItemsPerSlot(MAX_ATTESTATIONS_PER_SLOT));
}
self.map.insert(attestation_data_root, a.clone());
Ok(InsertOutcome::NewItemInserted { committee_index })
self.map
.insert(attestation_key_root, a.clone_as_attestation());
Ok(InsertOutcome::NewItemInserted {
committee_index: aggregation_bit,
})
}
}
@@ -177,11 +291,6 @@ impl<E: EthSpec> AggregateMap for AggregatedAttestationMap<E> {
&self.map
}
/// Returns an aggregated `Attestation` with the given `root`, if any.
fn get_by_root(&self, root: &Self::Key) -> Option<&Self::Value> {
self.map.get(root)
}
fn len(&self) -> usize {
self.map.len()
}
@@ -288,11 +397,6 @@ impl<E: EthSpec> AggregateMap for SyncContributionAggregateMap<E> {
&self.map
}
/// Returns an aggregated `SyncCommitteeContribution` with the given `root`, if any.
fn get_by_root(&self, root: &SyncDataRoot) -> Option<&SyncCommitteeContribution<E>> {
self.map.get(root)
}
fn len(&self) -> usize {
self.map.len()
}
@@ -336,12 +440,20 @@ impl<E: EthSpec> AggregateMap for SyncContributionAggregateMap<E> {
/// `current_slot - SLOTS_RETAINED` will be removed and any future item with a slot lower
/// than that will also be refused. Pruning is done automatically based upon the items it
/// receives and it can be triggered manually.
pub struct NaiveAggregationPool<T: AggregateMap> {
pub struct NaiveAggregationPool<T>
where
T: AggregateMap,
for<'a> <T::Value as AsReference>::Reference<'a>: SlotData,
{
lowest_permissible_slot: Slot,
maps: HashMap<Slot, T>,
}
impl<T: AggregateMap> Default for NaiveAggregationPool<T> {
impl<T> Default for NaiveAggregationPool<T>
where
T: AggregateMap,
for<'a> <T::Value as AsReference>::Reference<'a>: SlotData,
{
fn default() -> Self {
Self {
lowest_permissible_slot: Slot::new(0),
@@ -350,7 +462,11 @@ impl<T: AggregateMap> Default for NaiveAggregationPool<T> {
}
}
impl<T: AggregateMap> NaiveAggregationPool<T> {
impl<T> NaiveAggregationPool<T>
where
T: AggregateMap,
for<'a> <T::Value as AsReference>::Reference<'a>: SlotData,
{
/// Insert an item into `self`, aggregating it into the pool.
///
/// The given item must only have one signature and have an
@@ -358,7 +474,10 @@ impl<T: AggregateMap> NaiveAggregationPool<T> {
///
/// The pool may be pruned if the given item has a slot higher than any
/// previously seen.
pub fn insert(&mut self, item: &T::Value) -> Result<InsertOutcome, Error> {
pub fn insert(
&mut self,
item: <T::Value as AsReference>::Reference<'_>,
) -> Result<InsertOutcome, Error> {
let _timer = T::start_insert_timer();
let slot = item.get_slot();
let lowest_permissible_slot = self.lowest_permissible_slot;
@@ -412,13 +531,6 @@ impl<T: AggregateMap> NaiveAggregationPool<T> {
.and_then(|map| map.get(data))
}
/// Returns an aggregated `T::Value` with the given `slot` and `root`, if any.
pub fn get_by_slot_and_root(&self, slot: Slot, root: &T::Key) -> Option<T::Value> {
self.maps
.get(&slot)
.and_then(|map| map.get_by_root(root).cloned())
}
/// Iterate all items in all slots of `self`.
pub fn iter(&self) -> impl Iterator<Item = &T::Value> {
self.maps.values().flat_map(|map| map.get_map().values())
@@ -467,18 +579,30 @@ mod tests {
use super::*;
use ssz_types::BitList;
use store::BitVector;
use tree_hash::TreeHash;
use types::{
test_utils::{generate_deterministic_keypair, test_random_instance},
Fork, Hash256, SyncCommitteeMessage,
Attestation, AttestationBase, AttestationElectra, Fork, Hash256, SyncCommitteeMessage,
};
type E = types::MainnetEthSpec;
fn get_attestation(slot: Slot) -> Attestation<E> {
let mut a: Attestation<E> = test_random_instance();
fn get_attestation_base(slot: Slot) -> Attestation<E> {
let mut a: AttestationBase<E> = test_random_instance();
a.data.slot = slot;
a.aggregation_bits = BitList::with_capacity(4).expect("should create bitlist");
a
Attestation::Base(a)
}
fn get_attestation_electra(slot: Slot) -> Attestation<E> {
let mut a: AttestationElectra<E> = test_random_instance();
a.data.slot = slot;
a.aggregation_bits = BitList::with_capacity(4).expect("should create bitlist");
a.committee_bits = BitVector::new();
a.committee_bits
.set(0, true)
.expect("should set committee bit");
Attestation::Electra(a)
}
fn get_sync_contribution(slot: Slot) -> SyncCommitteeContribution<E> {
@@ -521,9 +645,16 @@ mod tests {
}
fn unset_attestation_bit(a: &mut Attestation<E>, i: usize) {
a.aggregation_bits
.set(i, false)
.expect("should unset aggregation bit")
match a {
Attestation::Base(ref mut att) => att
.aggregation_bits
.set(i, false)
.expect("should unset aggregation bit"),
Attestation::Electra(ref mut att) => att
.aggregation_bits
.set(i, false)
.expect("should unset aggregation bit"),
}
}
fn unset_sync_contribution_bit(a: &mut SyncCommitteeContribution<E>, i: usize) {
@@ -533,19 +664,19 @@ mod tests {
}
fn mutate_attestation_block_root(a: &mut Attestation<E>, block_root: Hash256) {
a.data.beacon_block_root = block_root
a.data_mut().beacon_block_root = block_root
}
fn mutate_attestation_slot(a: &mut Attestation<E>, slot: Slot) {
a.data.slot = slot
a.data_mut().slot = slot
}
fn attestation_block_root_comparator(a: &Attestation<E>, block_root: Hash256) -> bool {
a.data.beacon_block_root == block_root
a.data().beacon_block_root == block_root
}
fn key_from_attestation(a: &Attestation<E>) -> AttestationData {
a.data.clone()
fn key_from_attestation(a: &Attestation<E>) -> AttestationKey {
AttestationKey::from_attestation_ref(a.to_ref()).expect("should create attestation key")
}
fn mutate_sync_contribution_block_root(
@@ -570,6 +701,45 @@ mod tests {
SyncContributionData::from_contribution(a)
}
#[test]
fn attestation_key_tree_hash_tests() {
let attestation_base = get_attestation_base(Slot::new(42));
// for a base attestation, the tree_hash_root() of the key should be the same as the tree_hash_root() of the data
let attestation_key_base = AttestationKey::from_attestation_ref(attestation_base.to_ref())
.expect("should create attestation key");
assert_eq!(
attestation_key_base.tree_hash_root(),
attestation_base.data().tree_hash_root()
);
let mut attestation_electra = get_attestation_electra(Slot::new(42));
// for an electra attestation, the tree_hash_root() of the key should be different from the tree_hash_root() of the data
let attestation_key_electra =
AttestationKey::from_attestation_ref(attestation_electra.to_ref())
.expect("should create attestation key");
assert_ne!(
attestation_key_electra.tree_hash_root(),
attestation_electra.data().tree_hash_root()
);
// for an electra attestation, the tree_hash_root() of the key should be dependent on which committee bit is set
let committe_bits = attestation_electra
.committee_bits_mut()
.expect("should get committee bits");
committe_bits
.set(0, false)
.expect("should set committee bit");
committe_bits
.set(1, true)
.expect("should set committee bit");
let new_attestation_key_electra =
AttestationKey::from_attestation_ref(attestation_electra.to_ref())
.expect("should create attestation key");
// this new key should have a different tree_hash_root() than the previous key
assert_ne!(
attestation_key_electra.tree_hash_root(),
new_attestation_key_electra.tree_hash_root()
);
}
macro_rules! test_suite {
(
$mod_name: ident,
@@ -592,10 +762,10 @@ mod tests {
let mut a = $get_method_name(Slot::new(0));
let mut pool: NaiveAggregationPool<$map_type<E>> =
NaiveAggregationPool::default();
NaiveAggregationPool::<$map_type<E>>::default();
assert_eq!(
pool.insert(&a),
pool.insert(a.as_reference()),
Err(Error::NoAggregationBitsSet),
"should not accept item without any signatures"
);
@@ -603,12 +773,12 @@ mod tests {
$sign_method_name(&mut a, 0, Hash256::random());
assert_eq!(
pool.insert(&a),
pool.insert(a.as_reference()),
Ok(InsertOutcome::NewItemInserted { committee_index: 0 }),
"should accept new item"
);
assert_eq!(
pool.insert(&a),
pool.insert(a.as_reference()),
Ok(InsertOutcome::SignatureAlreadyKnown { committee_index: 0 }),
"should acknowledge duplicate signature"
);
@@ -621,7 +791,7 @@ mod tests {
$sign_method_name(&mut a, 1, Hash256::random());
assert_eq!(
pool.insert(&a),
pool.insert(a.as_reference()),
Err(Error::MoreThanOneAggregationBitSet(2)),
"should not accept item with multiple signatures"
);
@@ -637,15 +807,15 @@ mod tests {
$sign_method_name(&mut a_1, 1, genesis_validators_root);
let mut pool: NaiveAggregationPool<$map_type<E>> =
NaiveAggregationPool::default();
NaiveAggregationPool::<$map_type<E>>::default();
assert_eq!(
pool.insert(&a_0),
pool.insert(a_0.as_reference()),
Ok(InsertOutcome::NewItemInserted { committee_index: 0 }),
"should accept a_0"
);
assert_eq!(
pool.insert(&a_1),
pool.insert(a_1.as_reference()),
Ok(InsertOutcome::SignatureAggregated { committee_index: 1 }),
"should accept a_1"
);
@@ -655,7 +825,7 @@ mod tests {
.expect("should not error while getting attestation");
let mut a_01 = a_0.clone();
a_01.aggregate(&a_1);
a_01.aggregate(a_1.as_reference());
assert_eq!(retrieved, a_01, "retrieved item should be aggregated");
@@ -671,7 +841,7 @@ mod tests {
$block_root_mutator(&mut a_different, different_root);
assert_eq!(
pool.insert(&a_different),
pool.insert(a_different.as_reference()),
Ok(InsertOutcome::NewItemInserted { committee_index: 2 }),
"should accept a_different"
);
@@ -690,7 +860,7 @@ mod tests {
$sign_method_name(&mut base, 0, Hash256::random());
let mut pool: NaiveAggregationPool<$map_type<E>> =
NaiveAggregationPool::default();
NaiveAggregationPool::<$map_type<E>>::default();
for i in 0..SLOTS_RETAINED * 2 {
let slot = Slot::from(i);
@@ -698,7 +868,7 @@ mod tests {
$slot_mutator(&mut a, slot);
assert_eq!(
pool.insert(&a),
pool.insert(a.as_reference()),
Ok(InsertOutcome::NewItemInserted { committee_index: 0 }),
"should accept new item"
);
@@ -739,7 +909,7 @@ mod tests {
$sign_method_name(&mut base, 0, Hash256::random());
let mut pool: NaiveAggregationPool<$map_type<E>> =
NaiveAggregationPool::default();
NaiveAggregationPool::<$map_type<E>>::default();
for i in 0..=$item_limit {
let mut a = base.clone();
@@ -747,13 +917,13 @@ mod tests {
if i < $item_limit {
assert_eq!(
pool.insert(&a),
pool.insert(a.as_reference()),
Ok(InsertOutcome::NewItemInserted { committee_index: 0 }),
"should accept item below limit"
);
} else {
assert_eq!(
pool.insert(&a),
pool.insert(a.as_reference()),
Err(Error::ReachedMaxItemsPerSlot($item_limit)),
"should not accept item above limit"
);
@@ -765,8 +935,21 @@ mod tests {
}
test_suite! {
attestation_tests,
get_attestation,
attestation_tests_base,
get_attestation_base,
sign_attestation,
unset_attestation_bit,
mutate_attestation_block_root,
mutate_attestation_slot,
attestation_block_root_comparator,
key_from_attestation,
AggregatedAttestationMap,
MAX_ATTESTATIONS_PER_SLOT
}
test_suite! {
attestation_tests_electra,
get_attestation_electra,
sign_attestation,
unset_attestation_bit,
mutate_attestation_block_root,

View File

@@ -6,22 +6,33 @@ use ssz_types::{BitList, BitVector};
use std::collections::HashMap;
use std::marker::PhantomData;
use tree_hash::TreeHash;
use tree_hash_derive::TreeHash;
use types::consts::altair::{
SYNC_COMMITTEE_SUBNET_COUNT, TARGET_AGGREGATORS_PER_SYNC_SUBCOMMITTEE,
};
use types::slot_data::SlotData;
use types::{Attestation, EthSpec, Hash256, Slot, SyncCommitteeContribution};
use types::{
Attestation, AttestationData, AttestationRef, EthSpec, Hash256, Slot, SyncCommitteeContribution,
};
pub type ObservedSyncContributions<E> = ObservedAggregates<
SyncCommitteeContribution<E>,
E,
BitVector<<E as types::EthSpec>::SyncSubcommitteeSize>,
>;
pub type ObservedAggregateAttestations<E> = ObservedAggregates<
Attestation<E>,
E,
BitList<<E as types::EthSpec>::MaxValidatorsPerCommittee>,
>;
pub type ObservedAggregateAttestations<E> =
ObservedAggregates<Attestation<E>, E, BitList<<E as types::EthSpec>::MaxValidatorsPerSlot>>;
/// Attestation data augmented with committee index
///
/// This is hashed and used to key the map of observed aggregate attestations. This is important
/// post-Electra where the attestation data committee index is 0 and we want to avoid accidentally
/// comparing aggregation bits for *different* committees.
#[derive(TreeHash)]
pub struct ObservedAttestationKey {
pub committee_index: u64,
pub attestation_data: AttestationData,
}
/// A trait use to associate capacity constants with the type being stored in `ObservedAggregates`.
pub trait Consts {
@@ -35,7 +46,7 @@ pub trait Consts {
fn max_per_slot_capacity() -> usize;
}
impl<T: EthSpec> Consts for Attestation<T> {
impl<E: EthSpec> Consts for Attestation<E> {
/// Use 128 as it's the target committee size for the mainnet spec. This is perhaps a little
/// wasteful for the minimal spec, but considering it's approx. 128 * 32 bytes we're not wasting
/// much.
@@ -43,7 +54,7 @@ impl<T: EthSpec> Consts for Attestation<T> {
/// We need to keep attestations for each slot of the current epoch.
fn max_slot_capacity() -> usize {
T::slots_per_epoch() as usize
2 * E::slots_per_epoch() as usize
}
/// As a DoS protection measure, the maximum number of distinct `Attestations` or
@@ -62,7 +73,7 @@ impl<T: EthSpec> Consts for Attestation<T> {
}
}
impl<T: EthSpec> Consts for SyncCommitteeContribution<T> {
impl<E: EthSpec> Consts for SyncCommitteeContribution<E> {
/// Set to `TARGET_AGGREGATORS_PER_SYNC_SUBCOMMITTEE * SYNC_COMMITTEE_SUBNET_COUNT`. This is the
/// expected number of aggregators per slot across all subcommittees.
const DEFAULT_PER_SLOT_CAPACITY: usize =
@@ -75,7 +86,7 @@ impl<T: EthSpec> Consts for SyncCommitteeContribution<T> {
/// We should never receive more aggregates than there are sync committee participants.
fn max_per_slot_capacity() -> usize {
T::sync_committee_size()
E::sync_committee_size()
}
}
@@ -95,15 +106,61 @@ pub trait SubsetItem {
/// Returns the item that gets stored in `ObservedAggregates` for later subset
/// comparison with incoming aggregates.
fn get_item(&self) -> Self::Item;
fn get_item(&self) -> Result<Self::Item, Error>;
/// Returns a unique value that keys the object to the item that is being stored
/// in `ObservedAggregates`.
fn root(&self) -> Hash256;
fn root(&self) -> Result<Hash256, Error>;
}
impl<T: EthSpec> SubsetItem for Attestation<T> {
type Item = BitList<T::MaxValidatorsPerCommittee>;
impl<'a, E: EthSpec> SubsetItem for AttestationRef<'a, E> {
type Item = BitList<E::MaxValidatorsPerSlot>;
fn is_subset(&self, other: &Self::Item) -> bool {
match self {
Self::Base(att) => {
if let Ok(extended_aggregation_bits) = att.extend_aggregation_bits() {
return extended_aggregation_bits.is_subset(other);
}
false
}
Self::Electra(att) => att.aggregation_bits.is_subset(other),
}
}
fn is_superset(&self, other: &Self::Item) -> bool {
match self {
Self::Base(att) => {
if let Ok(extended_aggregation_bits) = att.extend_aggregation_bits() {
return other.is_subset(&extended_aggregation_bits);
}
false
}
Self::Electra(att) => other.is_subset(&att.aggregation_bits),
}
}
/// Returns the sync contribution aggregation bits.
fn get_item(&self) -> Result<Self::Item, Error> {
match self {
Self::Base(att) => att
.extend_aggregation_bits()
.map_err(|_| Error::GetItemError),
Self::Electra(att) => Ok(att.aggregation_bits.clone()),
}
}
/// Returns the hash tree root of the attestation data augmented with the committee index.
fn root(&self) -> Result<Hash256, Error> {
Ok(ObservedAttestationKey {
committee_index: self.committee_index().ok_or(Error::RootError)?,
attestation_data: self.data().clone(),
}
.tree_hash_root())
}
}
impl<'a, E: EthSpec> SubsetItem for &'a SyncCommitteeContribution<E> {
type Item = BitVector<E::SyncSubcommitteeSize>;
fn is_subset(&self, other: &Self::Item) -> bool {
self.aggregation_bits.is_subset(other)
}
@@ -113,40 +170,19 @@ impl<T: EthSpec> SubsetItem for Attestation<T> {
}
/// Returns the sync contribution aggregation bits.
fn get_item(&self) -> Self::Item {
self.aggregation_bits.clone()
}
/// Returns the hash tree root of the attestation data.
fn root(&self) -> Hash256 {
self.data.tree_hash_root()
}
}
impl<T: EthSpec> SubsetItem for SyncCommitteeContribution<T> {
type Item = BitVector<T::SyncSubcommitteeSize>;
fn is_subset(&self, other: &Self::Item) -> bool {
self.aggregation_bits.is_subset(other)
}
fn is_superset(&self, other: &Self::Item) -> bool {
other.is_subset(&self.aggregation_bits)
}
/// Returns the sync contribution aggregation bits.
fn get_item(&self) -> Self::Item {
self.aggregation_bits.clone()
fn get_item(&self) -> Result<Self::Item, Error> {
Ok(self.aggregation_bits.clone())
}
/// Returns the hash tree root of the root, slot and subcommittee index
/// of the sync contribution.
fn root(&self) -> Hash256 {
SyncCommitteeData {
fn root(&self) -> Result<Hash256, Error> {
Ok(SyncCommitteeData {
root: self.beacon_block_root,
slot: self.slot,
subcommittee_index: self.subcommittee_index,
}
.tree_hash_root()
.tree_hash_root())
}
}
@@ -173,6 +209,8 @@ pub enum Error {
expected: Slot,
attestation: Slot,
},
GetItemError,
RootError,
}
/// A `HashMap` that contains entries related to some `Slot`.
@@ -196,7 +234,7 @@ impl<I> SlotHashSet<I> {
/// Store the items in self so future observations recognise its existence.
pub fn observe_item<S: SlotData + SubsetItem<Item = I>>(
&mut self,
item: &S,
item: S,
root: Hash256,
) -> Result<ObserveOutcome, Error> {
if item.get_slot() != self.slot {
@@ -215,7 +253,7 @@ impl<I> SlotHashSet<I> {
// If true, we replace the new item with its existing subset. This allows us
// to hold fewer items in the list.
} else if item.is_superset(existing) {
*existing = item.get_item();
*existing = item.get_item()?;
return Ok(ObserveOutcome::New);
}
}
@@ -233,7 +271,7 @@ impl<I> SlotHashSet<I> {
return Err(Error::ReachedMaxObservationsPerSlot(self.max_capacity));
}
let item = item.get_item();
let item = item.get_item()?;
self.map.entry(root).or_default().push(item);
Ok(ObserveOutcome::New)
}
@@ -242,7 +280,7 @@ impl<I> SlotHashSet<I> {
/// the given root and slot.
pub fn is_known_subset<S: SlotData + SubsetItem<Item = I>>(
&self,
item: &S,
item: S,
root: Hash256,
) -> Result<bool, Error> {
if item.get_slot() != self.slot {
@@ -264,16 +302,43 @@ impl<I> SlotHashSet<I> {
}
}
/// Trait for observable items that can be observed from their reference type.
///
/// This is used to make observations for `Attestation`s from `AttestationRef`s.
pub trait AsReference {
type Reference<'a>
where
Self: 'a;
fn as_reference(&self) -> Self::Reference<'_>;
}
impl<E: EthSpec> AsReference for Attestation<E> {
type Reference<'a> = AttestationRef<'a, E>;
fn as_reference(&self) -> AttestationRef<'_, E> {
self.to_ref()
}
}
impl<E: EthSpec> AsReference for SyncCommitteeContribution<E> {
type Reference<'a> = &'a Self;
fn as_reference(&self) -> &Self {
self
}
}
/// Stores the roots of objects for some number of `Slots`, so we can determine if
/// these have previously been seen on the network.
pub struct ObservedAggregates<T: SlotData + Consts, E: EthSpec, I> {
pub struct ObservedAggregates<T: Consts + AsReference, E: EthSpec, I> {
lowest_permissible_slot: Slot,
sets: Vec<SlotHashSet<I>>,
_phantom_spec: PhantomData<E>,
_phantom_tree_hash: PhantomData<T>,
}
impl<T: SlotData + Consts, E: EthSpec, I> Default for ObservedAggregates<T, E, I> {
impl<T: Consts + AsReference, E: EthSpec, I> Default for ObservedAggregates<T, E, I> {
fn default() -> Self {
Self {
lowest_permissible_slot: Slot::new(0),
@@ -284,17 +349,22 @@ impl<T: SlotData + Consts, E: EthSpec, I> Default for ObservedAggregates<T, E, I
}
}
impl<T: SlotData + Consts + SubsetItem<Item = I>, E: EthSpec, I> ObservedAggregates<T, E, I> {
impl<T, E, I> ObservedAggregates<T, E, I>
where
T: Consts + AsReference,
E: EthSpec,
for<'a> T::Reference<'a>: SubsetItem<Item = I> + SlotData,
{
/// Store `item` in `self` keyed at `root`.
///
/// `root` must equal `item.root::<SubsetItem>()`.
pub fn observe_item(
&mut self,
item: &T,
item: T::Reference<'_>,
root_opt: Option<Hash256>,
) -> Result<ObserveOutcome, Error> {
let index = self.get_set_index(item.get_slot())?;
let root = root_opt.unwrap_or_else(|| item.root());
let root = root_opt.map_or_else(|| item.root(), Ok)?;
self.sets
.get_mut(index)
@@ -307,7 +377,11 @@ impl<T: SlotData + Consts + SubsetItem<Item = I>, E: EthSpec, I> ObservedAggrega
///
/// `root` must equal `item.root::<SubsetItem>()`.
#[allow(clippy::wrong_self_convention)]
pub fn is_known_subset(&mut self, item: &T, root: Hash256) -> Result<bool, Error> {
pub fn is_known_subset(
&mut self,
item: T::Reference<'_>,
root: Hash256,
) -> Result<bool, Error> {
let index = self.get_set_index(item.get_slot())?;
self.sets
@@ -399,14 +473,15 @@ impl<T: SlotData + Consts + SubsetItem<Item = I>, E: EthSpec, I> ObservedAggrega
#[cfg(not(debug_assertions))]
mod tests {
use super::*;
use types::{test_utils::test_random_instance, Hash256};
use types::{test_utils::test_random_instance, AttestationBase, Hash256};
type E = types::MainnetEthSpec;
fn get_attestation(slot: Slot, beacon_block_root: u64) -> Attestation<E> {
let mut a: Attestation<E> = test_random_instance();
a.data.slot = slot;
a.data.beacon_block_root = Hash256::from_low_u64_be(beacon_block_root);
let a: AttestationBase<E> = test_random_instance();
let mut a = Attestation::Base(a);
a.data_mut().slot = slot;
a.data_mut().beacon_block_root = Hash256::from_low_u64_be(beacon_block_root);
a
}
@@ -432,12 +507,15 @@ mod tests {
for a in &items {
assert_eq!(
store.is_known_subset(a, a.root()),
store.is_known_subset(
a.as_reference(),
a.as_reference().root().unwrap()
),
Ok(false),
"should indicate an unknown attestation is unknown"
);
assert_eq!(
store.observe_item(a, None),
store.observe_item(a.as_reference(), None),
Ok(ObserveOutcome::New),
"should observe new attestation"
);
@@ -445,12 +523,18 @@ mod tests {
for a in &items {
assert_eq!(
store.is_known_subset(a, a.root()),
store.is_known_subset(
a.as_reference(),
a.as_reference().root().unwrap()
),
Ok(true),
"should indicate a known attestation is known"
);
assert_eq!(
store.observe_item(a, Some(a.root())),
store.observe_item(
a.as_reference(),
Some(a.as_reference().root().unwrap())
),
Ok(ObserveOutcome::Subset),
"should acknowledge an existing attestation"
);

View File

@@ -24,18 +24,16 @@ use types::{Epoch, EthSpec, Hash256, Slot, Unsigned};
/// The maximum capacity of the `AutoPruningEpochContainer`.
///
/// Fits the next, current and previous epochs. We require the next epoch due to the
/// `MAXIMUM_GOSSIP_CLOCK_DISPARITY`. We require the previous epoch since the specification
/// declares:
/// If the current epoch is N, this fits epoch N + 1, N, N - 1, and N - 2. We require the next epoch due
/// to the `MAXIMUM_GOSSIP_CLOCK_DISPARITY`. We require the N - 2 epoch since the specification declares:
///
/// ```ignore
/// aggregate.data.slot + ATTESTATION_PROPAGATION_SLOT_RANGE
/// >= current_slot >= aggregate.data.slot
/// the epoch of `aggregate.data.slot` is either the current or previous epoch
/// ```
///
/// This means that during the current epoch we will always accept an attestation
/// from at least one slot in the previous epoch.
pub const MAX_CACHED_EPOCHS: u64 = 3;
/// This means that during the current epoch we will always accept an attestation from
/// at least one slot in the epoch prior to the previous epoch.
pub const MAX_CACHED_EPOCHS: u64 = 4;
pub type ObservedAttesters<E> = AutoPruningEpochContainer<EpochBitfield, E>;
pub type ObservedSyncContributors<E> =

View File

@@ -0,0 +1,430 @@
//! Provides the `ObservedBlobSidecars` struct which allows for rejecting `BlobSidecar`s
//! that we have already seen over the gossip network.
//! Only `BlobSidecar`s that have completed proposer signature verification can be added
//! to this cache to reduce DoS risks.
use crate::observed_block_producers::ProposalKey;
use std::collections::{HashMap, HashSet};
use std::marker::PhantomData;
use types::{BlobSidecar, EthSpec, Slot};
#[derive(Debug, PartialEq)]
pub enum Error {
/// The slot of the provided `BlobSidecar` is prior to finalization and should not have been provided
/// to this function. This is an internal error.
FinalizedBlob { slot: Slot, finalized_slot: Slot },
/// The blob sidecar contains an invalid blob index, the blob sidecar is invalid.
/// Note: The invalid blob should have been caught and flagged as an error much before reaching
/// here.
InvalidBlobIndex(u64),
}
/// Maintains a cache of seen `BlobSidecar`s that are received over gossip
/// and have been gossip verified.
///
/// The cache supports pruning based upon the finalized epoch. It does not automatically prune, you
/// must call `Self::prune` manually.
///
/// Note: To prevent DoS attacks, this cache must include only items that have received some DoS resistance
/// like checking the proposer signature.
pub struct ObservedBlobSidecars<E: EthSpec> {
finalized_slot: Slot,
/// Stores all received blob indices for a given `(ValidatorIndex, Slot)` tuple.
items: HashMap<ProposalKey, HashSet<u64>>,
_phantom: PhantomData<E>,
}
impl<E: EthSpec> Default for ObservedBlobSidecars<E> {
/// Instantiates `Self` with `finalized_slot == 0`.
fn default() -> Self {
Self {
finalized_slot: Slot::new(0),
items: HashMap::new(),
_phantom: PhantomData,
}
}
}
impl<E: EthSpec> ObservedBlobSidecars<E> {
/// Observe the `blob_sidecar` at (`blob_sidecar.block_proposer_index, blob_sidecar.slot`).
/// This will update `self` so future calls to it indicate that this `blob_sidecar` is known.
///
/// The supplied `blob_sidecar` **MUST** have completed proposer signature verification.
pub fn observe_sidecar(&mut self, blob_sidecar: &BlobSidecar<E>) -> Result<bool, Error> {
self.sanitize_blob_sidecar(blob_sidecar)?;
let blob_indices = self
.items
.entry(ProposalKey {
slot: blob_sidecar.slot(),
proposer: blob_sidecar.block_proposer_index(),
})
.or_insert_with(|| HashSet::with_capacity(E::max_blobs_per_block()));
let did_not_exist = blob_indices.insert(blob_sidecar.index);
Ok(!did_not_exist)
}
/// Returns `true` if the `blob_sidecar` has already been observed in the cache within the prune window.
pub fn proposer_is_known(&self, blob_sidecar: &BlobSidecar<E>) -> Result<bool, Error> {
self.sanitize_blob_sidecar(blob_sidecar)?;
let is_known = self
.items
.get(&ProposalKey {
slot: blob_sidecar.slot(),
proposer: blob_sidecar.block_proposer_index(),
})
.map_or(false, |blob_indices| {
blob_indices.contains(&blob_sidecar.index)
});
Ok(is_known)
}
fn sanitize_blob_sidecar(&self, blob_sidecar: &BlobSidecar<E>) -> Result<(), Error> {
if blob_sidecar.index >= E::max_blobs_per_block() as u64 {
return Err(Error::InvalidBlobIndex(blob_sidecar.index));
}
let finalized_slot = self.finalized_slot;
if finalized_slot > 0 && blob_sidecar.slot() <= finalized_slot {
return Err(Error::FinalizedBlob {
slot: blob_sidecar.slot(),
finalized_slot,
});
}
Ok(())
}
/// Prune `blob_sidecar` observations for slots less than or equal to the given slot.
pub fn prune(&mut self, finalized_slot: Slot) {
if finalized_slot == 0 {
return;
}
self.finalized_slot = finalized_slot;
self.items.retain(|k, _| k.slot > finalized_slot);
}
}
#[cfg(test)]
mod tests {
use super::*;
use bls::Hash256;
use std::sync::Arc;
use types::MainnetEthSpec;
type E = MainnetEthSpec;
fn get_blob_sidecar(slot: u64, proposer_index: u64, index: u64) -> Arc<BlobSidecar<E>> {
let mut blob_sidecar = BlobSidecar::empty();
blob_sidecar.signed_block_header.message.slot = slot.into();
blob_sidecar.signed_block_header.message.proposer_index = proposer_index;
blob_sidecar.index = index;
Arc::new(blob_sidecar)
}
#[test]
fn pruning() {
let mut cache = ObservedBlobSidecars::default();
assert_eq!(cache.finalized_slot, 0, "finalized slot is zero");
assert_eq!(cache.items.len(), 0, "no slots should be present");
// Slot 0, index 0
let proposer_index_a = 420;
let sidecar_a = get_blob_sidecar(0, proposer_index_a, 0);
assert_eq!(
cache.observe_sidecar(&sidecar_a),
Ok(false),
"can observe proposer, indicates proposer unobserved"
);
/*
* Preconditions.
*/
assert_eq!(cache.finalized_slot, 0, "finalized slot is zero");
assert_eq!(
cache.items.len(),
1,
"only one (validator_index, slot) tuple should be present"
);
let cached_blob_indices = cache
.items
.get(&ProposalKey::new(proposer_index_a, Slot::new(0)))
.expect("slot zero should be present");
assert_eq!(
cached_blob_indices.len(),
1,
"only one proposer should be present"
);
/*
* Check that a prune at the genesis slot does nothing.
*/
cache.prune(Slot::new(0));
assert_eq!(cache.finalized_slot, 0, "finalized slot is zero");
assert_eq!(cache.items.len(), 1, "only one slot should be present");
let cached_blob_indices = cache
.items
.get(&ProposalKey::new(proposer_index_a, Slot::new(0)))
.expect("slot zero should be present");
assert_eq!(
cached_blob_indices.len(),
1,
"only one proposer should be present"
);
/*
* Check that a prune empties the cache
*/
cache.prune(E::slots_per_epoch().into());
assert_eq!(
cache.finalized_slot,
Slot::from(E::slots_per_epoch()),
"finalized slot is updated"
);
assert_eq!(cache.items.len(), 0, "no items left");
/*
* Check that we can't insert a finalized sidecar
*/
// First slot of finalized epoch
let block_b = get_blob_sidecar(E::slots_per_epoch(), 419, 0);
assert_eq!(
cache.observe_sidecar(&block_b),
Err(Error::FinalizedBlob {
slot: E::slots_per_epoch().into(),
finalized_slot: E::slots_per_epoch().into(),
}),
"cant insert finalized sidecar"
);
assert_eq!(cache.items.len(), 0, "sidecar was not added");
/*
* Check that we _can_ insert a non-finalized block
*/
let three_epochs = E::slots_per_epoch() * 3;
// First slot of finalized epoch
let proposer_index_b = 421;
let block_b = get_blob_sidecar(three_epochs, proposer_index_b, 0);
assert_eq!(
cache.observe_sidecar(&block_b),
Ok(false),
"can insert non-finalized block"
);
assert_eq!(cache.items.len(), 1, "only one slot should be present");
let cached_blob_indices = cache
.items
.get(&ProposalKey::new(proposer_index_b, Slot::new(three_epochs)))
.expect("the three epochs slot should be present");
assert_eq!(
cached_blob_indices.len(),
1,
"only one proposer should be present"
);
/*
* Check that a prune doesnt wipe later blocks
*/
let two_epochs = E::slots_per_epoch() * 2;
cache.prune(two_epochs.into());
assert_eq!(
cache.finalized_slot,
Slot::from(two_epochs),
"finalized slot is updated"
);
assert_eq!(cache.items.len(), 1, "only one slot should be present");
let cached_blob_indices = cache
.items
.get(&ProposalKey::new(proposer_index_b, Slot::new(three_epochs)))
.expect("the three epochs slot should be present");
assert_eq!(
cached_blob_indices.len(),
1,
"only one proposer should be present"
);
}
#[test]
fn simple_observations() {
let mut cache = ObservedBlobSidecars::default();
// Slot 0, index 0
let proposer_index_a = 420;
let sidecar_a = get_blob_sidecar(0, proposer_index_a, 0);
assert_eq!(
cache.proposer_is_known(&sidecar_a),
Ok(false),
"no observation in empty cache"
);
assert_eq!(
cache.observe_sidecar(&sidecar_a),
Ok(false),
"can observe proposer, indicates proposer unobserved"
);
assert_eq!(
cache.proposer_is_known(&sidecar_a),
Ok(true),
"observed block is indicated as true"
);
assert_eq!(
cache.observe_sidecar(&sidecar_a),
Ok(true),
"observing again indicates true"
);
assert_eq!(cache.finalized_slot, 0, "finalized slot is zero");
assert_eq!(cache.items.len(), 1, "only one slot should be present");
let cached_blob_indices = cache
.items
.get(&ProposalKey::new(proposer_index_a, Slot::new(0)))
.expect("slot zero should be present");
assert_eq!(
cached_blob_indices.len(),
1,
"only one proposer should be present"
);
// Slot 1, proposer 0
let proposer_index_b = 421;
let sidecar_b = get_blob_sidecar(1, proposer_index_b, 0);
assert_eq!(
cache.proposer_is_known(&sidecar_b),
Ok(false),
"no observation for new slot"
);
assert_eq!(
cache.observe_sidecar(&sidecar_b),
Ok(false),
"can observe proposer for new slot, indicates proposer unobserved"
);
assert_eq!(
cache.proposer_is_known(&sidecar_b),
Ok(true),
"observed block in slot 1 is indicated as true"
);
assert_eq!(
cache.observe_sidecar(&sidecar_b),
Ok(true),
"observing slot 1 again indicates true"
);
assert_eq!(cache.finalized_slot, 0, "finalized slot is zero");
assert_eq!(cache.items.len(), 2, "two slots should be present");
let cached_blob_indices = cache
.items
.get(&ProposalKey::new(proposer_index_a, Slot::new(0)))
.expect("slot zero should be present");
assert_eq!(
cached_blob_indices.len(),
1,
"only one proposer should be present in slot 0"
);
let cached_blob_indices = cache
.items
.get(&ProposalKey::new(proposer_index_b, Slot::new(1)))
.expect("slot zero should be present");
assert_eq!(
cached_blob_indices.len(),
1,
"only one proposer should be present in slot 1"
);
// Slot 0, index 1
let sidecar_c = get_blob_sidecar(0, proposer_index_a, 1);
assert_eq!(
cache.proposer_is_known(&sidecar_c),
Ok(false),
"no observation for new index"
);
assert_eq!(
cache.observe_sidecar(&sidecar_c),
Ok(false),
"can observe new index, indicates sidecar unobserved for new index"
);
assert_eq!(
cache.proposer_is_known(&sidecar_c),
Ok(true),
"observed new sidecar is indicated as true"
);
assert_eq!(
cache.observe_sidecar(&sidecar_c),
Ok(true),
"observing new sidecar again indicates true"
);
assert_eq!(cache.finalized_slot, 0, "finalized slot is zero");
assert_eq!(cache.items.len(), 2, "two slots should be present");
let cached_blob_indices = cache
.items
.get(&ProposalKey::new(proposer_index_a, Slot::new(0)))
.expect("slot zero should be present");
assert_eq!(
cached_blob_indices.len(),
2,
"two blob indices should be present in slot 0"
);
// Create a sidecar sharing slot and proposer but with a different block root.
let mut sidecar_d: BlobSidecar<E> = BlobSidecar {
index: sidecar_c.index,
blob: sidecar_c.blob.clone(),
kzg_commitment: sidecar_c.kzg_commitment,
kzg_proof: sidecar_c.kzg_proof,
signed_block_header: sidecar_c.signed_block_header.clone(),
kzg_commitment_inclusion_proof: sidecar_c.kzg_commitment_inclusion_proof.clone(),
};
sidecar_d.signed_block_header.message.body_root = Hash256::repeat_byte(7);
assert_eq!(
cache.proposer_is_known(&sidecar_d),
Ok(true),
"there has been an observation for this proposer index"
);
assert_eq!(
cache.observe_sidecar(&sidecar_d),
Ok(true),
"indicates sidecar proposer was observed"
);
let cached_blob_indices = cache
.items
.get(&ProposalKey::new(proposer_index_a, Slot::new(0)))
.expect("slot zero should be present");
assert_eq!(
cached_blob_indices.len(),
2,
"two blob indices should be present in slot 0"
);
// Try adding an out of bounds index
let invalid_index = E::max_blobs_per_block() as u64;
let sidecar_d = get_blob_sidecar(0, proposer_index_a, invalid_index);
assert_eq!(
cache.observe_sidecar(&sidecar_d),
Err(Error::InvalidBlobIndex(invalid_index)),
"cannot add an index > MaxBlobsPerBlock"
);
}
}

View File

@@ -16,9 +16,15 @@ pub enum Error {
}
#[derive(Eq, Hash, PartialEq, Debug, Default)]
struct ProposalKey {
slot: Slot,
proposer: u64,
pub struct ProposalKey {
pub slot: Slot,
pub proposer: u64,
}
impl ProposalKey {
pub fn new(proposer: u64, slot: Slot) -> Self {
Self { slot, proposer }
}
}
/// Maintains a cache of observed `(block.slot, block.proposer)`.

View File

@@ -1,7 +1,6 @@
use derivative::Derivative;
use smallvec::{smallvec, SmallVec};
use ssz::{Decode, Encode};
use state_processing::{SigVerifiedOp, VerifyOperation, VerifyOperationAt};
use state_processing::{SigVerifiedOp, TransformPersist, VerifyOperation, VerifyOperationAt};
use std::collections::HashSet;
use std::marker::PhantomData;
use types::{
@@ -34,7 +33,7 @@ pub struct ObservedOperations<T: ObservableOperation<E>, E: EthSpec> {
/// Was the observed operation new and valid for further processing, or a useless duplicate?
#[derive(Debug, PartialEq, Eq, Clone)]
pub enum ObservationOutcome<T: Encode + Decode, E: EthSpec> {
pub enum ObservationOutcome<T: TransformPersist, E: EthSpec> {
New(SigVerifiedOp<T, E>),
AlreadyKnown,
}
@@ -62,15 +61,13 @@ impl<E: EthSpec> ObservableOperation<E> for ProposerSlashing {
impl<E: EthSpec> ObservableOperation<E> for AttesterSlashing<E> {
fn observed_validators(&self) -> SmallVec<[u64; SMALL_VEC_SIZE]> {
let attestation_1_indices = self
.attestation_1
.attesting_indices
.iter()
.attestation_1()
.attesting_indices_iter()
.copied()
.collect::<HashSet<u64>>();
let attestation_2_indices = self
.attestation_2
.attesting_indices
.iter()
.attestation_2()
.attesting_indices_iter()
.copied()
.collect::<HashSet<u64>>();
attestation_1_indices
@@ -153,6 +150,11 @@ impl<T: ObservableOperation<E>, E: EthSpec> ObservedOperations<T, E> {
self.current_fork = head_fork;
}
}
/// Reset the cache. MUST ONLY BE USED IN TESTS.
pub fn __reset_for_testing_only(&mut self) {
self.observed_validator_indices.clear();
}
}
impl<T: ObservableOperation<E> + VerifyOperationAt<E>, E: EthSpec> ObservedOperations<T, E> {

View File

@@ -0,0 +1,486 @@
//! Provides the `ObservedSlashable` struct which tracks slashable messages seen in
//! gossip or via RPC. Useful in supporting `broadcast_validation` in the Beacon API.
use crate::observed_block_producers::Error;
use std::collections::hash_map::Entry;
use std::collections::{HashMap, HashSet};
use std::marker::PhantomData;
use types::{EthSpec, Hash256, Slot, Unsigned};
#[derive(Eq, Hash, PartialEq, Debug, Default)]
pub struct ProposalKey {
pub slot: Slot,
pub proposer: u64,
}
/// Maintains a cache of observed `(block.slot, block.proposer)`.
///
/// The cache supports pruning based upon the finalized epoch. It does not automatically prune, you
/// must call `Self::prune` manually.
///
/// The maximum size of the cache is determined by `slots_since_finality *
/// VALIDATOR_REGISTRY_LIMIT`. This is quite a large size, so it's important that upstream
/// functions only use this cache for blocks with a valid signature. Only allowing valid signed
/// blocks reduces the theoretical maximum size of this cache to `slots_since_finality *
/// active_validator_count`, however in reality that is more like `slots_since_finality *
/// known_distinct_shufflings` which is much smaller.
pub struct ObservedSlashable<E: EthSpec> {
finalized_slot: Slot,
items: HashMap<ProposalKey, HashSet<Hash256>>,
_phantom: PhantomData<E>,
}
impl<E: EthSpec> Default for ObservedSlashable<E> {
/// Instantiates `Self` with `finalized_slot == 0`.
fn default() -> Self {
Self {
finalized_slot: Slot::new(0),
items: HashMap::new(),
_phantom: PhantomData,
}
}
}
impl<E: EthSpec> ObservedSlashable<E> {
/// Observe that the `header` was produced by `header.proposer_index` at `header.slot`. This will
/// update `self` so future calls to it indicate that this block is known.
///
/// The supplied `block` **MUST** be signature verified (see struct-level documentation).
///
/// ## Errors
///
/// - `header.proposer_index` is greater than `VALIDATOR_REGISTRY_LIMIT`.
/// - `header.slot` is equal to or less than the latest pruned `finalized_slot`.
pub fn observe_slashable(
&mut self,
slot: Slot,
proposer_index: u64,
block_root: Hash256,
) -> Result<(), Error> {
self.sanitize_header(slot, proposer_index)?;
let key = ProposalKey {
slot,
proposer: proposer_index,
};
let entry = self.items.entry(key);
match entry {
Entry::Occupied(mut occupied_entry) => {
let block_roots = occupied_entry.get_mut();
block_roots.insert(block_root);
}
Entry::Vacant(vacant_entry) => {
let block_roots = HashSet::from([block_root]);
vacant_entry.insert(block_roots);
}
}
Ok(())
}
/// Returns `Ok(true)` if the `block_root` is slashable, `Ok(false)` if not. Does not
/// update the cache, so calling this function multiple times will continue to return
/// `Ok(false)`, until `Self::observe_proposer` is called.
///
/// ## Errors
///
/// - `proposer_index` is greater than `VALIDATOR_REGISTRY_LIMIT`.
/// - `slot` is equal to or less than the latest pruned `finalized_slot`.
pub fn is_slashable(
&self,
slot: Slot,
proposer_index: u64,
block_root: Hash256,
) -> Result<bool, Error> {
self.sanitize_header(slot, proposer_index)?;
let key = ProposalKey {
slot,
proposer: proposer_index,
};
if let Some(block_roots) = self.items.get(&key) {
let no_prev_known_blocks =
block_roots.difference(&HashSet::from([block_root])).count() == 0;
Ok(!no_prev_known_blocks)
} else {
Ok(false)
}
}
/// Returns `Ok(())` if the given `header` is sane.
fn sanitize_header(&self, slot: Slot, proposer_index: u64) -> Result<(), Error> {
if proposer_index >= E::ValidatorRegistryLimit::to_u64() {
return Err(Error::ValidatorIndexTooHigh(proposer_index));
}
let finalized_slot = self.finalized_slot;
if finalized_slot > 0 && slot <= finalized_slot {
return Err(Error::FinalizedBlock {
slot,
finalized_slot,
});
}
Ok(())
}
/// Removes all observations of blocks equal to or earlier than `finalized_slot`.
///
/// Stores `finalized_slot` in `self`, so that `self` will reject any block that has a slot
/// equal to or less than `finalized_slot`.
///
/// No-op if `finalized_slot == 0`.
pub fn prune(&mut self, finalized_slot: Slot) {
if finalized_slot == 0 {
return;
}
self.finalized_slot = finalized_slot;
self.items.retain(|key, _| key.slot > finalized_slot);
}
}
#[cfg(test)]
mod tests {
use super::*;
use types::{BeaconBlock, Graffiti, MainnetEthSpec};
type E = MainnetEthSpec;
fn get_block(slot: u64, proposer: u64) -> BeaconBlock<E> {
let mut block = BeaconBlock::empty(&E::default_spec());
*block.slot_mut() = slot.into();
*block.proposer_index_mut() = proposer;
block
}
#[test]
fn pruning() {
let mut cache = ObservedSlashable::<E>::default();
assert_eq!(cache.finalized_slot, 0, "finalized slot is zero");
assert_eq!(cache.items.len(), 0, "no slots should be present");
// Slot 0, proposer 0
let block_a = get_block(0, 0);
let block_root = block_a.canonical_root();
assert_eq!(
cache.observe_slashable(block_a.slot(), block_a.proposer_index(), block_root),
Ok(()),
"can observe proposer"
);
/*
* Preconditions.
*/
assert_eq!(cache.finalized_slot, 0, "finalized slot is zero");
assert_eq!(cache.items.len(), 1, "only one slot should be present");
assert_eq!(
cache
.items
.get(&ProposalKey {
slot: Slot::new(0),
proposer: 0
})
.expect("slot zero should be present")
.len(),
1,
"only one proposer should be present"
);
/*
* Check that a prune at the genesis slot does nothing.
*/
cache.prune(Slot::new(0));
assert_eq!(cache.finalized_slot, 0, "finalized slot is zero");
assert_eq!(cache.items.len(), 1, "only one slot should be present");
assert_eq!(
cache
.items
.get(&ProposalKey {
slot: Slot::new(0),
proposer: 0
})
.expect("slot zero should be present")
.len(),
1,
"only one block root should be present"
);
/*
* Check that a prune empties the cache
*/
cache.prune(E::slots_per_epoch().into());
assert_eq!(
cache.finalized_slot,
Slot::from(E::slots_per_epoch()),
"finalized slot is updated"
);
assert_eq!(cache.items.len(), 0, "no items left");
/*
* Check that we can't insert a finalized block
*/
// First slot of finalized epoch, proposer 0
let block_b = get_block(E::slots_per_epoch(), 0);
let block_root_b = block_b.canonical_root();
assert_eq!(
cache.observe_slashable(block_b.slot(), block_b.proposer_index(), block_root_b),
Err(Error::FinalizedBlock {
slot: E::slots_per_epoch().into(),
finalized_slot: E::slots_per_epoch().into(),
}),
"cant insert finalized block"
);
assert_eq!(cache.items.len(), 0, "block was not added");
/*
* Check that we _can_ insert a non-finalized block
*/
let three_epochs = E::slots_per_epoch() * 3;
// First slot of finalized epoch, proposer 0
let block_b = get_block(three_epochs, 0);
assert_eq!(
cache.observe_slashable(block_b.slot(), block_b.proposer_index(), block_root_b),
Ok(()),
"can insert non-finalized block"
);
assert_eq!(cache.items.len(), 1, "only one slot should be present");
assert_eq!(
cache
.items
.get(&ProposalKey {
slot: Slot::new(three_epochs),
proposer: 0
})
.expect("the three epochs slot should be present")
.len(),
1,
"only one proposer should be present"
);
/*
* Check that a prune doesnt wipe later blocks
*/
let two_epochs = E::slots_per_epoch() * 2;
cache.prune(two_epochs.into());
assert_eq!(
cache.finalized_slot,
Slot::from(two_epochs),
"finalized slot is updated"
);
assert_eq!(cache.items.len(), 1, "only one slot should be present");
assert_eq!(
cache
.items
.get(&ProposalKey {
slot: Slot::new(three_epochs),
proposer: 0
})
.expect("the three epochs slot should be present")
.len(),
1,
"only one block root should be present"
);
}
#[test]
fn simple_observations() {
let mut cache = ObservedSlashable::<E>::default();
// Slot 0, proposer 0
let block_a = get_block(0, 0);
let block_root_a = block_a.canonical_root();
assert_eq!(
cache.is_slashable(
block_a.slot(),
block_a.proposer_index(),
block_a.canonical_root()
),
Ok(false),
"no observation in empty cache"
);
assert_eq!(
cache.observe_slashable(block_a.slot(), block_a.proposer_index(), block_root_a),
Ok(()),
"can observe proposer"
);
assert_eq!(
cache.is_slashable(
block_a.slot(),
block_a.proposer_index(),
block_a.canonical_root()
),
Ok(false),
"observed but unslashed block"
);
assert_eq!(
cache.observe_slashable(block_a.slot(), block_a.proposer_index(), block_root_a),
Ok(()),
"observing again"
);
assert_eq!(cache.finalized_slot, 0, "finalized slot is zero");
assert_eq!(cache.items.len(), 1, "only one slot should be present");
assert_eq!(
cache
.items
.get(&ProposalKey {
slot: Slot::new(0),
proposer: 0
})
.expect("slot zero should be present")
.len(),
1,
"only one block root should be present"
);
// Slot 1, proposer 0
let block_b = get_block(1, 0);
let block_root_b = block_b.canonical_root();
assert_eq!(
cache.is_slashable(
block_b.slot(),
block_b.proposer_index(),
block_b.canonical_root()
),
Ok(false),
"not slashable for new slot"
);
assert_eq!(
cache.observe_slashable(block_b.slot(), block_b.proposer_index(), block_root_b),
Ok(()),
"can observe proposer for new slot"
);
assert_eq!(
cache.is_slashable(
block_b.slot(),
block_b.proposer_index(),
block_b.canonical_root()
),
Ok(false),
"observed but not slashable block in slot 1"
);
assert_eq!(
cache.observe_slashable(block_b.slot(), block_b.proposer_index(), block_root_b),
Ok(()),
"observing slot 1 again"
);
assert_eq!(cache.finalized_slot, 0, "finalized slot is zero");
assert_eq!(cache.items.len(), 2, "two slots should be present");
assert_eq!(
cache
.items
.get(&ProposalKey {
slot: Slot::new(0),
proposer: 0
})
.expect("slot zero should be present")
.len(),
1,
"only one block root should be present in slot 0"
);
assert_eq!(
cache
.items
.get(&ProposalKey {
slot: Slot::new(1),
proposer: 0
})
.expect("slot zero should be present")
.len(),
1,
"only one block root should be present in slot 1"
);
// Slot 0, proposer 1
let block_c = get_block(0, 1);
let block_root_c = block_c.canonical_root();
assert_eq!(
cache.is_slashable(
block_c.slot(),
block_c.proposer_index(),
block_c.canonical_root()
),
Ok(false),
"not slashable due to new proposer"
);
assert_eq!(
cache.observe_slashable(block_c.slot(), block_c.proposer_index(), block_root_c),
Ok(()),
"can observe new proposer, indicates proposer unobserved"
);
assert_eq!(
cache.is_slashable(
block_c.slot(),
block_c.proposer_index(),
block_c.canonical_root()
),
Ok(false),
"not slashable due to new proposer"
);
assert_eq!(
cache.observe_slashable(block_c.slot(), block_c.proposer_index(), block_root_c),
Ok(()),
"observing new proposer again"
);
assert_eq!(cache.finalized_slot, 0, "finalized slot is zero");
assert_eq!(cache.items.len(), 3, "three slots should be present");
assert_eq!(
cache
.items
.iter()
.filter(|(k, _)| k.slot == cache.finalized_slot)
.count(),
2,
"two proposers should be present in slot 0"
);
assert_eq!(
cache
.items
.iter()
.filter(|(k, _)| k.slot == Slot::new(1))
.count(),
1,
"only one proposer should be present in slot 1"
);
// Slot 0, proposer 1 (again)
let mut block_d = get_block(0, 1);
*block_d.body_mut().graffiti_mut() = Graffiti::from(*b"this is slashable ");
let block_root_d = block_d.canonical_root();
assert_eq!(
cache.is_slashable(
block_d.slot(),
block_d.proposer_index(),
block_d.canonical_root()
),
Ok(true),
"slashable due to new proposer"
);
assert_eq!(
cache.observe_slashable(block_d.slot(), block_d.proposer_index(), block_root_d),
Ok(()),
"can observe new proposer, indicates proposer unobserved"
);
}
}

View File

@@ -119,10 +119,13 @@ pub fn start_otb_verification_service<T: BeaconChainTypes>(
pub fn load_optimistic_transition_blocks<T: BeaconChainTypes>(
chain: &BeaconChain<T>,
) -> Result<Vec<OptimisticTransitionBlock>, StoreError> {
process_results(chain.store.hot_db.iter_column(OTBColumn), |iter| {
iter.map(|(_, bytes)| OptimisticTransitionBlock::from_store_bytes(&bytes))
.collect()
})?
process_results(
chain.store.hot_db.iter_column::<Hash256>(OTBColumn),
|iter| {
iter.map(|(_, bytes)| OptimisticTransitionBlock::from_store_bytes(&bytes))
.collect()
},
)?
}
#[derive(Debug)]

View File

@@ -20,20 +20,20 @@ pub struct PersistedForkChoice {
pub fork_choice_store: PersistedForkChoiceStoreV17,
}
impl Into<PersistedForkChoice> for PersistedForkChoiceV11 {
fn into(self) -> PersistedForkChoice {
impl From<PersistedForkChoiceV11> for PersistedForkChoice {
fn from(from: PersistedForkChoiceV11) -> PersistedForkChoice {
PersistedForkChoice {
fork_choice: self.fork_choice,
fork_choice_store: self.fork_choice_store.into(),
fork_choice: from.fork_choice,
fork_choice_store: from.fork_choice_store.into(),
}
}
}
impl Into<PersistedForkChoiceV11> for PersistedForkChoice {
fn into(self) -> PersistedForkChoiceV11 {
impl From<PersistedForkChoice> for PersistedForkChoiceV11 {
fn from(from: PersistedForkChoice) -> PersistedForkChoiceV11 {
PersistedForkChoiceV11 {
fork_choice: self.fork_choice,
fork_choice_store: self.fork_choice_store.into(),
fork_choice: from.fork_choice,
fork_choice_store: from.fork_choice_store.into(),
}
}
}

View File

@@ -3,11 +3,13 @@ use itertools::process_results;
use lru::LruCache;
use parking_lot::Mutex;
use slog::debug;
use std::num::NonZeroUsize;
use std::time::Duration;
use types::non_zero_usize::new_non_zero_usize;
use types::Hash256;
const BLOCK_ROOT_CACHE_LIMIT: usize = 512;
const LOOKUP_LIMIT: usize = 8;
const BLOCK_ROOT_CACHE_LIMIT: NonZeroUsize = new_non_zero_usize(512);
const LOOKUP_LIMIT: NonZeroUsize = new_non_zero_usize(8);
const METRICS_TIMEOUT: Duration = Duration::from_millis(100);
/// Cache for rejecting attestations to blocks from before finalization.
@@ -78,7 +80,7 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
// 3. Check the network with a single block lookup.
cache.in_progress_lookups.put(block_root, ());
if cache.in_progress_lookups.len() == LOOKUP_LIMIT {
if cache.in_progress_lookups.len() == LOOKUP_LIMIT.get() {
// NOTE: we expect this to occur sometimes if a lot of blocks that we look up fail to be
// imported for reasons other than being pre-finalization. The cache will eventually
// self-repair in this case by replacing old entries with new ones until all the failed

View File

@@ -1,19 +1,16 @@
//! Utilities for managing database schema changes.
mod migration_schema_v12;
mod migration_schema_v13;
mod migration_schema_v14;
mod migration_schema_v15;
mod migration_schema_v16;
mod migration_schema_v17;
mod migration_schema_v18;
mod migration_schema_v19;
mod migration_schema_v20;
use crate::beacon_chain::{BeaconChainTypes, ETH1_CACHE_DB_KEY};
use crate::eth1_chain::SszEth1;
use crate::beacon_chain::BeaconChainTypes;
use crate::types::ChainSpec;
use slog::{warn, Logger};
use slog::Logger;
use std::sync::Arc;
use store::hot_cold_store::{HotColdDB, HotColdDBError};
use store::metadata::{SchemaVersion, CURRENT_SCHEMA_VERSION};
use store::{Error as StoreError, StoreItem};
use store::Error as StoreError;
/// Migrate the database from one schema version to another, applying all requisite mutations.
#[allow(clippy::only_used_in_recursion)] // spec is not used but likely to be used in future
@@ -56,92 +53,8 @@ pub fn migrate_schema<T: BeaconChainTypes>(
}
//
// Migrations from before SchemaVersion(11) are deprecated.
// Migrations from before SchemaVersion(16) are deprecated.
//
// Upgrade from v11 to v12 to store richer metadata in the attestation op pool.
(SchemaVersion(11), SchemaVersion(12)) => {
let ops = migration_schema_v12::upgrade_to_v12::<T>(db.clone(), log)?;
db.store_schema_version_atomically(to, ops)
}
// Downgrade from v12 to v11 to drop richer metadata from the attestation op pool.
(SchemaVersion(12), SchemaVersion(11)) => {
let ops = migration_schema_v12::downgrade_from_v12::<T>(db.clone(), log)?;
db.store_schema_version_atomically(to, ops)
}
(SchemaVersion(12), SchemaVersion(13)) => {
let mut ops = vec![];
if let Some(persisted_eth1_v1) = db.get_item::<SszEth1>(&ETH1_CACHE_DB_KEY)? {
let upgraded_eth1_cache =
match migration_schema_v13::update_eth1_cache(persisted_eth1_v1) {
Ok(upgraded_eth1) => upgraded_eth1,
Err(e) => {
warn!(log, "Failed to deserialize SszEth1CacheV1"; "error" => ?e);
warn!(log, "Reinitializing eth1 cache");
migration_schema_v13::reinitialized_eth1_cache_v13(
deposit_contract_deploy_block,
)
}
};
ops.push(upgraded_eth1_cache.as_kv_store_op(ETH1_CACHE_DB_KEY));
}
db.store_schema_version_atomically(to, ops)?;
Ok(())
}
(SchemaVersion(13), SchemaVersion(12)) => {
let mut ops = vec![];
if let Some(persisted_eth1_v13) = db.get_item::<SszEth1>(&ETH1_CACHE_DB_KEY)? {
let downgraded_eth1_cache = match migration_schema_v13::downgrade_eth1_cache(
persisted_eth1_v13,
) {
Ok(Some(downgraded_eth1)) => downgraded_eth1,
Ok(None) => {
warn!(log, "Unable to downgrade eth1 cache from newer version: reinitializing eth1 cache");
migration_schema_v13::reinitialized_eth1_cache_v1(
deposit_contract_deploy_block,
)
}
Err(e) => {
warn!(log, "Unable to downgrade eth1 cache from newer version: failed to deserialize SszEth1CacheV13"; "error" => ?e);
warn!(log, "Reinitializing eth1 cache");
migration_schema_v13::reinitialized_eth1_cache_v1(
deposit_contract_deploy_block,
)
}
};
ops.push(downgraded_eth1_cache.as_kv_store_op(ETH1_CACHE_DB_KEY));
}
db.store_schema_version_atomically(to, ops)?;
Ok(())
}
(SchemaVersion(13), SchemaVersion(14)) => {
let ops = migration_schema_v14::upgrade_to_v14::<T>(db.clone(), log)?;
db.store_schema_version_atomically(to, ops)
}
(SchemaVersion(14), SchemaVersion(13)) => {
let ops = migration_schema_v14::downgrade_from_v14::<T>(db.clone(), log)?;
db.store_schema_version_atomically(to, ops)
}
(SchemaVersion(14), SchemaVersion(15)) => {
let ops = migration_schema_v15::upgrade_to_v15::<T>(db.clone(), log)?;
db.store_schema_version_atomically(to, ops)
}
(SchemaVersion(15), SchemaVersion(14)) => {
let ops = migration_schema_v15::downgrade_from_v15::<T>(db.clone(), log)?;
db.store_schema_version_atomically(to, ops)
}
(SchemaVersion(15), SchemaVersion(16)) => {
let ops = migration_schema_v16::upgrade_to_v16::<T>(db.clone(), log)?;
db.store_schema_version_atomically(to, ops)
}
(SchemaVersion(16), SchemaVersion(15)) => {
let ops = migration_schema_v16::downgrade_from_v16::<T>(db.clone(), log)?;
db.store_schema_version_atomically(to, ops)
}
(SchemaVersion(16), SchemaVersion(17)) => {
let ops = migration_schema_v17::upgrade_to_v17::<T>(db.clone(), log)?;
db.store_schema_version_atomically(to, ops)
@@ -150,6 +63,30 @@ pub fn migrate_schema<T: BeaconChainTypes>(
let ops = migration_schema_v17::downgrade_from_v17::<T>(db.clone(), log)?;
db.store_schema_version_atomically(to, ops)
}
(SchemaVersion(17), SchemaVersion(18)) => {
let ops = migration_schema_v18::upgrade_to_v18::<T>(db.clone(), log)?;
db.store_schema_version_atomically(to, ops)
}
(SchemaVersion(18), SchemaVersion(17)) => {
let ops = migration_schema_v18::downgrade_from_v18::<T>(db.clone(), log)?;
db.store_schema_version_atomically(to, ops)
}
(SchemaVersion(18), SchemaVersion(19)) => {
let ops = migration_schema_v19::upgrade_to_v19::<T>(db.clone(), log)?;
db.store_schema_version_atomically(to, ops)
}
(SchemaVersion(19), SchemaVersion(18)) => {
let ops = migration_schema_v19::downgrade_from_v19::<T>(db.clone(), log)?;
db.store_schema_version_atomically(to, ops)
}
(SchemaVersion(19), SchemaVersion(20)) => {
let ops = migration_schema_v20::upgrade_to_v20::<T>(db.clone(), log)?;
db.store_schema_version_atomically(to, ops)
}
(SchemaVersion(20), SchemaVersion(19)) => {
let ops = migration_schema_v20::downgrade_from_v20::<T>(db.clone(), log)?;
db.store_schema_version_atomically(to, ops)
}
// Anything else is an error.
(_, _) => Err(HotColdDBError::UnsupportedSchemaVersion {
target_version: to,

View File

@@ -1,224 +0,0 @@
use crate::beacon_chain::{BeaconChainTypes, FORK_CHOICE_DB_KEY, OP_POOL_DB_KEY};
use crate::persisted_fork_choice::PersistedForkChoiceV11;
use operation_pool::{PersistedOperationPool, PersistedOperationPoolV12, PersistedOperationPoolV5};
use slog::{debug, info, Logger};
use state_processing::{
common::get_indexed_attestation, per_block_processing::is_valid_indexed_attestation,
VerifyOperation, VerifySignatures,
};
use std::sync::Arc;
use store::{Error, HotColdDB, KeyValueStoreOp, StoreItem};
pub fn upgrade_to_v12<T: BeaconChainTypes>(
db: Arc<HotColdDB<T::EthSpec, T::HotStore, T::ColdStore>>,
log: Logger,
) -> Result<Vec<KeyValueStoreOp>, Error> {
let spec = db.get_chain_spec();
// Load a V5 op pool and transform it to V12.
let PersistedOperationPoolV5 {
attestations_v5,
sync_contributions,
attester_slashings_v5,
proposer_slashings_v5,
voluntary_exits_v5,
} = if let Some(op_pool) = db.get_item(&OP_POOL_DB_KEY)? {
op_pool
} else {
debug!(log, "Nothing to do, no operation pool stored");
return Ok(vec![]);
};
// Load the persisted fork choice so we can grab the state of the justified block and use
// it to verify the stored attestations, slashings and exits.
let fork_choice = db
.get_item::<PersistedForkChoiceV11>(&FORK_CHOICE_DB_KEY)?
.ok_or_else(|| Error::SchemaMigrationError("fork choice missing from database".into()))?;
let justified_block_root = fork_choice
.fork_choice_store
.unrealized_justified_checkpoint
.root;
let justified_block = db
.get_blinded_block(&justified_block_root)?
.ok_or_else(|| {
Error::SchemaMigrationError(format!(
"unrealized justified block missing for migration: {justified_block_root:?}",
))
})?;
let justified_state_root = justified_block.state_root();
let mut state = db
.get_state(&justified_state_root, Some(justified_block.slot()))?
.ok_or_else(|| {
Error::SchemaMigrationError(format!(
"justified state missing for migration: {justified_state_root:?}"
))
})?;
state.build_all_committee_caches(spec).map_err(|e| {
Error::SchemaMigrationError(format!("unable to build committee caches: {e:?}"))
})?;
// Re-verify attestations while adding attesting indices.
let attestations = attestations_v5
.into_iter()
.flat_map(|(_, attestations)| attestations)
.filter_map(|attestation| {
let res = state
.get_beacon_committee(attestation.data.slot, attestation.data.index)
.map_err(Into::into)
.and_then(|committee| get_indexed_attestation(committee.committee, &attestation))
.and_then(|indexed_attestation| {
is_valid_indexed_attestation(
&state,
&indexed_attestation,
VerifySignatures::True,
spec,
)?;
Ok(indexed_attestation)
});
match res {
Ok(indexed) => Some((attestation, indexed.attesting_indices.into())),
Err(e) => {
debug!(
log,
"Dropping attestation on migration";
"err" => ?e,
"head_block" => ?attestation.data.beacon_block_root,
);
None
}
}
})
.collect::<Vec<_>>();
let attester_slashings = attester_slashings_v5
.iter()
.filter_map(|(slashing, _)| {
slashing
.clone()
.validate(&state, spec)
.map_err(|e| {
debug!(
log,
"Dropping attester slashing on migration";
"err" => ?e,
"slashing" => ?slashing,
);
})
.ok()
})
.collect::<Vec<_>>();
let proposer_slashings = proposer_slashings_v5
.iter()
.filter_map(|slashing| {
slashing
.clone()
.validate(&state, spec)
.map_err(|e| {
debug!(
log,
"Dropping proposer slashing on migration";
"err" => ?e,
"slashing" => ?slashing,
);
})
.ok()
})
.collect::<Vec<_>>();
let voluntary_exits = voluntary_exits_v5
.iter()
.filter_map(|exit| {
exit.clone()
.validate(&state, spec)
.map_err(|e| {
debug!(
log,
"Dropping voluntary exit on migration";
"err" => ?e,
"exit" => ?exit,
);
})
.ok()
})
.collect::<Vec<_>>();
debug!(
log,
"Migrated op pool";
"attestations" => attestations.len(),
"attester_slashings" => attester_slashings.len(),
"proposer_slashings" => proposer_slashings.len(),
"voluntary_exits" => voluntary_exits.len()
);
let v12 = PersistedOperationPool::V12(PersistedOperationPoolV12 {
attestations,
sync_contributions,
attester_slashings,
proposer_slashings,
voluntary_exits,
});
Ok(vec![v12.as_kv_store_op(OP_POOL_DB_KEY)])
}
pub fn downgrade_from_v12<T: BeaconChainTypes>(
db: Arc<HotColdDB<T::EthSpec, T::HotStore, T::ColdStore>>,
log: Logger,
) -> Result<Vec<KeyValueStoreOp>, Error> {
// Load a V12 op pool and transform it to V5.
let PersistedOperationPoolV12::<T::EthSpec> {
attestations,
sync_contributions,
attester_slashings,
proposer_slashings,
voluntary_exits,
} = if let Some(op_pool_v12) = db.get_item(&OP_POOL_DB_KEY)? {
op_pool_v12
} else {
debug!(log, "Nothing to do, no operation pool stored");
return Ok(vec![]);
};
info!(
log,
"Dropping attestations from pool";
"count" => attestations.len(),
);
let attester_slashings_v5 = attester_slashings
.into_iter()
.filter_map(|slashing| {
let fork_version = slashing.first_fork_verified_against()?;
Some((slashing.into_inner(), fork_version))
})
.collect::<Vec<_>>();
let proposer_slashings_v5 = proposer_slashings
.into_iter()
.map(|slashing| slashing.into_inner())
.collect::<Vec<_>>();
let voluntary_exits_v5 = voluntary_exits
.into_iter()
.map(|exit| exit.into_inner())
.collect::<Vec<_>>();
info!(
log,
"Migrated slashings and exits";
"attester_slashings" => attester_slashings_v5.len(),
"proposer_slashings" => proposer_slashings_v5.len(),
"voluntary_exits" => voluntary_exits_v5.len(),
);
let v5 = PersistedOperationPoolV5 {
attestations_v5: vec![],
sync_contributions,
attester_slashings_v5,
proposer_slashings_v5,
voluntary_exits_v5,
};
Ok(vec![v5.as_kv_store_op(OP_POOL_DB_KEY)])
}

View File

@@ -1,150 +0,0 @@
use crate::eth1_chain::SszEth1;
use eth1::{BlockCache, SszDepositCacheV1, SszDepositCacheV13, SszEth1CacheV1, SszEth1CacheV13};
use ssz::{Decode, Encode};
use state_processing::common::DepositDataTree;
use store::Error;
use types::DEPOSIT_TREE_DEPTH;
pub fn update_eth1_cache(persisted_eth1_v1: SszEth1) -> Result<SszEth1, Error> {
if persisted_eth1_v1.use_dummy_backend {
// backend_bytes is empty when using dummy backend
return Ok(persisted_eth1_v1);
}
let SszEth1 {
use_dummy_backend,
backend_bytes,
} = persisted_eth1_v1;
let ssz_eth1_cache_v1 = SszEth1CacheV1::from_ssz_bytes(&backend_bytes)?;
let SszEth1CacheV1 {
block_cache,
deposit_cache: deposit_cache_v1,
last_processed_block,
} = ssz_eth1_cache_v1;
let SszDepositCacheV1 {
logs,
leaves,
deposit_contract_deploy_block,
deposit_roots,
} = deposit_cache_v1;
let deposit_cache_v13 = SszDepositCacheV13 {
logs,
leaves,
deposit_contract_deploy_block,
finalized_deposit_count: 0,
finalized_block_height: deposit_contract_deploy_block.saturating_sub(1),
deposit_tree_snapshot: None,
deposit_roots,
};
let ssz_eth1_cache_v13 = SszEth1CacheV13 {
block_cache,
deposit_cache: deposit_cache_v13,
last_processed_block,
};
let persisted_eth1_v13 = SszEth1 {
use_dummy_backend,
backend_bytes: ssz_eth1_cache_v13.as_ssz_bytes(),
};
Ok(persisted_eth1_v13)
}
pub fn downgrade_eth1_cache(persisted_eth1_v13: SszEth1) -> Result<Option<SszEth1>, Error> {
if persisted_eth1_v13.use_dummy_backend {
// backend_bytes is empty when using dummy backend
return Ok(Some(persisted_eth1_v13));
}
let SszEth1 {
use_dummy_backend,
backend_bytes,
} = persisted_eth1_v13;
let ssz_eth1_cache_v13 = SszEth1CacheV13::from_ssz_bytes(&backend_bytes)?;
let SszEth1CacheV13 {
block_cache,
deposit_cache: deposit_cache_v13,
last_processed_block,
} = ssz_eth1_cache_v13;
let SszDepositCacheV13 {
logs,
leaves,
deposit_contract_deploy_block,
finalized_deposit_count,
finalized_block_height: _,
deposit_tree_snapshot,
deposit_roots,
} = deposit_cache_v13;
if finalized_deposit_count == 0 && deposit_tree_snapshot.is_none() {
// This tree was never finalized and can be directly downgraded to v1 without re-initializing
let deposit_cache_v1 = SszDepositCacheV1 {
logs,
leaves,
deposit_contract_deploy_block,
deposit_roots,
};
let ssz_eth1_cache_v1 = SszEth1CacheV1 {
block_cache,
deposit_cache: deposit_cache_v1,
last_processed_block,
};
return Ok(Some(SszEth1 {
use_dummy_backend,
backend_bytes: ssz_eth1_cache_v1.as_ssz_bytes(),
}));
}
// deposit cache was finalized; can't downgrade
Ok(None)
}
pub fn reinitialized_eth1_cache_v13(deposit_contract_deploy_block: u64) -> SszEth1 {
let empty_tree = DepositDataTree::create(&[], 0, DEPOSIT_TREE_DEPTH);
let deposit_cache_v13 = SszDepositCacheV13 {
logs: vec![],
leaves: vec![],
deposit_contract_deploy_block,
finalized_deposit_count: 0,
finalized_block_height: deposit_contract_deploy_block.saturating_sub(1),
deposit_tree_snapshot: empty_tree.get_snapshot(),
deposit_roots: vec![empty_tree.root()],
};
let ssz_eth1_cache_v13 = SszEth1CacheV13 {
block_cache: BlockCache::default(),
deposit_cache: deposit_cache_v13,
last_processed_block: None,
};
SszEth1 {
use_dummy_backend: false,
backend_bytes: ssz_eth1_cache_v13.as_ssz_bytes(),
}
}
pub fn reinitialized_eth1_cache_v1(deposit_contract_deploy_block: u64) -> SszEth1 {
let empty_tree = DepositDataTree::create(&[], 0, DEPOSIT_TREE_DEPTH);
let deposit_cache_v1 = SszDepositCacheV1 {
logs: vec![],
leaves: vec![],
deposit_contract_deploy_block,
deposit_roots: vec![empty_tree.root()],
};
let ssz_eth1_cache_v1 = SszEth1CacheV1 {
block_cache: BlockCache::default(),
deposit_cache: deposit_cache_v1,
last_processed_block: None,
};
SszEth1 {
use_dummy_backend: false,
backend_bytes: ssz_eth1_cache_v1.as_ssz_bytes(),
}
}

View File

@@ -1,125 +0,0 @@
use crate::beacon_chain::{BeaconChainTypes, OP_POOL_DB_KEY};
use operation_pool::{
PersistedOperationPool, PersistedOperationPoolV12, PersistedOperationPoolV14,
};
use slog::{debug, error, info, Logger};
use slot_clock::SlotClock;
use std::sync::Arc;
use std::time::Duration;
use store::{Error, HotColdDB, KeyValueStoreOp, StoreItem};
use types::{EthSpec, Hash256, Slot};
/// The slot clock isn't usually available before the database is initialized, so we construct a
/// temporary slot clock by reading the genesis state. It should always exist if the database is
/// initialized at a prior schema version, however we still handle the lack of genesis state
/// gracefully.
fn get_slot_clock<T: BeaconChainTypes>(
db: &HotColdDB<T::EthSpec, T::HotStore, T::ColdStore>,
log: &Logger,
) -> Result<Option<T::SlotClock>, Error> {
let spec = db.get_chain_spec();
let genesis_block = if let Some(block) = db.get_blinded_block(&Hash256::zero())? {
block
} else {
error!(log, "Missing genesis block");
return Ok(None);
};
let genesis_state =
if let Some(state) = db.get_state(&genesis_block.state_root(), Some(Slot::new(0)))? {
state
} else {
error!(log, "Missing genesis state"; "state_root" => ?genesis_block.state_root());
return Ok(None);
};
Ok(Some(T::SlotClock::new(
spec.genesis_slot,
Duration::from_secs(genesis_state.genesis_time()),
Duration::from_secs(spec.seconds_per_slot),
)))
}
pub fn upgrade_to_v14<T: BeaconChainTypes>(
db: Arc<HotColdDB<T::EthSpec, T::HotStore, T::ColdStore>>,
log: Logger,
) -> Result<Vec<KeyValueStoreOp>, Error> {
// Load a V12 op pool and transform it to V14.
let PersistedOperationPoolV12::<T::EthSpec> {
attestations,
sync_contributions,
attester_slashings,
proposer_slashings,
voluntary_exits,
} = if let Some(op_pool_v12) = db.get_item(&OP_POOL_DB_KEY)? {
op_pool_v12
} else {
debug!(log, "Nothing to do, no operation pool stored");
return Ok(vec![]);
};
// initialize with empty vector
let bls_to_execution_changes = vec![];
let v14 = PersistedOperationPool::V14(PersistedOperationPoolV14 {
attestations,
sync_contributions,
attester_slashings,
proposer_slashings,
voluntary_exits,
bls_to_execution_changes,
});
Ok(vec![v14.as_kv_store_op(OP_POOL_DB_KEY)])
}
pub fn downgrade_from_v14<T: BeaconChainTypes>(
db: Arc<HotColdDB<T::EthSpec, T::HotStore, T::ColdStore>>,
log: Logger,
) -> Result<Vec<KeyValueStoreOp>, Error> {
// We cannot downgrade from V14 once the Capella fork has been reached because there will
// be HistoricalSummaries stored in the database instead of HistoricalRoots and prior versions
// of Lighthouse can't handle that.
if let Some(capella_fork_epoch) = db.get_chain_spec().capella_fork_epoch {
let current_epoch = get_slot_clock::<T>(&db, &log)?
.and_then(|clock| clock.now())
.map(|slot| slot.epoch(T::EthSpec::slots_per_epoch()))
.ok_or(Error::SlotClockUnavailableForMigration)?;
if current_epoch >= capella_fork_epoch {
error!(
log,
"Capella already active: v14+ is mandatory";
"current_epoch" => current_epoch,
"capella_fork_epoch" => capella_fork_epoch,
);
return Err(Error::UnableToDowngrade);
}
}
// Load a V14 op pool and transform it to V12.
let PersistedOperationPoolV14::<T::EthSpec> {
attestations,
sync_contributions,
attester_slashings,
proposer_slashings,
voluntary_exits,
bls_to_execution_changes,
} = if let Some(op_pool) = db.get_item(&OP_POOL_DB_KEY)? {
op_pool
} else {
debug!(log, "Nothing to do, no operation pool stored");
return Ok(vec![]);
};
info!(
log,
"Dropping bls_to_execution_changes from pool";
"count" => bls_to_execution_changes.len(),
);
let v12 = PersistedOperationPoolV12 {
attestations,
sync_contributions,
attester_slashings,
proposer_slashings,
voluntary_exits,
};
Ok(vec![v12.as_kv_store_op(OP_POOL_DB_KEY)])
}

View File

@@ -1,76 +0,0 @@
use crate::beacon_chain::{BeaconChainTypes, OP_POOL_DB_KEY};
use operation_pool::{
PersistedOperationPool, PersistedOperationPoolV14, PersistedOperationPoolV15,
};
use slog::{debug, info, Logger};
use std::sync::Arc;
use store::{Error, HotColdDB, KeyValueStoreOp, StoreItem};
pub fn upgrade_to_v15<T: BeaconChainTypes>(
db: Arc<HotColdDB<T::EthSpec, T::HotStore, T::ColdStore>>,
log: Logger,
) -> Result<Vec<KeyValueStoreOp>, Error> {
// Load a V14 op pool and transform it to V15.
let PersistedOperationPoolV14::<T::EthSpec> {
attestations,
sync_contributions,
attester_slashings,
proposer_slashings,
voluntary_exits,
bls_to_execution_changes,
} = if let Some(op_pool_v14) = db.get_item(&OP_POOL_DB_KEY)? {
op_pool_v14
} else {
debug!(log, "Nothing to do, no operation pool stored");
return Ok(vec![]);
};
let v15 = PersistedOperationPool::V15(PersistedOperationPoolV15 {
attestations,
sync_contributions,
attester_slashings,
proposer_slashings,
voluntary_exits,
bls_to_execution_changes,
// Initialize with empty set
capella_bls_change_broadcast_indices: <_>::default(),
});
Ok(vec![v15.as_kv_store_op(OP_POOL_DB_KEY)])
}
pub fn downgrade_from_v15<T: BeaconChainTypes>(
db: Arc<HotColdDB<T::EthSpec, T::HotStore, T::ColdStore>>,
log: Logger,
) -> Result<Vec<KeyValueStoreOp>, Error> {
// Load a V15 op pool and transform it to V14.
let PersistedOperationPoolV15::<T::EthSpec> {
attestations,
sync_contributions,
attester_slashings,
proposer_slashings,
voluntary_exits,
bls_to_execution_changes,
capella_bls_change_broadcast_indices,
} = if let Some(op_pool) = db.get_item(&OP_POOL_DB_KEY)? {
op_pool
} else {
debug!(log, "Nothing to do, no operation pool stored");
return Ok(vec![]);
};
info!(
log,
"Forgetting address changes for Capella broadcast";
"count" => capella_bls_change_broadcast_indices.len(),
);
let v14 = PersistedOperationPoolV14 {
attestations,
sync_contributions,
attester_slashings,
proposer_slashings,
voluntary_exits,
bls_to_execution_changes,
};
Ok(vec![v14.as_kv_store_op(OP_POOL_DB_KEY)])
}

View File

@@ -1,46 +0,0 @@
use crate::beacon_chain::{BeaconChainTypes, FORK_CHOICE_DB_KEY};
use crate::persisted_fork_choice::PersistedForkChoiceV11;
use slog::{debug, Logger};
use std::sync::Arc;
use store::{Error, HotColdDB, KeyValueStoreOp, StoreItem};
pub fn upgrade_to_v16<T: BeaconChainTypes>(
db: Arc<HotColdDB<T::EthSpec, T::HotStore, T::ColdStore>>,
log: Logger,
) -> Result<Vec<KeyValueStoreOp>, Error> {
drop_balances_cache::<T>(db, log)
}
pub fn downgrade_from_v16<T: BeaconChainTypes>(
db: Arc<HotColdDB<T::EthSpec, T::HotStore, T::ColdStore>>,
log: Logger,
) -> Result<Vec<KeyValueStoreOp>, Error> {
drop_balances_cache::<T>(db, log)
}
/// Drop the balances cache from the fork choice store.
///
/// There aren't any type-level changes in this schema migration, however the
/// way that we compute the `JustifiedBalances` has changed due to:
/// https://github.com/sigp/lighthouse/pull/3962
pub fn drop_balances_cache<T: BeaconChainTypes>(
db: Arc<HotColdDB<T::EthSpec, T::HotStore, T::ColdStore>>,
log: Logger,
) -> Result<Vec<KeyValueStoreOp>, Error> {
let mut persisted_fork_choice = db
.get_item::<PersistedForkChoiceV11>(&FORK_CHOICE_DB_KEY)?
.ok_or_else(|| Error::SchemaMigrationError("fork choice missing from database".into()))?;
debug!(
log,
"Dropping fork choice balances cache";
"item_count" => persisted_fork_choice.fork_choice_store.balances_cache.items.len()
);
// Drop all items in the balances cache.
persisted_fork_choice.fork_choice_store.balances_cache = <_>::default();
let kv_op = persisted_fork_choice.as_kv_store_op(FORK_CHOICE_DB_KEY);
Ok(vec![kv_op])
}

View File

@@ -0,0 +1,119 @@
use crate::beacon_chain::BeaconChainTypes;
use slog::{error, info, warn, Logger};
use slot_clock::SlotClock;
use std::sync::Arc;
use std::time::Duration;
use store::{
get_key_for_col, metadata::BLOB_INFO_KEY, DBColumn, Error, HotColdDB, KeyValueStoreOp,
};
use types::{Epoch, EthSpec, Hash256, Slot};
/// The slot clock isn't usually available before the database is initialized, so we construct a
/// temporary slot clock by reading the genesis state. It should always exist if the database is
/// initialized at a prior schema version, however we still handle the lack of genesis state
/// gracefully.
fn get_slot_clock<T: BeaconChainTypes>(
db: &HotColdDB<T::EthSpec, T::HotStore, T::ColdStore>,
log: &Logger,
) -> Result<Option<T::SlotClock>, Error> {
let spec = db.get_chain_spec();
let Some(genesis_block) = db.get_blinded_block(&Hash256::zero())? else {
error!(log, "Missing genesis block");
return Ok(None);
};
let Some(genesis_state) = db.get_state(&genesis_block.state_root(), Some(Slot::new(0)))? else {
error!(log, "Missing genesis state"; "state_root" => ?genesis_block.state_root());
return Ok(None);
};
Ok(Some(T::SlotClock::new(
spec.genesis_slot,
Duration::from_secs(genesis_state.genesis_time()),
Duration::from_secs(spec.seconds_per_slot),
)))
}
fn get_current_epoch<T: BeaconChainTypes>(
db: &Arc<HotColdDB<T::EthSpec, T::HotStore, T::ColdStore>>,
log: &Logger,
) -> Result<Epoch, Error> {
get_slot_clock::<T>(db, log)?
.and_then(|clock| clock.now())
.map(|slot| slot.epoch(T::EthSpec::slots_per_epoch()))
.ok_or(Error::SlotClockUnavailableForMigration)
}
pub fn upgrade_to_v18<T: BeaconChainTypes>(
db: Arc<HotColdDB<T::EthSpec, T::HotStore, T::ColdStore>>,
log: Logger,
) -> Result<Vec<KeyValueStoreOp>, Error> {
db.heal_freezer_block_roots_at_split()?;
db.heal_freezer_block_roots_at_genesis()?;
info!(log, "Healed freezer block roots");
// No-op, even if Deneb has already occurred. The database is probably borked in this case, but
// *maybe* the fork recovery will revert the minority fork and succeed.
if let Some(deneb_fork_epoch) = db.get_chain_spec().deneb_fork_epoch {
let current_epoch = get_current_epoch::<T>(&db, &log)?;
if current_epoch >= deneb_fork_epoch {
warn!(
log,
"Attempting upgrade to v18 schema";
"info" => "this may not work as Deneb has already been activated"
);
} else {
info!(
log,
"Upgrading to v18 schema";
"info" => "ready for Deneb",
"epochs_until_deneb" => deneb_fork_epoch - current_epoch
);
}
} else {
info!(
log,
"Upgrading to v18 schema";
"info" => "ready for Deneb once it is scheduled"
);
}
Ok(vec![])
}
pub fn downgrade_from_v18<T: BeaconChainTypes>(
db: Arc<HotColdDB<T::EthSpec, T::HotStore, T::ColdStore>>,
log: Logger,
) -> Result<Vec<KeyValueStoreOp>, Error> {
// We cannot downgrade from V18 once the Deneb fork has been activated, because there will
// be blobs and blob metadata in the database that aren't understood by the V17 schema.
if let Some(deneb_fork_epoch) = db.get_chain_spec().deneb_fork_epoch {
let current_epoch = get_current_epoch::<T>(&db, &log)?;
if current_epoch >= deneb_fork_epoch {
error!(
log,
"Deneb already active: v18+ is mandatory";
"current_epoch" => current_epoch,
"deneb_fork_epoch" => deneb_fork_epoch,
);
return Err(Error::UnableToDowngrade);
} else {
info!(
log,
"Downgrading to v17 schema";
"info" => "you will need to upgrade before Deneb",
"epochs_until_deneb" => deneb_fork_epoch - current_epoch
);
}
} else {
info!(
log,
"Downgrading to v17 schema";
"info" => "you need to upgrade before Deneb",
);
}
let ops = vec![KeyValueStoreOp::DeleteKey(get_key_for_col(
DBColumn::BeaconMeta.into(),
BLOB_INFO_KEY.as_bytes(),
))];
Ok(ops)
}

View File

@@ -0,0 +1,65 @@
use crate::beacon_chain::BeaconChainTypes;
use slog::{debug, info, Logger};
use std::sync::Arc;
use store::{get_key_for_col, DBColumn, Error, HotColdDB, KeyValueStore, KeyValueStoreOp};
pub fn upgrade_to_v19<T: BeaconChainTypes>(
db: Arc<HotColdDB<T::EthSpec, T::HotStore, T::ColdStore>>,
log: Logger,
) -> Result<Vec<KeyValueStoreOp>, Error> {
let mut hot_delete_ops = vec![];
let mut blob_keys = vec![];
let column = DBColumn::BeaconBlob;
debug!(log, "Migrating from v18 to v19");
// Iterate through the blobs on disk.
for res in db.hot_db.iter_column_keys::<Vec<u8>>(column) {
let key = res?;
let key_col = get_key_for_col(column.as_str(), &key);
hot_delete_ops.push(KeyValueStoreOp::DeleteKey(key_col));
blob_keys.push(key);
}
let num_blobs = blob_keys.len();
debug!(log, "Collected {} blob lists to migrate", num_blobs);
let batch_size = 500;
let mut batch = Vec::with_capacity(batch_size);
for key in blob_keys {
let next_blob = db.hot_db.get_bytes(column.as_str(), &key)?;
if let Some(next_blob) = next_blob {
let key_col = get_key_for_col(column.as_str(), &key);
batch.push(KeyValueStoreOp::PutKeyValue(key_col, next_blob));
if batch.len() >= batch_size {
db.blobs_db.do_atomically(batch.clone())?;
batch.clear();
}
}
}
// Process the remaining batch if it's not empty
if !batch.is_empty() {
db.blobs_db.do_atomically(batch)?;
}
debug!(log, "Wrote {} blobs to the blobs db", num_blobs);
// Delete all the blobs
info!(log, "Upgrading to v19 schema");
Ok(hot_delete_ops)
}
pub fn downgrade_from_v19<T: BeaconChainTypes>(
_db: Arc<HotColdDB<T::EthSpec, T::HotStore, T::ColdStore>>,
log: Logger,
) -> Result<Vec<KeyValueStoreOp>, Error> {
// No-op
info!(
log,
"Downgrading to v18 schema";
);
Ok(vec![])
}

View File

@@ -0,0 +1,103 @@
use crate::beacon_chain::{BeaconChainTypes, OP_POOL_DB_KEY};
use operation_pool::{
PersistedOperationPool, PersistedOperationPoolV15, PersistedOperationPoolV20,
};
use slog::{debug, info, Logger};
use std::sync::Arc;
use store::{Error, HotColdDB, KeyValueStoreOp, StoreItem};
use types::Attestation;
pub fn upgrade_to_v20<T: BeaconChainTypes>(
db: Arc<HotColdDB<T::EthSpec, T::HotStore, T::ColdStore>>,
log: Logger,
) -> Result<Vec<KeyValueStoreOp>, Error> {
// Load a V15 op pool and transform it to V20.
let Some(PersistedOperationPoolV15::<T::EthSpec> {
attestations_v15,
sync_contributions,
attester_slashings_v15,
proposer_slashings,
voluntary_exits,
bls_to_execution_changes,
capella_bls_change_broadcast_indices,
}) = db.get_item(&OP_POOL_DB_KEY)?
else {
debug!(log, "Nothing to do, no operation pool stored");
return Ok(vec![]);
};
let attestations = attestations_v15
.into_iter()
.map(|(attestation, indices)| (Attestation::Base(attestation).into(), indices))
.collect();
let attester_slashings = attester_slashings_v15
.into_iter()
.map(|slashing| slashing.into())
.collect();
let v20 = PersistedOperationPool::V20(PersistedOperationPoolV20 {
attestations,
sync_contributions,
attester_slashings,
proposer_slashings,
voluntary_exits,
bls_to_execution_changes,
capella_bls_change_broadcast_indices,
});
Ok(vec![v20.as_kv_store_op(OP_POOL_DB_KEY)])
}
pub fn downgrade_from_v20<T: BeaconChainTypes>(
db: Arc<HotColdDB<T::EthSpec, T::HotStore, T::ColdStore>>,
log: Logger,
) -> Result<Vec<KeyValueStoreOp>, Error> {
// Load a V20 op pool and transform it to V15.
let Some(PersistedOperationPoolV20::<T::EthSpec> {
attestations,
sync_contributions,
attester_slashings,
proposer_slashings,
voluntary_exits,
bls_to_execution_changes,
capella_bls_change_broadcast_indices,
}) = db.get_item(&OP_POOL_DB_KEY)?
else {
debug!(log, "Nothing to do, no operation pool stored");
return Ok(vec![]);
};
let attestations_v15 = attestations
.into_iter()
.filter_map(|(attestation, indices)| {
if let Attestation::Base(attestation) = attestation.into() {
Some((attestation, indices))
} else {
info!(log, "Dropping attestation during downgrade"; "reason" => "not a base attestation");
None
}
})
.collect();
let attester_slashings_v15 = attester_slashings
.into_iter()
.filter_map(|slashing| match slashing.try_into() {
Ok(slashing) => Some(slashing),
Err(_) => {
info!(log, "Dropping attester slashing during downgrade"; "reason" => "not a base attester slashing");
None
}
})
.collect();
let v15 = PersistedOperationPool::V15(PersistedOperationPoolV15 {
attestations_v15,
sync_contributions,
attester_slashings_v15,
proposer_slashings,
voluntary_exits,
bls_to_execution_changes,
capella_bls_change_broadcast_indices,
});
Ok(vec![v15.as_kv_store_op(OP_POOL_DB_KEY)])
}

View File

@@ -268,9 +268,9 @@ impl BlockShufflingIds {
}
}
pub fn try_from_head<T: EthSpec>(
pub fn try_from_head<E: EthSpec>(
head_block_root: Hash256,
head_state: &BeaconState<T>,
head_state: &BeaconState<E>,
) -> Result<Self, String> {
let get_shuffling_id = |relative_epoch| {
AttestationShufflingId::new(head_block_root, head_state, relative_epoch).map_err(|e| {
@@ -339,7 +339,7 @@ mod test {
.clone();
let committee_b = state.committee_cache(RelativeEpoch::Next).unwrap().clone();
assert!(committee_a != committee_b);
(Arc::new(committee_a), Arc::new(committee_b))
(committee_a, committee_b)
}
/// Builds a deterministic but incoherent shuffling ID from a `u64`.

View File

@@ -1,523 +0,0 @@
use crate::BeaconSnapshot;
use itertools::process_results;
use std::cmp;
use std::sync::Arc;
use std::time::Duration;
use types::{
beacon_state::CloneConfig, BeaconState, BlindedPayload, ChainSpec, Epoch, EthSpec, Hash256,
SignedBeaconBlock, Slot,
};
/// The default size of the cache.
pub const DEFAULT_SNAPSHOT_CACHE_SIZE: usize = 4;
/// The minimum block delay to clone the state in the cache instead of removing it.
/// This helps keep block processing fast during re-orgs from late blocks.
fn minimum_block_delay_for_clone(seconds_per_slot: u64) -> Duration {
// If the block arrived at the attestation deadline or later, it might get re-orged.
Duration::from_secs(seconds_per_slot) / 3
}
/// This snapshot is to be used for verifying a child of `self.beacon_block`.
#[derive(Debug)]
pub struct PreProcessingSnapshot<T: EthSpec> {
/// This state is equivalent to the `self.beacon_block.state_root()` state that has been
/// advanced forward one slot using `per_slot_processing`. This state is "primed and ready" for
/// the application of another block.
pub pre_state: BeaconState<T>,
/// This value is only set to `Some` if the `pre_state` was *not* advanced forward.
pub beacon_state_root: Option<Hash256>,
pub beacon_block: SignedBeaconBlock<T, BlindedPayload<T>>,
pub beacon_block_root: Hash256,
}
impl<T: EthSpec> From<BeaconSnapshot<T>> for PreProcessingSnapshot<T> {
fn from(snapshot: BeaconSnapshot<T>) -> Self {
let beacon_state_root = Some(snapshot.beacon_state_root());
Self {
pre_state: snapshot.beacon_state,
beacon_state_root,
beacon_block: snapshot.beacon_block.clone_as_blinded(),
beacon_block_root: snapshot.beacon_block_root,
}
}
}
impl<T: EthSpec> CacheItem<T> {
pub fn new_without_pre_state(snapshot: BeaconSnapshot<T>) -> Self {
Self {
beacon_block: snapshot.beacon_block,
beacon_block_root: snapshot.beacon_block_root,
beacon_state: snapshot.beacon_state,
pre_state: None,
}
}
fn clone_to_snapshot_with(&self, clone_config: CloneConfig) -> BeaconSnapshot<T> {
BeaconSnapshot {
beacon_state: self.beacon_state.clone_with(clone_config),
beacon_block: self.beacon_block.clone(),
beacon_block_root: self.beacon_block_root,
}
}
pub fn into_pre_state(self) -> PreProcessingSnapshot<T> {
// Do not include the beacon state root if the state has been advanced.
let beacon_state_root =
Some(self.beacon_block.state_root()).filter(|_| self.pre_state.is_none());
PreProcessingSnapshot {
beacon_block: self.beacon_block.clone_as_blinded(),
beacon_block_root: self.beacon_block_root,
pre_state: self.pre_state.unwrap_or(self.beacon_state),
beacon_state_root,
}
}
pub fn clone_as_pre_state(&self) -> PreProcessingSnapshot<T> {
// Do not include the beacon state root if the state has been advanced.
let beacon_state_root =
Some(self.beacon_block.state_root()).filter(|_| self.pre_state.is_none());
PreProcessingSnapshot {
beacon_block: self.beacon_block.clone_as_blinded(),
beacon_block_root: self.beacon_block_root,
pre_state: self
.pre_state
.as_ref()
.map_or_else(|| self.beacon_state.clone(), |pre_state| pre_state.clone()),
beacon_state_root,
}
}
}
/// The information required for block production.
pub struct BlockProductionPreState<T: EthSpec> {
/// This state may or may not have been advanced forward a single slot.
///
/// See the documentation in the `crate::state_advance_timer` module for more information.
pub pre_state: BeaconState<T>,
/// This value will only be `Some` if `self.pre_state` was **not** advanced forward a single
/// slot.
///
/// This value can be used to avoid tree-hashing the state during the first call to
/// `per_slot_processing`.
pub state_root: Option<Hash256>,
}
pub enum StateAdvance<T: EthSpec> {
/// The cache does not contain the supplied block root.
BlockNotFound,
/// The cache contains the supplied block root but the state has already been advanced.
AlreadyAdvanced,
/// The cache contains the supplied block root and the state has not yet been advanced.
State {
state: Box<BeaconState<T>>,
state_root: Hash256,
block_slot: Slot,
},
}
/// The item stored in the `SnapshotCache`.
pub struct CacheItem<T: EthSpec> {
beacon_block: Arc<SignedBeaconBlock<T>>,
beacon_block_root: Hash256,
/// This state is equivalent to `self.beacon_block.state_root()`.
beacon_state: BeaconState<T>,
/// This state is equivalent to `self.beacon_state` that has had `per_slot_processing` applied
/// to it. This state assists in optimizing block processing.
pre_state: Option<BeaconState<T>>,
}
impl<T: EthSpec> Into<BeaconSnapshot<T>> for CacheItem<T> {
fn into(self) -> BeaconSnapshot<T> {
BeaconSnapshot {
beacon_state: self.beacon_state,
beacon_block: self.beacon_block,
beacon_block_root: self.beacon_block_root,
}
}
}
/// Provides a cache of `BeaconSnapshot` that is intended primarily for block processing.
///
/// ## Cache Queuing
///
/// The cache has a non-standard queue mechanism (specifically, it is not LRU).
///
/// The cache has a max number of elements (`max_len`). Until `max_len` is achieved, all snapshots
/// are simply added to the queue. Once `max_len` is achieved, adding a new snapshot will cause an
/// existing snapshot to be ejected. The ejected snapshot will:
///
/// - Never be the `head_block_root`.
/// - Be the snapshot with the lowest `state.slot` (ties broken arbitrarily).
pub struct SnapshotCache<T: EthSpec> {
max_len: usize,
head_block_root: Hash256,
snapshots: Vec<CacheItem<T>>,
}
impl<T: EthSpec> SnapshotCache<T> {
/// Instantiate a new cache which contains the `head` snapshot.
///
/// Setting `max_len = 0` is equivalent to setting `max_len = 1`.
pub fn new(max_len: usize, head: BeaconSnapshot<T>) -> Self {
Self {
max_len: cmp::max(max_len, 1),
head_block_root: head.beacon_block_root,
snapshots: vec![CacheItem::new_without_pre_state(head)],
}
}
/// The block roots of all snapshots contained in `self`.
pub fn beacon_block_roots(&self) -> Vec<Hash256> {
self.snapshots.iter().map(|s| s.beacon_block_root).collect()
}
/// The number of snapshots contained in `self`.
pub fn len(&self) -> usize {
self.snapshots.len()
}
/// Insert a snapshot, potentially removing an existing snapshot if `self` is at capacity (see
/// struct-level documentation for more info).
pub fn insert(
&mut self,
snapshot: BeaconSnapshot<T>,
pre_state: Option<BeaconState<T>>,
spec: &ChainSpec,
) {
let parent_root = snapshot.beacon_block.message().parent_root();
let item = CacheItem {
beacon_block: snapshot.beacon_block.clone(),
beacon_block_root: snapshot.beacon_block_root,
beacon_state: snapshot.beacon_state,
pre_state,
};
// Remove the grandparent of the block that was just inserted.
//
// Assuming it's unlikely to see re-orgs deeper than one block, this method helps keep the
// cache small by removing any states that already have more than one descendant.
//
// Remove the grandparent first to free up room in the cache.
let grandparent_result =
process_results(item.beacon_state.rev_iter_block_roots(spec), |iter| {
iter.map(|(_slot, root)| root)
.find(|root| *root != item.beacon_block_root && *root != parent_root)
});
if let Ok(Some(grandparent_root)) = grandparent_result {
let head_block_root = self.head_block_root;
self.snapshots.retain(|snapshot| {
let root = snapshot.beacon_block_root;
root == head_block_root || root != grandparent_root
});
}
if self.snapshots.len() < self.max_len {
self.snapshots.push(item);
} else {
let insert_at = self
.snapshots
.iter()
.enumerate()
.filter_map(|(i, snapshot)| {
if snapshot.beacon_block_root != self.head_block_root {
Some((i, snapshot.beacon_state.slot()))
} else {
None
}
})
.min_by_key(|(_i, slot)| *slot)
.map(|(i, _slot)| i);
if let Some(i) = insert_at {
self.snapshots[i] = item;
}
}
}
/// If available, returns a `CacheItem` that should be used for importing/processing a block.
/// The method will remove the block from `self`, carrying across any caches that may or may not
/// be built.
///
/// In the event the block being processed was observed late, clone the cache instead of
/// moving it. This allows us to process the next block quickly in the case of a re-org.
/// Additionally, if the slot was skipped, clone the cache. This ensures blocks that are
/// later than 1 slot still have access to the cache and can be processed quickly.
pub fn get_state_for_block_processing(
&mut self,
block_root: Hash256,
block_slot: Slot,
block_delay: Option<Duration>,
spec: &ChainSpec,
) -> Option<(PreProcessingSnapshot<T>, bool)> {
self.snapshots
.iter()
.position(|snapshot| snapshot.beacon_block_root == block_root)
.map(|i| {
if let Some(cache) = self.snapshots.get(i) {
// Avoid cloning the block during sync (when the `block_delay` is `None`).
if let Some(delay) = block_delay {
if delay >= minimum_block_delay_for_clone(spec.seconds_per_slot)
&& delay <= Duration::from_secs(spec.seconds_per_slot) * 4
|| block_slot > cache.beacon_block.slot() + 1
{
return (cache.clone_as_pre_state(), true);
}
}
}
(self.snapshots.remove(i).into_pre_state(), false)
})
}
/// If available, obtains a clone of a `BeaconState` that should be used for block production.
/// The clone will use `CloneConfig:all()`, ensuring any tree-hash cache is cloned too.
///
/// ## Note
///
/// This method clones the `BeaconState` (instead of removing it) since we assume that any block
/// we produce will soon be pushed to the `BeaconChain` for importing/processing. Keeping a copy
/// of that `BeaconState` in `self` will greatly help with import times.
pub fn get_state_for_block_production(
&self,
block_root: Hash256,
) -> Option<BlockProductionPreState<T>> {
self.snapshots
.iter()
.find(|snapshot| snapshot.beacon_block_root == block_root)
.map(|snapshot| {
if let Some(pre_state) = &snapshot.pre_state {
BlockProductionPreState {
pre_state: pre_state.clone_with(CloneConfig::all()),
state_root: None,
}
} else {
BlockProductionPreState {
pre_state: snapshot.beacon_state.clone_with(CloneConfig::all()),
state_root: Some(snapshot.beacon_block.state_root()),
}
}
})
}
/// If there is a snapshot with `block_root`, clone it and return the clone.
pub fn get_cloned(
&self,
block_root: Hash256,
clone_config: CloneConfig,
) -> Option<BeaconSnapshot<T>> {
self.snapshots
.iter()
.find(|snapshot| snapshot.beacon_block_root == block_root)
.map(|snapshot| snapshot.clone_to_snapshot_with(clone_config))
}
pub fn get_for_state_advance(&mut self, block_root: Hash256) -> StateAdvance<T> {
if let Some(snapshot) = self
.snapshots
.iter_mut()
.find(|snapshot| snapshot.beacon_block_root == block_root)
{
if snapshot.pre_state.is_some() {
StateAdvance::AlreadyAdvanced
} else {
let cloned = snapshot
.beacon_state
.clone_with(CloneConfig::committee_caches_only());
StateAdvance::State {
state: Box::new(std::mem::replace(&mut snapshot.beacon_state, cloned)),
state_root: snapshot.beacon_block.state_root(),
block_slot: snapshot.beacon_block.slot(),
}
}
} else {
StateAdvance::BlockNotFound
}
}
pub fn update_pre_state(&mut self, block_root: Hash256, state: BeaconState<T>) -> Option<()> {
self.snapshots
.iter_mut()
.find(|snapshot| snapshot.beacon_block_root == block_root)
.map(|snapshot| {
snapshot.pre_state = Some(state);
})
}
/// Removes all snapshots from the queue that are less than or equal to the finalized epoch.
pub fn prune(&mut self, finalized_epoch: Epoch) {
self.snapshots.retain(|snapshot| {
snapshot.beacon_state.slot() > finalized_epoch.start_slot(T::slots_per_epoch())
})
}
/// Inform the cache that the head of the beacon chain has changed.
///
/// The snapshot that matches this `head_block_root` will never be ejected from the cache
/// during `Self::insert`.
pub fn update_head(&mut self, head_block_root: Hash256) {
self.head_block_root = head_block_root
}
}
#[cfg(test)]
mod test {
use super::*;
use crate::test_utils::{BeaconChainHarness, EphemeralHarnessType};
use types::{
test_utils::generate_deterministic_keypair, BeaconBlock, Epoch, MainnetEthSpec,
SignedBeaconBlock, Slot,
};
fn get_harness() -> BeaconChainHarness<EphemeralHarnessType<MainnetEthSpec>> {
let harness = BeaconChainHarness::builder(MainnetEthSpec)
.default_spec()
.deterministic_keypairs(1)
.fresh_ephemeral_store()
.build();
harness.advance_slot();
harness
}
const CACHE_SIZE: usize = 4;
fn get_snapshot(i: u64) -> BeaconSnapshot<MainnetEthSpec> {
let spec = MainnetEthSpec::default_spec();
let beacon_state = get_harness().chain.head_beacon_state_cloned();
let signed_beacon_block = SignedBeaconBlock::from_block(
BeaconBlock::empty(&spec),
generate_deterministic_keypair(0)
.sk
.sign(Hash256::from_low_u64_be(42)),
);
BeaconSnapshot {
beacon_state,
beacon_block: Arc::new(signed_beacon_block),
beacon_block_root: Hash256::from_low_u64_be(i),
}
}
#[test]
fn insert_get_prune_update() {
let spec = MainnetEthSpec::default_spec();
let mut cache = SnapshotCache::new(CACHE_SIZE, get_snapshot(0));
// Insert a bunch of entries in the cache. It should look like this:
//
// Index Root
// 0 0 <--head
// 1 1
// 2 2
// 3 3
for i in 1..CACHE_SIZE as u64 {
let mut snapshot = get_snapshot(i);
// Each snapshot should be one slot into an epoch, with each snapshot one epoch apart.
*snapshot.beacon_state.slot_mut() =
Slot::from(i * MainnetEthSpec::slots_per_epoch() + 1);
cache.insert(snapshot, None, &spec);
assert_eq!(
cache.snapshots.len(),
i as usize + 1,
"cache length should be as expected"
);
assert_eq!(cache.head_block_root, Hash256::from_low_u64_be(0));
}
// Insert a new value in the cache. Afterwards it should look like:
//
// Index Root
// 0 0 <--head
// 1 42
// 2 2
// 3 3
assert_eq!(cache.snapshots.len(), CACHE_SIZE);
cache.insert(get_snapshot(42), None, &spec);
assert_eq!(cache.snapshots.len(), CACHE_SIZE);
assert!(
cache
.get_state_for_block_processing(
Hash256::from_low_u64_be(1),
Slot::new(0),
None,
&spec
)
.is_none(),
"the snapshot with the lowest slot should have been removed during the insert function"
);
assert!(cache
.get_cloned(Hash256::from_low_u64_be(1), CloneConfig::none())
.is_none());
assert_eq!(
cache
.get_cloned(Hash256::from_low_u64_be(0), CloneConfig::none())
.expect("the head should still be in the cache")
.beacon_block_root,
Hash256::from_low_u64_be(0),
"get_cloned should get the correct snapshot"
);
assert_eq!(
cache
.get_state_for_block_processing(
Hash256::from_low_u64_be(0),
Slot::new(0),
None,
&spec
)
.expect("the head should still be in the cache")
.0
.beacon_block_root,
Hash256::from_low_u64_be(0),
"get_state_for_block_processing should get the correct snapshot"
);
assert_eq!(
cache.snapshots.len(),
CACHE_SIZE - 1,
"get_state_for_block_processing should shorten the cache"
);
// Prune the cache. Afterwards it should look like:
//
// Index Root
// 0 2
// 1 3
cache.prune(Epoch::new(2));
assert_eq!(cache.snapshots.len(), 2);
cache.update_head(Hash256::from_low_u64_be(2));
// Over-fill the cache so it needs to eject some old values on insert.
for i in 0..CACHE_SIZE as u64 {
cache.insert(get_snapshot(u64::max_value() - i), None, &spec);
}
// Ensure that the new head value was not removed from the cache.
assert_eq!(
cache
.get_state_for_block_processing(
Hash256::from_low_u64_be(2),
Slot::new(0),
None,
&spec
)
.expect("the new head should still be in the cache")
.0
.beacon_block_root,
Hash256::from_low_u64_be(2),
"get_state_for_block_processing should get the correct snapshot"
);
}
}

View File

@@ -15,9 +15,7 @@
//! 2. There's a possibility that the head block is never built upon, causing wasted CPU cycles.
use crate::validator_monitor::HISTORIC_EPOCHS as VALIDATOR_MONITOR_HISTORIC_EPOCHS;
use crate::{
beacon_chain::{ATTESTATION_CACHE_LOCK_TIMEOUT, BLOCK_PROCESSING_CACHE_LOCK_TIMEOUT},
chain_config::FORK_CHOICE_LOOKAHEAD_FACTOR,
snapshot_cache::StateAdvance,
beacon_chain::ATTESTATION_CACHE_LOCK_TIMEOUT, chain_config::FORK_CHOICE_LOOKAHEAD_FACTOR,
BeaconChain, BeaconChainError, BeaconChainTypes,
};
use slog::{debug, error, warn, Logger};
@@ -27,9 +25,10 @@ use std::sync::{
atomic::{AtomicBool, Ordering},
Arc,
};
use store::KeyValueStore;
use task_executor::TaskExecutor;
use tokio::time::{sleep, sleep_until, Instant};
use types::{AttestationShufflingId, EthSpec, Hash256, RelativeEpoch, Slot};
use types::{AttestationShufflingId, BeaconStateError, EthSpec, Hash256, RelativeEpoch, Slot};
/// If the head slot is more than `MAX_ADVANCE_DISTANCE` from the current slot, then don't perform
/// the state advancement.
@@ -48,7 +47,10 @@ const MAX_FORK_CHOICE_DISTANCE: u64 = 256;
#[derive(Debug)]
enum Error {
BeaconChain(BeaconChainError),
HeadMissingFromSnapshotCache(Hash256),
// We don't use the inner value directly, but it's used in the Debug impl.
HeadMissingFromSnapshotCache(#[allow(dead_code)] Hash256),
BeaconState(#[allow(dead_code)] BeaconStateError),
Store(#[allow(dead_code)] store::Error),
MaxDistanceExceeded {
current_slot: Slot,
head_slot: Slot,
@@ -68,6 +70,18 @@ impl From<BeaconChainError> for Error {
}
}
impl From<BeaconStateError> for Error {
fn from(e: BeaconStateError) -> Self {
Self::BeaconState(e)
}
}
impl From<store::Error> for Error {
fn from(e: store::Error) -> Self {
Self::Store(e)
}
}
/// Provides a simple thread-safe lock to be used for task co-ordination. Practically equivalent to
/// `Mutex<()>`.
#[derive(Clone)]
@@ -113,14 +127,11 @@ async fn state_advance_timer<T: BeaconChainTypes>(
let slot_duration = slot_clock.slot_duration();
loop {
let duration_to_next_slot = match beacon_chain.slot_clock.duration_to_next_slot() {
Some(duration) => duration,
None => {
error!(log, "Failed to read slot clock");
// If we can't read the slot clock, just wait another slot.
sleep(slot_duration).await;
continue;
}
let Some(duration_to_next_slot) = beacon_chain.slot_clock.duration_to_next_slot() else {
error!(log, "Failed to read slot clock");
// If we can't read the slot clock, just wait another slot.
sleep(slot_duration).await;
continue;
};
// Run the state advance 3/4 of the way through the slot (9s on mainnet).
@@ -230,14 +241,18 @@ async fn state_advance_timer<T: BeaconChainTypes>(
// Prepare proposers so that the node can send payload attributes in the case where
// it decides to abandon a proposer boost re-org.
if let Err(e) = beacon_chain.prepare_beacon_proposer(current_slot).await {
warn!(
log,
"Unable to prepare proposer with lookahead";
"error" => ?e,
"slot" => next_slot,
);
}
beacon_chain
.prepare_beacon_proposer(current_slot)
.await
.unwrap_or_else(|e| {
warn!(
log,
"Unable to prepare proposer with lookahead";
"error" => ?e,
"slot" => next_slot,
);
None
});
// Use a blocking task to avoid blocking the core executor whilst waiting for locks
// in `ForkChoiceSignalTx`.
@@ -263,9 +278,9 @@ async fn state_advance_timer<T: BeaconChainTypes>(
}
}
/// Reads the `snapshot_cache` from the `beacon_chain` and attempts to take a clone of the
/// Reads the `state_cache` from the `beacon_chain` and attempts to take a clone of the
/// `BeaconState` of the head block. If it obtains this clone, the state will be advanced a single
/// slot then placed back in the `snapshot_cache` to be used for block verification.
/// slot then placed in the `state_cache` to be used for block verification.
///
/// See the module-level documentation for rationale.
fn advance_head<T: BeaconChainTypes>(
@@ -290,46 +305,42 @@ fn advance_head<T: BeaconChainTypes>(
}
}
let head_root = beacon_chain.head_beacon_block_root();
let (head_slot, head_state_root, mut state) = match beacon_chain
.snapshot_cache
.try_write_for(BLOCK_PROCESSING_CACHE_LOCK_TIMEOUT)
.ok_or(BeaconChainError::SnapshotCacheLockTimeout)?
.get_for_state_advance(head_root)
{
StateAdvance::AlreadyAdvanced => {
return Err(Error::StateAlreadyAdvanced {
block_root: head_root,
})
}
StateAdvance::BlockNotFound => return Err(Error::HeadMissingFromSnapshotCache(head_root)),
StateAdvance::State {
state,
state_root,
block_slot,
} => (block_slot, state_root, *state),
let (head_block_root, head_block_state_root) = {
let snapshot = beacon_chain.head_snapshot();
(snapshot.beacon_block_root, snapshot.beacon_state_root())
};
let (head_state_root, mut state) = beacon_chain
.store
.get_advanced_hot_state(head_block_root, current_slot, head_block_state_root)?
.ok_or(Error::HeadMissingFromSnapshotCache(head_block_root))?;
// Protect against advancing a state more than a single slot.
//
// Advancing more than one slot without storing the intermediate state would corrupt the
// database. Future works might store temporary, intermediate states inside this function.
match state.slot().cmp(&state.latest_block_header().slot) {
std::cmp::Ordering::Equal => (),
std::cmp::Ordering::Greater => {
return Err(Error::StateAlreadyAdvanced {
block_root: head_block_root,
});
}
std::cmp::Ordering::Less => {
return Err(Error::BadStateSlot {
_block_slot: state.latest_block_header().slot,
_state_slot: state.slot(),
});
}
}
let initial_slot = state.slot();
let initial_epoch = state.current_epoch();
let state_root = if state.slot() == head_slot {
Some(head_state_root)
} else {
// Protect against advancing a state more than a single slot.
//
// Advancing more than one slot without storing the intermediate state would corrupt the
// database. Future works might store temporary, intermediate states inside this function.
return Err(Error::BadStateSlot {
_block_slot: head_slot,
_state_slot: state.slot(),
});
};
// Advance the state a single slot.
if let Some(summary) = per_slot_processing(&mut state, state_root, &beacon_chain.spec)
.map_err(BeaconChainError::from)?
if let Some(summary) =
per_slot_processing(&mut state, Some(head_state_root), &beacon_chain.spec)
.map_err(BeaconChainError::from)?
{
// Expose Prometheus metrics.
if let Err(e) = summary.observe_metrics() {
@@ -363,7 +374,7 @@ fn advance_head<T: BeaconChainTypes>(
debug!(
log,
"Advanced head state one slot";
"head_root" => ?head_root,
"head_block_root" => ?head_block_root,
"state_slot" => state.slot(),
"current_slot" => current_slot,
);
@@ -382,14 +393,14 @@ fn advance_head<T: BeaconChainTypes>(
if initial_epoch < state.current_epoch() {
// Update the proposer cache.
//
// We supply the `head_root` as the decision block since the prior `if` statement guarantees
// We supply the `head_block_root` as the decision block since the prior `if` statement guarantees
// the head root is the latest block from the prior epoch.
beacon_chain
.beacon_proposer_cache
.lock()
.insert(
state.current_epoch(),
head_root,
head_block_root,
state
.get_beacon_proposer_indices(&beacon_chain.spec)
.map_err(BeaconChainError::from)?,
@@ -398,8 +409,9 @@ fn advance_head<T: BeaconChainTypes>(
.map_err(BeaconChainError::from)?;
// Update the attester cache.
let shuffling_id = AttestationShufflingId::new(head_root, &state, RelativeEpoch::Next)
.map_err(BeaconChainError::from)?;
let shuffling_id =
AttestationShufflingId::new(head_block_root, &state, RelativeEpoch::Next)
.map_err(BeaconChainError::from)?;
let committee_cache = state
.committee_cache(RelativeEpoch::Next)
.map_err(BeaconChainError::from)?;
@@ -412,7 +424,7 @@ fn advance_head<T: BeaconChainTypes>(
debug!(
log,
"Primed proposer and attester caches";
"head_root" => ?head_root,
"head_block_root" => ?head_block_root,
"next_epoch_shuffling_root" => ?shuffling_id.shuffling_decision_block,
"state_epoch" => state.current_epoch(),
"current_epoch" => current_slot.epoch(T::EthSpec::slots_per_epoch()),
@@ -422,22 +434,13 @@ fn advance_head<T: BeaconChainTypes>(
// Apply the state to the attester cache, if the cache deems it interesting.
beacon_chain
.attester_cache
.maybe_cache_state(&state, head_root, &beacon_chain.spec)
.maybe_cache_state(&state, head_block_root, &beacon_chain.spec)
.map_err(BeaconChainError::from)?;
let final_slot = state.slot();
// Insert the advanced state back into the snapshot cache.
beacon_chain
.snapshot_cache
.try_write_for(BLOCK_PROCESSING_CACHE_LOCK_TIMEOUT)
.ok_or(BeaconChainError::SnapshotCacheLockTimeout)?
.update_pre_state(head_root, state)
.ok_or(Error::HeadMissingFromSnapshotCache(head_root))?;
// If we have moved into the next slot whilst processing the state then this function is going
// to become ineffective and likely become a hindrance as we're stealing the tree hash cache
// from the snapshot cache (which may force the next block to rebuild a new one).
// to become ineffective.
//
// If this warning occurs very frequently on well-resourced machines then we should consider
// starting it earlier in the slot. Otherwise, it's a good indication that the machine is too
@@ -448,7 +451,7 @@ fn advance_head<T: BeaconChainTypes>(
warn!(
log,
"State advance too slow";
"head_root" => %head_root,
"head_block_root" => %head_block_root,
"advanced_slot" => final_slot,
"current_slot" => current_slot,
"starting_slot" => starting_slot,
@@ -456,10 +459,25 @@ fn advance_head<T: BeaconChainTypes>(
);
}
// Write the advanced state to the database with a temporary flag that will be deleted when
// a block is imported on top of this state. We should delete this once we bring in the DB
// changes from tree-states that allow us to prune states without temporary flags.
let advanced_state_root = state.update_tree_hash_cache()?;
let txn_lock = beacon_chain.store.hot_db.begin_rw_transaction();
let state_already_exists = beacon_chain
.store
.load_hot_state_summary(&advanced_state_root)?
.is_some();
let temporary = !state_already_exists;
beacon_chain
.store
.put_state_possibly_temporary(&advanced_state_root, &state, temporary)?;
drop(txn_lock);
debug!(
log,
"Completed state advance";
"head_root" => ?head_root,
"head_block_root" => ?head_block_root,
"advanced_slot" => final_slot,
"initial_slot" => initial_slot,
);

Some files were not shown because too many files have changed in this diff Show More