Commit Graph

900 Commits

Author SHA1 Message Date
João Oliveira
d9f8b13e36 remove no longer required #[allow(clippy::mutable_key_type)] (#6210)
* remove no longer required clippy::mutable_key_type lint
2024-08-01 00:26:26 +00:00
Jimmy Chen
96b00ef66c Replace lazy_static! with LazyLock (#6189)
* Replace `lazy_static` with `LazyLock`.

* Merge branch 'unstable' into remove-lazy-static

# Conflicts:
#	beacon_node/lighthouse_network/src/peer_manager/mod.rs

* Lint fixes.

* Merge branch 'unstable' into remove-lazy-static

# Conflicts:
#	beacon_node/beacon_chain/src/metrics.rs

* Moar lint fixes.

* Update rust version to 1.80.0.

* Merge branch 'unstable' into remove-lazy-static
2024-07-29 11:42:31 +00:00
Jimmy Chen
19b3ab39ee Data column gossip validation and error handling (#6181)
* Add gossip verification and error handling.

* Merge branch 'unstable' into das-gossip-validation

* Add inclusion proof verification and some renames for consistency
2024-07-29 05:52:10 +00:00
Age Manning
a3b1ef3129 Visualisation logging for sync batch states (#6034)
* Add visualization for batch states

* Replace icons with emojis

* Reviewers comments

* Change empty emoji and improve docs comments

* Fix lints

* Move to letters rather than emojis

* Replace 'V' with 'v'. Cargo update

* Merge latest unstable

* Improve docs around visualisation

* Merge branch 'unstable' into sync-batch-state
2024-07-29 05:52:07 +00:00
Mac L
cc55e610b9 Rust 1.80.0 lints (#6183)
* Fix lints
2024-07-25 15:56:22 +00:00
Jimmy Chen
4e5a363a4f Add DataColumnSidecar gossip topic and message handling (#6147)
* Add `DataColumnSidecar` gossip topic and verification (#5050 and #5783).

* Remove gossip verification changes (#5783).

* Merge branch 'unstable' into data-column-gossip

# Conflicts:
#	beacon_node/beacon_chain/src/data_column_verification.rs
#	beacon_node/beacon_chain/src/lib.rs

* Add gossip cache timeout for data columns. Rename data column metrics for consistency.

* Remove usage of `unimplemented!` and address review comments.

* Remove unnused `GossipDataColumnError` variants and address review comments.

* Merge branch 'unstable' into data-column-gossip

* Update Cargo.lock

* Arc `ChainSpec` in discovery to avoid performance regression when needing to clone it repeatedly.
2024-07-25 06:05:18 +00:00
Lion - dapplion
4bfca8251d Add range sync metrics to track efficiency (#6095)
* Add more range sync metrics to track efficiency

* Add ignored blocks metrics
2024-07-16 08:58:11 +10:00
Lion - dapplion
0c0b56d9e8 Bound max count of lookups (#6015)
* Bound max count of lookups

* Move up
2024-07-12 11:55:24 +00:00
Lion - dapplion
bde0428ac1 Add sync network context cache size metrics (#6049)
* Add sync network context cache size metrics
2024-07-09 16:31:26 +00:00
Lion - dapplion
2e2ccec9b5 Add randomization in sync retry batch peer selection (#5822)
* Add randomization in sync retry batch peer selection

* Use min

* Apply suggestions from code review

Co-authored-by: João Oliveira <hello@jxs.pt>

* Merge branch 'unstable' into peer-prio
2024-07-09 00:24:52 +00:00
João Oliveira
a59a61fef9 Remove generic Id param from RequestId (#6032)
* rename RequestId's for better context,

and move them to lighthouse_network crate.

* remove unrequired generic AppReqId from RequestID
2024-07-08 23:56:14 +00:00
Michael Sproul
94d55be6fa Increase penalty for old block gossip spam (#6050)
* Increase penalty for old block gossip spam
2024-07-05 04:30:15 +00:00
chonghe
8e0bc9a40b Record BEACON_BLOCK_DELAY_GOSSIP metric only after a bock is verified (#6046)
* Only record metrics after block verified
2024-07-05 02:37:54 +00:00
Michael Sproul
a64cee3af1 Merge remote-tracking branch 'origin/stable' into unstable 2024-06-28 14:07:00 +10:00
Lion - dapplion
784ef5fb43 Pass vec to range sync batch (#5710)
* Pass vec to range sync batch
2024-06-27 18:21:40 +00:00
Lion - dapplion
f14f21f37b Bound lookup parent chain length with tip extension (#5705)
* Bound lookup parent chain length with tip extension

* Add test
2024-06-27 18:21:37 +00:00
Lion - dapplion
b38019cb10 Attempt to continue lookups after adding peers (#5993)
* Attempt to continue lookups after adding peers
2024-06-26 23:53:55 +00:00
Pawan Dhananjay
bf4cbd3b0a Remove all batches related to a peer on disconnect (#5969)
* Remove all batches related to a peer on disconnect

* Cleanup map entries after disconnect

* Allow lookups to continue in case of disconnections

* Pretty response types

* fmt

* Fix lints

* Remove lookup if it cannot progress

* Fix tests

* Remove poll_close on rpc behaviour

* Remove redundant test

* Fix issue raised by lion

* Revert pretty response types

* Cleanup

* Fix test

* Merge remote-tracking branch 'origin/release-v5.2.1' into rpc-error-on-disconnect-revert

* Apply suggestions from joao

Co-authored-by: João Oliveira <hello@jxs.pt>

* Fix log

* update request status on no peers found

* Do not remove lookup after peer disconnection

* Add comments about expected event api

* Update single_block_lookup.rs

* Update mod.rs

* Merge branch 'rpc-error-on-disconnect-revert' into 5969-review

* Merge pull request #10 from dapplion/5969-review

Add comments about expected event api
2024-06-26 23:53:53 +00:00
ethDreamer
c52c598f69 Electra: Remaining Consensus Data Structures (#5712)
* Attestation superstruct changes for EIP 7549 (#5644)

* update

* experiment

* superstruct changes

* revert

* superstruct changes

* fix tests

* indexed attestation

* indexed attestation superstruct

* updated TODOs

* `superstruct` the `AttesterSlashing` (#5636)

* `superstruct` Attester Fork Variants

* Push a little further

* Deal with Encode / Decode of AttesterSlashing

* not so sure about this..

* Stop Encode/Decode Bounds from Propagating Out

* Tons of Changes..

* More Conversions to AttestationRef

* Add AsReference trait (#15)

* Add AsReference trait

* Fix some snafus

* Got it Compiling! :D

* Got Tests Building

* Get beacon chain tests compiling

---------

Co-authored-by: Michael Sproul <micsproul@gmail.com>

* Merge remote-tracking branch 'upstream/unstable' into electra_attestation_changes

* Make EF Tests Fork-Agnostic (#5713)

* Finish EF Test Fork Agnostic (#5714)

* Superstruct `AggregateAndProof` (#5715)

* Upgrade `superstruct` to `0.8.0`

* superstruct `AggregateAndProof`

* Merge remote-tracking branch 'sigp/unstable' into electra_attestation_changes

* cargo fmt

* Merge pull request #5726 from realbigsean/electra_attestation_changes

Merge unstable into Electra attestation changes

* EIP7549 `get_attestation_indices` (#5657)

* get attesting indices electra impl

* fmt

* get tests to pass

* fmt

* fix some beacon chain tests

* fmt

* fix slasher test

* fmt got me again

* fix more tests

* fix tests

* Some small changes (#5739)

* cargo fmt (#5740)

* Sketch op pool changes

* fix get attesting indices (#5742)

* fix get attesting indices

* better errors

* fix compile

* only get committee index once

* Ef test fixes (#5753)

* attestation related ef test fixes

* delete commented out stuff

* Fix Aggregation Pool for Electra (#5754)

* Fix Aggregation Pool for Electra

* Remove Outdated Interface

* fix ssz (#5755)

* Get `electra_op_pool` up to date (#5756)

* fix get attesting indices (#5742)

* fix get attesting indices

* better errors

* fix compile

* only get committee index once

* Ef test fixes (#5753)

* attestation related ef test fixes

* delete commented out stuff

* Fix Aggregation Pool for Electra (#5754)

* Fix Aggregation Pool for Electra

* Remove Outdated Interface

* fix ssz (#5755)

---------

Co-authored-by: realbigsean <sean@sigmaprime.io>

* Revert "Get `electra_op_pool` up to date (#5756)" (#5757)

This reverts commit ab9e58aa3d.

* Merge branch 'electra_attestation_changes' of https://github.com/sigp/lighthouse into electra_op_pool

* Compute on chain aggregate impl (#5752)

* add compute_on_chain_agg impl to op pool changes

* fmt

* get op pool tests to pass

* update the naive agg pool interface (#5760)

* Fix bugs in cross-committee aggregation

* Add comment to max cover optimisation

* Fix assert

* Merge pull request #5749 from sigp/electra_op_pool

Optimise Electra op pool aggregation

* update committee offset

* Fix Electra Fork Choice Tests (#5764)

* Subscribe to the correct subnets for electra attestations (#5782)

* subscribe to the correct att subnets for electra

* subscribe to the correct att subnets for electra

* cargo fmt

* fix slashing handling

* Merge remote-tracking branch 'upstream/unstable'

* Send unagg attestation based on fork

* Publish all aggregates

* just one more check bro plz..

* Merge pull request #5832 from ethDreamer/electra_attestation_changes_merge_unstable

Merge `unstable` into `electra_attestation_changes`

* Merge pull request #5835 from realbigsean/fix-validator-logic

Fix validator logic

* Merge pull request #5816 from realbigsean/electra-attestation-slashing-handling

Electra slashing handling

* Electra attestation changes rm decode impl (#5856)

* Remove Crappy Decode impl for Attestation

* Remove Inefficient Attestation Decode impl

* Implement Schema Upgrade / Downgrade

* Update beacon_node/beacon_chain/src/schema_change/migration_schema_v20.rs

Co-authored-by: Michael Sproul <micsproul@gmail.com>

---------

Co-authored-by: Michael Sproul <micsproul@gmail.com>

* Fix failing attestation tests and misc electra attestation cleanup (#5810)

* - get attestation related beacon chain tests to pass
- observed attestations are now keyed off of data + committee index
- rename op pool attestationref to compactattestationref
- remove unwraps in agg pool and use options instead
- cherry pick some changes from ef-tests-electra

* cargo fmt

* fix failing test

* Revert dockerfile changes

* make committee_index return option

* function args shouldnt be a ref to attestation ref

* fmt

* fix dup imports

---------

Co-authored-by: realbigsean <seananderson33@GMAIL.com>

* fix some todos (#5817)

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into electra_attestation_changes

* add consolidations to merkle calc for inclusion proof

* Remove Duplicate KZG Commitment Merkle Proof Code (#5874)

* Remove Duplicate KZG Commitment Merkle Proof Code

* s/tree_lists/fields/

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into electra_attestation_changes

* fix compile

* Fix slasher tests (#5906)

* Fix electra tests

* Add electra attestations to double vote tests

* Update superstruct to 0.8

* Merge remote-tracking branch 'origin/unstable' into electra_attestation_changes

* Small cleanup in slasher tests

* Clean up Electra observed aggregates (#5929)

* Use consistent key in observed_attestations

* Remove unwraps from observed aggregates

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into electra_attestation_changes

* De-dup attestation constructor logic

* Remove unwraps in Attestation construction

* Dedup match_attestation_data

* Remove outdated TODO

* Use ForkName Ord in fork-choice tests

* Use ForkName Ord in BeaconBlockBody

* Make to_electra not fallible

* Remove TestRandom impl for IndexedAttestation

* Remove IndexedAttestation faulty Decode impl

* Drop TestRandom impl

* Add PendingAttestationInElectra

* Indexed att on disk (#35)

* indexed att on disk

* fix lints

* Update slasher/src/migrate.rs

Co-authored-by: ethDreamer <37123614+ethDreamer@users.noreply.github.com>

---------

Co-authored-by: Lion - dapplion <35266934+dapplion@users.noreply.github.com>
Co-authored-by: ethDreamer <37123614+ethDreamer@users.noreply.github.com>

* add electra fork enabled fn to ForkName impl (#36)

* add electra fork enabled fn to ForkName impl

* remove inadvertent file

* Update common/eth2/src/types.rs

Co-authored-by: ethDreamer <37123614+ethDreamer@users.noreply.github.com>

* Dedup attestation constructor logic in attester cache

* Use if let Ok for committee_bits

* Dedup Attestation constructor code

* Diff reduction in tests

* Fix beacon_chain tests

* Diff reduction

* Use Ord for ForkName in pubsub

* Resolve into_attestation_and_indices todo

* Remove stale TODO

* Fix beacon_chain tests

* Test spec invariant

* Use electra_enabled in pubsub

* Remove get_indexed_attestation_from_signed_aggregate

* Use ok_or instead of if let else

* committees are sorted

* remove dup method `get_indexed_attestation_from_committees`

* Merge pull request #5940 from dapplion/electra_attestation_changes_lionreview

Electra attestations #5712 review

* update default persisted op pool deserialization

* ensure aggregate and proof uses serde untagged on ref

* Fork aware ssz static attestation tests

* Electra attestation changes from Lions review (#5971)

* dedup/cleanup and remove unneeded hashset use

* remove irrelevant TODOs

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into electra_attestation_changes

* Electra attestation changes sean review (#5972)

* instantiate empty bitlist in unreachable code

* clean up error conversion

* fork enabled bool cleanup

* remove a couple todos

* return bools instead of options in `aggregate` and use the result

* delete commented out code

* use map macros in simple transformations

* remove signers_disjoint_from

* get ef tests compiling

* get ef tests compiling

* update intentionally excluded files

* Avoid changing slasher schema for Electra

* Delete slasher schema v4

* Fix clippy

* Fix compilation of beacon_chain tests

* Update database.rs

* Add electra lightclient types

* Update slasher/src/database.rs

* fix imports

* Merge pull request #5980 from dapplion/electra-lightclient

Add electra lightclient types

* Merge pull request #5975 from michaelsproul/electra-slasher-no-migration

Avoid changing slasher schema for Electra

* Update beacon_node/beacon_chain/src/attestation_verification.rs

* Update beacon_node/beacon_chain/src/attestation_verification.rs
2024-06-24 21:08:07 +00:00
Michael Sproul
adb3f865ef Lower tolerance of stale blobs on gossip (#5935)
* Lower tolerance of stale blobs on gossip

* Drop to debug
2024-06-19 05:02:31 +00:00
Lion - dapplion
474c1b4486 Verify inclusion proof should not be fallible (#5787)
* Verify inclusion proof should not be fallible

* Add blob sidecar inclusion test (#33)

* Add blob sidecar inclusion test.

* Fix lint
2024-06-17 15:05:24 +00:00
Michael Sproul
21f3a191c5 Remove extern crate (#5922)
* Remove extern crate
2024-06-17 15:05:21 +00:00
realbigsean
a74098044a Rust 1.79 lints (#5927)
* max_value -> MAX

* remove unnecesary closures

* a couple more max_value -> MAX

* a couple more max_value -> MAX

* Revert "a couple more max_value -> MAX"

This reverts commit 807fe7cae9.

* unused spec field -> phantom data

* ignore some dead code warnings

* update kurtosis repo location
2024-06-13 23:04:30 +00:00
Age Manning
7b48b0b4a7 Errors for all RPC Requests (#5867)
* Return and error if peer has disconnected

* Report errors for rate limited requests

* Code improvement

* Bump rust version to 1.78

* Downgrade to 1.77

* Update beacon_node/lighthouse_network/src/service/mod.rs

Co-authored-by: João Oliveira <hello@jxs.pt>

* fix fmt

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into rpc-peer-disconnect-error

* update lockfile
2024-06-07 00:12:45 +00:00
Lion - dapplion
2c971fa9a1 Make beacon processor queue sizes dynamic (#5573)
* Make beacon processor queue sizes dynamic

* Update tests

* lint

* Review PR
2024-06-03 07:17:11 +00:00
Pawan Dhananjay
fb790decd6 Fix attestations not getting added to the aggregation pool (#5863)
* Remove from delay_map 2 slots after duty

* Cleanup
2024-05-31 04:08:43 +00:00
Lion - dapplion
17dc978760 Add peers to parent lookups (#5858)
* Add peers to parent lookups

* Add test

* lint

* Do not attempt to continue

* add_peers_to_lookup_and_ancestors can't drop
2024-05-30 17:07:32 +00:00
Lion - dapplion
6a7305a487 Add more info about why lookup is in AwaitingDownload (#5838)
* Add more info about why lookup is in AwaitingDownload

* Review comments

* Merge branch 'unstable' into awaiting-download-context

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into awaiting-download-context

* cargo fmt
2024-05-28 02:48:37 +00:00
Akihito Nakano
6f05863007 Remove unused dependencies (#5812)
* Remove unused dependencies from lighthouse_network

* Remove unused dependencies from gossipsub

* Remove unused dependencies from network

* Remove unused dependencies from consensus/types

* Remove unused dependencies from account_manager

* Remove unused dependencies from boot_node

* Remove unused dependencies from database_manager

* Remove unused dependencies from validator_manager

* Remove unused dependencies from testing/ef_tests

* Remove unused dependencies from testing/simulator

* Remove unused dependencies from testing/execution_engine_integration

* Remove unused dependencies from common/system_health

* Remove unused dependencies from common/eth2

* Remove unused dependencies from common/lighthouse_metrics

* Remove unused dependencies from common/eth2_network_config

* Remove unused dependencies from common/logging

* Remove unused dependencies from watch

* Remove unused dependencies from slasher

* Remove unused dependencies from lighthouse

* Remove unused dependencies from lcli

* Remove unused dependencies from beacon_node

* Remove unused dependencies from beacon_node/builder_client

* Remove unused dependencies from beacon_node/beacon_processor

* Remove unused dependencies from beacon_node/execution_layer

* Remove unused dependencies from beacon_node/eth1

* Remove unused dependencies from beacon_node/client

* Remove unused dependencies from beacon_node/store

* Fix missing deps which is used only for testing
2024-05-27 07:59:07 +00:00
Lion - dapplion
e4984217a6 Ensure lookup sync checks caches correctly (#5840)
* Ensure lookup sync checks caches correctly

* fix tests and remove unused method

* Simplify BlockProcessStatus
2024-05-25 14:56:51 +00:00
Lion - dapplion
f187ad8bb4 Do not drop lookups without peers while awaiting events (#5839)
* Do not drop lookups without peers while awaiting events
2024-05-24 21:28:37 +00:00
Lion - dapplion
7fda18bf49 Report sync of block import via blob import (#5836)
* Report sync of block import via blob import
2024-05-24 19:45:04 +00:00
Lion - dapplion
f1f1250784 Do not double log uknown lookup (#5837)
* Do not double log uknown lookup

* Fix log text, not necessarily a child lookup
2024-05-24 19:45:01 +00:00
Lion - dapplion
a1271bc839 Debounce UnknownBlockHashFromAttestation events (#5706)
* Debounce UnknownBlockHashFromAttestation events

* Merge branch 'unstable' into debounce-sync-block-unknown

* Re-add dropped comment
2024-05-24 19:17:22 +00:00
Lion - dapplion
17d9086df3 Drop stuck lookups (#5824)
* Drop stuck lookups
2024-05-23 12:46:05 +00:00
Lion - dapplion
2a87016d94 Fix lookup disconnect peer (#5815)
* Test lookup peer disconnect modes

* Fix lookup peer disconnected return early
2024-05-20 18:27:57 +00:00
Lion - dapplion
8006418d80 Type sync network context send errors (#5808)
* Type sync network context send errors

* Consisntent naming
2024-05-17 11:34:21 +00:00
Lion - dapplion
319b4a2467 Skip creating child lookup if parent is never created (#5803)
* Skip creating child lookup if parent is never created
2024-05-17 10:58:27 +00:00
Eitan Seri-Levi
6636167503 Log block import source (#5738)
* the default target peers is 100

* add some comments

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into track-block-import-source

* add block import source

* revert

* update logging text

* fix tests

* lint

* use % instaed of to_string
2024-05-15 09:17:06 +00:00
Lion - dapplion
6f45ad4534 Log stuck lookups (#5778)
* Log stuck lookups every interval

* Implement debug manually

* Add comment

* Do not print peers twice

* Add SYNC_LOOKUPS_STUCK metric

* Skip logging request root

* use derivative

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into log-stuck-lookups

* add req id to debug

* Merge remote-tracking branch 'sigp/unstable' into log-stuck-lookups

* Fix conflict with unstable
2024-05-14 17:34:26 +00:00
Lion - dapplion
683d9df63b Don't request block components until having block (#5774)
* Don't request block components until having block

* Update tests

* Resolve todo comment

* Merge branch 'unstable' into request-blocks-first
2024-05-14 14:50:38 +00:00
Lion - dapplion
ce66ab374e Enforce sync lookup receives a single result (#5777)
* Enforce sync lookup receives a single result
2024-05-14 10:12:48 +00:00
Lion - dapplion
f37ffe4b8d Do not request current child lookup peers (#5724)
* Do not request current child lookup peers

* Update tests
2024-05-13 15:13:32 +00:00
Lion - dapplion
93e0649abc Notify lookup sync of gossip processing results (#5722)
* Notify lookup sync of gossip processing results

* Add tests

* Add GossipBlockProcessResult event

* Re-add dropped comments

* Update beacon_node/network/src/network_beacon_processor/sync_methods.rs

* update test_lookup_disconnection_peer_left
2024-05-13 11:41:29 +00:00
Lion - dapplion
5135a77e38 Move sync lookup trait function to its caller (#5704)
* Move sync lookup trait function to its caller

* lint
2024-05-06 17:55:12 +00:00
Lion - dapplion
b87c36ac0e Report RPC Errors to the application on peer disconnections (#5680)
* Report RPC Errors to the application on peer disconnections

Co-authored-by: Age Manning <Age@AgeManning.com>

* Expect RPCError::Disconnect to fail ongoing requests

* Drop lookups after peer disconnect and not awaiting events

* Allow RPCError disconnect through network service

* Update beacon_node/lighthouse_network/src/service/mod.rs

Co-authored-by: Age Manning <Age@AgeManning.com>

* Merge branch 'unstable' into rpc-error-on-disconnect
2024-05-06 17:18:47 +00:00
Lion - dapplion
436d54e4bf Consistent logging of full root in lookup debug logs (#5700)
* Consistent logging of full root in lookup debug logs

* Tag sync log with service

* More logs

* Log when new peers are added

* Don't shortcircuit add_peer
2024-05-06 15:41:38 +00:00
Michael Sproul
d3d429ff5c Improve logging and metrics for block publication (#5699)
* Improve logging and metrics for block publication

* Add better buckets

* Bump SQL connection timeout for tests.
2024-05-03 08:24:49 +00:00
Lion - dapplion
01e4e3527e Check da_checker before doing a block lookup request (#5681)
* Check da_checker before doing a block lookup request

* Ensure consistent handling of lookup result

* use req resp pre import cache rather than da checker
2024-05-01 14:45:36 +00:00
realbigsean
b9655b658e Ensure block only range requests don't fail on download (#5675)
* ensure pruned blobs don't fail on download

* Typo
2024-05-01 06:10:15 +00:00