Commit Graph

866 Commits

Author SHA1 Message Date
Lion - dapplion
17d9086df3 Drop stuck lookups (#5824)
* Drop stuck lookups
2024-05-23 12:46:05 +00:00
Lion - dapplion
2a87016d94 Fix lookup disconnect peer (#5815)
* Test lookup peer disconnect modes

* Fix lookup peer disconnected return early
2024-05-20 18:27:57 +00:00
Lion - dapplion
8006418d80 Type sync network context send errors (#5808)
* Type sync network context send errors

* Consisntent naming
2024-05-17 11:34:21 +00:00
Lion - dapplion
319b4a2467 Skip creating child lookup if parent is never created (#5803)
* Skip creating child lookup if parent is never created
2024-05-17 10:58:27 +00:00
Eitan Seri-Levi
6636167503 Log block import source (#5738)
* the default target peers is 100

* add some comments

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into track-block-import-source

* add block import source

* revert

* update logging text

* fix tests

* lint

* use % instaed of to_string
2024-05-15 09:17:06 +00:00
Lion - dapplion
6f45ad4534 Log stuck lookups (#5778)
* Log stuck lookups every interval

* Implement debug manually

* Add comment

* Do not print peers twice

* Add SYNC_LOOKUPS_STUCK metric

* Skip logging request root

* use derivative

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into log-stuck-lookups

* add req id to debug

* Merge remote-tracking branch 'sigp/unstable' into log-stuck-lookups

* Fix conflict with unstable
2024-05-14 17:34:26 +00:00
Lion - dapplion
683d9df63b Don't request block components until having block (#5774)
* Don't request block components until having block

* Update tests

* Resolve todo comment

* Merge branch 'unstable' into request-blocks-first
2024-05-14 14:50:38 +00:00
Lion - dapplion
ce66ab374e Enforce sync lookup receives a single result (#5777)
* Enforce sync lookup receives a single result
2024-05-14 10:12:48 +00:00
Lion - dapplion
f37ffe4b8d Do not request current child lookup peers (#5724)
* Do not request current child lookup peers

* Update tests
2024-05-13 15:13:32 +00:00
Lion - dapplion
93e0649abc Notify lookup sync of gossip processing results (#5722)
* Notify lookup sync of gossip processing results

* Add tests

* Add GossipBlockProcessResult event

* Re-add dropped comments

* Update beacon_node/network/src/network_beacon_processor/sync_methods.rs

* update test_lookup_disconnection_peer_left
2024-05-13 11:41:29 +00:00
Lion - dapplion
5135a77e38 Move sync lookup trait function to its caller (#5704)
* Move sync lookup trait function to its caller

* lint
2024-05-06 17:55:12 +00:00
Lion - dapplion
b87c36ac0e Report RPC Errors to the application on peer disconnections (#5680)
* Report RPC Errors to the application on peer disconnections

Co-authored-by: Age Manning <Age@AgeManning.com>

* Expect RPCError::Disconnect to fail ongoing requests

* Drop lookups after peer disconnect and not awaiting events

* Allow RPCError disconnect through network service

* Update beacon_node/lighthouse_network/src/service/mod.rs

Co-authored-by: Age Manning <Age@AgeManning.com>

* Merge branch 'unstable' into rpc-error-on-disconnect
2024-05-06 17:18:47 +00:00
Lion - dapplion
436d54e4bf Consistent logging of full root in lookup debug logs (#5700)
* Consistent logging of full root in lookup debug logs

* Tag sync log with service

* More logs

* Log when new peers are added

* Don't shortcircuit add_peer
2024-05-06 15:41:38 +00:00
Michael Sproul
d3d429ff5c Improve logging and metrics for block publication (#5699)
* Improve logging and metrics for block publication

* Add better buckets

* Bump SQL connection timeout for tests.
2024-05-03 08:24:49 +00:00
Lion - dapplion
01e4e3527e Check da_checker before doing a block lookup request (#5681)
* Check da_checker before doing a block lookup request

* Ensure consistent handling of lookup result

* use req resp pre import cache rather than da checker
2024-05-01 14:45:36 +00:00
realbigsean
b9655b658e Ensure block only range requests don't fail on download (#5675)
* ensure pruned blobs don't fail on download

* Typo
2024-05-01 06:10:15 +00:00
realbigsean
beaa586d98 delete spammy log (#5672)
*  delete spammy log
2024-04-30 21:31:58 +00:00
Lion - dapplion
ce66582c16 Merge parent and current sync lookups (#5655)
* Drop lookup type trait for a simple arg

* Drop reconstructed for processing

* Send parent blocks one by one

* Merge current and parent lookups

* Merge current and parent lookups clean up todos

* Merge current and parent lookups tests

* Merge remote-tracking branch 'origin/unstable' into sync-merged-lookup

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into sync-merged-lookup

* fix compile after merge

* #5655 pr review (#26)

* fix compile after merge

* remove todos, fix typos etc

* fix compile

* stable rng

* delete TODO and unfilled out test

* make download result a struct

* enums instead of bools as params

* fix comment

* Various fixes

* Track ignored child components

* Track dropped lookup reason as metric

* fix test

* add comment describing behavior of avail check error

*  update ordering
2024-04-30 20:12:15 +00:00
realbigsean
c33edc82eb Beta compiler fix (#5659)
* fix beta compiler compilation

* remove unused import

* Revert "remove unused import"

This reverts commit 0bef36b05b.

* Revert "fix beta compiler compilation"

This reverts commit 23152cf4cc.

* rename ununsed fields

* allow dead code on some error variants

* remove unused blob download queue

* add back debug to backfill error

* more allow dead code on errors
2024-04-29 20:45:54 +00:00
Lion - dapplion
40d412629e Ignore gossip blob already imported (#5656)
* Ignore gossip blob already imported
2024-04-29 14:30:13 +00:00
Mac L
13f94ef0f3 Rename Merge to Bellatrix (#5601)
* Rename Merge to Bellatrix

* Remove tree-hash-cache which got readded from the rebase
2024-04-25 20:19:41 +00:00
Lion - dapplion
c38b05d640 Drop lookup type trait for a simple arg (#5620)
* Drop lookup type trait for a simple arg
2024-04-24 06:02:45 +00:00
Michael Sproul
72a33604b3 Add timing for block availability (#5510)
* Add timing for block availability

* Attestation metrics analysis

* Prettier printing

* Add some metrics and timings to track late blocks

* Update to latest unstable

* fmt

* Merge latest unstable

* Small tweaks

* Try pushing blob timing down into verification

* Simplify for clippy
2024-04-23 13:13:34 +00:00
Lion - dapplion
f7aca97a55 Handle sync lookup request streams in network context (#5583)
* by-root-stream-terminator

* Fix tests

* Resolve merge conflicts

* Log report reason

* Some lints and bugfixes (#23)

* fix lints

* bug fixes

* Fix tests

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into handle-sync-lookup-requests

* Pr 5583 review (#24)

* add bad state warn log

* add rust docs to new fields in `SyncNetworkContext`

* remove timestamp todo

* add back lookup verify error

* remove TODOs
2024-04-22 16:06:39 +00:00
ethDreamer
f68989815c Restore Log on Error & Spawn Blocking in Streamer (#5585)
* Restore Logging in Error Cases

* Use Spawn Blocking for Loading Blocks in Streamer

* Merge remote-tracking branch 'upstream/unstable' into request_logging_spawn_blocking

* Address Sean's Comments

* save a clone
2024-04-16 14:56:00 +00:00
Lion - dapplion
e5b8d1237d Use Action in single_block_component_processed (#5564)
* Use Action in single_block_component_processed

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into single_block_component_processed-action

* fix compile after merge

* add continue action for when we are awaiting other parts of missing components
2024-04-16 12:59:53 +00:00
Lion - dapplion
f5e0404fb8 Ensure proper ReqResp blocks_by_* response stream termination (#5582)
* Ensure proper ReqResp blocks_by_* response stream termination

* retrigger CI
2024-04-15 21:20:50 +00:00
Lion - dapplion
fee2ee9c08 Make SingleLookupRequestState fields private (#5563)
* Make SingleLookupRequestState fields private

* Fix tests
2024-04-15 13:47:08 +00:00
Lion - dapplion
b6a1c863a2 Use spawn_async in ByRoot handling workers (#5557)
* Use spawn_async in ByRoot handling workers

* box large variants
2024-04-12 19:30:04 +00:00
Lion - dapplion
116a55e8a5 Ensure proper ReqResp response stream termination (#5556)
* Ensure proper ReqResp response stream termination

* Update beacon_node/network/src/network_beacon_processor/rpc_methods.rs

* Update beacon_node/network/src/network_beacon_processor/rpc_methods.rs

* cargo fmt
2024-04-12 16:15:04 +00:00
Lion - dapplion
6fb0b2ed78 Sync lookup dedup range and blobs (#5561)
* Handle sync range blocks as blocks and blobs

* Merge range sync and backfill sync handling

* Update tests

* Add no_blobs_into_responses test

* Address @realbigsean comments

* Merge remote-tracking branch 'origin/unstable' into sync-lookup-dedup-range-and-blobs
2024-04-12 15:39:11 +00:00
Lion - dapplion
34dbb32610 Nest lookup type into request id SingleBlock and SingleBlob (#5562)
* Nest lookup type into block lookup RequestId
2024-04-11 16:32:06 +00:00
Lion - dapplion
54fbdda4d2 Add logging in UnknownBlockHashFromAttestation handling (#5546)
* Use should_search_for_block in unknown block root event

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into should_search_for_block
2024-04-10 20:52:22 +00:00
Lion - dapplion
30dc260472 Move processing cache out of DA (#5420)
* Move processing cache out of DA

* Merge branch 'sigp/unstable' into non-da-processing-cach

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into non-da-processing-cache

* remove unused file, remove outdated TODO, add is_deneb check to missing blob id calculations

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into non-da-processing-cache

* fix lints
2024-04-10 17:19:40 +00:00
Lion - dapplion
b1f9751a69 Event-based block lookup tests (#5534)
* WIP

* Initial working version of new sync tests.

* Remove sync traits and fix lints.

* Reduce internal method visibility and make test method instead. Remove extra beacon chain harness instance created in tests.

* Improve `SyncTester` api.

* Fix lint.

* Test example

* Lookup tests using rig

* Tests should interface with events only

* lint

* Skip deneb test pre-deneb

* Add more assertions

* Remove logging changes

* Address @jimmygchen comments

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into bn-p2p-tests

* remove unused assertions

* fix lint
2024-04-10 12:05:18 +00:00
Lion - dapplion
b74da14261 Improve block lookup logging (#5535)
* Improve block lookup logging
2024-04-09 14:25:18 +00:00
GeemoCandama
32be063f0f Support LightClientFinalityUpdate and LightClientOptimisticUpdate rpcs (#3849)
* add light client optimistic and finality update rpc

* Arc the updates in the response

* add conditional advertisement for both LightClientOptimisticUpdate and LightClientFinalityUpdate

* alter display for inboundrequest light client optimistic and finality updates

* remove LightClientOptimistic/FinalityReuest struct and some minor fixes

* rebase

* failing rpc_test for LightClientBootstrap and beginning of MockLib2pLightClient

* minor change

* added MockRPCHandler by importing everything except OutboundRequest. Need to implement the ConnectionHandler trait now should be copy pastable

* almost there but ran into issue where needed to implement BaseOutboundRequest.

* failing but running with a light client service of sorts

* small test change

* changed Protocol::LightClientBootstrap response limit

* deleted some stuff from ConnectionHandler Implementation for the mock light client if you need to make something with multiple requests work maybe check here

* deleted purging expired inbound/outbound streams code

* deleted drive inbound streams that need to be processed

* removed unused imports

* made things private again

* deleted inject_fully_negotiated_inbound

* made more things private again

* more

* turned the logger off in the test

* added failing test for new rpc

* add rate limit for new rpcs

* change InboundUpgrade function to use new rpcs. fmt. add test for LightClientFinalityUpdate

* rebasing fix

* add LightClientUpdate to handle_rpc functions

* added context bytes

* fmt

* use correct unsed_tcp4_port function

* fix for recent config changes and adding context_bytes for the light client protocols

* fix clippy complaint

* Merge branch 'unstable' into lc-reqresp

# Conflicts:
#	beacon_node/beacon_processor/src/lib.rs
#	beacon_node/lighthouse_network/src/peer_manager/mod.rs
#	beacon_node/lighthouse_network/src/rpc/codec/ssz_snappy.rs
#	beacon_node/lighthouse_network/src/rpc/config.rs
#	beacon_node/lighthouse_network/src/rpc/methods.rs
#	beacon_node/lighthouse_network/src/rpc/mod.rs
#	beacon_node/lighthouse_network/src/rpc/outbound.rs
#	beacon_node/lighthouse_network/src/rpc/protocol.rs
#	beacon_node/lighthouse_network/src/rpc/rate_limiter.rs
#	beacon_node/lighthouse_network/src/rpc/self_limiter.rs
#	beacon_node/lighthouse_network/src/service/api_types.rs
#	beacon_node/lighthouse_network/tests/common/mod.rs
#	beacon_node/lighthouse_network/tests/rpc_tests.rs
#	beacon_node/network/src/network_beacon_processor/rpc_methods.rs
#	beacon_node/network/src/router.rs

* Error handling updates and various cleanups.

* Moar minor clean ups.

* Do not ban peer for rate limiting light client requests

* Merge branch 'unstable' into lc-reqresp. Also removed the mock light client tests to make it compile (See #4940).

# Conflicts:
#	beacon_node/lighthouse_network/src/rpc/codec/ssz_snappy.rs
#	beacon_node/lighthouse_network/src/rpc/methods.rs
#	beacon_node/lighthouse_network/src/rpc/mod.rs
#	beacon_node/lighthouse_network/src/rpc/protocol.rs
#	beacon_node/lighthouse_network/src/service/api_types.rs
#	beacon_node/lighthouse_network/tests/common/mod.rs
#	beacon_node/network/src/network_beacon_processor/rpc_methods.rs
#	beacon_node/network/src/router.rs
#	consensus/types/src/light_client_bootstrap.rs
#	consensus/types/src/light_client_finality_update.rs
#	consensus/types/src/light_client_optimistic_update.rs

* Remove unnecessary changes

* Add missing light client queue handling.

* Merge branch 'unstable' into lc-reqresp

* Merge branch 'unstable' into lc-reqresp

# Conflicts:
#	beacon_node/lighthouse_network/src/rpc/codec/ssz_snappy.rs
#	beacon_node/lighthouse_network/src/service/api_types.rs
#	consensus/types/src/light_client_finality_update.rs
#	consensus/types/src/light_client_optimistic_update.rs

* Add context bytes for light client RPC responses.

* Add RPC limits for light client object.

* Fix lint

* Fix incorrect light client max size computation.

* Merge branch 'unstable' into lc-reqresp

# Conflicts:
#	beacon_node/lighthouse_network/src/rpc/codec/ssz_snappy.rs
#	beacon_node/lighthouse_network/src/rpc/protocol.rs
#	beacon_node/lighthouse_network/src/service/api_types.rs

* Remove unwanted local changes.

* Merge branch 'unstable' into lc-reqresp

* Replace `unimplemented` electra code path with deneb values.
2024-04-09 07:23:39 +00:00
Lion - dapplion
053525e281 Remove DataAvailabilityView trait from ChildComponents (#5421)
* Remove DataAvailabilityView trait from ChildComponents

* PR reviews

* Update beacon_node/network/src/sync/block_lookups/common.rs

Co-authored-by: realbigsean <seananderson33@GMAIL.com>

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into child_components_independent
2024-04-04 18:49:09 +00:00
Mac L
969d12dc6f Use E for EthSpec globally (#5264)
* Use `E` for `EthSpec` globally

* Fix tests

* Merge branch 'unstable' into e-ethspec

* Merge branch 'unstable' into e-ethspec

# Conflicts:
#	beacon_node/execution_layer/src/engine_api.rs
#	beacon_node/execution_layer/src/engine_api/http.rs
#	beacon_node/execution_layer/src/engine_api/json_structures.rs
#	beacon_node/execution_layer/src/test_utils/handle_rpc.rs
#	beacon_node/store/src/partial_beacon_state.rs
#	consensus/types/src/beacon_block.rs
#	consensus/types/src/beacon_block_body.rs
#	consensus/types/src/beacon_state.rs
#	consensus/types/src/config_and_preset.rs
#	consensus/types/src/execution_payload.rs
#	consensus/types/src/execution_payload_header.rs
#	consensus/types/src/light_client_optimistic_update.rs
#	consensus/types/src/payload.rs
#	lcli/src/parse_ssz.rs
2024-04-02 15:12:25 +00:00
Mac L
f8fdb71f50 Add Electra fork boilerplate (#5122)
* Add Electra fork boilerplate

* Remove electra from spec tests

* Fix tests

* Remove sneaky log file

* Fix more tests

* Fix even more tests and add suggestions

* Remove unrelated lcli addition

* Update more tests

* Merge branch 'unstable' into electra

* Add comment for test-suite lcli override

* Merge branch 'unstable' into electra

* Cleanup

* Merge branch 'unstable' into electra

* Apply suggestions

* Merge branch 'unstable' into electra

* Merge sigp/unstable into electra

* Merge branch 'unstable' into electra
2024-04-02 12:35:02 +00:00
realbigsean
9d24844f50 Lookup log improvements (#5491)
* log improvements
2024-03-27 13:24:04 +00:00
realbigsean
334aa2eabd Single lookup improvements (#5488)
* Fix unexpected `UnrequestedBlobId` and `ExtraBlocksReturned` errors due to race conditions.

* Continue chain segment processing and skip any blocks that are already known, rather than returning an error.

* more de-dup checking

* ensure we don't reset `requested_ids` during rpc download

* better fix

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into more-dup-lookup-fixes

* remove chain hash check

* Merge branch 'fix-block-lookup-race' of https://github.com/jimmygchen/lighthouse into sean-test-lookups

* remove block check

* add back tests

* Log and CI fixes

* undue extra check

* Merge branch 'sean-test-lookups' of https://github.com/realbigsean/lighthouse into sean-test-lookups

* log improvements

* Improve logging
2024-03-27 10:01:27 +00:00
Michael Sproul
250a5bdc27 Run fork choice after RPC blob import (#5475)
* Run fork choice after RPC blob import
2024-03-26 21:36:08 +00:00
João Oliveira
59ef564b1d Move gossipsub into a separate crate (#5401)
* move gossipsub into a separate crate

* Merge branch 'unstable' of github.com:sigp/lighthouse into separate-gossipsub

* address review 2

* clippy beta

* update logging to log gossipsub logs
2024-03-26 03:10:59 +00:00
Eitan Seri-Levi
e4d4e439cb Add Capella & Deneb light client support (#4946)
* rebase and add comment

* conditional test

* test

* optimistic chould be working now

* finality should be working now

* try again

* try again

* clippy fix

* add lc bootstrap beacon api

* add lc optimistic/finality update to events

* fmt

* That error isn't occuring on my computer but I think this should fix it

* Merge branch 'unstable' into light_client_beacon_api_1

# Conflicts:
#	beacon_node/beacon_chain/src/events.rs
#	beacon_node/http_api/src/lib.rs
#	beacon_node/http_api/src/test_utils.rs
#	beacon_node/http_api/tests/main.rs
#	beacon_node/lighthouse_network/src/rpc/codec/ssz_snappy.rs
#	beacon_node/lighthouse_network/src/rpc/methods.rs
#	beacon_node/lighthouse_network/src/service/api_types.rs
#	beacon_node/network/src/beacon_processor/worker/rpc_methods.rs
#	beacon_node/tests/test.rs
#	common/eth2/src/types.rs
#	lighthouse/src/main.rs

* Add missing test file

* Update light client types to comply with Altair light client spec.

* Fix test compilation

* Merge branch 'unstable' into light_client_beacon_api_1

* Support deserializing light client structures for the Bellatrix fork

* Move `get_light_client_bootstrap` logic to `BeaconChain`. `LightClientBootstrap` API to return `ForkVersionedResponse`.

* Misc fixes.
- log cleanup
- move http_api config mutation to `config::get_config` for consistency
- fix light client API responses

* Add light client bootstrap API test and fix existing ones.

* Merge branch 'unstable' into light_client_beacon_api_1

* Fix test for `light-client-server` http api config.

* Appease clippy

* Add Altair light client SSZ tests

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into light_client_beacon_api_1

* updates to light client header

* light client header from signed beacon block

* using options

* implement helper functions

* placeholder conversion from vec hash256 to exec branch

* add deneb

* using fixed vector

* remove unwraps

* by epoch

* compute merkle proof

* merkle proof

* update comments

* resolve merge conflicts

* linting

* Merge branch 'unstable' into light-client-ssz-tests

# Conflicts:
#	beacon_node/beacon_chain/src/beacon_chain.rs
#	consensus/types/src/light_client_bootstrap.rs
#	consensus/types/src/light_client_header.rs

* superstruct attempt

* superstruct changes

* lint

* altair

* update

* update

* changes to light_client_optimistic_ and finality

* merge unstable

* refactor

* resolved merge conflicts

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into capella_deneb_light_client_types

* block_to_light_client_header fork aware

* fmt

* comment fix

* comment fix

* include merge fork, update deserialize_by_fork, refactor

* fmt

* pass by ref to prevent clone

* rename merkle proof fn

* add FIXME

* LightClientHeader TestRandom

* fix comments

* fork version deserialize

* merge unstable

* move fn arguments, fork name calc

* use task executor

* remove unneeded fns

* remove dead code

* add manual ssz decoding/encoding and add ssz_tests_by_fork macro

* merge deneb types with tests

* merge ssz tests, revert code deletion, cleanup

* move chainspec

* update ssz tests

* fmt

* light client ssz tests

* change to superstruct

* changes from feedback

* linting

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into capella_deneb_light_client_types

* test fix

* cleanup

* Remove unused `derive`.

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into capella_deneb_light_client_types

* beta compiler fix

* merge
2024-03-25 11:01:56 +00:00
realbigsean
21cdc64bfe Improve parent lookup logging (#5451)
* upgrade parent lookup result processing logs to debug, use display instead of debug for BlockError in case a blob parent unknown error is hit, add block root to BlockIsAlreadyKnown

* fix compile

* fix compile

* fix compile
2024-03-22 18:16:13 +00:00
Eitan Seri-Levi
01ec42e75a Fix Rust beta compiler errors 1.78.0-beta.1 (#5439)
* remove redundant imports

* fix test

* contains key

* fmt

* Merge branch 'unstable' into fix-beta-compiler
2024-03-20 05:17:02 +00:00
realbigsean
4449627c7c Dedup parent blob requests (#5432)
* de dup parent blob requests

* add new line
2024-03-20 00:19:26 +00:00
Pawan Dhananjay
84a902a589 Reduce load on validator subscription channels (#5311)
* Fix tests

* Merge branch 'unstable' into unclog-channels

* Avoid reallocations

* Reduce subscription load on beacon node
2024-03-07 12:32:27 +00:00
João Oliveira
65c4ff0775 remove exit-future (#5183)
* remove exit-future usage,

as it is non maintained, and replace with async-channel which is already in the repo.

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into remove-exit-future

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into remove-exit-future
2024-02-27 22:12:44 +00:00