Commit Graph

412 Commits

Author SHA1 Message Date
Lion - dapplion
ff15c78ced Implement data columns by network boilerplate (#6224)
* Implement data columns by network boilerplate

* Use correct quota values

* Address PR review

* Update currently_supported

* Merge remote-tracking branch 'sigp/unstable' into peerdas-network-boilerplate

* PR reviews

* Fix data column rpc request not being sent due to incorrect limits set. (#6000)
2024-08-13 00:16:14 +00:00
Age Manning
a3b1ef3129 Visualisation logging for sync batch states (#6034)
* Add visualization for batch states

* Replace icons with emojis

* Reviewers comments

* Change empty emoji and improve docs comments

* Fix lints

* Move to letters rather than emojis

* Replace 'V' with 'v'. Cargo update

* Merge latest unstable

* Improve docs around visualisation

* Merge branch 'unstable' into sync-batch-state
2024-07-29 05:52:07 +00:00
Mac L
cc55e610b9 Rust 1.80.0 lints (#6183)
* Fix lints
2024-07-25 15:56:22 +00:00
Jimmy Chen
4e5a363a4f Add DataColumnSidecar gossip topic and message handling (#6147)
* Add `DataColumnSidecar` gossip topic and verification (#5050 and #5783).

* Remove gossip verification changes (#5783).

* Merge branch 'unstable' into data-column-gossip

# Conflicts:
#	beacon_node/beacon_chain/src/data_column_verification.rs
#	beacon_node/beacon_chain/src/lib.rs

* Add gossip cache timeout for data columns. Rename data column metrics for consistency.

* Remove usage of `unimplemented!` and address review comments.

* Remove unnused `GossipDataColumnError` variants and address review comments.

* Merge branch 'unstable' into data-column-gossip

* Update Cargo.lock

* Arc `ChainSpec` in discovery to avoid performance regression when needing to clone it repeatedly.
2024-07-25 06:05:18 +00:00
Lion - dapplion
4bfca8251d Add range sync metrics to track efficiency (#6095)
* Add more range sync metrics to track efficiency

* Add ignored blocks metrics
2024-07-16 08:58:11 +10:00
Lion - dapplion
0c0b56d9e8 Bound max count of lookups (#6015)
* Bound max count of lookups

* Move up
2024-07-12 11:55:24 +00:00
Lion - dapplion
bde0428ac1 Add sync network context cache size metrics (#6049)
* Add sync network context cache size metrics
2024-07-09 16:31:26 +00:00
Lion - dapplion
2e2ccec9b5 Add randomization in sync retry batch peer selection (#5822)
* Add randomization in sync retry batch peer selection

* Use min

* Apply suggestions from code review

Co-authored-by: João Oliveira <hello@jxs.pt>

* Merge branch 'unstable' into peer-prio
2024-07-09 00:24:52 +00:00
João Oliveira
a59a61fef9 Remove generic Id param from RequestId (#6032)
* rename RequestId's for better context,

and move them to lighthouse_network crate.

* remove unrequired generic AppReqId from RequestID
2024-07-08 23:56:14 +00:00
Michael Sproul
a64cee3af1 Merge remote-tracking branch 'origin/stable' into unstable 2024-06-28 14:07:00 +10:00
Lion - dapplion
784ef5fb43 Pass vec to range sync batch (#5710)
* Pass vec to range sync batch
2024-06-27 18:21:40 +00:00
Lion - dapplion
f14f21f37b Bound lookup parent chain length with tip extension (#5705)
* Bound lookup parent chain length with tip extension

* Add test
2024-06-27 18:21:37 +00:00
Lion - dapplion
b38019cb10 Attempt to continue lookups after adding peers (#5993)
* Attempt to continue lookups after adding peers
2024-06-26 23:53:55 +00:00
Pawan Dhananjay
bf4cbd3b0a Remove all batches related to a peer on disconnect (#5969)
* Remove all batches related to a peer on disconnect

* Cleanup map entries after disconnect

* Allow lookups to continue in case of disconnections

* Pretty response types

* fmt

* Fix lints

* Remove lookup if it cannot progress

* Fix tests

* Remove poll_close on rpc behaviour

* Remove redundant test

* Fix issue raised by lion

* Revert pretty response types

* Cleanup

* Fix test

* Merge remote-tracking branch 'origin/release-v5.2.1' into rpc-error-on-disconnect-revert

* Apply suggestions from joao

Co-authored-by: João Oliveira <hello@jxs.pt>

* Fix log

* update request status on no peers found

* Do not remove lookup after peer disconnection

* Add comments about expected event api

* Update single_block_lookup.rs

* Update mod.rs

* Merge branch 'rpc-error-on-disconnect-revert' into 5969-review

* Merge pull request #10 from dapplion/5969-review

Add comments about expected event api
2024-06-26 23:53:53 +00:00
Lion - dapplion
474c1b4486 Verify inclusion proof should not be fallible (#5787)
* Verify inclusion proof should not be fallible

* Add blob sidecar inclusion test (#33)

* Add blob sidecar inclusion test.

* Fix lint
2024-06-17 15:05:24 +00:00
Lion - dapplion
17dc978760 Add peers to parent lookups (#5858)
* Add peers to parent lookups

* Add test

* lint

* Do not attempt to continue

* add_peers_to_lookup_and_ancestors can't drop
2024-05-30 17:07:32 +00:00
Lion - dapplion
6a7305a487 Add more info about why lookup is in AwaitingDownload (#5838)
* Add more info about why lookup is in AwaitingDownload

* Review comments

* Merge branch 'unstable' into awaiting-download-context

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into awaiting-download-context

* cargo fmt
2024-05-28 02:48:37 +00:00
Lion - dapplion
e4984217a6 Ensure lookup sync checks caches correctly (#5840)
* Ensure lookup sync checks caches correctly

* fix tests and remove unused method

* Simplify BlockProcessStatus
2024-05-25 14:56:51 +00:00
Lion - dapplion
f187ad8bb4 Do not drop lookups without peers while awaiting events (#5839)
* Do not drop lookups without peers while awaiting events
2024-05-24 21:28:37 +00:00
Lion - dapplion
f1f1250784 Do not double log uknown lookup (#5837)
* Do not double log uknown lookup

* Fix log text, not necessarily a child lookup
2024-05-24 19:45:01 +00:00
Lion - dapplion
a1271bc839 Debounce UnknownBlockHashFromAttestation events (#5706)
* Debounce UnknownBlockHashFromAttestation events

* Merge branch 'unstable' into debounce-sync-block-unknown

* Re-add dropped comment
2024-05-24 19:17:22 +00:00
Lion - dapplion
17d9086df3 Drop stuck lookups (#5824)
* Drop stuck lookups
2024-05-23 12:46:05 +00:00
Lion - dapplion
2a87016d94 Fix lookup disconnect peer (#5815)
* Test lookup peer disconnect modes

* Fix lookup peer disconnected return early
2024-05-20 18:27:57 +00:00
Lion - dapplion
8006418d80 Type sync network context send errors (#5808)
* Type sync network context send errors

* Consisntent naming
2024-05-17 11:34:21 +00:00
Lion - dapplion
319b4a2467 Skip creating child lookup if parent is never created (#5803)
* Skip creating child lookup if parent is never created
2024-05-17 10:58:27 +00:00
Lion - dapplion
6f45ad4534 Log stuck lookups (#5778)
* Log stuck lookups every interval

* Implement debug manually

* Add comment

* Do not print peers twice

* Add SYNC_LOOKUPS_STUCK metric

* Skip logging request root

* use derivative

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into log-stuck-lookups

* add req id to debug

* Merge remote-tracking branch 'sigp/unstable' into log-stuck-lookups

* Fix conflict with unstable
2024-05-14 17:34:26 +00:00
Lion - dapplion
683d9df63b Don't request block components until having block (#5774)
* Don't request block components until having block

* Update tests

* Resolve todo comment

* Merge branch 'unstable' into request-blocks-first
2024-05-14 14:50:38 +00:00
Lion - dapplion
ce66ab374e Enforce sync lookup receives a single result (#5777)
* Enforce sync lookup receives a single result
2024-05-14 10:12:48 +00:00
Lion - dapplion
f37ffe4b8d Do not request current child lookup peers (#5724)
* Do not request current child lookup peers

* Update tests
2024-05-13 15:13:32 +00:00
Lion - dapplion
93e0649abc Notify lookup sync of gossip processing results (#5722)
* Notify lookup sync of gossip processing results

* Add tests

* Add GossipBlockProcessResult event

* Re-add dropped comments

* Update beacon_node/network/src/network_beacon_processor/sync_methods.rs

* update test_lookup_disconnection_peer_left
2024-05-13 11:41:29 +00:00
Lion - dapplion
5135a77e38 Move sync lookup trait function to its caller (#5704)
* Move sync lookup trait function to its caller

* lint
2024-05-06 17:55:12 +00:00
Lion - dapplion
b87c36ac0e Report RPC Errors to the application on peer disconnections (#5680)
* Report RPC Errors to the application on peer disconnections

Co-authored-by: Age Manning <Age@AgeManning.com>

* Expect RPCError::Disconnect to fail ongoing requests

* Drop lookups after peer disconnect and not awaiting events

* Allow RPCError disconnect through network service

* Update beacon_node/lighthouse_network/src/service/mod.rs

Co-authored-by: Age Manning <Age@AgeManning.com>

* Merge branch 'unstable' into rpc-error-on-disconnect
2024-05-06 17:18:47 +00:00
Lion - dapplion
436d54e4bf Consistent logging of full root in lookup debug logs (#5700)
* Consistent logging of full root in lookup debug logs

* Tag sync log with service

* More logs

* Log when new peers are added

* Don't shortcircuit add_peer
2024-05-06 15:41:38 +00:00
Michael Sproul
d3d429ff5c Improve logging and metrics for block publication (#5699)
* Improve logging and metrics for block publication

* Add better buckets

* Bump SQL connection timeout for tests.
2024-05-03 08:24:49 +00:00
Lion - dapplion
01e4e3527e Check da_checker before doing a block lookup request (#5681)
* Check da_checker before doing a block lookup request

* Ensure consistent handling of lookup result

* use req resp pre import cache rather than da checker
2024-05-01 14:45:36 +00:00
realbigsean
b9655b658e Ensure block only range requests don't fail on download (#5675)
* ensure pruned blobs don't fail on download

* Typo
2024-05-01 06:10:15 +00:00
realbigsean
beaa586d98 delete spammy log (#5672)
*  delete spammy log
2024-04-30 21:31:58 +00:00
Lion - dapplion
ce66582c16 Merge parent and current sync lookups (#5655)
* Drop lookup type trait for a simple arg

* Drop reconstructed for processing

* Send parent blocks one by one

* Merge current and parent lookups

* Merge current and parent lookups clean up todos

* Merge current and parent lookups tests

* Merge remote-tracking branch 'origin/unstable' into sync-merged-lookup

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into sync-merged-lookup

* fix compile after merge

* #5655 pr review (#26)

* fix compile after merge

* remove todos, fix typos etc

* fix compile

* stable rng

* delete TODO and unfilled out test

* make download result a struct

* enums instead of bools as params

* fix comment

* Various fixes

* Track ignored child components

* Track dropped lookup reason as metric

* fix test

* add comment describing behavior of avail check error

*  update ordering
2024-04-30 20:12:15 +00:00
realbigsean
c33edc82eb Beta compiler fix (#5659)
* fix beta compiler compilation

* remove unused import

* Revert "remove unused import"

This reverts commit 0bef36b05b.

* Revert "fix beta compiler compilation"

This reverts commit 23152cf4cc.

* rename ununsed fields

* allow dead code on some error variants

* remove unused blob download queue

* add back debug to backfill error

* more allow dead code on errors
2024-04-29 20:45:54 +00:00
Lion - dapplion
c38b05d640 Drop lookup type trait for a simple arg (#5620)
* Drop lookup type trait for a simple arg
2024-04-24 06:02:45 +00:00
Lion - dapplion
f7aca97a55 Handle sync lookup request streams in network context (#5583)
* by-root-stream-terminator

* Fix tests

* Resolve merge conflicts

* Log report reason

* Some lints and bugfixes (#23)

* fix lints

* bug fixes

* Fix tests

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into handle-sync-lookup-requests

* Pr 5583 review (#24)

* add bad state warn log

* add rust docs to new fields in `SyncNetworkContext`

* remove timestamp todo

* add back lookup verify error

* remove TODOs
2024-04-22 16:06:39 +00:00
Lion - dapplion
e5b8d1237d Use Action in single_block_component_processed (#5564)
* Use Action in single_block_component_processed

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into single_block_component_processed-action

* fix compile after merge

* add continue action for when we are awaiting other parts of missing components
2024-04-16 12:59:53 +00:00
Lion - dapplion
fee2ee9c08 Make SingleLookupRequestState fields private (#5563)
* Make SingleLookupRequestState fields private

* Fix tests
2024-04-15 13:47:08 +00:00
Lion - dapplion
6fb0b2ed78 Sync lookup dedup range and blobs (#5561)
* Handle sync range blocks as blocks and blobs

* Merge range sync and backfill sync handling

* Update tests

* Add no_blobs_into_responses test

* Address @realbigsean comments

* Merge remote-tracking branch 'origin/unstable' into sync-lookup-dedup-range-and-blobs
2024-04-12 15:39:11 +00:00
Lion - dapplion
34dbb32610 Nest lookup type into request id SingleBlock and SingleBlob (#5562)
* Nest lookup type into block lookup RequestId
2024-04-11 16:32:06 +00:00
Lion - dapplion
54fbdda4d2 Add logging in UnknownBlockHashFromAttestation handling (#5546)
* Use should_search_for_block in unknown block root event

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into should_search_for_block
2024-04-10 20:52:22 +00:00
Lion - dapplion
30dc260472 Move processing cache out of DA (#5420)
* Move processing cache out of DA

* Merge branch 'sigp/unstable' into non-da-processing-cach

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into non-da-processing-cache

* remove unused file, remove outdated TODO, add is_deneb check to missing blob id calculations

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into non-da-processing-cache

* fix lints
2024-04-10 17:19:40 +00:00
Lion - dapplion
b1f9751a69 Event-based block lookup tests (#5534)
* WIP

* Initial working version of new sync tests.

* Remove sync traits and fix lints.

* Reduce internal method visibility and make test method instead. Remove extra beacon chain harness instance created in tests.

* Improve `SyncTester` api.

* Fix lint.

* Test example

* Lookup tests using rig

* Tests should interface with events only

* lint

* Skip deneb test pre-deneb

* Add more assertions

* Remove logging changes

* Address @jimmygchen comments

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into bn-p2p-tests

* remove unused assertions

* fix lint
2024-04-10 12:05:18 +00:00
Lion - dapplion
b74da14261 Improve block lookup logging (#5535)
* Improve block lookup logging
2024-04-09 14:25:18 +00:00
Lion - dapplion
053525e281 Remove DataAvailabilityView trait from ChildComponents (#5421)
* Remove DataAvailabilityView trait from ChildComponents

* PR reviews

* Update beacon_node/network/src/sync/block_lookups/common.rs

Co-authored-by: realbigsean <seananderson33@GMAIL.com>

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into child_components_independent
2024-04-04 18:49:09 +00:00