* add light client optimistic and finality update rpc
* Arc the updates in the response
* add conditional advertisement for both LightClientOptimisticUpdate and LightClientFinalityUpdate
* alter display for inboundrequest light client optimistic and finality updates
* remove LightClientOptimistic/FinalityReuest struct and some minor fixes
* rebase
* failing rpc_test for LightClientBootstrap and beginning of MockLib2pLightClient
* minor change
* added MockRPCHandler by importing everything except OutboundRequest. Need to implement the ConnectionHandler trait now should be copy pastable
* almost there but ran into issue where needed to implement BaseOutboundRequest.
* failing but running with a light client service of sorts
* small test change
* changed Protocol::LightClientBootstrap response limit
* deleted some stuff from ConnectionHandler Implementation for the mock light client if you need to make something with multiple requests work maybe check here
* deleted purging expired inbound/outbound streams code
* deleted drive inbound streams that need to be processed
* removed unused imports
* made things private again
* deleted inject_fully_negotiated_inbound
* made more things private again
* more
* turned the logger off in the test
* added failing test for new rpc
* add rate limit for new rpcs
* change InboundUpgrade function to use new rpcs. fmt. add test for LightClientFinalityUpdate
* rebasing fix
* add LightClientUpdate to handle_rpc functions
* added context bytes
* fmt
* use correct unsed_tcp4_port function
* fix for recent config changes and adding context_bytes for the light client protocols
* fix clippy complaint
* Merge branch 'unstable' into lc-reqresp
# Conflicts:
# beacon_node/beacon_processor/src/lib.rs
# beacon_node/lighthouse_network/src/peer_manager/mod.rs
# beacon_node/lighthouse_network/src/rpc/codec/ssz_snappy.rs
# beacon_node/lighthouse_network/src/rpc/config.rs
# beacon_node/lighthouse_network/src/rpc/methods.rs
# beacon_node/lighthouse_network/src/rpc/mod.rs
# beacon_node/lighthouse_network/src/rpc/outbound.rs
# beacon_node/lighthouse_network/src/rpc/protocol.rs
# beacon_node/lighthouse_network/src/rpc/rate_limiter.rs
# beacon_node/lighthouse_network/src/rpc/self_limiter.rs
# beacon_node/lighthouse_network/src/service/api_types.rs
# beacon_node/lighthouse_network/tests/common/mod.rs
# beacon_node/lighthouse_network/tests/rpc_tests.rs
# beacon_node/network/src/network_beacon_processor/rpc_methods.rs
# beacon_node/network/src/router.rs
* Error handling updates and various cleanups.
* Moar minor clean ups.
* Do not ban peer for rate limiting light client requests
* Merge branch 'unstable' into lc-reqresp. Also removed the mock light client tests to make it compile (See #4940).
# Conflicts:
# beacon_node/lighthouse_network/src/rpc/codec/ssz_snappy.rs
# beacon_node/lighthouse_network/src/rpc/methods.rs
# beacon_node/lighthouse_network/src/rpc/mod.rs
# beacon_node/lighthouse_network/src/rpc/protocol.rs
# beacon_node/lighthouse_network/src/service/api_types.rs
# beacon_node/lighthouse_network/tests/common/mod.rs
# beacon_node/network/src/network_beacon_processor/rpc_methods.rs
# beacon_node/network/src/router.rs
# consensus/types/src/light_client_bootstrap.rs
# consensus/types/src/light_client_finality_update.rs
# consensus/types/src/light_client_optimistic_update.rs
* Remove unnecessary changes
* Add missing light client queue handling.
* Merge branch 'unstable' into lc-reqresp
* Merge branch 'unstable' into lc-reqresp
# Conflicts:
# beacon_node/lighthouse_network/src/rpc/codec/ssz_snappy.rs
# beacon_node/lighthouse_network/src/service/api_types.rs
# consensus/types/src/light_client_finality_update.rs
# consensus/types/src/light_client_optimistic_update.rs
* Add context bytes for light client RPC responses.
* Add RPC limits for light client object.
* Fix lint
* Fix incorrect light client max size computation.
* Merge branch 'unstable' into lc-reqresp
# Conflicts:
# beacon_node/lighthouse_network/src/rpc/codec/ssz_snappy.rs
# beacon_node/lighthouse_network/src/rpc/protocol.rs
# beacon_node/lighthouse_network/src/service/api_types.rs
* Remove unwanted local changes.
* Merge branch 'unstable' into lc-reqresp
* Replace `unimplemented` electra code path with deneb values.
* move gossipsub into a separate crate
* Merge branch 'unstable' of github.com:sigp/lighthouse into separate-gossipsub
* address review 2
* clippy beta
* update logging to log gossipsub logs
* Downgrade duplicate publish logs
* Maintain backwards compatiblity, deprecate flag
* The tests had to go, because there's no config to test against
* Update help_bn.md
* switch libp2p source to sigp fork
* Shift the connection closing inside RPC behaviour
* Tag specific commits
* Add slow peer scoring
* Fix test
* Use default yamux config
* Pin discv5 to our libp2p fork and cargo update
* Upgrade libp2p to enable yamux gains
* Add a comment specifying the branch being used
* cleanup build output from within container
(prevents CI warnings related to fs permissions)
* Remove revision tags add branches for testing, will revert back once we're happy
* Update to latest rust-libp2p version
* Pin forks
* Update cargo.lock
* Re-pin to panic-free rust
---------
Co-authored-by: Age Manning <Age@AgeManning.com>
Co-authored-by: Pawan Dhananjay <pawandhananjay@gmail.com>
Co-authored-by: antondlr <anton@delaruelle.net>
Co-authored-by: Michael Sproul <michael@sigmaprime.io>
* add runtime variable list type
* add configs to ChainSpec
* git rid of max request blocks type
* fix tests and lints
* remove todos
* git rid of old const usage
* fix decode impl
* add new config to `Config` api struct
* add docs fix compilt
* move methods for per-fork-spec to chainspec
* get values off chain spec
* fix compile
* remove min by root size
* add tests for runtime var list
---------
Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>
* update libp2p and address compiler errors
* remove bandwidth logging from transport
* use libp2p registry
* make clippy happy
* use rust 1.73
* correct rpc keep alive
* remove comments and obsolte code
* remove libp2p prefix
* make clippy happy
* use quic under facade
* remove fast msg id
* bubble up close statements
* fix wrong comment
## Issue Addressed
updates libp2p to the latest version and uses the new `SwarmBuilder`. Superseeds https://github.com/sigp/lighthouse/pull/4695/
CC @mxinden I don't think we can use both `bandwidth_loggers` with the new syntax right?
## Issue Addressed
Following the conversation on https://github.com/libp2p/rust-libp2p/pull/3666 the changes introduced in this PR will allow us to give more insights if the bandwidth limitations happen at the transport level, namely if quic helps vs yamux and it's [window size limitation](https://github.com/libp2p/rust-yamux/issues/162) or if the bottleneck is at the gossipsub level.
## Proposed Changes
introduce new quic and tcp bandwidth metric gauges.
cc @mxinden (turned out to be easier, Thomas gave me a hint)
## Issue Addressed
This PR closes https://github.com/sigp/lighthouse/issues/3237
## Proposed Changes
Remove topic weight of old topics when the fork happens.
## Additional Info
- Divided `NetworkService::start()` into `NetworkService::build()` and `NetworkService::start()` for ease of testing.
## Issue Addressed
https://github.com/sigp/lighthouse/issues/4543
## Proposed Changes
- Removes `NotBanned` from `BanResult`, implements `Display` and `std::error::Error` for `BanResult` and changes `ban_result` return type to `Option<BanResult>` which helps returning `BanResult` on `handle_established_inbound_connection`
- moves the check from for banned peers from `on_connection_established` to `handle_established_inbound_connection` to start addressing #4543.
- Removes `allow_block_list` as it's now redundant? Not sure about this one but if `PeerManager` keeps track of the banned peers, no need to send a `Swarm` event for `alow_block_list` to also keep that list right?
## Questions
- #4543 refers:
> More specifically, implement the connection limit behaviour inside the peer manager.
@AgeManning do you mean copying `libp2p::connection_limits::Behaviour`'s code into `PeerManager`/ having it as an inner `NetworkBehaviour` of `PeerManager`/other? If it's the first two, I think it probably makes more sense to have it as it is as it's less code to maintain.
> Also implement the banning of peers inside the behaviour, rather than passing messages back up to the swarm.
I tried to achieve this, but we still need to pass the `PeerManagerEvent::Banned` swarm event as `DiscV5` handles it's node and ip management internally and I did not find a method to query if a peer is banned. Is there anything else we can do from here?
3397612160/beacon_node/lighthouse_network/src/discovery/mod.rs (L931-L940)
Same as the question above, I did not find a way to check if `DiscV5` has the peer banned, so that we could check here and avoid sending `Swarm` events
3397612160/beacon_node/lighthouse_network/src/peer_manager/network_behaviour.rs (L168-L178)
Is there a chance we try to dial a peer that has been banned previously?
Thanks!
## Issue Addressed
#4402
## Proposed Changes
This PR adds QUIC support to Lighthouse. As this is not officially spec'd this will only work between lighthouse <-> lighthouse connections. We attempt a QUIC connection (if the node advertises it) and if it fails we fallback to TCP.
This should be a backwards compatible modification. We want to test this functionality on live networks to observe any improvements in bandwidth/latency.
NOTE: This also removes the websockets transport as I believe no one is really using it. It should be mentioned in our release however.
Co-authored-by: João Oliveira <hello@jxs.pt>
* increase the max topic subscriptions #4581
* make the max_subscription limitation based off constants / configuration
* format
* wording & add deneb topic array
* reduce max_subscriptions_per_request to 2x
* format
* update comment
* remove protoc and token from network tests github action
* delete unused beacon chain methods
* downgrade writing blobs to store log
* reduce diff in block import logic
* remove some todo's and deneb built in network
* remove unnecessary error, actually use some added metrics
* remove some metrics, fix missing components on publish funcitonality
* fix status tests
* rename sidecar by root to blobs by root
* clean up some metrics
* remove unnecessary feature gate from attestation subnet tests, clean up blobs by range response code
* pawan's suggestion in `protocol_info`, peer score in matching up batch sync block and blobs
* fix range tests for deneb
* pub block and blob db cache behind the same mutex
* remove unused errs and an empty file
* move sidecar trait to new file
* move types from payload to eth2 crate
* update comment and add flag value name
* make function private again, remove allow unused
* use reth rlp for tx decoding
* fix compile after merge
* rename kzg commitments
* cargo fmt
* remove unused dep
* Update beacon_node/execution_layer/src/lib.rs
Co-authored-by: Pawan Dhananjay <pawandhananjay@gmail.com>
* Update beacon_node/beacon_processor/src/lib.rs
Co-authored-by: Pawan Dhananjay <pawandhananjay@gmail.com>
* pawan's suggestiong for vec capacity
* cargo fmt
* Revert "use reth rlp for tx decoding"
This reverts commit 5181837d81.
* remove reth rlp
---------
Co-authored-by: Pawan Dhananjay <pawandhananjay@gmail.com>
## Issue Addressed
Addresses [#4401](https://github.com/sigp/lighthouse/issues/4401)
## Proposed Changes
Shift some constants into ```ChainSpec``` and remove the constant values from code space.
## Additional Info
I mostly used ```MainnetEthSpec::default_spec()``` for getting ```ChainSpec```. I wonder Did I make a mistake about that.
Co-authored-by: armaganyildirak <armaganyildirak@gmail.com>
Co-authored-by: Paul Hauner <paul@paulhauner.com>
Co-authored-by: Age Manning <Age@AgeManning.com>
Co-authored-by: Diva M <divma@protonmail.com>
## Issue Addressed
Upgrade libp2p to v0.52
## Proposed Changes
- **Workflows**: remove installation of `protoc`
- **Book**: remove installation of `protoc`
- **`Dockerfile`s and `cross`**: remove custom base `Dockerfile` for cross since it's no longer needed. Remove `protoc` from remaining `Dockerfiles`s
- **Upgrade `discv5` to `v0.3.1`:** we have some cool stuff in there: no longer needs `protoc` and faster ip updates on cold start
- **Upgrade `prometheus` to `0.21.0`**, now it no longer needs encoding checks
- **things that look like refactors:** bunch of api types were renamed and need to be accessed in a different (clearer) way
- **Lighthouse network**
- connection limits is now a behaviour
- banned peers no longer exist on the swarm level, but at the behaviour level
- `connection_event_buffer_size` now is handled per connection with a buffer size of 4
- `mplex` is deprecated and was removed
- rpc handler now logs the peer to which it belongs
## Additional Info
Tried to keep as much behaviour unchanged as possible. However, there is a great deal of improvements we can do _after_ this upgrade:
- Smart connection limits: Connection limits have been checked only based on numbers, we can now use information about the incoming peer to decide if we want it
- More powerful peer management: Dial attempts from other behaviours can be rejected early
- Incoming connections can be rejected early
- Banning can be returned exclusively to the peer management: We should not get connections to banned peers anymore making use of this
- TCP Nat updates: We might be able to take advantage of confirmed external addresses to check out tcp ports/ips
Co-authored-by: Age Manning <Age@AgeManning.com>
Co-authored-by: Akihito Nakano <sora.akatsuki@gmail.com>
## Issue Addressed
On deneb devnetv5, lighthouse keeps rate limiting peers which makes it harder to bootstrap new nodes as there are very few peers in the network. This PR adds an option to disable the inbound rate limiter for testnets.
Added an option to configure inbound rate limits as well.
Co-authored-by: Diva M <divma@protonmail.com>
## Issue Addressed
N/A
## Proposed Changes
Adds a flag for disabling peer scoring. This is useful for local testing and testing small networks for new features.
It is possible that when we go to ban a peer, there is already an unbanned message in the queue. It could lead to the case that we ban and immediately unban a peer leaving us in a state where a should-be banned peer is unbanned.
If this banned peer connects to us in this faulty state, we currently do not attempt to re-ban it. This PR does correct this also, so if we do see this error, it will now self-correct (although we shouldn't see the error in the first place).
I have also incremented the severity of not supporting protocols as I see peers ultimately get banned in a few steps and it seems to make sense to just ban them outright, rather than have them linger.
## Issue Addressed
Add support for ipv6 and dual stack in lighthouse.
## Proposed Changes
From an user perspective, now setting an ipv6 address, optionally configuring the ports should feel exactly the same as using an ipv4 address. If listening over both ipv4 and ipv6 then the user needs to:
- use the `--listen-address` two times (ipv4 and ipv6 addresses)
- `--port6` becomes then required
- `--discovery-port6` can now be used to additionally configure the ipv6 udp port
### Rough list of code changes
- Discovery:
- Table filter and ip mode set to match the listening config.
- Ipv6 address, tcp port and udp port set in the ENR builder
- Reported addresses now check which tcp port to give to libp2p
- LH Network Service:
- Can listen over Ipv6, Ipv4, or both. This uses two sockets. Using mapped addresses is disabled from libp2p and it's the most compatible option.
- NetworkGlobals:
- No longer stores udp port since was not used at all. Instead, stores the Ipv4 and Ipv6 TCP ports.
- NetworkConfig:
- Update names to make it clear that previous udp and tcp ports in ENR were Ipv4
- Add fields to configure Ipv6 udp and tcp ports in the ENR
- Include advertised enr Ipv6 address.
- Add type to model Listening address that's either Ipv4, Ipv6 or both. A listening address includes the ip, udp port and tcp port.
- UPnP:
- Kept only for ipv4
- Cli flags:
- `--listen-addresses` now can take up to two values
- `--port` will apply to ipv4 or ipv6 if only one listening address is given. If two listening addresses are given it will apply only to Ipv4.
- `--port6` New flag required when listening over ipv4 and ipv6 that applies exclusively to Ipv6.
- `--discovery-port` will now apply to ipv4 and ipv6 if only one listening address is given.
- `--discovery-port6` New flag to configure the individual udp port of ipv6 if listening over both ipv4 and ipv6.
- `--enr-udp-port` Updated docs to specify that it only applies to ipv4. This is an old behaviour.
- `--enr-udp6-port` Added to configure the enr udp6 field.
- `--enr-tcp-port` Updated docs to specify that it only applies to ipv4. This is an old behaviour.
- `--enr-tcp6-port` Added to configure the enr tcp6 field.
- `--enr-addresses` now can take two values.
- `--enr-match` updated behaviour.
- Common:
- rename `unused_port` functions to specify that they are over ipv4.
- add functions to get unused ports over ipv6.
- Testing binaries
- Updated code to reflect network config changes and unused_port changes.
## Additional Info
TODOs:
- use two sockets in discovery. I'll get back to this and it's on https://github.com/sigp/discv5/pull/160
- lcli allow listening over two sockets in generate_bootnodes_enr
- add at least one smoke flag for ipv6 (I have tested this and works for me)
- update the book
