Fix clippy lints for `rustc` 1.87
clippy complains about `BeaconChainError` being too large. I went on a bit of a boxing spree because of this. We may instead want to `Box` some of the `BeaconChainError` variants?
This PR relates to:
- https://github.com/sigp/lighthouse/pull/7199
- -> workspace_filter has been enabled (dependency logging has been disabled)
- https://github.com/sigp/lighthouse/pull/7394
- -> file logging has been optionally enabled
Building on these, this PR enables dependency logging for the simulators. The logs are written to separate files.
The libp2p/discv5 logs:
- are saved to the directory specified with `--log-dir`
- respects the `RUST_LOG` environment variable for log level configuration
Beacon logs in the simulator are printed only to stdout. The logs are usually large, so persisting them would be helpful for debugging.
Added `--log-dir` parameter to the simulators and a step to upload the logs to Artifacts.
(Update)
Added `--disable-stdout-logging` to disable stdout logging, making the CI page cleaner.
This PR adds transitions to Electra ~~and Fulu~~ fork epochs in the simulator tests.
~~It also covers blob inclusion verification and data column syncing on a full node in Fulu.~~
UPDATE: Remove fulu fork from sim tests due to https://github.com/sigp/lighthouse/pull/7199#issuecomment-2852281176
- Create trait `ValidatorStore` with all functions used by the `validator_services`
- Make `validator_services` generic on `S: ValidatorStore`
- Introduce `LighthouseValidatorStore`, which has identical functionality to the old `ValidatorStore`
- Remove dependencies (especially `environment`) from `validator_services` and `beacon_node_fallback` in order to be able to cleanly use them in Anchor
#7153#7146#7147#7148 -> Thanks to @ackintosh
This PR does the following:
1. Disable logging to file when using either `--logfile-max-number 0` or `--logfile-max-size 0`. Note that disabling the log file in this way will also disable `discv5` and `libp2p` logging.
1. `discv5` and `libp2p` logging will be disabled by default unless running `beacon_node` or `boot_node`. This also should fix the VC panic we were seeing.
1. Removes log rotation and compression from `libp2p` and `discv5` logs. It is now limited to 1 file and will rotate based on the value of the `--logfile-max-size` flag. We could potentially add flags specifically to control the size/number of these, however I felt a single log file was sufficient. Perhaps @AgeManning has opinions about this?
1. Removes all dependency logging and references to `dep_log`.
1. Introduces workspace filtering to file and stdout. This explicitly allows logs from members of the Lighthouse workspace, disallowing all others. It uses a proc macro which pulls the member list from cargo metadata at compile time. This might be over-engineered but my hope is that this list will not require maintenance.
1. Unifies file and stdout JSON format. With slog, the formats were slightly different. @threehrsleep worked to maintain that format difference, to ensure there was no breaking changes. If these format differences are actually problematic we can restore it, however I felt the added complexity wasn't worth it.
1. General code improvements and cleanup.
Backport of:
- https://github.com/sigp/lighthouse/pull/7067
For:
- https://github.com/sigp/lighthouse/issues/7039
- Prevent writing to state cache when migrating the database
- Add `state-cache-headroom` flag to control pruning
- Prune old epoch boundary states ahead of mid-epoch states
- Never prune head block's state
- Avoid caching ancestor states unless they are on an epoch boundary
- Log when states enter/exit the cache
Co-authored-by: Eitan Seri-Levi <eserilev@ucsc.edu>
https://github.com/sigp/lighthouse/issues/7146
Removes `filter_layer` from the builder as this was acting as a "global minimum". We don't actually need this, since we are using more granular control in `stdout_logging_layer` and `file_logging_layer`. Removing this restores control of the logfiles level back to the `--logfile-debug-level` flag (and defaults to debug).
- #6452 (partially)
Remove dependencies on `store` and `lighthouse_network` from `eth2`. This was achieved as follows:
- depend on `enr` and `multiaddr` directly instead of using `lighthouse_network`'s reexports.
- make `lighthouse_network` responsible for converting between API and internal types.
- in two cases, remove complex internal types and use the generic `serde_json::Value` instead - this is not ideal, but should be fine for now, as this affects two internal non-spec endpoints which are meant for debugging, unstable, and subject to change without notice anyway. Inspired by #6679. The alternative is to move all relevant types to `eth2` or `types` instead - what do you think?
Partly addresses
- https://github.com/sigp/lighthouse/issues/6959
Use the `enable_light_client_server` field from the beacon chain config in the HTTP API. I think we can make this the single source of truth, as I think the network crate also has access to the beacon chain config.
N/A
2 changes:
1. Replace Option::map_or(true, ...) with is_none_or(...)
2. Remove unnecessary `Into::into` blocks where the type conversion is apparent from the types
Fix a regression introduced in this PR:
- https://github.com/sigp/lighthouse/pull/6361
We were indexing into the `MerkleTree` with raw generalized indices, which was incorrect and triggering `debug_assert` failures, as described here:
- https://github.com/sigp/lighthouse/issues/7005
- Convert `generalized_index` to the correct leaf index prior to proof generation.
- Add sanity checks on indices used in `BeaconState::generate_proof`.
- Remove debug asserts from `MerkleTree::generate_proof` in favour of actual errors. This would have caught the bug earlier.
- Refactor the EF tests so that the merkle validity tests are actually run. They were misconfigured in a way that resulted in them running silently with 0 test cases, and the `check_all_files_accessed.py` script still had an ignore that covered the test files, so this omission wasn't detected.
Update spec tests for recent v1.5.0-beta.2 release. There are no substantial changes for Electra and earlier, and the Fulu test updates to be made are tracked here:
- https://github.com/sigp/lighthouse/issues/6957
- Add `SingleAttestation` SSZ tests
- Add new `deposit_with_reorg` fork choice tests
- Update tag to v1.5.0-beta.2
- Ignore Fulu tests
A temporary workaround for the failing execution tests for geth.
https://github.com/sigp/lighthouse/actions/runs/13192297954
The test is broken due to the following breaking changes in geth that requires updating our tests:
1. removal of `personal` namespace in v1.14.12: See #30704
2. removal of `totalDifficulty` field from RPC in v1.14.11. See #30386.
Using an older version for now (` 1.14.10`) as we need to get things merged for the upcoming release.
Will create a separate issue to fix this.
Addresses #6706
This PR activates PeerDAS at the Fulu fork epoch instead of `EIP_7594_FORK_EPOCH`. This means we no longer support testing PeerDAS with Deneb / Electrs, as it's now part of a hard fork.
No substantial changes in v1.5.0-beta.1, this PR just updates the tests.
The optimisation described in this PR is already implemented in our single-pass epoch processing:
- https://github.com/ethereum/consensus-specs/pull/4081
* Refactor spec testing for features and simplify usage.
* Fix `SszStatic` tests for PeerDAS: exclude eip7594 test vectors when testing Electra types.
* Merge branch 'unstable' into refactor-ef-tests-features
* First pass
* Add restrictions to RuntimeVariableList api
* Use empty_uninitialized and fix warnings
* Fix some todos
* Merge branch 'unstable' into max-blobs-preset
* Fix take impl on RuntimeFixedList
* cleanup
* Fix test compilations
* Fix some more tests
* Fix test from unstable
* Merge branch 'unstable' into max-blobs-preset
* Merge remote-tracking branch 'origin/unstable' into max-blobs-preset
* Remove footgun function
* Minor simplifications
* Move from preset to config
* Fix typo
* Revert "Remove footgun function"
This reverts commit de01f923c7.
* Try fixing tests
* Thread through ChainSpec
* Fix release tests
* Move RuntimeFixedVector into module and rename
* Add test
* Remove empty RuntimeVarList awefullness
* Fix tests
* Simplify BlobSidecarListFromRoot
* Merge remote-tracking branch 'origin/unstable' into max-blobs-preset
* Bump quota to account for new target (6)
* Remove clone
* Fix issue from review
* Try to remove ugliness
* Merge branch 'unstable' into max-blobs-preset
* Fix max value
* Fix doctest
* Fix formatting
* Fix max check
* Delete hardcoded max_blobs_per_block in RPC limits
* Merge remote-tracking branch 'origin/unstable' into max-blobs-preset
* Fix partial withdrawals count
* Remove get_active_balance
* Remove queue_entire_balance_and_reset_validator
* Switch to compounding when consolidating with source==target
* Queue deposit requests and apply them during epoch processing
* Fix ef tests
* Clear todos
* Fix engine api formatting issues
* Merge branch 'unstable' into electra-alpha7
* Make add_validator_to_registry more in line with the spec
* Address some review comments
* Cleanup
* Update initialize_beacon_state_from_eth1
* Merge branch 'unstable' into electra-alpha7
* Fix rpc decoding for blobs by range/root
* Fix block hash computation
* Fix process_deposits bug
* Merge branch 'unstable' into electra-alpha7
* Fix topup deposit processing bug
* Update builder api for electra
* Refactor mock builder to separate functionality
* Merge branch 'unstable' into electra-alpha7
* Address review comments
* Use copied for reference rather than cloned
* Optimise and simplify PendingDepositsContext::new
* Merge remote-tracking branch 'origin/unstable' into electra-alpha7
* Fix processing of deposits with invalid signatures
* Remove redundant code in genesis init
* Revert "Refactor mock builder to separate functionality"
This reverts commit 6d10456912.
* Revert "Update builder api for electra"
This reverts commit c5c9aca6db.
* Simplify pre-activation sorting
* Fix stale validators used in upgrade_to_electra
* Merge branch 'unstable' into electra-alpha7
* Add lighthouse/key_legacy.p12 for macOS
* Specify `-days 825` to meet Apple's requirements for TLS server certificates
* Remove `-aes256` as it's ignored on exporting
The following warning will appear:
Warning: output encryption option -aes256 ignored with -export
* Update certificates and keys
* Get blobs from EL.
Co-authored-by: Michael Sproul <michael@sigmaprime.io>
* Avoid cloning blobs after fetching blobs.
* Address review comments and refactor code.
* Fix lint.
* Move blob computation metric to the right spot.
* Merge branch 'unstable' into das-fetch-blobs
* Merge branch 'unstable' into das-fetch-blobs
# Conflicts:
# beacon_node/beacon_chain/src/beacon_chain.rs
# beacon_node/beacon_chain/src/block_verification.rs
# beacon_node/beacon_chain/src/data_availability_checker/overflow_lru_cache.rs
* Merge branch 'unstable' into das-fetch-blobs
# Conflicts:
# beacon_node/beacon_chain/src/beacon_chain.rs
* Gradual publication of data columns for supernodes.
* Recompute head after importing block with blobs from the EL.
* Fix lint
* Merge branch 'unstable' into das-fetch-blobs
* Use blocking task instead of async when computing cells.
* Merge branch 'das-fetch-blobs' of github.com:jimmygchen/lighthouse into das-fetch-blobs
* Merge remote-tracking branch 'origin/unstable' into das-fetch-blobs
* Fix semantic conflicts
* Downgrade error log.
* Merge branch 'unstable' into das-fetch-blobs
# Conflicts:
# beacon_node/beacon_chain/src/data_availability_checker.rs
# beacon_node/beacon_chain/src/data_availability_checker/overflow_lru_cache.rs
# beacon_node/execution_layer/src/engine_api.rs
# beacon_node/execution_layer/src/engine_api/json_structures.rs
# beacon_node/network/src/network_beacon_processor/gossip_methods.rs
# beacon_node/network/src/network_beacon_processor/mod.rs
# beacon_node/network/src/network_beacon_processor/sync_methods.rs
* Merge branch 'unstable' into das-fetch-blobs
* Publish block without waiting for blob and column proof computation.
* Address review comments and refactor.
* Merge branch 'unstable' into das-fetch-blobs
* Fix test and docs.
* Comment cleanups.
* Merge branch 'unstable' into das-fetch-blobs
* Address review comments and cleanup
* Address review comments and cleanup
* Refactor to de-duplicate gradual publication logic.
* Add more logging.
* Merge remote-tracking branch 'origin/unstable' into das-fetch-blobs
# Conflicts:
# Cargo.lock
* Fix incorrect comparison on `num_fetched_blobs`.
* Implement gradual blob publication.
* Merge branch 'unstable' into das-fetch-blobs
* Inline `publish_fn`.
* Merge branch 'das-fetch-blobs' of github.com:jimmygchen/lighthouse into das-fetch-blobs
* Gossip verify blobs before publishing
* Avoid queries for 0 blobs and error for duplicates
* Gossip verified engine blob before processing them, and use observe cache to detect duplicates before publishing.
* Merge branch 'das-fetch-blobs' of github.com:jimmygchen/lighthouse into das-fetch-blobs
# Conflicts:
# beacon_node/network/src/network_beacon_processor/mod.rs
* Merge branch 'unstable' into das-fetch-blobs
* Fix invalid commitment inclusion proofs in blob sidecars created from EL blobs.
* Only publish EL blobs triggered from gossip block, and not RPC block.
* Downgrade gossip blob log to `debug`.
* Merge branch 'unstable' into das-fetch-blobs
* Merge branch 'unstable' into das-fetch-blobs
* Grammar
* Rework Validator Client fallback mechanism
* Add CI workflow for fallback simulator
* Tie-break with sync distance for non-synced nodes
* Fix simulator
* Cleanup unused code
* More improvements
* Add IsOptimistic enum for readability
* Use configurable sync distance tiers
* Fix tests
* Combine status and health and improve logging
* Fix nodes not being marked as available
* Fix simulator
* Fix tests again
* Increase fallback simulator tolerance
* Add http api endpoint
* Fix todos and tests
* Update simulator
* Merge branch 'unstable' into vc-fallback
* Add suggestions
* Add id to ui endpoint
* Remove unnecessary clones
* Formatting
* Merge branch 'unstable' into vc-fallback
* Merge branch 'unstable' into vc-fallback
* Fix flag tests
* Merge branch 'unstable' into vc-fallback
* Merge branch 'unstable' into vc-fallback
* Fix conflicts
* Merge branch 'unstable' into vc-fallback
* Remove unnecessary pubs
* Simplify `compute_distance_tier` and reduce notifier awaits
* Use the more descriptive `user_index` instead of `id`
* Combine sync distance tolerance flags into one
* Merge branch 'unstable' into vc-fallback
* Merge branch 'unstable' into vc-fallback
* wip
* Use new simulator from unstable
* Fix cli text
* Remove leftover files
* Remove old commented code
* Merge branch 'unstable' into vc-fallback
* Update cli text
* Silence candidate errors when pre-genesis
* Merge branch 'unstable' into vc-fallback
* Merge branch 'unstable' into vc-fallback
* Retry on failure
* Merge branch 'unstable' into vc-fallback
* Merge branch 'unstable' into vc-fallback
* Remove disable_run_on_all
* Remove unused error variant
* Fix out of date comment
* Merge branch 'unstable' into vc-fallback
* Remove unnecessary as_u64
* Remove more out of date comments
* Use tokio RwLock and remove parking_lot
* Merge branch 'unstable' into vc-fallback
* Formatting
* Ensure nodes are still added to total when not available
* Allow VC to detect when BN comes online
* Fix ui endpoint
* Don't have block_service as an Option
* Merge branch 'unstable' into vc-fallback
* Clean up lifetimes and futures
* Revert "Don't have block_service as an Option"
This reverts commit b5445a09e9.
* Merge branch 'unstable' into vc-fallback
* Merge branch 'unstable' into vc-fallback
* Improve rwlock sanitation using clones
* Merge branch 'unstable' into vc-fallback
* Drop read lock immediately by cloning the vec.
* add fork_name_enabled fn to Forkname impl
* refactor codebase to use new fork_enabled fn
* fmt
* Merge branch 'unstable' of https://github.com/sigp/lighthouse into fork-ord-impl
* small code cleanup
* resolve merge conflicts
* fix beacon chain test
* merge conflicts
* fix ef test issue
* resolve merge conflicts
* start splitting gossip verification
* WIP
* Gossip verify separate (#7)
* save
* save
* make ProvenancedBlock concrete
* delete into gossip verified block contents
* get rid of IntoBlobSidecar trait
* remove IntoGossipVerified trait
* get tests compiling
* don't check sidecar slashability in publish
* remove second publish closure
* drop blob bool. also prefer using message index over index of position in list
* Merge remote-tracking branch 'origin/unstable' into gossip-verify-separate
* Fix low-hanging tests
* Fix tests and clean up
* Clean up imports
* more cleanup
* Merge remote-tracking branch 'origin/unstable' into gossip-verify-separate
* Further refine behaviour and add tests
* Merge remote-tracking branch 'origin/unstable' into gossip-verify-separate
* Merge remote-tracking branch 'origin/unstable' into gossip-verify-separate
* Remove empty line
* Fix test (block is not fully imported just gossip verified)
* Merge remote-tracking branch 'origin/unstable' into gossip-verify-separate
* Update for unstable & use empty blob list
* Update comment
* Add test for duplicate block case
* Merge remote-tracking branch 'origin/unstable' into gossip-verify-separate
* Clarify unreachable case
* Fix another publish_block case
* Remove unreachable case in filter chain segment
* Revert unrelated blob optimisation
* Merge remote-tracking branch 'origin/unstable' into gossip-verify-separate
* Merge remote-tracking branch 'origin/unstable' into gossip-verify-separate
* Fix merge conflicts
* Merge remote-tracking branch 'origin/unstable' into gossip-verify-separate
* Fix some compilation issues. Impl is fucked though
* Support peerDAS
* Fix tests
* Merge remote-tracking branch 'origin/unstable' into gossip-verify-separate
* Fix conflict
* Merge remote-tracking branch 'origin/unstable' into gossip-verify-separate
* Address review comments
* Merge remote-tracking branch 'origin/unstable' into gossip-verify-separate
