[Missing values in /eth/v1/config/spec #8571
](https://github.com/sigp/lighthouse/issues/8571) - there will be follow up PR for the re org props
1. As per above issue from EF dev ops, I added
```
"EPOCHS_PER_SUBNET_SUBSCRIPTION": "256",
"ATTESTATION_SUBNET_COUNT": "64",
"ATTESTATION_SUBNET_EXTRA_BITS": "0",
"UPDATE_TIMEOUT": "8192",
"DOMAIN_BLS_TO_EXECUTION_CHANGE": "0x0a000000"
```
to `/eth/v1/config/spec`
2. Had to change the minimal config for UPDATE_TIMEOUT to get currents tests to pass. This is ok given UPDATE_TIMEOUT is not used in lighthouse as this config for light client spec from altair
3. ATTESTATION_SUBNET_PREFIX_BITS is now dynamically calculated and shimmed into the /eth/v1/config/spec output as advised by @michaelsproul
Co-Authored-By: Joseph Patchen <josephmipatchen@gmail.com>
Which issue # does this PR address?
#8586
Please list or describe the changes introduced by this PR.
Remove `service_name` from `TaskExecutor`
Co-Authored-By: Abhivansh <31abhivanshj@gmail.com>
This PR relates to:
- https://github.com/sigp/lighthouse/pull/7199
- -> workspace_filter has been enabled (dependency logging has been disabled)
- https://github.com/sigp/lighthouse/pull/7394
- -> file logging has been optionally enabled
Building on these, this PR enables dependency logging for the simulators. The logs are written to separate files.
The libp2p/discv5 logs:
- are saved to the directory specified with `--log-dir`
- respects the `RUST_LOG` environment variable for log level configuration
Resolves#6811
Rename `GOSSIP_MAX_SIZE` to `MAX_PAYLOAD_SIZE` and remove `MAX_CHUNK_SIZE` in accordance with the spec.
The spec also "clarifies" the message size limits at different levels. The rpc limits are equivalent to what we had before imo.
The gossip limits have additional checks.
I have gotten rid of the `is_bellatrix_enabled` checks that used a lower limit (1mb) pre-merge. Since all networks we run start from the merge, I don't think this will break any setups.
#7153#7146#7147#7148 -> Thanks to @ackintosh
This PR does the following:
1. Disable logging to file when using either `--logfile-max-number 0` or `--logfile-max-size 0`. Note that disabling the log file in this way will also disable `discv5` and `libp2p` logging.
1. `discv5` and `libp2p` logging will be disabled by default unless running `beacon_node` or `boot_node`. This also should fix the VC panic we were seeing.
1. Removes log rotation and compression from `libp2p` and `discv5` logs. It is now limited to 1 file and will rotate based on the value of the `--logfile-max-size` flag. We could potentially add flags specifically to control the size/number of these, however I felt a single log file was sufficient. Perhaps @AgeManning has opinions about this?
1. Removes all dependency logging and references to `dep_log`.
1. Introduces workspace filtering to file and stdout. This explicitly allows logs from members of the Lighthouse workspace, disallowing all others. It uses a proc macro which pulls the member list from cargo metadata at compile time. This might be over-engineered but my hope is that this list will not require maintenance.
1. Unifies file and stdout JSON format. With slog, the formats were slightly different. @threehrsleep worked to maintain that format difference, to ensure there was no breaking changes. If these format differences are actually problematic we can restore it, however I felt the added complexity wasn't worth it.
1. General code improvements and cleanup.
https://github.com/sigp/lighthouse/issues/7146
Removes `filter_layer` from the builder as this was acting as a "global minimum". We don't actually need this, since we are using more granular control in `stdout_logging_layer` and `file_logging_layer`. Removing this restores control of the logfiles level back to the `--logfile-debug-level` flag (and defaults to debug).
* 1D PeerDAS prototype: Data format and Distribution (#5050)
* Build and publish column sidecars. Add stubs for gossip.
* Add blob column subnets
* Add `BlobColumnSubnetId` and initial compute subnet logic.
* Subscribe to blob column subnets.
* Introduce `BLOB_COLUMN_SUBNET_COUNT` based on DAS configuration parameter changes.
* Fix column sidecar type to use `VariableList` for data.
* Fix lint errors.
* Update types and naming to latest consensus-spec #3574.
* Fix test and some cleanups.
* Merge branch 'unstable' into das
* Merge branch 'unstable' into das
* Merge branch 'unstable' into das
# Conflicts:
# consensus/types/src/chain_spec.rs
* Add `DataColumnSidecarsByRoot ` req/resp protocol (#5196)
* Add stub for `DataColumnsByRoot`
* Add basic implementation of serving RPC data column from DA checker.
* Store data columns in early attester cache and blobs db.
* Apply suggestions from code review
Co-authored-by: Eitan Seri-Levi <eserilev@gmail.com>
Co-authored-by: Jacob Kaufmann <jacobkaufmann18@gmail.com>
* Fix build.
* Store `DataColumnInfo` in database and various cleanups.
* Update `DataColumnSidecar` ssz max size and remove panic code.
---------
Co-authored-by: Eitan Seri-Levi <eserilev@gmail.com>
Co-authored-by: Jacob Kaufmann <jacobkaufmann18@gmail.com>
* feat: add DAS KZG in data col construction (#5210)
* feat: add DAS KZG in data col construction
* refactor data col sidecar construction
* refactor: add data cols to GossipVerifiedBlockContents
* Disable windows tests for `das` branch. (c-kzg doesn't build on windows)
* Formatting and lint changes only.
* refactor: remove iters in construction of data cols
* Update vec capacity and error handling.
* Add `data_column_sidecar_computation_seconds` metric.
---------
Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>
* Merge branch 'unstable' into das
# Conflicts:
# .github/workflows/test-suite.yml
# beacon_node/lighthouse_network/src/types/topics.rs
* fix: update data col subnet count from 64 to 32 (#5413)
* feat: add peerdas custody field to ENR (#5409)
* feat: add peerdas custody field to ENR
* add hash prefix step in subnet computation
* refactor test and fix possible u64 overflow
* default to min custody value if not present in ENR
* Merge branch 'unstable' into das
* Merge branch 'unstable' into das-unstable-merge-0415
# Conflicts:
# Cargo.lock
# beacon_node/beacon_chain/src/data_availability_checker.rs
# beacon_node/beacon_chain/src/data_availability_checker/availability_view.rs
# beacon_node/beacon_chain/src/data_availability_checker/overflow_lru_cache.rs
# beacon_node/beacon_chain/src/data_availability_checker/processing_cache.rs
# beacon_node/lighthouse_network/src/rpc/methods.rs
# beacon_node/network/src/network_beacon_processor/mod.rs
# beacon_node/network/src/sync/block_lookups/tests.rs
# crypto/kzg/Cargo.toml
* Merge remote-tracking branch 'sigp/unstable' into das
* Merge remote-tracking branch 'sigp/unstable' into das
* Fix merge conflicts.
* Send custody data column to `DataAvailabilityChecker` for determining block importability (#5570)
* Only import custody data columns after publishing a block.
* Add `subscribe-all-data-column-subnets` and pass custody column count to `availability_cache`.
* Add custody requirement checks to `availability_cache`.
* Fix config not being passed to DAChecker and add more logging.
* Introduce `peer_das_epoch` and make blobs and columns mutually exclusive.
* Add DA filter for PeerDAS.
* Fix data availability check and use test_logger in tests.
* Fix subscribe to all data column subnets not working correctly.
* Fix tests.
* Only publish column sidecars if PeerDAS is activated. Add `PEER_DAS_EPOCH` chain spec serialization.
* Remove unused data column index in `OverflowKey`.
* Fix column sidecars incorrectly produced when there are no blobs.
* Re-instate index to `OverflowKey::DataColumn` and downgrade noisy debug log to `trace`.
* DAS sampling on sync (#5616)
* Data availability sampling on sync
* Address @jimmygchen review
* Trigger sampling
* Address some review comments and only send `SamplingBlock` sync message after PEER_DAS_EPOCH.
---------
Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>
* Merge branch 'unstable' into das
# Conflicts:
# Cargo.lock
# Cargo.toml
# beacon_node/beacon_chain/src/block_verification.rs
# beacon_node/http_api/src/publish_blocks.rs
# beacon_node/lighthouse_network/src/rpc/codec/ssz_snappy.rs
# beacon_node/lighthouse_network/src/rpc/protocol.rs
# beacon_node/lighthouse_network/src/types/pubsub.rs
# beacon_node/network/src/sync/block_lookups/single_block_lookup.rs
# beacon_node/store/src/hot_cold_store.rs
# consensus/types/src/beacon_state.rs
# consensus/types/src/chain_spec.rs
# consensus/types/src/eth_spec.rs
* Merge branch 'unstable' into das
* Re-process early sampling requests (#5569)
* Re-process early sampling requests
# Conflicts:
# beacon_node/beacon_processor/src/work_reprocessing_queue.rs
# beacon_node/lighthouse_network/src/rpc/methods.rs
# beacon_node/network/src/network_beacon_processor/rpc_methods.rs
* Update beacon_node/beacon_processor/src/work_reprocessing_queue.rs
Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>
* Add missing var
* Beta compiler fixes and small typo fixes.
* Remove duplicate method.
---------
Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>
* Merge remote-tracking branch 'sigp/unstable' into das
* Fix merge conflict.
* Add data columns by root to currently supported protocol list (#5678)
* Add data columns by root to currently supported protocol list.
* Add missing data column by roots handling.
* Merge branch 'unstable' into das
# Conflicts:
# Cargo.lock
# Cargo.toml
# beacon_node/network/src/sync/block_lookups/tests.rs
# beacon_node/network/src/sync/manager.rs
* Fix simulator tests on `das` branch (#5731)
* Bump genesis delay in sim tests as KZG setup takes longer for DAS.
* Fix incorrect YAML spacing.
* DataColumnByRange boilerplate (#5353)
* add boilerplate
* fmt
* PeerDAS custody lookup sync (#5684)
* Implement custody sync
* Lint
* Fix tests
* Fix rebase issue
* Add data column kzg verification and update `c-kzg`. (#5701)
* Add data column kzg verification and update `c-kzg`.
* Fix incorrect `Cell` size.
* Add kzg verification on rpc blocks.
* Add kzg verification on rpc data columns.
* Rename `PEER_DAS_EPOCH` to `EIP7594_FORK_EPOCH` for client interop. (#5750)
* Fetch custody columns in range sync (#5747)
* Fetch custody columns in range sync
* Clean up todos
* Remove `BlobSidecar` construction and publish after PeerDAS activated (#5759)
* Avoid building and publishing blob sidecars after PeerDAS.
* Ignore gossip blobs with a slot greater than peer das activation epoch.
* Only attempt to verify blob count and import blobs before PeerDAS.
* #5684 review comments (#5748)
* #5684 review comments.
* Doc and message update only.
* Fix incorrect condition when constructing `RpcBlock` with `DataColumn`s
* Make sampling tests deterministic (#5775)
* PeerDAS spec tests (#5772)
* Add get_custody_columns spec tests.
* Add kzg merkle proof spec tests.
* Add SSZ spec tests.
* Add remaining KZG tests
* Load KZG only once per process, exclude electra tests and add missing SSZ tests.
* Fix lint and missing changes.
* Ignore macOS generated file.
* Merge remote branch 'sigp/unstable' into das
* Merge remote tracking branch 'origin/unstable' into das
* Implement unconditional reconstruction for supernodes (#5781)
* Implement unconditional reconstruction for supernodes
* Move code into KzgVerifiedCustodyDataColumn
* Remove expect
* Add test
* Thanks justin
* Add withhold attack mode for interop (#5788)
* Add withhold attack mode
* Update readme
* Drop added readmes
* Undo styling changes
* Add column gossip verification and handle unknown parent block (#5783)
* Add column gossip verification and handle missing parent for columns.
* Review PR
* Fix rebase issue
* more lint issues :)
---------
Co-authored-by: dapplion <35266934+dapplion@users.noreply.github.com>
* Trigger sampling on sync events (#5776)
* Trigger sampling on sync events
* Update beacon_chain.rs
* Fix tests
* Fix tests
* PeerDAS parameter changes for devnet-0 (#5779)
* Update PeerDAS parameters to latest values.
* Lint fix
* Fix lint.
* Update hardcoded subnet count to 64 (#5791)
* Fix incorrect columns per subnet and config cleanup (#5792)
* Tidy up PeerDAS preset and config values.
* Fix broken config
* Fix DAS branch CI (#5793)
* Fix invalid syntax.
* Update cli doc. Ignore get_custody_columns test temporarily.
* Fix failing test and add verify inclusion test.
* Undo accidentally removed code.
* Only attempt reconstruct columns once. (#5794)
* Re-enable precompute table for peerdas kzg (#5795)
* Merge branch 'unstable' into das
* Update subscription filter. (#5797)
* Remove penalty for duplicate columns (expected due to reconstruction) (#5798)
* Revert DAS config for interop testing. Optimise get_custody_columns function. (#5799)
* Don't perform reconstruction for proposer node as it already has all the columns. (#5806)
* Multithread compute_cells_and_proofs (#5805)
* Multi-thread reconstruct data columns
* Multi-thread path for block production
* Merge branch 'unstable' into das
# Conflicts:
# .github/workflows/test-suite.yml
# beacon_node/network/src/sync/block_lookups/mod.rs
# beacon_node/network/src/sync/block_lookups/single_block_lookup.rs
# beacon_node/network/src/sync/network_context.rs
* Fix CI errors.
* Move PeerDAS type-level config to configurable `ChainSpec` (#5828)
* Move PeerDAS type level config to `ChainSpec`.
* Fix tests
* Misc custody lookup improvements (#5821)
* Improve custody requests
* Type DataColumnsByRootRequestId
* Prioritize peers and load balance
* Update tests
* Address PR review
* Merge branch 'unstable' into das
* Rename deploy_block in network config (`das` branch) (#5852)
* Rename deploy_block.txt to deposit_contract_block.txt
* fmt
---------
Co-authored-by: Pawan Dhananjay <pawandhananjay@gmail.com>
* Merge branch 'unstable' into das
* Fix CI and merge issues.
* Merge branch 'unstable' into das
# Conflicts:
# beacon_node/beacon_chain/src/data_availability_checker/overflow_lru_cache.rs
# lcli/src/main.rs
* Store data columns individually in store and caches (#5890)
* Store data columns individually in store and caches
* Implement data column pruning
* Merge branch 'unstable' into das
# Conflicts:
# Cargo.lock
* Update reconstruction benches to newer criterion version. (#5949)
* Merge branch 'unstable' into das
# Conflicts:
# .github/workflows/test-suite.yml
* chore: add `recover_cells_and_compute_proofs` method (#5938)
* chore: add recover_cells_and_compute_proofs method
* Introduce type alias `CellsAndKzgProofs` to address type complexity.
---------
Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>
* Update `csc` format in ENR and spec tests for devnet-1 (#5966)
* Update `csc` format in ENR.
* Add spec tests for `recover_cells_and_kzg_proofs`.
* Add tests for ENR.
* Fix failing tests.
* Add protection against invalid csc value in ENR.
* Fix lint
* Fix csc encoding and decoding (#5997)
* Fix data column rpc request not being sent due to incorrect limits set. (#6000)
* Fix incorrect inbound request count causing rate limiting. (#6025)
* Merge branch 'stable' into das
# Conflicts:
# beacon_node/network/src/sync/block_lookups/tests.rs
# beacon_node/network/src/sync/block_sidecar_coupling.rs
# beacon_node/network/src/sync/manager.rs
# beacon_node/network/src/sync/network_context.rs
# beacon_node/network/src/sync/network_context/requests.rs
* Merge remote-tracking branch 'unstable' into das
* Add kurtosis config for DAS testing (#5968)
* Add kurtosis config for DAS testing.
* Fix invalid yaml file
* Update network parameter files.
* chore: add rust PeerdasKZG crypto library for peerdas functionality and rollback c-kzg dependency to 4844 version (#5941)
* chore: add recover_cells_and_compute_proofs method
* chore: add rust peerdas crypto library
* chore: integrate peerdaskzg rust library into kzg crate
* chore(multi):
- update `ssz_cell_to_crypto_cell`
- update conversion from the crypto cell type to a Vec<u8>. Since the Rust library defines them as references to an array, the conversion is simply `to_vec`
* chore(multi):
- update rest of code to handle the new crypto `Cell` type
- update test case code to no longer use the Box type
* chore: cleanup of superfluous conversions
* chore: revert c-kzg dependency back to v1
* chore: move dependency into correct order
* chore: update rust dependency
- This version includes a new method `PeerDasContext::with_num_threads`
* chore: remove Default initialization of PeerDasContext and explicitly set the parameters in `new_from_trusted_setup`
* chore: cleanup exports
* chore: commit updated cargo.lock
* Update Cargo.toml
Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>
* chore: rename dependency
* chore: update peerdas lib
- sets the blst version to 0.3 so that it matches whatever lighthouse is using. Although 0.3.12 is latest, lighthouse is pinned to 0.3.3
* chore: fix clippy lifetime
- Rust doesn't allow you to elide the lifetime on type aliases
* chore: cargo clippy fix
* chore: cargo fmt
* chore: update lib to add redundant checks (these will be removed in consensus-specs PR 3819)
* chore: update dependency to ignore proofs
* chore: update peerdas lib to latest
* update lib
* chore: remove empty proof parameter
---------
Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>
* Update PeerDAS interop testnet config (#6069)
* Update interop testnet config.
* Fix typo and remove target peers
* Avoid retrying same sampling peer that previously failed. (#6084)
* Various fixes to custody range sync (#6004)
* Only start requesting batches when there are good peers across all custody columns to avoid spaming block requests.
* Add custody peer check before mutating `BatchInfo` to avoid inconsistent state.
* Add check to cover a case where batch is not processed while waiting for custody peers to become available.
* Fix lint and logic error
* Fix `good_peers_on_subnet` always returning false for `DataColumnSubnet`.
* Add test for `get_custody_peers_for_column`
* Revert epoch parameter refactor.
* Fall back to default custody requiremnt if peer ENR is not present.
* Add metrics and update code comment.
* Add more debug logs.
* Use subscribed peers on subnet before MetaDataV3 is implemented. Remove peer_id matching when injecting error because multiple peers are used for range requests. Use randomized custodial peer to avoid repeatedly sending requests to failing peers. Batch by range request where possible.
* Remove unused code and update docs.
* Add comment
* chore: update peerdas-kzg library (#6118)
* chore: update peerDAS lib
* chore: update library
* chore: update library to version that include "init context" benchmarks and optional validation checks
* chore: (can remove) -- Add benchmarks for init context
* Prevent continuous searchers for low-peer networks (#6162)
* Merge branch 'unstable' into das
* Fix merge conflicts
* Add cli flag to enable sampling and disable by default. (#6209)
* chore: Use reference to an array representing a blob instead of an owned KzgBlob (#6179)
* add KzgBlobRef type
* modify code to use KzgBlobRef
* clippy
* Remove Deneb blob related changes to maintain compatibility with `c-kzg-4844`.
---------
Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>
* Store computed custody subnets in PeerDB and fix custody lookup test (#6218)
* Fix failing custody lookup tests.
* Store custody subnets in PeerDB, fix custody lookup test and refactor some methods.
* Merge branch 'unstable' into das
# Conflicts:
# beacon_node/beacon_chain/src/beacon_chain.rs
# beacon_node/beacon_chain/src/block_verification_types.rs
# beacon_node/beacon_chain/src/builder.rs
# beacon_node/beacon_chain/src/data_availability_checker.rs
# beacon_node/beacon_chain/src/data_availability_checker/overflow_lru_cache.rs
# beacon_node/beacon_chain/src/data_column_verification.rs
# beacon_node/beacon_chain/src/early_attester_cache.rs
# beacon_node/beacon_chain/src/historical_blocks.rs
# beacon_node/beacon_chain/tests/store_tests.rs
# beacon_node/lighthouse_network/src/discovery/enr.rs
# beacon_node/network/src/service.rs
# beacon_node/src/cli.rs
# beacon_node/store/src/hot_cold_store.rs
# beacon_node/store/src/lib.rs
# lcli/src/generate_bootnode_enr.rs
* Fix CI failures after merge.
* Batch sampling requests by peer (#6256)
* Batch sampling requests by peer
* Fix clippy errors
* Fix tests
* Add column_index to error message for ease of tracing
* Remove outdated comment
* Fix range sync never evaluating request as finished, causing it to get stuck. (#6276)
* Merge branch 'unstable' into das-0821-merge
# Conflicts:
# Cargo.lock
# Cargo.toml
# beacon_node/beacon_chain/src/beacon_chain.rs
# beacon_node/beacon_chain/src/data_availability_checker.rs
# beacon_node/beacon_chain/src/data_availability_checker/overflow_lru_cache.rs
# beacon_node/beacon_chain/src/data_column_verification.rs
# beacon_node/beacon_chain/src/kzg_utils.rs
# beacon_node/beacon_chain/src/metrics.rs
# beacon_node/beacon_processor/src/lib.rs
# beacon_node/lighthouse_network/src/rpc/codec/ssz_snappy.rs
# beacon_node/lighthouse_network/src/rpc/config.rs
# beacon_node/lighthouse_network/src/rpc/methods.rs
# beacon_node/lighthouse_network/src/rpc/outbound.rs
# beacon_node/lighthouse_network/src/rpc/rate_limiter.rs
# beacon_node/lighthouse_network/src/service/api_types.rs
# beacon_node/lighthouse_network/src/types/globals.rs
# beacon_node/network/src/network_beacon_processor/mod.rs
# beacon_node/network/src/network_beacon_processor/rpc_methods.rs
# beacon_node/network/src/network_beacon_processor/sync_methods.rs
# beacon_node/network/src/sync/block_lookups/common.rs
# beacon_node/network/src/sync/block_lookups/mod.rs
# beacon_node/network/src/sync/block_lookups/single_block_lookup.rs
# beacon_node/network/src/sync/block_lookups/tests.rs
# beacon_node/network/src/sync/manager.rs
# beacon_node/network/src/sync/network_context.rs
# consensus/types/src/data_column_sidecar.rs
# crypto/kzg/Cargo.toml
# crypto/kzg/benches/benchmark.rs
# crypto/kzg/src/lib.rs
* Fix custody tests and load PeerDAS KZG instead.
* Fix ef tests and bench compilation.
* Fix failing sampling test.
* Merge pull request #6287 from jimmygchen/das-0821-merge
Merge `unstable` into `das` 20240821
* Remove get_block_import_status
* Merge branch 'unstable' into das
* Re-enable Windows release tests.
* Address some review comments.
* Address more review comments and cleanups.
* Comment out peer DAS KZG EF tests for now
* Address more review comments and fix build.
* Merge branch 'das' of github.com:sigp/lighthouse into das
* Unignore Electra tests
* Fix metric name
* Address some of Pawan's review comments
* Merge remote-tracking branch 'origin/unstable' into das
* Update PeerDAS network parameters for peerdas-devnet-2 (#6290)
* update subnet count & custody req
* das network params
* update ef tests
---------
Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>
* remove exit-future usage,
as it is non maintained, and replace with async-channel which is already in the repo.
* Merge branch 'unstable' of https://github.com/sigp/lighthouse into remove-exit-future
* Merge branch 'unstable' of https://github.com/sigp/lighthouse into remove-exit-future
* Lint fixes
* More fixes for beta compiler.
* Format fixes
* Move `#[allow(dead_code)]` to field level.
* Remove old comment.
* Update beacon_node/execution_layer/src/test_utils/mod.rs
Co-authored-by: João Oliveira <hello@jxs.pt>
* remove duplicate line
* add new spec field to spec
* add spec changes
* fix bug and update expected subnets with validation from pyspec
* fix values and make test prettier (?)
* fix style
---------
Co-authored-by: Lion - dapplion <35266934+dapplion@users.noreply.github.com>
## Issue Addressed
Synchronize dependencies and edition on the workspace `Cargo.toml`
## Proposed Changes
with https://github.com/rust-lang/cargo/issues/8415 merged it's now possible to synchronize details on the workspace `Cargo.toml` like the metadata and dependencies.
By only having dependencies that are shared between multiple crates aligned on the workspace `Cargo.toml` it's easier to not miss duplicate versions of the same dependency and therefore ease on the compile times.
## Additional Info
this PR also removes the no longer required direct dependency of the `serde_derive` crate.
should be reviewed after https://github.com/sigp/lighthouse/pull/4639 get's merged.
closes https://github.com/sigp/lighthouse/issues/4651
Co-authored-by: Michael Sproul <michael@sigmaprime.io>
Co-authored-by: Michael Sproul <micsproul@gmail.com>
## Issue Addressed
Addresses [#4401](https://github.com/sigp/lighthouse/issues/4401)
## Proposed Changes
Shift some constants into ```ChainSpec``` and remove the constant values from code space.
## Additional Info
I mostly used ```MainnetEthSpec::default_spec()``` for getting ```ChainSpec```. I wonder Did I make a mistake about that.
Co-authored-by: armaganyildirak <armaganyildirak@gmail.com>
Co-authored-by: Paul Hauner <paul@paulhauner.com>
Co-authored-by: Age Manning <Age@AgeManning.com>
Co-authored-by: Diva M <divma@protonmail.com>
This PR adds the ability to read the Lighthouse logs from the HTTP API for both the BN and the VC.
This is done in such a way to as minimize any kind of performance hit by adding this feature.
The current design creates a tokio broadcast channel and mixes is into a form of slog drain that combines with our main global logger drain, only if the http api is enabled.
The drain gets the logs, checks the log level and drops them if they are below INFO. If they are INFO or higher, it sends them via a broadcast channel only if there are users subscribed to the HTTP API channel. If not, it drops the logs.
If there are more than one subscriber, the channel clones the log records and converts them to json in their independent HTTP API tasks.
Co-authored-by: Michael Sproul <micsproul@gmail.com>
## Proposed Changes
Decouple the stdout and logfile formats by adding the `--logfile-format` CLI flag.
This behaves identically to the existing `--log-format` flag, but instead will only affect the logs written to the logfile.
The `--log-format` flag will no longer have any effect on the contents of the logfile.
## Additional Info
This avoids being a breaking change by causing `logfile-format` to default to the value of `--log-format` if it is not provided.
This means that users who were previously relying on being able to use a JSON formatted logfile will be able to continue to use `--log-format JSON`.
Users who want to use JSON on stdout and default logs in the logfile, will need to pass the following flags: `--log-format JSON --logfile-format DEFAULT`
## Issue Addressed
#3732
## Proposed Changes
Add a CLI flag to allow users to opt out of the restrictive permissions of the log files.
## Additional Info
This is not recommended for most users. The log files can contain sensitive information such as validator indices, public keys and API tokens (see #2438). However some users using a multi-user setup may find this helpful if they understand the risks involved.
## Issue Addressed
<!--Which issue # does this PR address?-->
Removed some unnecessary `mut`. 🙂
<!--
## Proposed Changes
Please list or describe the changes introduced by this PR.
-->
<!--
## Additional Info
Please provide any additional information. For example, future considerations
or information useful for reviewers.
-->
## Issue Addressed
Adding CLI tests for logging flags: log-color and disable-log-timestamp
Which issue # does this PR address?
#3588
## Proposed Changes
Add CLI tests for logging flags as described in #3588
Please list or describe the changes introduced by this PR.
Added logger_config to client::Config as suggested. Implemented Default for LoggerConfig based on what was being done elsewhere in the repo. Created 2 tests for each flag addressed.
## Additional Info
Please provide any additional information. For example, future considerations
or information useful for reviewers.
Add flag 'log-color' which preserves colors of log when stdout is redirected to a file.
This is my first lighthouse PR, please let me know if I'm not following contribution guidelines, I welcome meta-feeback (feedback on git commit messages, git branch naming, and the contents of the description of this PR.)
## Issue Addressed
Solves https://github.com/sigp/lighthouse/issues/3527
## Proposed Changes
Adding a flag which enables log color preserving when stdout is redirected to a file.
### Usage
Below I demonstrate current behaviour (without using the new flag) and the new behaviur (when using new flag).
In the screenshot below, I have to panes, one on top running `lighthouse` which redirects to file `~/Desktop/test.log` and one pane in the bottom which runs `tail -f ~/Desktop/test.log`.
#### Current behaviour
```sh
lighthouse --network prater vc |& tee -a ~/Desktop/test.log
```
**Result is no colors**
<img width="1624" alt="current" src="https://user-images.githubusercontent.com/864410/188258226-bfcf8271-4c9e-474c-848e-ac92a60df25c.png">
#### New behaviour
```sh
lighthouse --network prater vc --log-color |& tee -a ~/Desktop/test.log
```
**Result is colors** 🔴🟢🔵🟡
<img width="1624" alt="new" src="https://user-images.githubusercontent.com/864410/188258223-7d9ecf09-92c8-4cba-8f24-bd4d88fc0353.png">
## Additional Info
I chose American spelling of "color" instead of Brittish "colour' since that was aligned with `slog`'s API - method`force_color()`, let me know if you prefer spelling "colour" instead. I also chose to let it be an arg not taking any argument, just like `logfile-compress` flag, rather than having to write `--log-color true`.
# Description
Since the `TaskExecutor` currently requires a `Weak<Runtime>`, it's impossible to use it in an async test where the `Runtime` is created outside our scope. Whilst we *could* create a new `Runtime` instance inside the async test, dropping that `Runtime` would cause a panic (you can't drop a `Runtime` in an async context).
To address this issue, this PR creates the `enum Handle`, which supports either:
- A `Weak<Runtime>` (for use in our production code)
- A `Handle` to a runtime (for use in testing)
In theory, there should be no change to the behaviour of our production code (beyond some slightly different descriptions in HTTP 500 errors), or even our tests. If there is no change, you might ask *"why bother?"*. There are two PRs (#3070 and #3175) that are waiting on these fixes to introduce some new tests. Since we've added the EL to the `BeaconChain` (for the merge), we are now doing more async stuff in tests.
I've also added a `RuntimeExecutor` to the `BeaconChainTestHarness`. Whilst that's not immediately useful, it will become useful in the near future with all the new async testing.
## Proposed Changes
Lots of lint updates related to `flat_map`, `unwrap_or_else` and string patterns. I did a little more creative refactoring in the op pool, but otherwise followed Clippy's suggestions.
## Additional Info
We need this PR to unblock CI.
## Issue Addressed
Closes#2990
## Proposed Changes
Add a check to see if the `--validators-dir` CLI flag is set and if so store validator logs into it.
Ensure that if the log directory cannot be created, emit a `WARN` and disable file logging rather than panicking.
## Additional Info
Panics associated with logfiles can still occur in these scenarios:
1. The `$datadir/validators/logs` directory already exists with the wrong permissions (or was changed after creation).
1. The logfile already exists with the wrong permissions (or was changed after creation).
> These panics are cosmetic only since only the logfile thread panics. Following the panics, LH will continue to function as normal.
I believe this is due to the use of [`slog::Fuse`](https://docs.rs/slog/latest/slog/struct.Fuse.html) when initializing the logger.
I'm not sure if there a better way of handling logfile errors?
I think ideally, rather than panicking, we would emit a `WARN` to the stdout logger with the panic reason, then exit the logfile thread gracefully.
## Proposed Changes
Add a new hardcoded spec for the Gnosis Beacon Chain.
Ideally, official Lighthouse executables will be able to connect to the gnosis beacon chain from now on, using `--network gnosis` CLI option.
## Proposed Changes
Change the canonical fork name for the merge to Bellatrix. Keep other merge naming the same to avoid churn.
I've also fixed and enabled the `fork` and `transition` tests for Bellatrix, and the v1.1.7 fork choice tests.
Additionally, the `BellatrixPreset` has been added with tests. It gets served via the `/config/spec` API endpoint along with the other presets.
## Issue Addressed
Resolves: https://github.com/sigp/lighthouse/issues/2741
Includes: https://github.com/sigp/lighthouse/pull/2853 so that we can get ssz static tests passing here on v1.1.6. If we want to merge that first, we can make this diff slightly smaller
## Proposed Changes
- Changes the `justified_epoch` and `finalized_epoch` in the `ProtoArrayNode` each to an `Option<Checkpoint>`. The `Option` is necessary only for the migration, so not ideal. But does allow us to add a default logic to `None` on these fields during the database migration.
- Adds a database migration from a legacy fork choice struct to the new one, search for all necessary block roots in fork choice by iterating through blocks in the db.
- updates related to https://github.com/ethereum/consensus-specs/pull/2727
- We will have to update the persisted forkchoice to make sure the justified checkpoint stored is correct according to the updated fork choice logic. This boils down to setting the forkchoice store's justified checkpoint to the justified checkpoint of the block that advanced the finalized checkpoint to the current one.
- AFAICT there's no migration steps necessary for the update to allow applying attestations from prior blocks, but would appreciate confirmation on that
- I updated the consensus spec tests to v1.1.6 here, but they will fail until we also implement the proposer score boost updates. I confirmed that the previously failing scenario `new_finalized_slot_is_justified_checkpoint_ancestor` will now pass after the boost updates, but haven't confirmed _all_ tests will pass because I just quickly stubbed out the proposer boost test scenario formatting.
- This PR now also includes proposer boosting https://github.com/ethereum/consensus-specs/pull/2730
## Additional Info
I realized checking justified and finalized roots in fork choice makes it more likely that we trigger this bug: https://github.com/ethereum/consensus-specs/pull/2727
It's possible the combination of justified checkpoint and finalized checkpoint in the forkchoice store is different from in any block in fork choice. So when trying to startup our store's justified checkpoint seems invalid to the rest of fork choice (but it should be valid). When this happens we get an `InvalidBestNode` error and fail to start up. So I'm including that bugfix in this branch.
Todo:
- [x] Fix fork choice tests
- [x] Self review
- [x] Add fix for https://github.com/ethereum/consensus-specs/pull/2727
- [x] Rebase onto Kintusgi
- [x] Fix `num_active_validators` calculation as @michaelsproul pointed out
- [x] Clean up db migrations
Co-authored-by: realbigsean <seananderson33@gmail.com>
## Issue Addressed
Closes#1996
## Proposed Changes
Run a second `Logger` via `sloggers` which logs to a file in the background with:
- separate `debug-level` for background and terminal logging
- the ability to limit log size
- rotation through a customizable number of log files
- an option to compress old log files (`.gz` format)
Add the following new CLI flags:
- `--logfile-debug-level`: The debug level of the log files
- `--logfile-max-size`: The maximum size of each log file
- `--logfile-max-number`: The number of old log files to store
- `--logfile-compress`: Whether to compress old log files
By default background logging uses the `debug` log level and saves logfiles to:
- Beacon Node: `$HOME/.lighthouse/$network/beacon/logs/beacon.log`
- Validator Client: `$HOME/.lighthouse/$network/validators/logs/validator.log`
Or, when using the `--datadir` flag:
`$datadir/beacon/logs/beacon.log` and `$datadir/validators/logs/validator.log`
Once rotated, old logs are stored like so: `beacon.log.1`, `beacon.log.2` etc.
> Note: `beacon.log.1` is always newer than `beacon.log.2`.
## Additional Info
Currently the default value of `--logfile-max-size` is 200 (MB) and `--logfile-max-number` is 5.
This means that the maximum storage space that the logs will take up by default is 1.2GB.
(200MB x 5 from old log files + <200MB the current logfile being written to)
Happy to adjust these default values to whatever people think is appropriate.
It's also worth noting that when logging to a file, we lose our custom `slog` formatting. This means the logfile logs look like this:
```
Oct 27 16:02:50.305 INFO Lighthouse started, version: Lighthouse/v2.0.1-8edd9d4+, module: lighthouse:413
Oct 27 16:02:50.305 INFO Configured for network, name: prater, module: lighthouse:414
```
