Fixes the error `fatal: No names found, cannot describe anything.` that occurs when running `make`
commands in CI (GitHub Actions).
https://github.com/sigp/lighthouse/actions/runs/19839541042/job/56844781126#step:5:13
> fatal: No names found, cannot describe anything.
Changed the `GIT_TAG` variable assignment in the Makefile from immediate evaluation to lazy evaluation:
```diff
- GIT_TAG := $(shell git describe --tags --candidates 1)
+ GIT_TAG = $(shell git describe --tags --candidates 1)
```
This change ensures that git describe is only executed when `GIT_TAG` is actually used (in the `build-release-tarballs` target), rather than on every Makefile invocation.
Co-Authored-By: ackintosh <sora.akatsuki@gmail.com>
This pull request introduces workflows and updates to ensure reproducible builds for the Lighthouse project. It adds two GitHub Actions workflows for building and testing reproducible Docker images and binaries, updates the `Makefile` to streamline reproducible build configurations, and modifies the `Dockerfile.reproducible` to align with the new build process. Additionally, it removes the `reproducible` profile from `Cargo.toml`.
### New GitHub Actions Workflows:
* [`.github/workflows/docker-reproducible.yml`](diffhunk://#diff-222af23bee616920b04f5b92a83eb5106fce08abd885cd3a3b15b8beb5e789c3R1-R145): Adds a workflow to build and push reproducible multi-architecture Docker images for releases, including support for dry runs without pushing an image.
### Build Configuration Updates:
* [`Makefile`](diffhunk://#diff-76ed074a9305c04054cdebb9e9aad2d818052b07091de1f20cad0bbac34ffb52L85-R143): Refactors reproducible build targets, centralizes environment variables for reproducibility, and updates Docker build arguments for `x86_64` and `aarch64` architectures.
* [`Dockerfile.reproducible`](diffhunk://#diff-587298ff141278ce3be7c54a559f9f31472cc5b384e285e2105b3dee319ba31dL1-R24): Updates the base Rust image to version 1.86, removes hardcoded reproducibility settings, and delegates build logic to the `Makefile`.
* Switch to using jemalloc-sys from Debian repos instead of building it from source. A Debian version is [reproducible](https://tests.reproducible-builds.org/debian/rb-pkg/trixie/amd64/jemalloc.html) which is [hard to achieve](https://github.com/NixOS/nixpkgs/issues/380852) if you build it from source.
### Profile Removal:
* [`Cargo.toml`](diffhunk://#diff-2e9d962a08321605940b5a657135052fbcef87b5e360662bb527c96d9a615542L289-L295): Removes the `reproducible` profile, simplifying build configurations and relying on external tooling for reproducibility.
Co-Authored-By: Moe Mahhouk <mohammed-mahhouk@hotmail.com>
Co-Authored-By: chonghe <44791194+chong-he@users.noreply.github.com>
Co-Authored-By: Michael Sproul <michaelsproul@users.noreply.github.com>
Partially addresses #8248
Run the beacon chain, http and network tests only for recent forks instead of everything from phase 0.
Also added gloas also to the recent forks list. I thought that would be a good way to know if changes in the current fork affect future forks.
Not completely sure if we should run for future forks, but added it so that we can discuss here.
Co-Authored-By: Pawan Dhananjay <pawandhananjay@gmail.com>
Co-Authored-By: Jimmy Chen <jchen.tc@gmail.com>
Adds the required boilerplate code for the Gloas (Glamsterdam) hard fork. This allows PRs testing Gloas-candidate features to test fork transition.
This also includes de-duplication of post-Bellatrix readiness notifiers from #6797 (credit to @dapplion)
Follow-up to:
- https://github.com/sigp/lighthouse/pull/7764
The `heaptrack` feature added in my previous PR was ineffective, because the jemalloc feature was turned on by the Linux target-specific dependency.
This PR tweaks the features such that:
- The jemalloc feature is just used to control whether jemalloc is compiled in. It is enabled on Linux by the target-specific dependency (see `lighthouse/Cargo.toml`), and completely disabled on Windows.
- If the `sysmalloc` feature is set on Linux then it overrides jemalloc when selecting an allocator, _even if_ the jemalloc feature is enabled (and the jemalloc dep was compiled).
This PR fixes a bug where wrong columns could get processed immediately after a CGC increase.
Scenario:
- The node's CGC increased due to additional validators attached to it (lets say from 10 to 11)
- The new CGC is advertised and new subnets are subscribed immediately, however the change won't be effective in the data availability check until the next epoch (See [this](ab0e8870b4/beacon_node/beacon_chain/src/validator_custody.rs (L93-L99))). Data availability checker still only require 10 columns for the current epoch.
- During this time, data columns for the additional custody column (lets say column 11) may arrive via gossip as we're already subscribed to the topic, and it may be incorrectly used to satisfy the existing data availability requirement (10 columns), and result in this additional column (instead of a required one) getting persisted, resulting in database inconsistency.
In #7743, rust version was bumped:
- msrv to 1.87
- `Dockerfile` to 1.88
We also need to bump the other docker images as well, and might as well keep them all consistent at 1.88.
Closes#6855
Add PeerDAS broadcast validation tests and fix a small bug where `sampling_columns_indices` is none (indicating that we've already sampled the necessary columns) and `process_gossip_data_columns` gets called
This PR adds the ability to download [nightly reference tests from the consensus-specs repo](https://github.com/ethereum/consensus-specs/actions/workflows/generate_vectors.yml). This will be used by spec maintainers to ensure that there are no unexpected test failures prior to new releases. Also, we will keep track of test compliance with [this website](https://jtraglia.github.io/nyx/); eventually this will be integrated into Hive.
* A new script (`download_test_vectors.sh`) is added to handle downloads.
* The logic for downloading GitHub artifacts is a bit complex.
* Rename the variables which store test versions:
* `TESTS_TAG` to `CONSENSUS_SPECS_TEST_VERSION`.
* `BLS_TEST_TAG` to `BLS_TEST_VERSION`, for consistency.
* Delete tarballs after extracting them.
* I see no need to keep these; they just use extra disk.
* Consolidate `clean` rules into a single rule.
* Do `clean` prior to downloading/extracting tests.
* Remove `CURL` variable with GitHub token; don't need it for downloading releases.
* Do `mkdir -p` when creating directories.
* Probably more small stuff...
Which issue # does this PR address?
This PR addresses reproducible builds. The current dockerfile builds the lighthouse binary but not reproducibly.
You can verify that by following these steps:
```
docker build --no-cache --output=. .
mv usr/local/bin/lighthouse lighthouse1
rm usr/local/bin/lighthouse
docker build --no-cache --output=. .
mv usr/local/bin/lighthouse lighthouse2
sha256sum lighthouse1 lighthouse2
```
You will notice that each one of the binaries has a different checksum upon each build. This is critical for systems that depends on requiring reproducible builds, such as running lighthouse in confidential computing, like Intel TDX.
This PR adds a new build profile as well as a Dockerfile.reproducible that enables building the lighthouse binary reproducibly.
By following the steps I listed above, you will be able to verify that the resulted binary has the same hash upon several subsequent builds for the same version.
How to test it:
```
mkdir output1 output2
docker build --no-cache -f Dockerfile.reproducible --output=output1 .
docker build --no-cache -f Dockerfile.reproducible --output=output2 .
sha256sum output1/lighthouse output2/lighthouse
# hashes should be identical
rm -rf output1 output2
```
Lighthouse currently lacks support for cross-compilation targeting the `riscv64` architecture.
This PR introduces initial support for cross-compiling Lighthouse to `riscv64`. The following changes were made:
- **Makefile**: Updated to support `cross` with `riscv64` as a target.
- **Cross.toml**: Added configuration specific to `riscv64`.
- **Documentation**: List 'build-riscv64' in `book/src/installation_cross_compiling.md`.
Resolves#7091
The `prometheus` crate pulls in `protobuf 2.x` which fails cargo audit. We actually dont use any `protobuf` related features in LH. By disabling default features for `prometheus`, we no longer pull in the `protobuf` crate
Run `cargo update` to address [RUSTSEC-2025-0004](https://rustsec.org/advisories/RUSTSEC-2025-0004), a vulnerability in `openssl`. I don't think we are affected, but this PR is required for us to pass `cargo audit` and unblock CI.
#5244
Pass `JEMALLOC_SYS_WITH_LG_PAGE=16` env to aarch64 cross-compilation to support systems with up to 64-KiB page sizes. This is backwards-compatible for the current (most usual) 4-KiB systems.
* Update kurtosis-cli
* Fix name of Kurtosis artefact used in doppelganger tests
* Ignore idna vuln
* Set Java Version to 21 (required since Web3Signer 24.12.0).
* Remove portable in book
* Add back modern in docker.md
* Update Makefile
* Update release.yml
* Update docker.yml
* Update book
* Minor revision
* delete portable tarballs
* delete portable in Make command
* Fix link in book
* mdlint
* Remove modernity in docker
* Merge branch 'unstable' into update-book-portable
* Merge branch 'unstable' into update-book-portable
* Remove `-dev` docker images
* Merge remote-tracking branch 'origin/unstable' into update-book-portable
* initial redb impl
* redb impl
* remove phantom data
* fixed table definition
* fighting the borrow checker
* a rough draft that doesnt cause lifetime issues
* refactoring
* refactor
* refactor
* passing unit tests
* refactor
* refactor
* refactor
* commit
* move everything to one database
* remove panics, ready for a review
* merge
* a working redb impl
* passing a ref of txn to cursor
* this tries to create a second write transaction when initializing cursor. breaks everything
* Use 2 lifetimes and subtyping
Also fixes a bug in last_key caused by rev and next_back cancelling out
* Move table into cursor
* Merge remote-tracking branch 'origin/unstable' into redb-slasher-backend-impl
* changes based on feedback
* update lmdb
* fix lifetime issues
* moving everything from Cursor to Transaction
* update
* upgrade to redb 2.0
* Merge branch 'unstable' of https://github.com/sigp/lighthouse into redb-slasher-backend-impl
* bring back cursor
* Merge branch 'unstable' of https://github.com/sigp/lighthouse into redb-slasher-backend-impl
* fix delete while
* linting
* linting
* switch to lmdb
* update redb to v2.1
* build fixes, remove unwrap or default
* another build error
* hopefully this is the last build error
* fmt
* cargo.toml
* fix mdbx
* Merge branch 'unstable' of https://github.com/sigp/lighthouse into redb-slasher-backend-impl
* Remove a collect
* Merge remote-tracking branch 'origin/unstable' into redb-slasher-backend-impl
* Merge branch 'redb-slasher-backend-impl' of https://github.com/eserilev/lighthouse into redb-slasher-backend-impl
* re-enable test
* fix failing slasher test
* Merge remote-tracking branch 'origin/unstable' into redb-slasher-backend-impl
* Rename DB file to `slasher.redb`
* Enable jemalloc by default on non windows target.
* Update `allocator_name` function to check for `target_os` instead as we've deprecated `jemalloc` feature.
* chore: Change Into trait impl for KzgProof to From trait impl
* chore: change `impl Into <T> for U` to `impl From<U> for T`
* chore: remove `from-over-into` clippy lint exception
* fix lib.rs and tests.rs
* update decode.rs
* auto-delete in Cargo.lock
* delete milagro in cargo.toml
* remove milagro from makefile
* remove milagro from the name
* delete milagro in comment
* delete milagro in cargo.toml
* delete in /testing/ef_tests/cargo.toml
* delete milagro in the logical OR
* delete milagro in /lighthouse/src/main.rs
* delete milagro in /crypto/bls/tests/tests.rs
* delete milagro in comment
* delete milagro in /testing//ef_test/src//cases/bls_eth_aggregate_pubkeys.rs
* delete milagro
* delete more in lib.rs
* delete more in lib.rs
* delete more in lib.rs
* delete milagro in /crypto/bls/src/lib.rs
* delete milagro in crypto/bls/src/mod.rs
* delete milagro.rs
* switch libp2p source to sigp fork
* Shift the connection closing inside RPC behaviour
* Tag specific commits
* Add slow peer scoring
* Fix test
* Use default yamux config
* Pin discv5 to our libp2p fork and cargo update
* Upgrade libp2p to enable yamux gains
* Add a comment specifying the branch being used
* cleanup build output from within container
(prevents CI warnings related to fs permissions)
* Remove revision tags add branches for testing, will revert back once we're happy
* Update to latest rust-libp2p version
* Pin forks
* Update cargo.lock
* Re-pin to panic-free rust
---------
Co-authored-by: Age Manning <Age@AgeManning.com>
Co-authored-by: Pawan Dhananjay <pawandhananjay@gmail.com>
Co-authored-by: antondlr <anton@delaruelle.net>
Co-authored-by: Michael Sproul <michael@sigmaprime.io>
* Add cli.sh file
* update bash script
* update Makefile
* update
* modified test-suite
* fix path
* Fix cli.sh permissions
* update cmd
* cli_manual
* Revise to update
* Update directory in Github
* Correct cli.txt directory
* test old cli_manual
* change exit 1
* Update cli and makefile
* Move cli.sh
* remove files
* fix permission
* Indentation and revision
* Fixed permission
* Create new cli folder
* remove dummy
* put a dummy file
* Revise cli.sh
* comment
* function
* remove vm.md
* test make cli
* test
* testing
* testing
* update
* update
* test
* test
* add vm
* change back non-debug mode
* add exist and update for future debug
* revise
* remove troubleshooting part
* update
* add summary.md
* test
* test
* Update Makefile
Co-authored-by: Mac L <mjladson@pm.me>
* Update Makefile
Co-authored-by: Mac L <mjladson@pm.me>
* Remove help-cli.md and rearrange
* Remove help-cli.md
* Update scripts/cli.sh
Co-authored-by: Mac L <mjladson@pm.me>
* Update scripts/cli.sh
Co-authored-by: Mac L <mjladson@pm.me>
* remove maxperf
* move then to same line as if
* Fix indent and echo file not found
* To be explicit in replacing the old file
* Add logging when there are changes
* Add local variables
* spacing
* remove cargo fmt
* update .md files
* Edit exit message to avoid confusion
* Remove am and add vm subcommands
* Add cargo-fmt
* Revise test-suite.yml
* Update SUMMARY.md
* Add set -e
* Add vm
* Fix
* Add vm
* set -e
* Remove return 1 and add :
* Small revision
* Fix typo
* Update scripts/cli.sh
Co-authored-by: Mac L <mjladson@pm.me>
* Indent
* Update scripts/cli.sh
Co-authored-by: Mac L <mjladson@pm.me>
* Remove .exe in Windows
* Fix period with \.
* test
* check diff
* linux commit
* Add cli.sh file
* update bash script
* update Makefile
* update
* modified test-suite
* fix path
* Fix cli.sh permissions
* update cmd
* cli_manual
* Revise to update
* Update directory in Github
* Correct cli.txt directory
* test old cli_manual
* change exit 1
* Update cli and makefile
* Move cli.sh
* remove files
* fix permission
* Indentation and revision
* Fixed permission
* Create new cli folder
* remove dummy
* put a dummy file
* Revise cli.sh
* comment
* function
* remove vm.md
* test make cli
* test
* testing
* testing
* update
* update
* test
* test
* add vm
* change back non-debug mode
* add exist and update for future debug
* revise
* remove troubleshooting part
* update
* add summary.md
* test
* test
* Update Makefile
Co-authored-by: Mac L <mjladson@pm.me>
* Update Makefile
Co-authored-by: Mac L <mjladson@pm.me>
* Remove help-cli.md and rearrange
* Remove help-cli.md
* Update scripts/cli.sh
Co-authored-by: Mac L <mjladson@pm.me>
* Update scripts/cli.sh
Co-authored-by: Mac L <mjladson@pm.me>
* remove maxperf
* move then to same line as if
* Fix indent and echo file not found
* To be explicit in replacing the old file
* Add logging when there are changes
* Add local variables
* spacing
* remove cargo fmt
* Edit exit message to avoid confusion
* update .md files
* Remove am and add vm subcommands
* Add cargo-fmt
* Revise test-suite.yml
* Update SUMMARY.md
* Add set -e
* Add vm
* Fix
* Add vm
* set -e
* Remove return 1 and add :
* Small revision
* Fix typo
* Update scripts/cli.sh
Co-authored-by: Mac L <mjladson@pm.me>
* Indent
* Update scripts/cli.sh
Co-authored-by: Mac L <mjladson@pm.me>
* Remove .exe in Windows
* Fix period with \.
* test
* check diff
* Revert "Merge branch 'book-cli' of https://github.com/chong-he/lighthouse into book-cli"
This reverts commit 314005d3f8, reversing
changes made to a007f61378.
* update
* update
* Remove echo diff
* Dockerize
* Remove `-ti`
* take ownership inside container
* fix mistake
* proper escaping, restore ownership afterwards
* try without taking ownership of repo
* update
* add diff for troubleshooting
* binary
* update using linux
* binary
* make file
* remove diff
* add diff
* update progressive balance help text
* Remove diff
---------
Co-authored-by: Mac L <mjladson@pm.me>
Co-authored-by: antondlr <anton@delaruelle.net>
