Commit Graph

16 Commits

Author SHA1 Message Date
Jimmy Chen
8e54f6e1a8 Fix md format (#8434)
The merge queue is failing due to md lint changes:

https://github.com/sigp/lighthouse/actions/runs/19491272535/job/55783746002

This PR fixes the lint. I'm targeting the release branch so we can get things merged for release tomorrow, and we'll merge back down to `unstable`.


  


Co-Authored-By: Jimmy Chen <jchen.tc@gmail.com>

Co-Authored-By: Michael Sproul <michael@sigmaprime.io>
2025-11-19 11:00:36 +00:00
chonghe
3f6c11db0e Some updates to Lighthouse book (#6995)
* #6447


  - Move some deprecated pages to a new section under `Archived`

- Remove fallback log in mev as the log will not be present after VC using `/eth/v3/validator/blocks` endpoint by default
- Add warning against using Btrfs file system (thank you @ChosunOne for the report)
- Add data shared by @mcdee  on tree states API queries time
- Rename partial withdrawals to validator sweep to differentiate it from the upcoming execution layer partial withdrawals
- Update NAT API response
- Update docs on IPv6

- Rename .md files to follow a standard prefix section name, e.g., installation_*.md, advanced_*.md
- Standardise .md files using underscore `_` instead of hyphen `-` to be consistent with other files naming conventions.
2025-03-24 02:30:20 +00:00
chonghe
1de498340c Add spell check and update Lighthouse book (#6627)
* spellcheck config

* Merge remote-tracking branch 'origin/unstable' into spellcheck

* spellcheck update

* update spellcheck

* spell check passes

* Remove ignored and add other md files

* Remove some words in wordlist

* CI

* test spell check CI

* correct spell check

* Merge branch 'unstable' into spellcheck

* minor fix

* Merge branch 'spellcheck' of https://github.com/chong-he/lighthouse into spellcheck

* Update book

* mdlint

* delete previous_epoch_active_gwei

* Merge branch 'unstable' into spellcheck

* Tweak "container runtime" wording

* Try `BeaconState`s
2024-12-17 07:26:59 +00:00
Michael Sproul
9fdd53df56 Hierarchical state diffs (#5978)
* Start extracting freezer changes for tree-states

* Remove unused config args

* Add comments

* Remove unwraps

* Subjective more clear implementation

* Clean up hdiff

* Update xdelta3

* Tree states archive metrics (#6040)

* Add store cache size metrics

* Add compress timer metrics

* Add diff apply compute timer metrics

* Add diff buffer cache hit metrics

* Add hdiff buffer load times

* Add blocks replayed metric

* Move metrics to store

* Future proof some metrics

---------

Co-authored-by: Michael Sproul <michael@sigmaprime.io>

* Port and clean up forwards iterator changes

* Add and polish hierarchy-config flag

* Merge remote-tracking branch 'origin/unstable' into tree-states-archive

* Cleaner errors

* Fix beacon_chain test compilation

* Merge remote-tracking branch 'origin/unstable' into tree-states-archive

* Patch a few more freezer block roots

* Fix genesis block root bug

* Fix test failing due to pending updates

* Beacon chain tests passing

* Merge remote-tracking branch 'origin/unstable' into tree-states-archive

* Merge remote-tracking branch 'origin/unstable' into tree-states-archive

* Fix doc lint

* Implement DB schema upgrade for hierarchical state diffs (#6193)

* DB upgrade

* Add flag

* Delete RestorePointHash

* Update docs

* Update docs

* Implement hierarchical state diffs config migration (#6245)

* Implement hierarchical state diffs config migration

* Review PR

* Remove TODO

* Set CURRENT_SCHEMA_VERSION correctly

* Fix genesis state loading

* Re-delete some PartialBeaconState stuff

---------

Co-authored-by: Michael Sproul <michael@sigmaprime.io>

* Merge remote-tracking branch 'origin/unstable' into tree-states-archive

* Fix test compilation

* Update schema downgrade test

* Fix tests

* Fix null anchor migration

* Merge remote-tracking branch 'origin/unstable' into tree-states-archive

* Fix tree states upgrade migration (#6328)

* Towards crash safety

* Fix compilation

* Move cold summaries and state roots to new columns

* Rename StateRoots chunked field

* Update prune states

* Clean hdiff CLI flag and metrics

* Fix "staged reconstruction"

* Merge remote-tracking branch 'origin/unstable' into tree-states-archive

* Fix alloy issues

* Fix staged reconstruction logic

* Prevent weird slot drift

* Remove "allow" flag

* Update CLI help

* Remove FIXME about downgrade

* Merge remote-tracking branch 'origin/unstable' into tree-states-archive

* Remove some unnecessary error variants

* Fix new test

* Tree states archive - review comments and metrics (#6386)

* Review PR comments and metrics

* Comments

* Add anchor metrics

* drop prev comment

* Update metadata.rs

* Apply suggestions from code review

---------

Co-authored-by: Michael Sproul <micsproul@gmail.com>

* Update beacon_node/store/src/hot_cold_store.rs

Co-authored-by: Lion - dapplion <35266934+dapplion@users.noreply.github.com>

* Merge remote-tracking branch 'origin/unstable' into tree-states-archive

* Clarify comment and remove anchor_slot garbage

* Simplify database anchor (#6397)

* Simplify database anchor

* Update beacon_node/store/src/reconstruct.rs

* Add migration for anchor

* Fix and simplify light_client store tests

* Fix incompatible config test

* Merge remote-tracking branch 'origin/unstable' into tree-states-archive

* Merge remote-tracking branch 'origin/unstable' into tree-states-archive

* More metrics

* Merge remote-tracking branch 'origin/unstable' into tree-states-archive

* New historic state cache (#6475)

* New historic state cache

* Add more metrics

* State cache hit rate metrics

* Fix store metrics

* More logs and metrics

* Fix logger

* Ensure cached states have built caches :O

* Replay blocks in preference to diffing

* Two separate caches

* Distribute cache build time to next slot

* Re-plumb historic-state-cache flag

* Clean up metrics

* Update book

* Update beacon_node/store/src/hdiff.rs

Co-authored-by: Lion - dapplion <35266934+dapplion@users.noreply.github.com>

* Update beacon_node/store/src/historic_state_cache.rs

Co-authored-by: Lion - dapplion <35266934+dapplion@users.noreply.github.com>

---------

Co-authored-by: Lion - dapplion <35266934+dapplion@users.noreply.github.com>

* Update database docs

* Update diagram

* Merge remote-tracking branch 'origin/unstable' into tree-states-archive

* Update lockbud to work with bindgen/etc

* Correct pkg name for Debian

* Remove vestigial epochs_per_state_diff

* Merge remote-tracking branch 'origin/unstable' into tree-states-archive

* Markdown lint

* Merge remote-tracking branch 'origin/unstable' into tree-states-archive

* Address Jimmy's review comments

* Simplify ReplayFrom case

* Fix and document genesis_state_root

* Typo

Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>

* Merge branch 'unstable' into tree-states-archive

* Compute diff of validators list manually (#6556)

* Split hdiff computation

* Dedicated logic for historical roots and summaries

* Benchmark against real states

* Mutated source?

* Version the hdiff

* Add lighthouse DB config for hierarchy exponents

* Tidy up hierarchy exponents flag

* Apply suggestions from code review

Co-authored-by: Michael Sproul <micsproul@gmail.com>

* Address PR review

* Remove hardcoded paths in benchmarks

* Delete unused function in benches

* lint

---------

Co-authored-by: Michael Sproul <michael@sigmaprime.io>

* Test hdiff binary format stability (#6585)

* Merge remote-tracking branch 'origin/unstable' into tree-states-archive

* Add deprecation warning for SPRP

* Update xdelta to get rid of duplicate deps

* Document test
2024-11-18 01:51:44 +00:00
chonghe
3070cb7c39 Markdown linter (#5494)
* linter

* Add markdown linter

* add env

* only check markdown

* Add token

* Update .github/workflows/test-suite.yml

* Markdown linter

* Exit code

* Update script

* rename

* mdlint

* Add an empty line after end of file

* Testing disable

* add text

* update mdlint.sh

* ori validator inclusion

* Add config yml file

* Remove MD041 and fix advanced-datadir file

* FIx validator inclusion file conflict

* Merge branch 'unstable' into markdown-linter

* change files

* Merge branch 'markdown-linter' of https://github.com/chong-he/lighthouse into markdown-linter

* mdlint

* Remove MD025

* Remove MD036

* Remove MD045

* Removr MD001

* Set MD028 to false

* Remove MD024

* Remove MD055

* Remove MD029

* Remove MD040

* Set MD040 to false

* Set MD033 to false

* Set MD013 to false

* Rearrange yml file

* Update mdlint.sh and test

* Test remove fix

* Test with fix

* Test with space

* Fix summary indentation

* Test mdlint.sh

* Update mdlint.sh

* Test

* Update

* Test fix

* Test again

* Fix

* merge into check-code

* Update scripts/mdlint.sh

Co-authored-by: Mac L <mjladson@pm.me>

* Update scripts/mdlint.sh

Co-authored-by: Mac L <mjladson@pm.me>

* Remove set -e

* Add comment

* Merge pull request #7 from chong-he/unstable

Merge unstable to markdown branch

* mdlint

* Merge branch 'unstable' into markdown-linter

* mdlint
2024-05-24 02:45:19 +00:00
chonghe
eab3672c6d Add attestation simulator, blobs info and some updates to Lighthouse Book (#5364)
* Apply suggestions from code review

* Revise attestation simulator doc

* Revise blobs.md

* Summary

* Add blobs

* Simulator docs

* Revise attestation simulator

* minor formatting

* Revise vm node

* Update faq

* Update faq

* Add link to v4.6.0

* Remove minification in the docs

* Update Goerli to Holesky

* Add a note on moved vm validator monitor

* Update Rpi 4 note

* Revise attestation simulator doc

* Add docs for attestation simulator

* update database table

* Update faq on resources used

* Fix and update table
2024-03-14 06:12:25 +00:00
chonghe
ab6a6e0741 Some updates in Lighthouse Book (#5000)
* Add jq in api-bn

* Update beaconstate size

* Add fields to web3signer API

* Link web3signer API

* Update /lighthouse/logs in table

* plural

* update slasher doc

* update FAQ

* Add link in validator section

* Add more info on state pruning

* Update database size

* Merge branch 'unstable' into book-update

* Revise Siren for vc to connect bn

* Merge branch 'book-update' of https://github.com/chong-he/lighthouse into book-update

* Corrections to siren faq

* Fix typos

* Update release date for 4.6.0

* Merge branch 'unstable' into book-update
2024-01-31 18:11:54 +00:00
chonghe
cb275e746d Update Lighthouse book FAQ (#4510)
Some updates in the FAQ based on issues seen on Discord. Additionally, corrected the disk usage on the default SPRP as the previously provided value is not correct. 

Co-authored-by: chonghe <44791194+chong-he@users.noreply.github.com>
2023-07-31 23:51:38 +00:00
chonghe
2548be3e66 Minor revision in Lighthouse book (#4385)
## Proposed Changes

Correct some typos in the book, also update information about withdrawals since the Mainnet will be having 700K validators in about a month
2023-06-13 13:12:56 +00:00
chonghe
6c769ed86c Update Lighthouse Book API and Advanced Usage section (#4300)
## Issue Addressed

Update Information in Lighthouse Book

## Proposed Changes

- move Validator Graffiti from Advanced Usage to Validator Management
- update API response and command
- some items that aren't too sure I put it in comment, which can be seen in raw/review format but not live


## Additional Info

Please provide any additional information. For example, future considerations
or information useful for reviewers.


Co-authored-by: chonghe <44791194+chong-he@users.noreply.github.com>
2023-06-02 03:17:36 +00:00
int88
6d8d212da8 use state cache to optimise historical state lookup (#4228)
## Issue Addressed

#3873

## Proposed Changes

add a cache to optimise historical state lookup.

## Additional Info

N/A


Co-authored-by: Michael Sproul <micsproul@gmail.com>
2023-05-05 00:51:57 +00:00
Justin Traglia
e29765e118 Reformat tables and add borders (#3377)
## Proposed Changes

This PR reformats Markdown tables and ensures all tables have borders.
2022-07-27 00:51:07 +00:00
Michael Sproul
375e2b49b3 Conserve disk space by raising default SPRP (#3137)
## Proposed Changes

Increase the default `--slots-per-restore-point` to 8192 for a 4x reduction in freezer DB disk usage.

Existing nodes that use the previous default of 2048 will be left unchanged. Newly synced nodes (with or without checkpoint sync) will use the new 8192 default. 

Long-term we could do away with the freezer DB entirely for validator-only nodes, but this change is much simpler and grants us some extra space in the short term. We can also roll it out gradually across our nodes by purging databases one by one, while keeping the Ansible config the same.

## Additional Info

We ignore a change from 2048 to 8192 if the user hasn't set the 8192 explicitly. We fire a debug log in the case where we do ignore:

```
DEBG Ignoring slots-per-restore-point config in favour of on-disk value, on_disk: 2048, config: 8192
```
2022-04-01 07:16:25 +00:00
Emilia Hane
2aabcaaaed Correct typos book (#3099)
## Issue Addressed

No issue

## Proposed Changes

Correct typos in book

## Additional Info

Nothing to add


Co-authored-by: Emilia Hane <58548332+emhane@users.noreply.github.com>
2022-03-20 22:48:15 +00:00
Michael Sproul
9667dc2f03 Implement checkpoint sync (#2244)
## Issue Addressed

Closes #1891
Closes #1784

## Proposed Changes

Implement checkpoint sync for Lighthouse, enabling it to start from a weak subjectivity checkpoint.

## Additional Info

- [x] Return unavailable status for out-of-range blocks requested by peers (#2561)
- [x] Implement sync daemon for fetching historical blocks (#2561)
- [x] Verify chain hashes (either in `historical_blocks.rs` or the calling module)
- [x] Consistency check for initial block + state
- [x] Fetch the initial state and block from a beacon node HTTP endpoint
- [x] Don't crash fetching beacon states by slot from the API
- [x] Background service for state reconstruction, triggered by CLI flag or API call.

Considered out of scope for this PR:

- Drop the requirement to provide the `--checkpoint-block` (this would require some pretty heavy refactoring of block verification)


Co-authored-by: Diva M <divma@protonmail.com>
2021-09-22 00:37:28 +00:00
Michael Sproul
4632e9ce52 Document the freezer DB space-time trade-off (#808) 2020-01-20 10:01:58 +11:00