Michael Sproul
|
836c39efaa
|
Shrink persisted fork choice data (#7805)
Closes:
- https://github.com/sigp/lighthouse/issues/7760
- [x] Remove `balances_cache` from `PersistedForkChoiceStore` (~65 MB saving on mainnet)
- [x] Remove `justified_balances` from `PersistedForkChoiceStore` (~16 MB saving on mainnet)
- [x] Remove `balances` from `ProtoArray`/`SszContainer`.
- [x] Implement zstd compression for votes
- [x] Fix bug in justified state usage
- [x] Bump schema version to V28 and implement migration.
|
2025-08-18 06:03:28 +00:00 |
|
chonghe
|
522bd9e9c6
|
Update Rust Edition to 2024 (#7766)
* #7749
Thanks @dknopik and @michaelsproul for your help!
|
2025-08-13 03:04:31 +00:00 |
|
Lion - dapplion
|
dd98534158
|
Hierarchical state diffs in hot DB (#6750)
This PR implements https://github.com/sigp/lighthouse/pull/5978 (tree-states) but on the hot DB. It allows Lighthouse to massively reduce its disk footprint during non-finality and overall I/O in all cases.
Closes https://github.com/sigp/lighthouse/issues/6580
Conga into https://github.com/sigp/lighthouse/pull/6744
### TODOs
- [x] Fix OOM in CI https://github.com/sigp/lighthouse/pull/7176
- [x] optimise store_hot_state to avoid storing a duplicate state if the summary already exists (should be safe from races now that pruning is cleaner)
- [x] mispelled: get_ancenstor_state_root
- [x] get_ancestor_state_root should use state summaries
- [x] Prevent split from changing during ancestor calc
- [x] Use same hierarchy for hot and cold
### TODO Good optimization for future PRs
- [ ] On the migration, if the latest hot snapshot is aligned with the cold snapshot migrate the diffs instead of the full states.
```
align slot time
10485760 Nov-26-2024
12582912 Sep-14-2025
14680064 Jul-02-2026
```
### TODO Maybe things good to have
- [ ] Rename anchor_slot https://github.com/sigp/lighthouse/compare/tree-states-hot-rebase-oom...dapplion:lighthouse:tree-states-hot-anchor-slot-rename?expand=1
- [ ] Make anchor fields not public such that they must be mutated through a method. To prevent un-wanted changes of the anchor_slot
### NOTTODO
- [ ] Use fork-choice and a new method [`descendants_of_checkpoint`](ca2388e196 (diff-046fbdb517ca16b80e4464c2c824cf001a74a0a94ac0065e635768ac391062a8)) to filter only the state summaries that descend of finalized checkpoint]
|
2025-06-19 02:43:25 +00:00 |
|
Michael Sproul
|
7d792e615c
|
Fix xdelta3 output buffer issue (#7174)
* Fix xdelta3 output buffer issue
* Fix buckets
* Update commit hash to `main`
* Tag TODO(hdiff)
* Update cargo lock
|
2025-03-27 13:25:50 +11:00 |
|
Michael Sproul
|
9fdd53df56
|
Hierarchical state diffs (#5978)
* Start extracting freezer changes for tree-states
* Remove unused config args
* Add comments
* Remove unwraps
* Subjective more clear implementation
* Clean up hdiff
* Update xdelta3
* Tree states archive metrics (#6040)
* Add store cache size metrics
* Add compress timer metrics
* Add diff apply compute timer metrics
* Add diff buffer cache hit metrics
* Add hdiff buffer load times
* Add blocks replayed metric
* Move metrics to store
* Future proof some metrics
---------
Co-authored-by: Michael Sproul <michael@sigmaprime.io>
* Port and clean up forwards iterator changes
* Add and polish hierarchy-config flag
* Merge remote-tracking branch 'origin/unstable' into tree-states-archive
* Cleaner errors
* Fix beacon_chain test compilation
* Merge remote-tracking branch 'origin/unstable' into tree-states-archive
* Patch a few more freezer block roots
* Fix genesis block root bug
* Fix test failing due to pending updates
* Beacon chain tests passing
* Merge remote-tracking branch 'origin/unstable' into tree-states-archive
* Merge remote-tracking branch 'origin/unstable' into tree-states-archive
* Fix doc lint
* Implement DB schema upgrade for hierarchical state diffs (#6193)
* DB upgrade
* Add flag
* Delete RestorePointHash
* Update docs
* Update docs
* Implement hierarchical state diffs config migration (#6245)
* Implement hierarchical state diffs config migration
* Review PR
* Remove TODO
* Set CURRENT_SCHEMA_VERSION correctly
* Fix genesis state loading
* Re-delete some PartialBeaconState stuff
---------
Co-authored-by: Michael Sproul <michael@sigmaprime.io>
* Merge remote-tracking branch 'origin/unstable' into tree-states-archive
* Fix test compilation
* Update schema downgrade test
* Fix tests
* Fix null anchor migration
* Merge remote-tracking branch 'origin/unstable' into tree-states-archive
* Fix tree states upgrade migration (#6328)
* Towards crash safety
* Fix compilation
* Move cold summaries and state roots to new columns
* Rename StateRoots chunked field
* Update prune states
* Clean hdiff CLI flag and metrics
* Fix "staged reconstruction"
* Merge remote-tracking branch 'origin/unstable' into tree-states-archive
* Fix alloy issues
* Fix staged reconstruction logic
* Prevent weird slot drift
* Remove "allow" flag
* Update CLI help
* Remove FIXME about downgrade
* Merge remote-tracking branch 'origin/unstable' into tree-states-archive
* Remove some unnecessary error variants
* Fix new test
* Tree states archive - review comments and metrics (#6386)
* Review PR comments and metrics
* Comments
* Add anchor metrics
* drop prev comment
* Update metadata.rs
* Apply suggestions from code review
---------
Co-authored-by: Michael Sproul <micsproul@gmail.com>
* Update beacon_node/store/src/hot_cold_store.rs
Co-authored-by: Lion - dapplion <35266934+dapplion@users.noreply.github.com>
* Merge remote-tracking branch 'origin/unstable' into tree-states-archive
* Clarify comment and remove anchor_slot garbage
* Simplify database anchor (#6397)
* Simplify database anchor
* Update beacon_node/store/src/reconstruct.rs
* Add migration for anchor
* Fix and simplify light_client store tests
* Fix incompatible config test
* Merge remote-tracking branch 'origin/unstable' into tree-states-archive
* Merge remote-tracking branch 'origin/unstable' into tree-states-archive
* More metrics
* Merge remote-tracking branch 'origin/unstable' into tree-states-archive
* New historic state cache (#6475)
* New historic state cache
* Add more metrics
* State cache hit rate metrics
* Fix store metrics
* More logs and metrics
* Fix logger
* Ensure cached states have built caches :O
* Replay blocks in preference to diffing
* Two separate caches
* Distribute cache build time to next slot
* Re-plumb historic-state-cache flag
* Clean up metrics
* Update book
* Update beacon_node/store/src/hdiff.rs
Co-authored-by: Lion - dapplion <35266934+dapplion@users.noreply.github.com>
* Update beacon_node/store/src/historic_state_cache.rs
Co-authored-by: Lion - dapplion <35266934+dapplion@users.noreply.github.com>
---------
Co-authored-by: Lion - dapplion <35266934+dapplion@users.noreply.github.com>
* Update database docs
* Update diagram
* Merge remote-tracking branch 'origin/unstable' into tree-states-archive
* Update lockbud to work with bindgen/etc
* Correct pkg name for Debian
* Remove vestigial epochs_per_state_diff
* Merge remote-tracking branch 'origin/unstable' into tree-states-archive
* Markdown lint
* Merge remote-tracking branch 'origin/unstable' into tree-states-archive
* Address Jimmy's review comments
* Simplify ReplayFrom case
* Fix and document genesis_state_root
* Typo
Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>
* Merge branch 'unstable' into tree-states-archive
* Compute diff of validators list manually (#6556)
* Split hdiff computation
* Dedicated logic for historical roots and summaries
* Benchmark against real states
* Mutated source?
* Version the hdiff
* Add lighthouse DB config for hierarchy exponents
* Tidy up hierarchy exponents flag
* Apply suggestions from code review
Co-authored-by: Michael Sproul <micsproul@gmail.com>
* Address PR review
* Remove hardcoded paths in benchmarks
* Delete unused function in benches
* lint
---------
Co-authored-by: Michael Sproul <michael@sigmaprime.io>
* Test hdiff binary format stability (#6585)
* Merge remote-tracking branch 'origin/unstable' into tree-states-archive
* Add deprecation warning for SPRP
* Update xdelta to get rid of duplicate deps
* Document test
|
2024-11-18 01:51:44 +00:00 |
|