99e53b88c3
* Remove use of ethers_core::RlpStream
* Merge branch 'unstable' of https://github.com/sigp/lighthouse into remove_use_of_ethers_core
* Remove old code
* Simplify keccak call
* Remove unused package
* Merge branch 'unstable' of https://github.com/ethDreamer/lighthouse into remove_use_of_ethers_core
* Merge branch 'unstable' into remove_use_of_ethers_core
* Run clippy
* Merge branch 'remove_use_of_ethers_core' of https://github.com/dospore/lighthouse into remove_use_of_ethers_core
* Check all cargo fmt
* migrate to alloy primitives init
* fix deps
* integrate alloy-primitives
* resolve dep issues
* more changes based on dep changes
* add TODOs
* Merge branch 'unstable' of https://github.com/sigp/lighthouse into remove_use_of_ethers_core
* Revert lock
* Add BeaconBlocksByRange v3
* continue migration
* Revert "Add BeaconBlocksByRange v3"
This reverts commit e3ce7fc5ea.
* impl hash256 extended trait
* revert some uneeded diffs
* merge conflict resolved
* fix subnet id rshift calc
* rename to FixedBytesExtended
* debugging
* Merge branch 'unstable' of https://github.com/sigp/lighthouse into migrate-to-alloy-primitives
* fix failed test
* fixing more tests
* Merge branch 'unstable' of https://github.com/sigp/lighthouse into remove_use_of_ethers_core
* introduce a shim to convert between the two u256 types
* move alloy to wrokspace
* align alloy versions
* update
* update web3signer test certs
* refactor
* resolve failing tests
* linting
* fix graffiti string test
* fmt
* fix ef test
* resolve merge conflicts
* remove udep and revert cert
* cargo patch
* cyclic dep
* fix build error
* Merge branch 'unstable' of https://github.com/sigp/lighthouse into migrate-to-alloy-primitives
* resolve conflicts, update deps
* merge unstable
* fmt
* fix deps
* Merge branch 'unstable' of https://github.com/sigp/lighthouse into migrate-to-alloy-primitives
* resolve merge conflicts
* resolve conflicts, make necessary changes
* Remove patch
* fmt
* remove file
* merge conflicts
* sneaking in a smol change
* bump versions
* Merge remote-tracking branch 'origin/unstable' into migrate-to-alloy-primitives
* Updates for peerDAS
* Update ethereum_hashing to prevent dupe
* updated alloy-consensus, removed TODOs
* cargo update
* endianess fix
* Merge branch 'unstable' of https://github.com/sigp/lighthouse into migrate-to-alloy-primitives
* fmt
* fix merge
* fix test
* fixed_bytes crate
* minor fixes
* convert u256 to i64
* panic free mixin to_low_u64_le
* from_str_radix
* computbe_subnet api and ensuring we use big-endian
* Merge branch 'unstable' of https://github.com/sigp/lighthouse into migrate-to-alloy-primitives
* fix test
* Simplify subnet_id test
* Simplify some more tests
* Add tests to fixed_bytes crate
* Merge branch 'unstable' into migrate-to-alloy-primitives
Database Schema Migrations
This document is an attempt to record some best practices and design conventions for applying database schema migrations within Lighthouse.
General Structure
If you make a breaking change to an on-disk data structure you need to increment the
SCHEMA_VERSION in beacon_node/store/src/metadata.rs and add a migration from the previous
version to the new version.
The entry-point for database migrations is in schema_change.rs, not migrate.rs (which deals
with finalization). Supporting code for a specific migration may be added in
schema_change/migration_schema_vX.rs, where X is the version being migrated to.
Combining Schema Changes
Schema changes may be combined if they are part of the same pull request to
unstable. Once a schema version is defined in unstable we should not apply changes to it
without incrementing the version. This prevents conflicts between versions that appear to be the
same. This allows us to deploy unstable to nodes without having to worry about needing to resync
because of a sneaky schema change.
Changing the on-disk structure for a version before it is merged to unstable is OK. You will
just have to handle manually resyncing any test nodes (use checkpoint sync).
Naming Conventions
Prefer to name versions of structs by the version at which the change was introduced. For example
if you add a field to Foo in v9, call the previous version FooV1 (assuming this is Foo's first
migration) and write a schema change that migrates from FooV1 to FooV9.
Prefer to use explicit version names in schema_change.rs and the schema_change module. To
interface with the outside either:
- Define a type alias to the latest version, e.g.
pub type Foo = FooV9, or - Define a mapping from the latest version to the version used elsewhere, e.g.
impl From<FooV9> for Foo {}
Avoid names like:
LegacyFooOldFooFooWithoutX
First-version vs Last-version
Previously the schema migration code would name types by the last version at which they were
valid. For example if Foo changed in V9 then we would name the two variants FooV8 and FooV9.
The problem with this scheme is that if Foo changes again in the future at say v12 then FooV9 would
need to be renamed to FooV11, which is annoying. Using the first valid version as described
above does not have this issue.
Using SuperStruct
If possible, consider using superstruct to handle data
structure changes between versions.
- Use
superstruct(no_enum)to avoid generating an unnecessary top-level enum.
Example
A field is added to Foo in v9, and there are two variants: FooV1 and FooV9. There is a
migration from FooV1 to FooV9. Foo is aliased to FooV9.
Some time later another field is added to Foo in v12. A new FooV12 is created, along with a
migration from FooV9 to FooV12. The primary Foo type gets re-aliased to FooV12. The previous
migration from V1 to V9 shouldn't break because the schema migration refers to FooV9 explicitly
rather than Foo. Due to the re-aliasing (or re-mapping) the compiler will check every usage
of Foo to make sure that it still makes sense with FooV12.