395 Commits

Author SHA1 Message Date
Jimmy Chen
02a88f0704 Add KZG proof and blob validation 2023-03-15 15:15:46 +11:00
Diva M
d93753cc88 Merge branch 'unstable' into off-4844 2023-03-02 15:38:00 -05:00
Divma
047c7544e3 Clean capella (#4019)
## Issue Addressed

Cleans up all the remnants of 4844 in capella. This makes sure when 4844 is reviewed there is nothing we are missing because it got included here 

## Proposed Changes

drop a bomb on every 4844 thing 

## Additional Info

Merge process I did (locally) is as follows:
- squash merge to produce one commit
- in new branch off unstable with the squashed commit create a `git revert HEAD` commit
- merge that new branch onto 4844 with `--strategy ours`
- compare local 4844 to remote 4844 and make sure the diff is empty
- enjoy

Co-authored-by: Paul Hauner <paul@paulhauner.com>
2023-03-01 03:19:02 +00:00
Michael Sproul
0b6850221e Fix Capella schema downgrades (#4004) 2023-02-20 17:50:42 +11:00
Michael Sproul
18c8cab4da Merge remote-tracking branch 'origin/unstable' into capella-merge 2023-02-14 12:07:27 +11:00
realbigsean
28702c9d5d merge upstream, add back get_blobs logic 2023-02-13 16:29:21 -05:00
Michael Sproul
2f456ff9eb Fix regression in DB write atomicity (#3931)
## Issue Addressed

Fix a bug introduced by #3696. The bug is not expected to occur frequently, so releasing this PR is non-urgent.

## Proposed Changes

* Add a variant to `StoreOp` that allows a raw KV operation to be passed around.
* Return to using `self.store.do_atomically` rather than `self.store.hot_db.do_atomically`. This streamlines the write back into a single call and makes our auto-revert work again.
* Prevent `import_block_update_shuffling_cache` from failing block import. This is an outstanding bug from before v3.4.0 which may have contributed to some random unexplained database corruption.

## Additional Info

In #3696 I split the database write into two calls, one to convert the `StoreOp`s to `KeyValueStoreOp`s and one to write them. This had the unfortunate side-effect of damaging our atomicity guarantees in case of a write error. If the first call failed, we would be left with the block in fork choice but not on-disk (or the snapshot cache), which would prevent us from processing any descendant blocks. On `unstable` the first call is very unlikely to fail unless the disk is full, but on `tree-states` the conversion is more involved and a user reported database corruption after it failed in a way that should have been recoverable.

Additionally, as @emhane observed, #3696 also inadvertently removed the import of the new block into the block cache. Although this seems like it could have negatively impacted performance, there are several mitigating factors:

- For regular block processing we should almost always load the parent block (and state) from the snapshot cache.
- We often load blinded blocks, which bypass the block cache anyway.
- Metrics show no noticeable increase in the block cache miss rate with v3.4.0.

However, I expect the block cache _will_ be useful again in `tree-states`, so it is restored to use by this PR.
2023-02-13 03:32:01 +00:00
Emilia Hane
28e9f07746 Fix lint for prune blobs pr 2023-02-10 16:23:04 +01:00
Emilia Hane
6a37e84399 fixup! Fix regression in DB write atomicity 2023-02-08 11:44:46 +01:00
Michael Sproul
ac4b5b580c Fix regression in DB write atomicity 2023-02-08 11:44:45 +01:00
Emilia Hane
9d919917f5 Removed unused code 2023-02-08 11:44:45 +01:00
Emilia Hane
d7eb9441cf Reorder loading of db metadata from disk to allow for future changes to schema 2023-02-08 11:44:45 +01:00
Emilia Hane
d599e41f3d Remove debug comment
Co-authored-by: Michael Sproul <micsproul@gmail.com>
2023-02-08 11:44:44 +01:00
Emilia Hane
577262ccbf Improve use of whitespace
Co-authored-by: Michael Sproul <micsproul@gmail.com>
2023-02-08 11:44:44 +01:00
Emilia Hane
b2abec5d35 Verify StoreConfig 2023-02-08 11:44:44 +01:00
Emilia Hane
00ca21e84c Make implementation of BlobInfo more coder friendly 2023-02-08 11:44:43 +01:00
Emilia Hane
8f137df02e fixup! Allow user to set an epoch margin for pruning 2023-02-08 11:44:43 +01:00
Emilia Hane
a2eda76291 Correct comment 2023-02-08 11:44:43 +01:00
Emilia Hane
9ee9b6df76 Remove unused stuff 2023-02-08 11:44:42 +01:00
Emilia Hane
6dff69bde9 Atomically update blob info with pruned blobs 2023-02-08 11:44:42 +01:00
Emilia Hane
5d2480c762 Improve naming 2023-02-08 11:44:42 +01:00
Emilia Hane
43c3c74a48 fixup! Fix blobs store bug 2023-02-08 11:44:41 +01:00
Emilia Hane
c50f83116e Fix wording
Co-authored-by: Michael Sproul <micsproul@gmail.com>
2023-02-08 11:44:41 +01:00
Emilia Hane
f6346f89c1 Clarify comment
Co-authored-by: Michael Sproul <micsproul@gmail.com>
2023-02-08 11:44:41 +01:00
Emilia Hane
e4b447395a Clarify wording
Co-authored-by: Michael Sproul <micsproul@gmail.com>
2023-02-08 11:44:40 +01:00
Emilia Hane
756c881857 Keep uniform size small keys
Co-authored-by: Michael Sproul <micsproul@gmail.com>
2023-02-08 11:44:40 +01:00
Emilia Hane
4de523fb75 fixup! Allow user to set an epoch margin for pruning 2023-02-08 11:44:40 +01:00
Emilia Hane
1812301c9c Allow user to set an epoch margin for pruning 2023-02-08 11:44:40 +01:00
Emilia Hane
d7fc24a9d5 Plug in running blob pruning in migrator, related bug fixes and add todos 2023-02-08 11:44:40 +01:00
Emilia Hane
a875bec5f2 Fix blobs store bug 2023-02-08 11:44:39 +01:00
Emilia Hane
3bede06c9b Fix typo 2023-02-08 11:44:38 +01:00
Emilia Hane
54699f808c fixup! Clarify hybrid blob prune solution and fix error handling 2023-02-08 11:44:38 +01:00
Emilia Hane
83a9520761 Clarify hybrid blob prune solution and fix error handling 2023-02-08 11:44:38 +01:00
Emilia Hane
3d93dad0e2 Fix type bug
Co-authored-by: realbigsean <seananderson33@GMAIL.com>
2023-02-08 11:44:37 +01:00
Emilia Hane
44ec331452 fixup! Simplify conceptual design 2023-02-08 11:44:37 +01:00
Emilia Hane
20567750c1 fixup! Simplify conceptual design 2023-02-08 11:44:37 +01:00
Emilia Hane
7103a257ce Simplify conceptual design 2023-02-08 11:44:37 +01:00
Emilia Hane
0d13932663 Fix epoch constructor misconception 2023-02-08 11:44:37 +01:00
Emilia Hane
b5abfe620a Convert epochs_per_blob_prune to Epoch once 2023-02-08 11:44:36 +01:00
Emilia Hane
d58a30b3de fixup! Store orphan block roots 2023-02-08 11:44:36 +01:00
Emilia Hane
6346c30158 Enable skipping blob pruning at each epoch 2023-02-08 11:44:35 +01:00
Emilia Hane
8752deeced Store orphan block roots 2023-02-08 11:44:35 +01:00
Emilia Hane
94aa2cef67 Log info loaded from disk 2023-02-08 11:44:34 +01:00
Emilia Hane
a2b8c6ee69 Save fetching state for blobs pruning 2023-02-08 11:44:33 +01:00
Emilia Hane
6f5ca02ac9 Improve syntax
Co-authored-by: Michael Sproul <micsproul@gmail.com>
2023-02-08 11:44:33 +01:00
Emilia Hane
667cca5cf2 Fix try_prune_blobs to use state root 2023-02-08 11:44:33 +01:00
Emilia Hane
d67468d737 Prune blobs on migration in addition to start-up 2023-02-08 11:44:32 +01:00
Emilia Hane
ce2db355de Fix rebase conflict 2023-02-08 11:44:32 +01:00
Emilia Hane
a211e6afee Fix rebase conflict 2023-02-08 11:44:31 +01:00
Emilia Hane
28e1e635c3 Fix rebase conflict 2023-02-08 11:44:31 +01:00