gnosis/lighthouse
1
0
Fork 0
mirror of https://github.com/sigp/lighthouse.git synced 2026-04-30 03:03:45 +00:00
Code Issues Packages Projects Releases Wiki Activity

Optimise slasher DB layout and switch to MDBX (#2776)

Browse Source 
## Issue Addressed

Closes #2286
Closes #2538
Closes #2342

## Proposed Changes

Part II of major slasher optimisations after #2767

These changes will be backwards-incompatible due to the move to MDBX (and the schema change) 😱 

* [x] Shrink attester keys from 16 bytes to 7 bytes.
* [x] Shrink attester records from 64 bytes to 6 bytes.
* [x] Separate `DiskConfig` from regular `Config`.
* [x] Add configuration for the LRU cache size.
* [x] Add a "migration" that deletes any legacy LMDB database.
This commit is contained in:
Michael Sproul
2021-12-21 08:23:17 +00:00
parent a290a3c537
commit 3b61ac9cbf
26 changed files with 963 additions and 566 deletions
Download Patch File Download Diff File Expand all files Collapse all files

164
book/src/installation-source.md
View File

@@ -1,85 +1,107 @@
# Installation: Build from Source
# Build from Source
Lighthouse builds on Linux, macOS, and Windows (native Windows support in
BETA, we also support Windows via [WSL][]).
Lighthouse builds on Linux, macOS, and Windows. Install the [Dependencies](#dependencies) using
the instructions below, and then proceed to [Building Lighthouse](#build-lighthouse).
Compilation should be easy. In fact, if you already have Rust and the build
dependencies installed, all you need is:
## Dependencies
- `git clone https://github.com/sigp/lighthouse.git`
- `cd lighthouse`
- `git checkout stable`
- `make`
First, **install Rust** using [rustup](https://rustup.rs/). The rustup installer provides an easy way
to update the Rust compiler, and works on all platforms.
If this doesn't work or is not clear enough, see the [Detailed
Instructions](#detailed-instructions) below. If you have further issues, see
[Troubleshooting](#troubleshooting). If you'd prefer to use Docker, see the
[Docker Guide](./docker.md).
With Rust installed, follow the instructions below to install dependencies relevant to your
operating system.
## Updating lighthouse
#### Ubuntu
Install the following packages:
```bash
sudo apt install -y git gcc g++ make cmake pkg-config llvm-dev libclang-dev clang
```
#### macOS
1. Install the [Homebrew][] package manager.
1. Install CMake using Homebrew:
```
brew install cmake
```
[Homebrew]: https://brew.sh/
#### Windows
1. Install [Git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git).
1. Install the [Chocolatey](https://chocolatey.org/install) package manager for Windows.
1. Install Make, CMake and LLVM using Chocolatey:
```
choco install make
```
```
choco install cmake --installargs 'ADD_CMAKE_TO_PATH=System'
```
```
choco install llvm
```
These dependencies are for compiling Lighthouse natively on Windows, which is currently in beta
testing. Lighthouse can also run successfully under the [Windows Subsystem for Linux (WSL)][WSL].
If using Ubuntu under WSL, you should follow the instructions for Ubuntu listed in the [Dependencies
(Ubuntu)](#ubuntu) section.
[WSL]: https://docs.microsoft.com/en-us/windows/wsl/about
## Build Lighthouse
Once you have Rust and the build dependencies you're ready to build Lighthouse:
```
git clone https://github.com/sigp/lighthouse.git
```
```
cd lighthouse
```
```
git checkout stable
```
```
make
```
Compilation may take around 10 minutes. Installation was successful if `lighthouse --help` displays
the command-line documentation.
If you run into any issues, please check the [Troubleshooting](#troubleshooting) section, or reach
out to us on [Discord](https://discord.gg/cyAszAh).
## Update Lighthouse
You can update Lighthouse to a specific version by running the commands below. The `lighthouse`
directory will be the location you cloned Lighthouse to during the installation process.
`${VERSION}` will be the version you wish to build in the format `vX.X.X`.
- `cd lighthouse`
- `git fetch`
- `git checkout ${VERSION}`
- `make`
## Detailed Instructions
1. Install the build dependencies for your platform
- Check the [Dependencies](#dependencies) section for additional
information.
1. Clone the Lighthouse repository.
- Run `$ git clone https://github.com/sigp/lighthouse.git`
- Change into the newly created directory with `$ cd lighthouse`
1. Build Lighthouse with `$ make`.
1. Installation was successful if `$ lighthouse --help` displays the command-line documentation.
> First time compilation may take several minutes. If you experience any
> failures, please reach out on [discord](https://discord.gg/cyAszAh) or
> [create an issue](https://github.com/sigp/lighthouse/issues/new).
## Dependencies
#### Installing Rust
The best way to install Rust (regardless of platform) is usually with [rustup](https://rustup.rs/)
- Use the `stable` toolchain (it's the default).
#### Windows Support
These instructions are for compiling or running Lighthouse natively on Windows, which is currently in
BETA testing. Lighthouse can also run successfully under the [Windows Subsystem for Linux (WSL)][WSL].
If using Ubuntu under WSL, you should follow the instructions for Ubuntu listed in the
[Dependencies (Ubuntu)](#ubuntu) section.
[WSL]: https://docs.microsoft.com/en-us/windows/wsl/about
1. Install [Git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git)
1. Install [Chocolatey](https://chocolatey.org/install) Package Manager for Windows
- Install `make` via `choco install make`
- Install `cmake` via `choco install cmake --installargs 'ADD_CMAKE_TO_PATH=System'`
#### Ubuntu
Several dependencies may be required to compile Lighthouse. The following
packages may be required in addition a base Ubuntu Server installation:
```bash
sudo apt install -y git gcc g++ make cmake pkg-config
```
cd lighthouse
```
#### macOS
```
git fetch
```
You will need `cmake`. You can install via homebrew:
brew install cmake
```
git checkout ${VERSION}
```
```
make
```
## Troubleshooting
@@ -93,12 +115,12 @@ See ["Configuring the `PATH` environment variable"
### Compilation error
Make sure you are running the latest version of Rust. If you have installed Rust using rustup, simply type `$ rustup update`.
Make sure you are running the latest version of Rust. If you have installed Rust using rustup, simply type `rustup update`.
If compilation fails with `(signal: 9, SIGKILL: kill)`, this could mean your machine ran out of
memory during compilation. If you are on a resource-constrained device you can
look into [cross compilation](./cross-compiling.md).
look into [cross compilation](./cross-compiling.md), or use a [pre-built
binary](./installation-binaries.md).
If compilation fails with `error: linking with cc failed: exit code: 1`, try running `cargo clean`.
[WSL]: https://docs.microsoft.com/en-us/windows/wsl/about

43
book/src/slasher.md
View File

@@ -12,7 +12,6 @@ of the immaturity of the slasher UX and the extra resources required.
* Quad-core CPU
* 16 GB RAM
* 256 GB solid state storage (in addition to space for the beacon node DB)
* ⚠️ **If you are running natively on Windows**: LMDB will pre-allocate the entire 256 GB for the slasher database
## How to Run
@@ -66,24 +65,29 @@ changed after initialization.
* Argument: maximum size of the database in gigabytes
* Default: 256 GB
The slasher uses LMDB as its backing store, and LMDB will consume up to the maximum amount of disk
space allocated to it. By default the limit is set to accomodate the default history length and
around 150K validators but you can set it lower if running with a reduced history length. The space
required scales approximately linearly in validator count and history length, i.e. if you halve
either you can halve the space required.
The slasher uses MDBX as its backing store, which places a hard limit on the size of the database
file. You can use the `--slasher-max-db-size` flag to set this limit. It can be adjusted after
initialization if the limit is reached.
If you want a better estimate you can use this formula:
By default the limit is set to accomodate the default history length and around 300K validators but
you can set it lower if running with a reduced history length. The space required scales
approximately linearly in validator count and history length, i.e. if you halve either you can halve
the space required.
If you want an estimate of the database size you can use this formula:
```
360 * V * N + (16 * V * N)/(C * K) + 15000 * N
4.56 GB * (N / 256) * (V / 250000)
```
where
where `V` is the validator count and `N` is the history length.
* `V` is the validator count
* `N` is the history length
* `C` is the chunk size
* `K` is the validator chunk size
You should set the maximum size higher than the estimate to allow room for growth in the validator
count.
> NOTE: In Lighthouse v2.1.0 the slasher database was switched from LMDB to MDBX. Unlike LMDB, MDBX
> does garbage collection of free pages and is capable of shrinking the database file and preventing
> it from growing indefinitely.
### Update Period
@@ -138,6 +142,19 @@ about [how the slasher works][design-notes], and/or reading the source code.
[design-notes]: https://hackmd.io/@sproul/min-max-slasher
### Attestation Root Cache Size
* Flag: `--slasher-att-cache-size COUNT`
* Argument: number of attestations
* Default: 100,000
The number of attestation data roots to cache in memory. The cache is an LRU cache used to map
indexed attestation IDs to the tree hash roots of their attestation data. The cache prevents reading
whole indexed attestations from disk to determine whether they are slashable.
Each value is very small (38 bytes) so the entire cache should fit in around 4 MB of RAM. Decreasing
the cache size is not recommended, and the size is set so as to be large enough for future growth.
### Short-Range Example
If you would like to run a lightweight slasher that just checks blocks and attestations within