mirror of
https://github.com/sigp/lighthouse.git
synced 2026-05-07 16:55:46 +00:00
Add beacon.watch (#3362)
> This is currently a WIP and all features are subject to alteration or removal at any time. ## Overview The successor to #2873. Contains the backbone of `beacon.watch` including syncing code, the initial API, and several core database tables. See `watch/README.md` for more information, requirements and usage.
This commit is contained in:
Mac L
460
watch/README.md
Normal file
460
watch/README.md
Normal file
@@ -0,0 +1,460 @@
|
||||
## beacon.watch
|
||||
|
||||
>beacon.watch is pre-MVP and still under active development and subject to change.
|
||||
|
||||
beacon.watch is an Ethereum Beacon Chain monitoring platform whose goal is to provide fast access to
|
||||
data which is:
|
||||
1. Not already stored natively in the Beacon Chain
|
||||
2. Too specialized for Block Explorers
|
||||
3. Too sensitive for public Block Explorers
|
||||
|
||||
|
||||
### Requirements
|
||||
- `git`
|
||||
- `rust` : https://rustup.rs/
|
||||
- `libpg` : https://www.postgresql.org/download/
|
||||
- `diesel_cli` :
|
||||
```
|
||||
cargo install diesel_cli --no-default-features --features postgres
|
||||
```
|
||||
- `docker` : https://docs.docker.com/engine/install/
|
||||
- `docker-compose` : https://docs.docker.com/compose/install/
|
||||
|
||||
### Setup
|
||||
1. Setup the database:
|
||||
```
|
||||
cd postgres_docker_compose
|
||||
docker-compose up
|
||||
```
|
||||
|
||||
1. Ensure the tests pass:
|
||||
```
|
||||
cargo test --release
|
||||
```
|
||||
|
||||
1. Drop the database (if it already exists) and run the required migrations:
|
||||
```
|
||||
diesel database reset --database-url postgres://postgres:postgres@localhost/dev
|
||||
```
|
||||
|
||||
1. Ensure a synced Lighthouse beacon node with historical states is available
|
||||
at `localhost:5052`.
|
||||
The smaller the value of `--slots-per-restore-point` the faster beacon.watch
|
||||
will be able to sync to the beacon node.
|
||||
|
||||
1. Run the updater daemon:
|
||||
```
|
||||
cargo run --release -- run-updater
|
||||
```
|
||||
|
||||
1. Start the HTTP API server:
|
||||
```
|
||||
cargo run --release -- serve
|
||||
```
|
||||
|
||||
1. Ensure connectivity:
|
||||
```
|
||||
curl "http://localhost:5059/v1/slots/highest"
|
||||
```
|
||||
|
||||
> Functionality on MacOS has not been tested. Windows is not supported.
|
||||
|
||||
|
||||
### Configuration
|
||||
beacon.watch can be configured through the use of a config file.
|
||||
Available options can be seen in `config.yaml.default`.
|
||||
|
||||
You can specify a config file during runtime:
|
||||
```
|
||||
cargo run -- run-updater --config path/to/config.yaml
|
||||
cargo run -- serve --config path/to/config.yaml
|
||||
```
|
||||
|
||||
You can specify only the parts of the config file which you need changed.
|
||||
Missing values will remain as their defaults.
|
||||
|
||||
For example, if you wish to run with default settings but only wish to alter `log_level`
|
||||
your config file would be:
|
||||
```yaml
|
||||
# config.yaml
|
||||
log_level = "info"
|
||||
```
|
||||
|
||||
### Available Endpoints
|
||||
As beacon.watch continues to develop, more endpoints will be added.
|
||||
|
||||
> In these examples any data containing information from blockprint has either been redacted or fabricated.
|
||||
|
||||
#### `/v1/slots/{slot}`
|
||||
```bash
|
||||
curl "http://localhost:5059/v1/slots/4635296"
|
||||
```
|
||||
```json
|
||||
{
|
||||
"slot": "4635296",
|
||||
"root": "0xf7063a9d6c663682e59bd0b41d29ce80c3ff0b089049ff8676d6f9ee79622c62",
|
||||
"skipped": false,
|
||||
"beacon_block": "0xf7063a9d6c663682e59bd0b41d29ce80c3ff0b089049ff8676d6f9ee79622c62"
|
||||
}
|
||||
```
|
||||
|
||||
#### `/v1/slots?start_slot={}&end_slot={}`
|
||||
```bash
|
||||
curl "http://localhost:5059/v1/slots?start_slot=4635296&end_slot=4635297"
|
||||
```
|
||||
```json
|
||||
[
|
||||
{
|
||||
"slot": "4635297",
|
||||
"root": "0x04ad2e963811207e344bebeba5b1217805bcc3a9e2ed9fcf2205d491778c6182",
|
||||
"skipped": false,
|
||||
"beacon_block": "0x04ad2e963811207e344bebeba5b1217805bcc3a9e2ed9fcf2205d491778c6182"
|
||||
},
|
||||
{
|
||||
"slot": "4635296",
|
||||
"root": "0xf7063a9d6c663682e59bd0b41d29ce80c3ff0b089049ff8676d6f9ee79622c62",
|
||||
"skipped": false,
|
||||
"beacon_block": "0xf7063a9d6c663682e59bd0b41d29ce80c3ff0b089049ff8676d6f9ee79622c62"
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
#### `/v1/slots/lowest`
|
||||
```bash
|
||||
curl "http://localhost:5059/v1/slots/lowest"
|
||||
```
|
||||
```json
|
||||
{
|
||||
"slot": "4635296",
|
||||
"root": "0xf7063a9d6c663682e59bd0b41d29ce80c3ff0b089049ff8676d6f9ee79622c62",
|
||||
"skipped": false,
|
||||
"beacon_block": "0xf7063a9d6c663682e59bd0b41d29ce80c3ff0b089049ff8676d6f9ee79622c62"
|
||||
}
|
||||
```
|
||||
|
||||
#### `/v1/slots/highest`
|
||||
```bash
|
||||
curl "http://localhost:5059/v1/slots/highest"
|
||||
```
|
||||
```json
|
||||
{
|
||||
"slot": "4635358",
|
||||
"root": "0xe9eff13560688f1bf15cf07b60c84963d4d04a4a885ed0eb19ceb8450011894b",
|
||||
"skipped": false,
|
||||
"beacon_block": "0xe9eff13560688f1bf15cf07b60c84963d4d04a4a885ed0eb19ceb8450011894b"
|
||||
}
|
||||
```
|
||||
|
||||
#### `v1/slots/{slot}/block`
|
||||
```bash
|
||||
curl "http://localhost:5059/v1/slots/4635296/block"
|
||||
```
|
||||
```json
|
||||
{
|
||||
"slot": "4635296",
|
||||
"root": "0xf7063a9d6c663682e59bd0b41d29ce80c3ff0b089049ff8676d6f9ee79622c62",
|
||||
"parent_root": "0x7c4860b420a23de9d126da71f9043b3744af98c847efd9e1440f2da8fbf7f31b"
|
||||
}
|
||||
```
|
||||
|
||||
#### `/v1/blocks/{block_id}`
|
||||
```bash
|
||||
curl "http://localhost:5059/v1/blocks/4635296"
|
||||
# OR
|
||||
curl "http://localhost:5059/v1/blocks/0xf7063a9d6c663682e59bd0b41d29ce80c3ff0b089049ff8676d6f9ee79622c62"
|
||||
```
|
||||
```json
|
||||
{
|
||||
"slot": "4635296",
|
||||
"root": "0xf7063a9d6c663682e59bd0b41d29ce80c3ff0b089049ff8676d6f9ee79622c62",
|
||||
"parent_root": "0x7c4860b420a23de9d126da71f9043b3744af98c847efd9e1440f2da8fbf7f31b"
|
||||
}
|
||||
```
|
||||
|
||||
#### `/v1/blocks?start_slot={}&end_slot={}`
|
||||
```bash
|
||||
curl "http://localhost:5059/v1/blocks?start_slot=4635296&end_slot=4635297"
|
||||
```
|
||||
```json
|
||||
[
|
||||
{
|
||||
"slot": "4635297",
|
||||
"root": "0x04ad2e963811207e344bebeba5b1217805bcc3a9e2ed9fcf2205d491778c6182",
|
||||
"parent_root": "0xf7063a9d6c663682e59bd0b41d29ce80c3ff0b089049ff8676d6f9ee79622c62"
|
||||
},
|
||||
{
|
||||
"slot": "4635296",
|
||||
"root": "0xf7063a9d6c663682e59bd0b41d29ce80c3ff0b089049ff8676d6f9ee79622c62",
|
||||
"parent_root": "0x7c4860b420a23de9d126da71f9043b3744af98c847efd9e1440f2da8fbf7f31b"
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
#### `/v1/blocks/{block_id}/previous`
|
||||
```bash
|
||||
curl "http://localhost:5059/v1/blocks/4635297/previous"
|
||||
# OR
|
||||
curl "http://localhost:5059/v1/blocks/0x04ad2e963811207e344bebeba5b1217805bcc3a9e2ed9fcf2205d491778c6182/previous"
|
||||
```
|
||||
```json
|
||||
{
|
||||
"slot": "4635296",
|
||||
"root": "0xf7063a9d6c663682e59bd0b41d29ce80c3ff0b089049ff8676d6f9ee79622c62",
|
||||
"parent_root": "0x7c4860b420a23de9d126da71f9043b3744af98c847efd9e1440f2da8fbf7f31b"
|
||||
}
|
||||
```
|
||||
|
||||
#### `/v1/blocks/{block_id}/next`
|
||||
```bash
|
||||
curl "http://localhost:5059/v1/blocks/4635296/next"
|
||||
# OR
|
||||
curl "http://localhost:5059/v1/blocks/0xf7063a9d6c663682e59bd0b41d29ce80c3ff0b089049ff8676d6f9ee79622c62/next"
|
||||
```
|
||||
```json
|
||||
{
|
||||
"slot": "4635297",
|
||||
"root": "0x04ad2e963811207e344bebeba5b1217805bcc3a9e2ed9fcf2205d491778c6182",
|
||||
"parent_root": "0xf7063a9d6c663682e59bd0b41d29ce80c3ff0b089049ff8676d6f9ee79622c62"
|
||||
}
|
||||
```
|
||||
|
||||
#### `/v1/blocks/lowest`
|
||||
```bash
|
||||
curl "http://localhost:5059/v1/blocks/lowest"
|
||||
```
|
||||
```json
|
||||
{
|
||||
"slot": "4635296",
|
||||
"root": "0xf7063a9d6c663682e59bd0b41d29ce80c3ff0b089049ff8676d6f9ee79622c62",
|
||||
"parent_root": "0x7c4860b420a23de9d126da71f9043b3744af98c847efd9e1440f2da8fbf7f31b"
|
||||
}
|
||||
```
|
||||
|
||||
#### `/v1/blocks/highest`
|
||||
```bash
|
||||
curl "http://localhost:5059/v1/blocks/highest"
|
||||
```
|
||||
```json
|
||||
{
|
||||
"slot": "4635358",
|
||||
"root": "0xe9eff13560688f1bf15cf07b60c84963d4d04a4a885ed0eb19ceb8450011894b",
|
||||
"parent_root": "0xb66e05418bb5b1d4a965c994e1f0e5b5f0d7b780e0df12f3f6321510654fa1d2"
|
||||
}
|
||||
```
|
||||
|
||||
#### `/v1/blocks/{block_id}/proposer`
|
||||
```bash
|
||||
curl "http://localhost:5059/v1/blocks/4635296/proposer"
|
||||
# OR
|
||||
curl "http://localhost:5059/v1/blocks/0xf7063a9d6c663682e59bd0b41d29ce80c3ff0b089049ff8676d6f9ee79622c62/proposer"
|
||||
|
||||
```
|
||||
```json
|
||||
{
|
||||
"slot": "4635296",
|
||||
"proposer_index": 223126,
|
||||
"graffiti": ""
|
||||
}
|
||||
```
|
||||
|
||||
#### `/v1/blocks/{block_id}/rewards`
|
||||
```bash
|
||||
curl "http://localhost:5059/v1/blocks/4635296/reward"
|
||||
# OR
|
||||
curl "http://localhost:5059/v1/blocks/0xf7063a9d6c663682e59bd0b41d29ce80c3ff0b089049ff8676d6f9ee79622c62/reward"
|
||||
|
||||
```
|
||||
```json
|
||||
{
|
||||
"slot": "4635296",
|
||||
"total": 25380059,
|
||||
"attestation_reward": 24351867,
|
||||
"sync_committee_reward": 1028192
|
||||
}
|
||||
```
|
||||
|
||||
#### `/v1/blocks/{block_id}/packing`
|
||||
```bash
|
||||
curl "http://localhost:5059/v1/blocks/4635296/packing"
|
||||
# OR
|
||||
curl "http://localhost:5059/v1/blocks/0xf7063a9d6c663682e59bd0b41d29ce80c3ff0b089049ff8676d6f9ee79622c62/packing"
|
||||
|
||||
```
|
||||
```json
|
||||
{
|
||||
"slot": "4635296",
|
||||
"available": 16152,
|
||||
"included": 13101,
|
||||
"prior_skip_slots": 0
|
||||
}
|
||||
```
|
||||
|
||||
#### `/v1/validators/{validator}`
|
||||
```bash
|
||||
curl "http://localhost:5059/v1/validators/1"
|
||||
# OR
|
||||
curl "http://localhost:5059/v1/validators/0xa1d1ad0714035353258038e964ae9675dc0252ee22cea896825c01458e1807bfad2f9969338798548d9858a571f7425c"
|
||||
```
|
||||
```json
|
||||
{
|
||||
"index": 1,
|
||||
"public_key": "0xa1d1ad0714035353258038e964ae9675dc0252ee22cea896825c01458e1807bfad2f9969338798548d9858a571f7425c",
|
||||
"status": "active_ongoing",
|
||||
"client": null,
|
||||
"activation_epoch": 0,
|
||||
"exit_epoch": null
|
||||
}
|
||||
```
|
||||
|
||||
#### `/v1/validators/{validator}/attestation/{epoch}`
|
||||
```bash
|
||||
curl "http://localhost:5059/v1/validators/1/attestation/144853"
|
||||
# OR
|
||||
curl "http://localhost:5059/v1/validators/0xa1d1ad0714035353258038e964ae9675dc0252ee22cea896825c01458e1807bfad2f9969338798548d9858a571f7425c/attestation/144853"
|
||||
```
|
||||
```json
|
||||
{
|
||||
"index": 1,
|
||||
"epoch": "144853",
|
||||
"source": true,
|
||||
"head": true,
|
||||
"target": true
|
||||
}
|
||||
```
|
||||
|
||||
#### `/v1/validators/missed/{vote}/{epoch}`
|
||||
```bash
|
||||
curl "http://localhost:5059/v1/validators/missed/head/144853"
|
||||
```
|
||||
```json
|
||||
[
|
||||
63,
|
||||
67,
|
||||
98,
|
||||
...
|
||||
]
|
||||
```
|
||||
|
||||
#### `/v1/validators/missed/{vote}/{epoch}/graffiti`
|
||||
```bash
|
||||
curl "http://localhost:5059/v1/validators/missed/head/144853/graffiti"
|
||||
```
|
||||
```json
|
||||
{
|
||||
"Mr F was here": 3,
|
||||
"Lighthouse/v3.1.0-aa022f4": 5,
|
||||
...
|
||||
}
|
||||
```
|
||||
|
||||
#### `/v1/clients/missed/{vote}/{epoch}`
|
||||
```bash
|
||||
curl "http://localhost:5059/v1/clients/missed/source/144853"
|
||||
```
|
||||
```json
|
||||
{
|
||||
"Lighthouse": 100,
|
||||
"Lodestar": 100,
|
||||
"Nimbus": 100,
|
||||
"Prysm": 100,
|
||||
"Teku": 100,
|
||||
"Unknown": 100
|
||||
}
|
||||
```
|
||||
|
||||
#### `/v1/clients/missed/{vote}/{epoch}/percentages`
|
||||
Note that this endpoint expresses the following:
|
||||
```
|
||||
What percentage of each client implementation missed this vote?
|
||||
```
|
||||
|
||||
```bash
|
||||
curl "http://localhost:5059/v1/clients/missed/target/144853/percentages"
|
||||
```
|
||||
```json
|
||||
{
|
||||
"Lighthouse": 0.51234567890,
|
||||
"Lodestar": 0.51234567890,
|
||||
"Nimbus": 0.51234567890,
|
||||
"Prysm": 0.09876543210,
|
||||
"Teku": 0.09876543210,
|
||||
"Unknown": 0.05647382910
|
||||
}
|
||||
```
|
||||
|
||||
#### `/v1/clients/missed/{vote}/{epoch}/percentages/relative`
|
||||
Note that this endpoint expresses the following:
|
||||
```
|
||||
For the validators which did miss this vote, what percentage of them were from each client implementation?
|
||||
```
|
||||
You can check these values against the output of `/v1/clients/percentages` to see any discrepancies.
|
||||
|
||||
```bash
|
||||
curl "http://localhost:5059/v1/clients/missed/target/144853/percentages/relative"
|
||||
```
|
||||
```json
|
||||
{
|
||||
"Lighthouse": 11.11111111111111,
|
||||
"Lodestar": 11.11111111111111,
|
||||
"Nimbus": 11.11111111111111,
|
||||
"Prysm": 16.66666666666667,
|
||||
"Teku": 16.66666666666667,
|
||||
"Unknown": 33.33333333333333
|
||||
}
|
||||
|
||||
```
|
||||
|
||||
#### `/v1/clients`
|
||||
```bash
|
||||
curl "http://localhost:5059/v1/clients"
|
||||
```
|
||||
```json
|
||||
{
|
||||
"Lighthouse": 5000,
|
||||
"Lodestar": 5000,
|
||||
"Nimbus": 5000,
|
||||
"Prysm": 5000,
|
||||
"Teku": 5000,
|
||||
"Unknown": 5000
|
||||
}
|
||||
```
|
||||
|
||||
#### `/v1/clients/percentages`
|
||||
```bash
|
||||
curl "http://localhost:5059/v1/clients/percentages"
|
||||
```
|
||||
```json
|
||||
{
|
||||
"Lighthouse": 16.66666666666667,
|
||||
"Lodestar": 16.66666666666667,
|
||||
"Nimbus": 16.66666666666667,
|
||||
"Prysm": 16.66666666666667,
|
||||
"Teku": 16.66666666666667,
|
||||
"Unknown": 16.66666666666667
|
||||
}
|
||||
```
|
||||
|
||||
### Future work
|
||||
- New tables
|
||||
- `skip_slots`?
|
||||
|
||||
|
||||
- More API endpoints
|
||||
- `/v1/proposers?start_epoch={}&end_epoch={}` and similar
|
||||
- `/v1/validators/{status}/count`
|
||||
|
||||
|
||||
- Concurrently backfill and forwards fill, so forwards fill is not bottlenecked by large backfills.
|
||||
|
||||
|
||||
- Better/prettier (async?) logging.
|
||||
|
||||
|
||||
- Connect to a range of beacon_nodes to sync different components concurrently.
|
||||
Generally, processing certain api queries such as `block_packing` and `attestation_performance` take the longest to sync.
|
||||
|
||||
|
||||
### Architecture
|
||||
Connection Pooling:
|
||||
- 1 Pool for Updater (read and write)
|
||||
- 1 Pool for HTTP Server (should be read only, although not sure if we can enforce this)
|
||||
Reference in New Issue
Block a user