BitBoxBase Supervisor
Every running system needs to be managed. On a services level, this is the job of systemd which starts application services in the right order and restarts them on failure. On top of this default service management, the BitBoxBase Supervisor (bbbsupervisor) monitors the system using custom logic for the many intricacies of the various application components. It follows application logs and checks system metrics in Prometheus, detects predefined events and when changes to the system become necessary, and is then able to trigger custom actions.
Technical documentation
Scope
| System / Monitor | Action to take | Comment | OK |
|---|---|---|---|
| disk space full: check Prometheus metrics against threshold | Compact log files, user warning | Disk space ok for 5+ years | ☐ |
| ramdisk full: check Prometheus metric | restart device | active logrotate should prevent that | ☐ |
swap file: periodically check swapon -s | recreate swap file | critical on new ssd setup, no issues expected after | ☐ |
| System temperature too high | none, part of bbbfancontrol | ✅ | |
| Middleware | |||
auth to bitcoind fails: GetBlockChainInfo rpc call failed | restart bbbmiddleware.service | bitcoin .cookie auth out of sync | ✅ |
log Failed to start c-lightning daemon. and ‘restart counter’ over threshold. | user warning | OK during Bitcoin IBD | ☐ |
| Bitcoin | |||
| change in “initial blockchain download” flag | update bitcoin:ibd in Redis | other services check if IBD is finished | ✅ |
| -“- | change dbcache to either 2000 (during IBD) or 300 MB | speed up IBD, free up memory otherwise | ☐ |
log No space left on device | user warning, SSD full | Disk space ok for 5+ years | ☐ |
log Failed to start Bitcoin daemon. and ‘restart counter’ over threshold. | user warning | OK during Bitcoin IBD | ☐ |
| Lightning | |||
| channel backup | recurring (not possible yet) | ☐ | |
log Failed to start c-lightning daemon. and ‘restart counter’ over threshold. | user warning | OK during Bitcoin IBD | ☐ |
| Electrs | |||
log finished full compaction | restart electrs.service | free memory after initial indexing | ☐ |
auth to bitcoind fails: log reconnecting to bitcoind: no reply from daemon | restart electrs.service to update auth info | bitcoin .cookie auth out of sync | ☐ |
log Failed to start Electrs server daemon. and ‘restart counter’ over threshold. | user warning | OK during Bitcoin IBD | ☐ |