Incident Report

in witness-category •  7 years ago 


This evening at 7:45 pm local time, I was notified by Telegram bot that my Peerplays witness node was missing blocks. I could not log in to my server, nor even get to the VPS company main page. The VPS reported a connectivity issue and was back online 5 minutes later.

I could connect to my node again, but it was still reporting missing blocks, even the logs were reporting pushing blocks. I noted that I was not getting blocks for other witnesses and decided my node had forked. I restarted and replayed the blockchain. The replay process took 5 minutes, and it took another couple of minutes to catch up missing blocks.

By 8:06 pm we were signing blocks again. In this time, we missed 30 blocks (with only 15 active witnesses at the moment, block slots are coming around more frequently).

I run a testnet node on a different server. I have now set up a backup main node on this server also. Switching will currently be manual, but will investigate automated options, such a port of jesta's script.

Authors get paid when people like you upvote their post.
If you enjoyed what you read here, create your account today and start earning FREE STEEM!
Sort Order:  

Congratulations @scissors-witness! You received a personal award!

Happy Birthday! - You are on the Steem blockchain for 1 year!

Click here to view your Board

Support SteemitBoard's project! Vote for its witness and get one more award!

Congratulations @scissors-witness! You received a personal award!

Happy Birthday! - You are on the Steem blockchain for 2 years!

You can view your badges on your Steem Board and compare to others on the Steem Ranking

Vote for @Steemitboard as a witness to get one more award and increased upvotes!