Here's a quick video update of what happened last night when the STEEM blockchain froze.
Getting three text messages all at once was certainly a freak out moment!
Finding new error messages in the logs I've never seen before was also quite interesting:
I was quick to jump into the Steemdevs Slack:
As well as the #witness steem chat:
I'm very thankful @andrarchy was available to send out this tweet to our community:
As I mentioned in the video, the witnesses and Steemit, inc did a great job working hard to get this chain back up and running as quickly as possible. Improvements I'd still like to see from Steemit, inc include a professional site down page that is immediately in place (instead of a generic 503 Error Response page) along with a status page people can go to with regular updates in real time as to what is going on. Some of this was mentioned 8 months ago. That said, they did Tweet this time and stay in close contact with the Witnesses as things were happening in real time. That's a big improvement.
Thank you all for watching/reading and for your support of my witness account!
As of this post, I'm currently position 10!
If you want to know more about what witnesses do check out:
You can read my witness announcement post here:
My previous witness updates:
- EOS Is Alive! Object Computing + STEEM? (June 14th, 2018)
- Updated lukestokes.mhth Witness Account Delegations (January 6th, 2018)
- Witness Delegation Plans for lukestokes.mhth (December 6th, 2017)
- Witness Update - lukestokes.mhth - December 1st, 2017
- Witness Update - lukestokes.mhth - October 26th, 2017
- Witness Update - lukestokes.mhth - September 2nd, 2017
- Witness Update - lukestokes.mhth - July 19th, 2017
- Witness Update - lukestokes.mhth - July 14th, 2017
- Witness Update - lukestokes.mhth - June 19th, 2017
- Witness Update - 6/6/2017 - More Human Than Human
I only post witness updates when I have something worth sharing, but I'm always available if you have a question or concern. I can be reached on Discord or Rocketchat.
What an intense and draining night it was. So many witnesses had personal problems, traveling, in hospitals. I had a damn leaky pipe at 3am (when I discovered it), luckily it didn't burst. I had to cut off the main water valve for the house, then went back at my computer to follow the situation. Shitty times. Yet, despite everything, we did our best to be around to keep the blockchain alive. Teamwork at its finest.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Thanks for the information. I hope the platform is secure and stable over the long term. Questions are being raised about EOS, but I hope that's just a case of EOS being new with the initial kinks needing to be worked out.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
So, the blockchain froze as the consensus was lost. This is a good thing then, which means that it did what it was supposed to do, right? Except the fact that buggy part of the software did allow the input of wrong value the rest went as it is supposed to……………. that’s good to know. Maybe you are one of the few witnesses that I follow so this the only post in my feed that gave me a level of assurance as to what went down wasn’t anything lethal.
Although apart from the general gist, I have really no idea about the technical differences between a replay and a restart. Except maybe that replay lasts for hours while a restart can be a matter of minutes. I have heard that the unlucky ones had to do a replay 😊. I never knew that missing the blocks maybe out of your hands then (I actually thought that it had a lot to with guys being lazy)
Thanks for sharing all this man. I spend quite some time and energy on Steemit so had a bit of scare, not much but it was compounded by the events of my personal life. I hope you drop by my latest post about my experience of the scare.
PS: Nice green hair 😊. Just a few days I ago I gave my first witness vote to you. Second was for @dragosroua. I’ll take a little more time with rest of the twenty-eight.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Thank you for being available all the time everywhere and helping out with information flow. Yes, response from the community in a crisis mode was absolutely amazing.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Thanks for the update. Communication timing is great.
Green is often spill-over from the background being too close or too brightly lit. If you can push it back, that is easiest. Otherwise, lowering the green lighting but keeping the background evenly list is tricky.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Huobi announces suspension of Steem deposit and withdrawal?
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Well done, Luke.
You were the first witness I voted for, and I know I made the right decision. Thanks for staying up all night, fixing thing and reporting back promptly.
While your bedroom looks like it just survived nuclear winter, I'm glad the Steem blockchain had a far easier time of it. ;)
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
I was freaking out... not really but didnt made any post yesterday.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
For several days now I cannot upvote any post or comment without getting an error. Have you seen this before? Do you know if anyone else is experiencing this?
Thanks for the explanation of what happened last night. I though maybe it was connected with the issue I'm having with upvote errors, but it was a bigger system issue.
If you can help me with why I can't upvote, I will then be able to upvote this post!
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
If you're getting the slider before the error: find something you want to upvote 100%. Drag the slider down a ways and then back to 100, then vote. After that you should be able to vote normally.
If you're not getting the slider I don't know that I can help; when it happens to me I glare at it for a while and eventually get the slider.
It's not related to yesterday's issues; @m-sargon has had this off an on for a month, and it hit my own account once.
Also when it doesn't work on Steemit I've been able to vote through Busy.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Perfect! I'm back up and able to upvote again. Thank you!!!
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
What error? Would help to know what it is in order to know what problem you’re seeing. You might also want to log out and log back in again (maybe you’re using your memo key instead of your posting key)?
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
I have only ever used my posting key and it's saved in my password keychain on my iMac. Below is a screenshot of the error. Thank you for taking time to respond to me.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
See the reply from @tcpolymath which may help. Looks like it may be related to this? https://github.com/steemit/condenser/issues/2926
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Yes, that's the bug I'm seeing.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
well, thanks for all the hard work
I saw the blockchain down and as part of a witness team my first call was to our tech genius to let him know as well. As we are only around place 100 with @swisswitness we only missed one block but indeed the number of missed blocks is not always a way to see how good one is as a witness. But the pressure of being one of the top 20 is something else but then the rewards are also better...
I guess as witnesses the only thing we have to do is be vigilant and stick together to keep this blockchain up and running.
liking the hulk hair though.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Thanks so much for the explanations. This shows how vibrant our community is
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
you guys jumped in an handled it very well. great attitude! gives comfort to know you guys are out there working so hard for this site. much love
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
I saw all this last night, saw the tweet, went to bed confident in those behind the scene to correct whatever it was. Will it happen again? My guess is the back door is being bolted.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Thanks @lukestokes really useful to get the news update - I've learnt some new bits there - and sounds like a great response to an emergency and it's impressive to get all the block producers onto the rollout so quickly. You have my vote and this update just gives me further confidence that it is a vote well cast. It's a good point about the missed block stats and I will also keep this in mind. Good work on the comms. Great leadership.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
I think the whole issue has been seen as positive as the price of STEEM remained consistent throughout the resolution of the situation. This could be used as an indication of the confidence in place for the STEEM blockchain and its governance process. Thanks for the explanation and transparency.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Frankly, I was pleasantly surprised regarding how calm and well everyone handled. Nice job!
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Thanks for the update on the situation. It's great to see how quick a fix was implemented and everyones funds are safe and the chain didnt fork.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Hey Luke, thanks for your hard work and awesome communication! Keep it up!
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Thanks Dan!
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
I vote for you, you vote for me. I like your post, your post is very good.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
If everyone just traded votes (follow for follow, vote for vote) then the quality of the whole platform would go down. Read the white paper. It should be based on value, not vote trading. You've given me no reason to vote for you as you've added no value to this conversation.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Thank you, @lukestokes, I feel better about Steem.it today!!!
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Thanks for your unquestioning efforts, Luke!
I think these type of events really create strong bonds between all participants. Even though we dislike software bugs and short of sleep, there's nothing better than solving issues as a team and being able to say: "We made it."
Thanks for your dedication and for keeping us posted!
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Thanks Marly! It's encouraging to get such positive responses from people like yourself who have put so much into this platform.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
@lukestokes
The freeze scared me for a while, until I realized it was for all
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
As one of the newer users around here (26 days according to Steemd), it was interesting to see what unfolded here yesterday. I'm still not sure that I understand exactly what happened, but I have been reading everything factual that I can find about it, and your post and video here were a great help to my understanding, @lukestokes.
What I am curious about is how this decentralised platform had a single point of failure that took down all of the Witness servers at once? I thought the idea of decentralisation is meant to prevent such things from happening?
My thanks to all the Witnesses, and to Steem Inc., for getting the blockchain back up and running. It sure looked like a mammoth task, coordinating everyone to update their servers like that.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Let's think about it this way:
Decentralized means many individual nodes, yes? If all those nodes are running the same software and they all have the same bug, then they will all respond in the same way. This is by design. All the nodes froze because the transaction that was being processed was invalid and should not have been allowed. So it's not a matter of more decentralization would have fixed this or not. The BitShares chain froze about a year ago also. Bitcoin had a chain fork in 2013 which I remember well. These things happen.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Thank you for taking the time to explain that, @lukestokes. I'm coming from a background of having used (as an end-user only) decentralised social networks like Diaspora* and GNUSocial/StatusNet (and now Mastodon), so I compare it to one of the "pods" going down, but not taking the entire network down with it.
But it seems that what you're saying here is that if all pods were running the same software, and there was a bug in that underlying software, then the whole lot could possibly fail, especially on a sync event, am I right? I guess the crypto side of things here (which neither D* nor GNUSocial etc have to worry about) means that all nodes in the Witness network for SteemIt must be running exactly the same software, and exactly the same version, huh?
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Yep, that's essentially it.
Ideally we're all following the same protocol, but there could be multiple implementations of it in multiple programming languages. We're no where close that yet at this point though.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
It was very surprising... And there was a lot of people saying crazy things like "Steem is a scam" or "Steemconneeeect" lol. Many of us found out what was happening because of you, for example I spread your message in several communities, thank you for that.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Thanks for the breakdown/information 👍
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Hi @lukestokes ! I imagined your mind when you got those messages and more over when you saw the issues were things never happened before. Great it was quickly solved by the dev team!
I wanted to let you know about an idea I had, and your thoughts are very welcomed:
A Token as SMT for the Steem Ambassador and Promo-Steem Project
Regards, @gold84
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Appreciate the straightforward explanation, as well as all your hard work! I can't even imagine what it's like to be the core of multiple social sites like these, coordinated between people all around the globe.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit