Hello Steemians, we apologize for the interruptions in service you experienced yesterday which are now resolved. Thanks to the blockchain’s built-in safety measures, at no time were funds at risk. The interruptions were due to an issue that arose out of the process of transitioning to Hardfork 20. In our last post we announced that we published our Hardfork 20 release and that witnesses and exchanges should begin running version 0.20.0 of steemd at their earliest convenience. When a witness runs the new code, it signals that they are voting in favor of those changes. If a super-majority of witnesses are running Hardfork 20 on September 25th, the blockchain will fork onto the new chain.
Blockchain Halting
Until the date of the Hardfork, both versions produce blocks on the same chain. This gives the witnesses time to test the code as well as cast their vote for the new changes by running the new version. Yesterday, that chain unexpectedly forked when the 0.20.0 nodes produced blocks that did not match the consensus of nodes running version 19 of steemd. This activated one of the many safeguards built into the blockchain which pause block production due to the presence of some unforeseen error. These safeguards guaranteed that no important information, including token balances, was at risk.
The Problem
The specific cause of the problem was a bug in the HF20 code. A constant was changed that should not have been, and this caused the initial fork. Your witnesses did a fantastic job of quickly spotting the suspicious behavior and began reverting to a stable build of the Steem blockchain. This exposed another bug that resided in our fork database logic that was present due to the infrequency that this logic is activated, and which caused the minority fork to halt as well.
Our blockchain developers worked extremely hard to identify the problems quickly and issue a patch that would enable the witnesses to resume block production. They assessed that the 19 fork (version 0.19.X of steemd) was not the problem and advised the witnesses to revert to that version and restart their nodes. 19 of the witnesses responded by restarting their nodes, returning stability to the network. Steemit Inc.’s production nodes are now fully functioning as is steemit.com.
Next Steps
We have already performed a post-mortem on the event and have drafted an attack plan for developing the tools and resources which will ensure that events like this will not happen again. But before we can get to work on those, we’ll be spending the rest of today investigating the fork database code in order to ensure that it’s prepared for the hardfork date of September 25th.
The Cost of Innovation
When we launched the Steem blockchain it was with the goal of creating a protocol that could power real applications that real people use every day. That goal, along with the governance mechanism we chose, has enabled us‒as a community‒to understand what changes need to be made and act on that information by upgrading the software through hardforks. So far we have successfully hardforked 19 times in just 2 years.
This rapid rate of iteration is one of Steem’s uniquely valuable attributes and is virtually unheard of in the blockchain space, which itself is still in a highly experimental phase of development. The cost of this rapid innovation is incidents like the one that occurred yesterday, and the fact that so many people were unable to use the apps they love, is a testament to how successful Steem has been at fostering the development of functional applications that real people use every single day. Yesterday also demonstrated, once again, that our community has the ability to work together to handle these issues so that we can continue to push blockchain technology forward at a more rapid pace.
Thank You
We want to express our gratitude to Steem’s amazing community of witnesses, developers, and users. Blockchain-halting events are pretty much as bad as it gets for a blockchain, but everyone worked together to ensure service was returned as rapidly as possible. It’s a testament to the power of this community that such an event was handled so effectively. Many people outside of Steemit Inc.‒especially the witnesses‒shared vital information that enabled us to respond more rapidly, and return the platform to proper operation.
You don’t lose if you get knocked down; you lose if you stay down. - Muhammad Ali
Steemit Team
For details on all changes in HF20 :
https://steemit.com/steem/@steemitblog/steem-velocity-hardfork-hardfork-20
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Not all is fixed
Far as I can tell, Nodes that exchanges use are down. This means no deposits and withdrawals. One can send STEEM to the exchange, but the exchanges won't know and user's STEEM won't show up on the exchange.
Here is a list of Broken Exchanges
What is Steemit Inc doing about this problem?
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
https://market.rudex.org/ works well, with 0% trading fee for STEEM/SBD btw
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
In a course I took about creating product loyalty, we were taught that having a problem and handling it well often creates more loyalty, than not having a problem at all.
Yesterday was a showcase in how many people are engaged and interested in the site, even those who voice frustration were concerned and interested in the outcome.
I know I could be over communicating this message, but I really do see improvement in the communication coming from SteemIt, Inc.
Thank you for the information.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
As a point of improvement, I believe at least https://steemit.com/ temporarily should have been displaying some information during such an outage.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
All the more reason why top 20's should be developers, or have someone on their team who is! Nice work everyone!
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Exactly. And the servers should also be hosted by the witness himself/herself or somebody in the team.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
That is a no-brainer. I'd be stunned to hear it was otherwise...is it otherwise?
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
It is otherwise. :) There is a large witness that rolls out witness servers for other witnesses.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
steem power inequality asides, it's everyone's imperative to vote for witnesses that they feel is best suited for the job.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
the voting mechanism should be more transparent within the steem website. it has been 4 month that I am using steem and I still don't know how and where to vote....
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Go here:
https://steemit.com/~witnesses
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
thanks! was about to link there too.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Is that the same as someone who deliberately creates a problem, and then handles it well?
(thinking cap on)... in history, who do we know who does that? ::chuckle::
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Actually, when you are breaking new ground and developing new industries, mistakes happen. You deal with them, learn, and move on.
That is not the same thing at all as purposely creating/pointing out problems and pretending to fix them. Which is also a strategy used by many. Especially politicians and companies that profit from fixing things. :) The impact is close though.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Politicians. Hmm... I was waiting for that to appear. Yes. I agree with you.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Thank you Steemit, for making my point with every blockchain seizure.
I can't wait for the other "edge cases" to manifest themselves as more "functionality" is bolted on to the shuddering PoS chain.
I know, I know - everyone loves Steemit and anything to the contrary is blasphemy.
I'll see myself out.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
It isn't blasphemy at all. It is just really weird to hang aroundthe place if you don't like it and don't trust it and aren't willing to assume the risks.
I am often confused. I don't like myspace, but I never, ever, log in to tell them what I don't like about it.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Seriously? I hang out at myspace every day and tell them what they are doing wrong. Hopefully, one day they will get the message!
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
I see someone is emotionally invested.
If you can't handle some criticism of Steemit, then what are you going to do when it seizes up like a rusty lawnmower next time?
I'm sure it will be instructive.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
lol
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
haha
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Why not enjoy the decline? Being proven right is its own reward, after all.
Also, why would you care? Isn't Steemit glorious and amazing - so much so that you don't have time to reply to me? What? It isn't? Well, whaddaya know...
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
I reply to most of my comments, not all. check it.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Well said @whatsup!
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
While we'd rather the incident hadn't happened, such events do show off the strength of the ecosystem. I think people use these incidents to gauge the degree of antifragility within a system. They are demonstrations of its ability to grow stronger from volatility. Good communications can highlight this aspect of the system.
The engagement and interest you mention is, IMO, the secret sauce of any such system. So many projects out there think that the solution to all of their problems are algorithmic-first, and if they can just design the right algorithms, that will solve all of their problems when in reality networks of people solve problems. The most elegantly designed "on-chain governance" will crumble if no one uses the software and likely will collapse when they do use it because then it will be forced to interact with unpredictable and irrational (not in a bad way) human actors. The only way to develop the right algorithms, build the right software, and establish functional governance mechanisms, is to build a network of engaged and interested people who are constantly kicking the tires, providing feedback, and influencing change in a positive direction. We don't just have that, the Steem community is years ahead of the competition. People often make the mistake of thinking that Steem should be defined as what it has been in the past, when the real power of Steem is what we will work together to make it in the future.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
That’s well said.
Also see:
How to always win on Steemit!
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Bitcoin has also had one major outage which needed manual intervention from the miners - https://github.com/bitcoin/bips/blob/master/bip-0050.mediawiki
Bitcoin used to have a centralized alert system, where urgent messages could be broadcast over the Bitcoin network by anyone having the private keys for the alert system. This alert system has later been removed from the Bitcoin Core client. I think maybe it would make sense to set up some alert system where any of the top 20 active witnesses could send an alert. I don't believe false alerts would be a big problem, since such an alert message (hopefully) would cause the witness to lose votes.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Thanks for the feedback. I don't think witness responsiveness has ever been, or was in this case, an issue but it certainly might be worth considering. It's important to remember that any time that is spent developing one solution is time not spent developing another. As Steve Jobs famously said, "People think focus means saying yes to the thing you've got to focus on. But that's not what it means at all. It means saying no to the hundred other good ideas that there are."
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
I wish the success of the platform! Hoping you release more youtube videos Andrew; you're a great spokesman.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Yes. @andrarchy
Why not create a video explaining how this problem was fixed?
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Thanks! I'll work on it!
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
True. The alert-system is probably not worth pursuing - but I do believe it's rather important that nodes stays up and serves read-only-requests even if the block production is halted.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Many witnesses have alarms set up if they have missed blocks.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
I agree!!
It was nice having the Steemit Twitter page to stay up to date on the status!
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Thanks for mentioning the twitter page, I was unaware
Posted using Partiko iOS
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
I was happy that there the problem did arose. We are the only blockchain that encounters problem because a) there are so many users using the blockchain,
b) The inevitability of stumbling bugs at rate of upgrades and forks we are expereincing.
It's something that we in fact can be proud. As you said, it's better than not facing any problem at all.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Well said boss
Posted using Partiko Android
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
It seems like all the nodes went offline and all interfaces to Steem (like steemit.com) got broken. Wouldn't it be a lot better if the nodes could continue serving content up until the "uncle block", and that the interfaces could issue a status message, something like "blockchain temporary frozen due to technical problems - posting and voting won't work"?
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
imho, it maybe a lot more complicated than that, but yeah sure there could be more sophistication but that requires prioritization and surely more available quality resources and that will all happen in due time
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
It's not the first time the network freezes, and I believe it's not the last time either. As I see it, this is a major trust issue - all of a sudden, both busy, steemit, steempeak, eSteem, etc stops working - and people are even concerned that their steem funds may be lost.
I haven't studied the software, but it should not be a great problem to change the node behaviour from "shutdown" to "stop producing blocks and ignore all newly produced blocks".
Showing the end-users nice error messages explaining that the network has gone down into a "fail-safe read-only-mode while some technical problem is investigated" is of lesser importance, but that's a job done by busy.org, steemit.com, steempeak.com, eSteem, etc.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
i'm with you, thanks, but your middle paragraph is imho not that simple or even sensible, because imho if a catastrophic error occurs (which did) NO activity should be allowed, ALL user activity should cease hence shutdown
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
In my opinion, no matter how catastrophic the error is, anyone ought to be allowed to inspect the historical blockchain, that includes browsing old articles and replies on sites like steemit.com, as well as checking the wallet. This is important for user confidence, and also for search engines, etc.
As no new blocks are produced, it will of course be impossible with any actions that are to be recorded in the blockchain, that includes placing orders on the internal market, voting, writing replies or posts, etc. Ideally some alert should be shown even before the reader considers voting on anything, but a generic "an error has occurred" when trying to vote on something should eventually suffice. I believe the latter is roughly how steemit.com works today if an action cannot be recorded in the blockchain for whatever reason; one gets some error message in red text.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
agreed, it will possibly all happen when STEEMIT is declared out of beta and big investors allow for / deliver the needed quality resources - hopefully sooner than later!
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
I actually think this was a very good thing to have happen.
We have a lot of smart people around here and to read how all came together to offset a potentially devastating event provides a great deal of confidence.
Bugs are a matter of course in the programming world. Having them is not the problem, what is done once they are found is. It was reassuring to see the team get the chain going in a rather quick period of time. Down for a number of hours seems like a lot but think about how long Microsoft leaves bugs operating.
The fact that many, including myself, were lost without STEEM shows how committed we are as a community. This is something that might be unmatched in this arena at this point.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
I am disappointed @abit wasn't mentioned. He was a total rockstar yesterday and without him, things would have taken far longer.
The first thing I did when I could get back on Steemit was to vote for @abit. We need more witnesses that can show up and make a difference.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Good point you make - I heard in various chats how awesome @abit was - I will probably re-vote him as witness. Thought he was not really active anymore.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Is that @abit another of your stupid accounts?
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
You do a very good job making Steemit smaller and smaller. You can go brag about it to your bosses. :-)
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
That will show em...
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
so you and bernie and marky are 3 of a kind - good to know
and nextgencrypto = helpie + all the other shit accounts LOL
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
oh i see you are a loving person yourself. :) awesome
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Until reading this post, and the comments, I didn't realize what a good thing it was for the site to crash. Now I know that crashes like that are healthy, and we should probably have them at least once a week.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Well observed, @vantocan! Probably you are a European. We analyze every word along its full meaning and also read the white space in between the lines of sentences. Not a long time ago, the months of delay of the SMT has also been celebrated as a big victory from the heroes of the Steemit Inc., NY. That's fault management as usual in PR, unfortunately not on real intellectual eye level for Europeans. Every PR–scentence from NY sounds like parents are talking to their children. It's a matter of culture.
You don't have to wonder much about, that your intelligence feels a little bit insulted. It is nothing serious at all, just Public Relation!
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Thank you for staying on the top of this & keeping us updated. @SteemitBlog
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
AKA
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
I am so happy to be a part of this wonderful community! :)
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
I was waiting for some report to find out about the events of yesterday, excellent work.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
It's good you got it fixed. So the hardfork date was only a target? You were basically trying to hardfork early?
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
No, the hardfork will be activated on September 25 when a majority of the witness are running the new version of the software, but there was a mistake in the part that was supposed to be compatible with the current version.
Installing the new software at the latest moment and activating it immediately would create more chaos.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Thanks for the update.. many of us non-nerds were just sitting around frustrated ..luckily I was able to reach out to a witness (@themarkymark) who informed me what was going on.. I certainly hope everything is fixed..
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
You can also follow @SteemNetwork on twitter for official updates on the Steem network.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Will do thanks.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Maintaining the September 25th hardfork date isn't ideal.
For political and marketing reasons it might be...
I recommend delaying it an extra 10 days... and rescheduling it for October 5th.... give witnesses a time to role out a little slower, and more stable... and run additional testing...
Knowing we're trying again in only a week seems a little soon.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Only up hill from here
Posted using Partiko Android
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Is there such a thing as a Steem status webpage?
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
So we had a unexpected hard fork, damn no wonder everything crashed. At least the stop mechanism ensured that errors were resolved.
Thank you for explaining the reason for the delay and issues yesterday it was hard to figure out when Steemit itself is down when explaining to a friend DAPPs :p
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Okay. I'm glad that the problem was identified and that the blockchain came up as quickly as it could. For me, that was 11 hours. Don't know what it meant to others, but it could have been less.
Realistically, I might have lost two to three STEEM from it being down. No big deal.
Maybe it meant more to others. I don't want to downplay it's significance for anyone else.
I guess the main thing that sits wrong is the inevitability of it all. Yes, the blockchain is highly complex. Yes, an error in coding is bound to happen. Yes, it would take way too long to test everything that could possibly go wrong, even if you could anticipate everything that could go wrong.
But what is it about these incidences (this is the second blockchain stoppage since I've been here—don't know if there were any or not before January 1, 2018), that is meant to instill confidence that they won't happen again? The fact that the community came together and resolved the issue? I'm glad it happened and things weren't prolonged. I'd be much happier if it never happened.
But that apparently can't be. So, when the next unlikely, unique, shouldn't happen anomaly takes place, and then the next, and the next, how often do you think this can go on before you lose the confidence of the users?
And maybe it's not so a big a deal now, with 50,000-60,000 daily users that are largely scraping out earnings through their social media efforts, but what happens when it becomes SMTs and after that, entire industries built on the STEEM blockchain, as in a worldwide digital economy?
As I've commented elsewhere, I hope the inevitability of this, sooner than later, becomes totally avoidable. That the impossible to foresee or catch or prevent now becomes absolutely possible, if not automatic. That means resources and money I suppose, and technology and knowledge that doesn't yet exist. Maybe that will be much later than sooner and we'll just all have to cross our fingers that another stoppage won't happen for a while longer.
It's great we've got people who can troubleshoot and resolve things. I'm glad people can work together. It's got to be frustrating for all of those involved since it can largely be a thankless job when something like this happens. I'm just hoping the commitment is there to avoid the likelihood of an all day (in my case) downtime, and if there are alternatives for allowing people to work, cache it somewhere or whatever the proper terminology might be, and then get it going on the blockchain when it's up and running again would be great. I don't know what the solution is. I just know there needs to be something better than it's impossible or it's inevitable and great work guys when it's restored.
I hope I've been able to communicate what I'm feeling and trying to say. I'm not trying to ruffle feathers or point fingers. I'm just trying to express a point of view in hopes that it will be heard and that someone will understand it. If anything could be done about it now or in the new future, that would be ultimate, but right now I'd settle for someone getting what I'm saying and being okay with the fact that I said it.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Imagine somebody just invented the steam engine (see what I did there?) and people are worried it might derail or explode. It's due to arrive at the station at 3pm and at 4pm there's no sign of it.
If you're just a traveler on the platform you're going to worry that any number of terrible things have happened.
This post is the train finally arriving at the station, and the conductor announcing that somebody pulled on the emergency brake, which did what it was designed to do and brought the train to a screeching halt.
Most of the delay was the investigation by the engineers as they made sure they'd thoroughly investigated the cause of the problem, and double checked the timetable to make sure that restarting after the delay wouldn't put the engine on a collision course with any other trains.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Hey, @mattclarke.
I have no problem with everything that occurred the moment the bug/bugs came to play. I'm good with all that.
I'm more concerned with what happened to cause the emergency brake (ie the bugs). I'm understanding that it's impossible to account for every instance of why the brake might be pulled. Right now there aren't that many trains out there, at least not running the same length of track, nor are that many people waiting for the train, so the inconveniences might not be so bad.
Soon, however, we're hoping for many more people to be at the station waiting for the train to arrive (ie velocity), and we're anticipating that there will be many more trains providing service on that stretch of track (ie SMTs, entire industries). We're also understanding there will be other track built and trains on them (competitors) that will take away customers if we can't provide on time service all the time.
The one that's going to be able to get the most people from and to their destinations on time is going to be the one which stands.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
As we said in the post, we have gained a lot of knowledge from this incident and know a lot of tools we want to develop to ensure this won't happen again. While the process might be unpleasant for you, and for that we apologize, this process of dealing with the unknowns and developing solutions is precisely how problems are solved for good. Building blockchains is still a new thing. Building blockchains that power real applications is something only we do. And building such a blockchain and updating it frequently is yet another thing only we do. We have no one to look to for guidance, no established model for anything. If someone tells you they can develop bug free code in such an environment, they're just a liar. What we can promise is that we will never release code that puts your important information at risk, and we have demonstrated that over and over again. This event was a consequence of that commitment as the blockchain stopped producing blocks specifically to safeguard what matters most. Those who come after us will be able to benefit from our solutions, but we do not have the same privilege. I am sorry we live in a world where we have to be the one's blazing the trail on this, but that's also why we are so committed to what we do. We've been doing this for over 2 years and our engineers and witnesses still respond at all hours and put in countless hours to fix the problems that arise. If you don't think that it is commitment, I don't know what is. And yes, for the people who do the real heavy lifting (i.e. not me) it is totally thankless.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Hey, @andrarchy.
I'm glad to hear all of this. It is all pioneering on your part, and the analogy you gave me the last time of building the airplane as it's flying has stuck with me. It does describe what you're doing, and attempting to do. Something that is basically impossible to do (at least building a commercial jet in flight would be).
I'm glad to hear about the dedication and expertise employed after the fact. I've seen it in action. I know it exists. I'm glad to hear about the tools that you all want to develop to prevent such things from happening.
I guess since it's nearly ten years since Bitcoin rolled out I figured there might be a little more on blockchain technology out there, along with project management, product testing, etc. I thought there would be more to it as far as common practices go. If not specifically for the blockchain, at least adapting from other industries.
I know the STEEM blockchain is unique in many aspects from other blockchains that aren't designed to do what it does and will do. I can see the uncharted territory there, but I was expecting there would be something else to draw on, rather than it basically being put it together as you go. That's more than just building the blockchain. That's also needing to build all the procedures and protocols surrounding how it gets built. I figured there would be a lot of that in place by now.
I've never heard of anyone saying they can create bug free code. Developers are pretty united on that front, even ones who think they can do it better. So, wasn't trying to say that at all.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Of course we use established project management techniques.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
So happy I can enjoy the idiot spam comments of @badcontent on my posts again! Good Job!
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Awsome :D
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Thanks for being on top of the situation
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Just test reply passing through :)
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Great to see an official update on this and I am glad it was resolved reasonably quickly. As more and more DApps come online (inc SMT's) this sort of down time will become harder for independent business who rely on the blockchain to tolerate.
I do feel that communication to other stakeholders (ie the general public, non top 20 witnesses, DApp holders) could have been much better however. This issues with this was expressed well by @therealwoolf in his post this morning so I won't repeat all his points here however I hope the team gets a chance to read it.
https://steempeak.com/witness-category/@therealwolf/the-day-the-blockchain-stood-still
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
The scariest part is not knowing what is happening while everything is down. Maybe I am in the minority, but I am not well connected with many of the higher ups in the know. So I have to assume the worst and cross my fingers that everybody's accounts are safe.
I am glad it is back, and the glitches are being ironed out.
Hopefully fixing the fork database will prevent this kind of downage again. It is concerning that witnesses testing the new fork can cause the main blockchain to hit a stop error.
Having some communication representatives available from official sources would also be appreciated in the future. Rumors start flaring up like wildfire because most of us have no direct communication lines, and that is bad for value of Steem on the exchanges. I heard there was a twitter post, and that was it. Maybe this can be improved too. I'm sure some of the witnesses would be happy to be trusted with spreading official news through popular discord channels, and on their app sites.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
You can follow @SteemNetwork for live updates. We understand how much you care about Steem and the Steem applications you rely on and are continuously working to develop better communication systems so that you can remain up to date, while still allowing the engineers who have to work together to diagnose the problem and develop a fix to do their jobs.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Yesterday during the 12hour / 12second issue, this was actually not true. I found it sad that all of the information about the hardfork that I would have considered relevant and helpful was not accessible through normal Steem front-ends. I am not saying its bad to eat your own dogfood, but if there's suddenly no dog-food to eat then its a hungry pack of wolves... or something.
Perhaps it makes sense to consider IPFS'ing important announcements about things like hardforks as a type of insurance against this kind of thing happening in the future.
Anyway, thanks to all the awesome witnesses who got us back online, their devs and the whole steem community. Great to be here.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
I definitely agree with this, but I will point out that myself and many others were indeed spreading the word through a number of the servers where we're active. We do our best to make sure we can explain as clearly as we can wherever we can, while juggling the background stuff, but you're absolutely right that we can't reach every community or even most people as effectively as official channels.
Having compounding errors is one of those things that testing might miss, no matter how much is done — in this case, almost a month of running testnet, plus two simulated hardforks, and a week or so of quite a few witnesses running the new code on mainnet all managed to sneak by before the perfect storm of nodes and transactions hit to kick this all of. It definitely sucks, and goes to show we can basically never sleep... but I'm bolstered working on a chain designed to stop before it starts double spending or compromising wallets. If there's a community that you're in that you feel needs a liaison, please reach out to me on steem.chat or discord and I'm happy to add it to the places I roll through when I'm updating.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
I agree that the communication has improved considerably and thanks to all the new staff who was hired (I’m assuming new people are doing much of the outward facing communication).... we are noticing a difference so keep it up.
Posted using Partiko iOS
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
I was actually expecting to see the HF20 being completed already. I saw some earlier comments on Steemit,Inc never delivering on time. So delivering too early could have been awesome.
I'd be great if witnesses had their own alert system so that they can be woken up from their beds and get into action fast. The cost of decentralization is that everybody lives at a different time zone.
Good Luck With Everything!
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Most of us do have alert systems, and we were woken up and got to work right away as soon as the issue was detected.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Glad to know :-)
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Most of us do, and
We were woken up and got
To work right away.
- timcliff
I'm a bot. I detect haiku.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Actually executing a hardfork early is the one thing you don't want to do early! I get what you're saying, and in most cases you would be right, but Hardforks are scheduled in at the blockchain level. The blockchain team released the Hardfork code last week, which you could consider "early," but inside that code was the specification that it would occur on the 25th. The only way to change that would be to go back and change the hardfork code, which would only give the witnesses less time to review the code and exchanges less time to fire up their own version of steemd 0.20.0. This very incident proves why it's so important not to rush hardforks and the firm date ensures that everyone can switch over at the same time.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Thanks for clarifying.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Just out of curiosity, what is undergoing the HF exactly - Steem, the Steem blockchain, or Steemit? Many simply do not understand the gist of it all. If the crypto Steem were HF'ing, wouldn't that mean more crypto (like BTC to BCH)?
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
All of that information is discussed in recent @steemitblog posts. The short version is that the Steem blockchain is hardforking, but thanks to Steem's governance mechanism, when Steem hardforks it is more like an "upgrade" and so no "sister-chains" like BTC or BCH are created. Steem moves to a new fork and the old fork is abandoned. Cryptocurrencies themselves do not fork, only blockchains fork. New "sister-currencies" are only created when a blockchain forks, but the old fork of the chain continues. So BTC exists because someone created a new fork of Bitcoin while the old fork also continued to run. Since there were 2 blockchains they needed two different cryptocurrencies. Since only one fork of Steem will continue after the hardfork, the currency of that blockchain will remain STEEM. This hardfork contains no changes to the STEEM currency.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Thanks for such a concise explanation. I know I learned something today!
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
please upvote my comment as compensation for the hf20 mishap....
just kidding..
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
lol
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
so that's a firm NO on the upvote ?
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Sorry, company policy :)
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
can I talk to your manager ?
@ned
can you quit your dancing and take my call ?
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Any idea why Steem auto doesn't work anymore?
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
You would have to join their discord for updates. https://discord.gg/jpN3RwF
They may run their own RPC node and it may be down currently or replaying blocks. I'm not sure the setup. You would have to ask the owner of steemauto. He is often active in Steem.chat as well.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Who is the owner of Steem auto?
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Thank you for keeping us informed. This is what team leaders do! Thank you for coming out in front of some rumors that hurt the platform. Similarly, I do not know if you know that after yesterday there have been some pernicious phishing that seek to destabilize the platform and some accounts. Greetings
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
We are aware of the situation, but thanks for checking
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Bravo team Steem! You guys are awesome!
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Amazing quote at the end. Well done team!
Posted using Partiko Android
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
I appreciated the robot graphic on the main page, which let me know this problem was recognized, and being worked on. Thank you for all your hard work guys!
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Thanks for the feedback! Good to know!
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Just a small bump on the long road ahead.. Great work!
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Thanks for being on top of the situation
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Thanks for being so transparent about the issues and working hard to resolve them. The Steemit platform really is unique in the blockchain space and you guys are really pushing it to the next level.
Keep up the great work!
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
The final Muhammad Ali’s quote it’s like the cherry 🍒 on the cake: really nice and appropriated!
Thanks guys for all your kind effort! 🙏🏻
A huge hug from @amico! 🤗
Posted using Partiko iOS
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Great job Steemit team and all the witnesses!!!
I know I poke my nose where it don’t belong but consider my interest to get the correct information my way of intercepting any accusations that may be thrown around, I aim to defuse and reinstall trust nothing else!
Good to hear that you guys have the safeguards in place to ensure the safety of everyone’s accounts. Going forward could we not install a system that automatically returns back to the last stable version in a situation like this?
Posted using Partiko iOS
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
It's not quite as simple as that, but effectively those are the types of solutions we're working on.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Perfect, yes I air on the side of simplicity... how it functions is a little over my head but I understand the need for a system for these situations and it’s good to hear that your on it!
Posted using Partiko iOS
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Thank you to all involved. Awaiting the 20.x fix.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Well played :)
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Realmente en los 7 meses que tengo dentro de la plataforma , me parece que es una plataforma estable, casi no he frecuentado caídas como la de ayer, pero fue muy rápida su resolución.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Much appreciated 'team', keep up the good work!
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Where is my hardfork...? I know you've got yours...
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Apology well accepted, we all appreciate your good work. Please keep it up
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Upvoted.
DISCLAIMER: Your post is upvoted based on curation algorithm configured to find good articles e.g. stories, arts, photography, health, etc. This is to reward you (authors) for sharing good content using the Steem platform especially newbies.
If you're a dolphin or whales, and wish not to be included in future selection, please let me know so I can exclude your account. And if you find the upvoted post is inappropriate, FLAG if you must. This will help a better selection of post.
Keep steeming good content.
@Yehey
Posted using https://Steeming.com condenser site.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Kudos to Steemit and everyone involved in this fix! I am proud to be a part of this community 🤗
Definitely Resteemed!!!!👍
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Thx for the update and I am so glad Steem is up and running again, looking forward to HF 20 :)
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
It's OK .........,
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Thanks for your development
In case of any downtime on the steem, you should add some information about that downtime to the steemit.com
IT IS IMPORTANT
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Kudos to the Steemit team (developers & witnesses) who worked tirelessly to ensure service is restored!
Posted using Partiko Android
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Thank you for this update.
Posted using Partiko iOS
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
It was disconcerting to think that my blog was removed and no longer existed ... there needs to be a general broadcast page that says “ Steemit is experiencing Momentary Network Problems and we are working to fix them” ... I think I had a small heart attack thinking that I did something wrong and my page as deleted ... just my opinion.
Posted using Partiko iOS
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Good to have an explanation, so thanks for that!
Also a nice thing that we now at least have the "tinkering robot" as the default page when there are issues causing the site to be offline.
As a non-nerd, I want to point to the support of the large number of Discord communities that have formed all by themselves, aside from any planned "communities" by Steemit, Inc. They enable "regular folks" to get some information — often from the technically inclined — in "plain English." And that speaks highly of the cohesiveness of the core of the Steemit community.
I did not come across even a single "I've had it!" rage-quitting type post/comment there.
=^..^=
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Absolutely.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Thanks for taking care of all this. The community is amazing.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Thank you for this information and professional presentation. This lends credibility to the platform when you have a clear an thorough explanation to the recent events that all steem members are concerned about. Thank you.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
howdy there steemitblog! Thanks so much for this information and explanation of what went down and why, greatly appreciated!
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Congratulations @steemitblog!
Your post was mentioned in the Steemit Hit Parade in the following category:
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
You don''t know what you got till it's gone. I don't understand the magic the team of programmers do in the background to keep this thing working. But good job to everyone involved and thanks for the update.
Posted using Partiko Android
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Thanks for the update. @steemitblog is doing a good job these days of keeping us informed of what is happening around the place.
I think the confidence of those on the platform regarding the direction of Steem is certainly growing. This should spread to those not yet invested.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Many thanks to all the witnesses who worked tirelessly to run steemit once again :)
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Could we can face similar issue on implementation of HF20 velocity on 25 September.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
The bug that occurred has now been patched, so it is now impossible for the same thing to happen.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Best quote, by far...and that's why I show up everyday.
It's also why I will continue to hodl SP; as a direct result of this community which supports each other.
Respectfully,
Rebecca
Thank you.
Thanks for
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Thanks for your support!
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
No one said it was going to be easy and well done to all those involved in fixing the problem. Good luck with HardFork 20. Stephen
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Eagerly waiting for HF20
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Good work, you guys seem resilient to hop back up from trouble.
And it seems that the community isn't salty but in the contrary more helpful than I thought 😄
Posted using Partiko Android
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
I must say I felt totally peaceful with this.
Is not the very first time this Blackout happens and Devs & Witnesses always find a solution.
Cheers for the Chain!
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
https://busy.org/@rabok/bananaprogram-partiko-week-6-dr5gf4p4
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
I am not a developer, I don't understand about some specific details yet of blockchain, but I find this post very interesting. I'd also like to share something with you, guys, that was happening to me last week. I am not sure if your test for the new hardfork was being run the last week, but this is what happened to me:
I know that curation rewards only reflects in our wallets after redeeming or Transfer To Balance. My partner has access to my account in his Smart Phone, but last week he didn't redeem any of my curation rewards, and what I saw is that each of my curation rewards were being added to my SP before and after redeeming. It was like: I have 245,.03. The next time I see the "Transfer To Balance" button I have 245.004 before redeeming, and after redeeming 245.005.
I don't know if you have another case like this. I am almost sure about this fact. I didn't know you were running a test. I don't know if it could have happened to any amount of people.
Anyway, it's not a bad thing for me, but I had the chance to see that happening. I don't know if it represents an important fact.
Thanks for keeping us informed, @steemitblog.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
One thing that should be improved is the comunication.
Steemit.com only showed tge message "ups something went wrong" , ¿on my system, the frobt end ,the blockchain? It took me a while to figure out what was going on
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Yes. Yesterday was a testament to this. And it's been said before, but really good job communicating about this so clearly :-) Thanks for all the good work keeping steemit and the community alive :-)
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
In my country we say, that in the difficult moments that we can see the true friends. For those of us who believe in the steemit project, we always hope that everything that happens is to make us stronger. Good energy for everyone and for better times to come.
Good vibes.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Very well said. As someone who has been invested in this community since the beginning, I have seen how true that is many times now. It's easy to come in here, benefit from the advancements that have been made and the risks that other people have taken, only to jump ship the moment one stops receiving rewards or encounters a temporary setback. There's a reason why they say that friends stick with you through through both thick and thin.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
That is why sometimes when difficult situations happen, you can take the positive aspect. Because we can see the true face of the people. I have little time on this platform, but as I said in several opportunities, since I started, it was like love at first sight and here I am learning and trying to collaborate as I can, besides betting on a long-term investment, while still being clear that to achieve things, you have to do it with effort and work there is no other way, because I do not believe in easy money, to obtain profits from something you always have to work hard. I was pleased to read your response, sincere thanks for your time.
Good vibes @andrarchy
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit