suggestions and battle plan for keeping steemit websites up and running.steemCreated with Sketch.

in steemit •  7 years ago  (edited)

Perception is Everything when it comes to Websites.

The majority of people using the steemit platform do not realise the underlying system of how it works, at least in a basic way — so let me try and explain a little to those that might not know, we have the frontend website (condenser) that connects to a backend blockchain, the blockchain part is decentralised in that it can’t be taken down it’s distributed across many different parts of the world on different hosts by the witnesses, without these guys we don’t have decentalization (so give ya witnesses some extra props today because they rock)

Now what’s been happening is that steemit.com and I presume the main steemit node has been attacked with a ddos attack, this is often called a distributed denial of service attack, think of it as the other way around from what the witnesses do for having copies of the blockchain but instead of a blockchain all the ddos attack does is throttle or overload requests to a service on the web, in our case it’s the steemit.com website and probably one of the main nodes that steemit uses — sure steemit is decentalized in operation of the blockchain but we still have issues with the way that condenser is hosted (condenser is the frontend) which we see that connects to the steemit blockchain backend. That make sense?

Anyway, some cranky, triggered, strung out on g-fuel script kiddy has decided to aim it’s packet spamming nerd cannon over in our general direction. Well ain’t that some shiz? Tbh, it happens a lot on the internet, think of it as a mardy tantrum of someone wanting to get attention or at least reduce the attention of an audience (i.e steemians) to consider them over the platform. The point of a ddos is to do one thing, limit access. Most of the time it’s some power hungry, end of the rope kinda person that’s just trying anything at all possible to gain some kind of control over their live by pretending they are in control by doing this. It’s lame, it’s not even a hack, it’s just noise, white noise, mostly untraceable and the tool of the script kiddie, government agencies, competition and people with an axe to grind.

And take a deep breath. That was a lot right?

Sorry, I decided to throw my thoughts into the ring (I realise the irony of putting up a post on a platform that’s up and down so much today) about some of the things that maybe we should be doing, forming a team around, having a debate about, asking some of the whales who have done well ‘thank you very much’ to dip their hand into their steem and pitch in for a good solid attack and battle plan to mitigate not just similar downtime but also to construct a respectable social media narrative for the users and potential users that heard about the site in press and came today and got a 503 error message instead. Hardly inspiring for them is it?

cloudflare or something similar?


Now I realise that this might not even be possible or even help fix anything because of the way that condenser connects to the blockchain but maybe we can learn from the attack vectors of this last attack and start working with some of the teams of these mitigation experts, find out a cost, see what they can help with, maybe have some kind of early warning system when the data traffic inbound starts to peak and it’s randomly data — I don’t know the vector of the attack all I know is it’s a ddos so it could be on the blockchain master node itself or elsewhere. Either way, we need a good firewall guy and mitigation specialist — find one, give him a ton of steem and let’s rest a little easier that we have someone fighting out corner with us.

round robin between steemit, busy and chain


The first thing I did was to check to see if I could get on busy.org and that was down as well, that’s when I knew that things were pretty serious because I’m assuming that busy uses the master node as well, so it was either a broad ddos attack over certain ranges and sites or it was on a source that both were using — which leads me to a few questions and potential outcomes.

Why are we not setting up a round robin in this times to go to the ‘next available front-end’ or partner of the network. We all use the same blockchain right so doesn’t it make sense from the users perspective that we offer automatically an alternative until the main site can be restored after all we are building not only loyalty to the steemit site but also care about the availability and uptime of the service. That goes a long way with users when they are used to services going down.

We could have a message on the top of these sites when it’s in a round robin when issues are detected to let new users who maybe don’t understand that we have many sites that connect to the steemit blockchain feel a little bit more confident about the reliability but trust of the platform as a whole, also these third party interfaces get some props in the process, after all we are a collective hub talking on this social blockchain right?

Another question I have and I had this before when we had that massive east coast aws issue (not steemit) was why are we not using multiple cloud providers and actually doing a proper routing exercise to build in as many backups to the choke points as possible, if we pick the right places that have 1/10gb connectivity we could be taking super fast transfers and copies of the blockchain and pushing it to multiple edges. I know I make it sound oh so simple but that’s just a suggestion, let’s not tie ourselves into one cloud provider — think of it like a condenser for multiple cloud instances, you could even then go down to doing look ups (if the user is cool) to find nearest steemit blockchain a bit like how CDN’s (content delivery network) work for media — can’t we design a BDN? A blockchain delivery network?

status.steemit.com - build an offsite status page


This is a bit of a no brainer. But why don’t we have this?
Statuspage | Hosted Status Pages for Your Company

I’m sure this would be super useful for the whole steemit community if all the tools, services, websites, api’s and everything else that goes a long with it (smt’s later on) and such like with a team of people who could be alerted and activated when issues happen or even when latency or issues start to develop — maybe I’m being disrespectful and they already have this on lockdown already but I think having something for the general public to be redirected too on twitter or facebook or the many other places that people are spreading the ‘wrong information’ could be actually informed and stop the spread of fake news, bad messaging and bad image for the platform, let’s just cut that down by having a status place to go — replace the 503 page with an automatic redirect to this.

torrent style app to sync copy of steemit blockchain


Ok, hear me out on this one. This is a long shot but it’s just an idea that popped in my mind that just might work - what if, we had a packaged condenser frontend wrapped in a say electron app, that was self updating, automatically pulled down via torrent technology from other users that were in the seed pool (like the witnesses) a copy of the blockchain locally, and whenever those users were on for a period of time could be listed as a virtual witness, maybe even create a route and easy payment system to pay with steem or steem dollar to host a witness full time with the ability to us that blockchain (offline) in a sense on their machine and then sync later when connectivity to the web - think google gears in approach to offline technology — I know it’s out there, but a peer to peer local copy of the steemit blockchain with a self updating condenser frontend, maybe as a sandboxed docker instance?

educate masses about security of the steemit blockchain, because they don't know the difference.


One thing is obvious is that people don’t understand the different between the website being down and the contents of the blockchain that it connects to being safe. People just react to what they see — it’s been hacked, it’s under ddos attack, criminals are trying to break in, etc. for the non technical old and new users they are instantly transported to a place of concern about their information, is the platform safe, is my money/funds safe, is my machine safe — will the site ever be the same again.

The knock on effect is having lots of people trying to constantly reload the website, spread misinformation on social networks and generally turn against everything the platform has managed to achieve up until the next time the site takes a dump (for whatever reason) — we don’t need to just mitigation attacks we need to mitigation public and social opinion by doing the best we can to keep the user base ahead of the loop — and look, as someone that once had a 47u rack server full of game servers and had a script kiddy flood my whole ip range taking them all offline I understand your pain, you’re in the midst of it. Fighting it off the best you can with the tools you have — your literally in a swordfight with someone with a thousand more swords, you don’t have time to be doing the social part, which is I why I suggest we form. ..

.. .a social media response team (@steemithelp)


Either on twitter, facebook, instagram wherever people want to find out information about the platform, wherever you want to point them too, we need a collection of advocates around the globe that can be on call around the clock to feed back from a master update system, be it a channel on a discord server or wherever so we can spread that news out to the masses. I’m happy to throw my name into the hat because I’m pretty much online most of the time (apart from sleeping!) and so happy to craft and update people when things go awry.

conclusion


Just wanted to throw my $0.02 into the ring on it, I hope these are some useful ideas generation and that something might come from discussing them here, if not for the improvement of comms to the steemit user base or to the higher up whales and developers that already do a fantastic job of keeping the good ship steemit floating in the right direction.

Peace!
Teamhumble X


'to protect and serve ya updates'

TRUST AND AUTHENTICATION. .

http://socialgandalf.com/legacypinterest epic wins pinboard → i was a brand advocate for nokia, 1000heads, verisign → won a vloggie for involvement in node666 (san francisco 2006) → on television for time team history hunters 1999 (burton on trent) → sold me.dm to evan williams (twitter) in april 2011 for 5 figures → went to phil campbell, alabama to help repair/recovery/help raise money month after a tornado ripped through the town (was on sky news, bbc news, al jazeeza, nightly news, alabama weather channel?!) → CNN for sxsw 2013 about austin south by southwest event → video chat with robert scoble from rackspace → i was in a music video can you spot me? → won the digital derry contest for 5k euros to kickstart web tv studio → did project work with sleepydog indirectly for nesta (responder) → crowdfunded digital signage concept called the pi street (do it for steem?)

☂ᗴᗩᙢᖺᘮᙢᕊᒪᗴ
together we win

» TWEET ME «
» YOUTUBES «
» INSTARARA «
» PODCAST «
Authors get paid when people like you upvote their post.
If you enjoyed what you read here, create your account today and start earning FREE STEEM!
Sort Order:  

I would hope they can use some sort of distributed solution to make the service more resistant to attack. I wonder if the publicity in Wired encouraged someone to attack it and possibly try to demand money. I'm using Busy today and it generally works.

Thanks my friends

Wow thanks @teamhumble i did not know, i am using esteem and of course its working fine. Be well...

Great post. Thanks for raising awareness about this issue. As you saw from my conversation with @sneak, it seems they do not share our view of how important this stuff is.

well i would'nt say that. we don't know what work they have to do behind the scenes. if anything, it's just pure trust in more people and they probably don't have enough people in the mix. it's a lot to deal with. i just hope they can chip away at some of these important tasks for it to scale.

Help me understand this conversation then. What am I missing? If they can trust a vendor like AWS for hosting, ELBs, etc... why not for Cloudfront or something similar?

not the same things. one cloud is not a like, not when it comes to hosting vs defense. also i'm not aware of the actual attack vector if just the website or nodes too but amazon have spent a large amount on firewall protection and engineers that work around the clock doing what @sneak is doing PLUS extra keeping latency low by optimizing routes and routing tables often. i don't know how well a cloudfront solution fits into the steemit system otherwise i agreed i'd say go for a big package, stick the steemit.com website behind it, also, you have to wonder why it's the main site and not the other sites (busy,chain) are not effected, do they know about them? is this about reputation damaging or just some ex-developer with an axe to grind. either way, yes. of course some kind of cloudfront would be something i would look at, even just get an 1hr consult with someone there and map out the current networked assets to see if they have a service or a plan or something to offer. it would make the most sense. it's not like the whales or the people with the most steem/money can't afford it --- i would certainly do that as a first step. i can't read that conversation right now btw, i'm currently crushing hourlies with my client. maybe later. cheers!

Rock on. If you do get through that one eventually, there's another conversation here as well. Maybe I'm missing something obvious, but I don't see what it is right now.

eeek, ok quickly.. scanned it. i don't know what HSTS is for a start. i last had a DDOS on my game server network clanlife over a decade ago, i'm afraid my network FU is weaksauce these days. i understand your frustrated and that you've had similar (not the same issues) with ddos in the past for an ecommerce site from what i read, when it comes to blockchain and the way that updates, i have no idea, the less links in the chain the better i guess. mitigation of ddos attacks are getting heavier to deal with with the sheer volume of them, especially with crappy chinese routers and shit like that on the market, anyone can build a bot network to take people offline, maybe we need to invest something with GOLEM and some AI to build a reactive forcefield style enterprise battle armour that goes around the steemit services. but then, they will only go and use a similar service. it's war in the trenches, which then means you are taken away from actually improving and building up the service, from the outside world people just assume it's bad service when really it's just angry headerless zombie armies. it's the world of the lamers out there. they love it! :) - maybe it's a BIG discussion to be had at steemfest 2. are you going? if not i'll try and have a chat with sneak there so i can understand the limitation a little better.

Yeah, I'll be at steamfest2.

awesome. let's get around a table then and discuss it, gotta be some points to raise, questions to ask of other providers, ways to mitigate or at least handle the social elements that can cause the general public confusion better and maybe we can get that ironed out together.

ACTUALLY first thing that needs to happen is ROUND ROBIN REDIRECTION TO BUSY or CHAINBB ;) - automated, automatic, with little boxes at the top saying we are suffering from downtime or a ddos and it's being mitigated. better than a 503 or an animated thing (as much as i love the shark) for the overall image of the platform. ok, that's enough drama for me today. ;)

I understand the idea behind that, but steemit, inc is a company. busy.org and chainbb.com are separate, independent ventures. Routing to them would be really confusing, IMO. A simple site down page along with a status page makes the most sense to me. I could see it linking to those other sites, but that introduces other privacy and security concerns steemit, inc probably doesn't want to tackle. They aren't responsible for those other sites.

shame they can't work together. it would certainly do a lot of the rep of the platform and make it easier for newbies to understand that it's not a website, it's websites connecting to the steemit blockchain. guess they are not newbies forever thou eh -- maybe on the holding page? dunno. i always go for what the general public might think but i can see the issues with different ventures sure.

  ·  7 years ago Reveal Comment

was no video on it. muted.