Heavy duty witness node infrastructure

As you know, witness nodes are an essential part of the STEEM universe.

If I have a witness node with as much as (or as little as) 3% of witness approval, I can choose the right time to shut it down, listen to my favourite music album, walk my cat, and then start it up again, most probably without missing a block.
A full-time witness, or one of the top19 witnesses, needs to produce blocks 1600 times more frequently.
Maintaining such a server in the long run is not an easy task.

Witnesses are vulnerable to DDoS attacks. An attacker who knows the network location of your witness node can saturate your uplink to a level that makes you unable to produce blocks on time. Effectively, this means voting the witness out using a network-level attack. Finding the IP address of your witness node is easier than you might think. Even if you follow the guidelines for setting up your public seed node on a different machine than the one you used to get your witness running, your witness still needs to connect to other nodes in the network.
So what does this mean? Take a look at: http://seeds.quisquis.de/steem.html
(service provided by cyrano.witness)

Does any of these IP addresses look familiar?
The attacker still doesn't know which of them is yours, right?

An attack that makes the target IP unavailable for 3 seconds? Not exactly a difficult thing to do, right? So the attacker needs to make a guess by choosing from a set of suspected IP addresses he has obtained. Guessing doesn't cost him much. Neither does a high volume, short period DDoS attack. After that, he can see that your total_missed count increases, and he knows he guessed correctly. And then you are out.

A concept of infrastructure:

2 witness nodes (primary and backup)
4 dedicated seed nodes

One witness node has your current signing key and does its job. The other is for failover, backup purposes. For example, whenever you need to upgrade your witness binary, you do that on the backup witness node, then change your signing key, so the backup node and the primary node switch their roles.
(Never leave your current signing key on two nodes at the same time!)

These 4 seed nodes mentioned here only work on behalf of your witness i.e. they don’t provide seed service on public network interfaces.
Your public seed node is not within the scope of this document.

Your witness node connects ONLY to your seed nodes using VLAN or another private, low latency network solution of your choice. All other incoming/outgoing connections should be blocked except for the secure shell, preferably limited to access only from your IPs.

Each seed node should be in a separate datacenter, ideally in a different part of the world, never in the same location as any of your (primary/backup) witness nodes. The purpose is to minimize the effects of a successful DDoS attack.

Please note that setting up all seed nodes in a single data center won’t help. A DDoS attack targeted at that particular network will bring all your nodes down anyway.

                      +---+
         +  +----+    |   |
         |  |    |<-->| B |
         +--+ S1 |<-->| i |
         |  |    |<-->| g |
         |  +----+    |   |
+------+ |            | B |
|      +-+  +----+    | a |
| W(p) | |  |    |<-->| d |
|      | +--+ S2 |<-->|   |
+------+ |  |    |<-->| U |
         |  +----+    | g |
     VLAN|            | l |
         |  +----+    | y |
+------+ |  |    |<-->|   |
|      | +--+ S3 |<-->| I |
| W(b) | |  |    |<-->| n |
|      +-+  +----+    | t |
+------+ |            | e |
         |  +----+    | r |
         |  |    |<-->| n |
         +--+ S4 |<-->| e |
         |  |    |<-->| t |
         +  +----+    |   |
                      +---+

W(p) - primary witness
W(b) - backup witness
S1-4 - seed nodes

Of course, you need to keep your seed nodes (at least one of them) running all the time, otherwise your witness will not be able to sync the blockchain.
But if you do the math, you will see that the odds that all your seed nodes stop working at the same time are much lower than the chances that this happens to your witness node.

If one of your seed nodes falls victim to a DDoS attack, you can, depending on the features offered by your infrastructure provider:

• do nothing (until most of them are targeted at the same time)
• change its external interface IP address, routing the old one to a black hole
• set up a new, additional, temporary seed node somewhere else
• replace that seed node with a new one in another location

If you believe this idea is of use and value to Steem, please vote for me as a witness
either on Steemit's Witnesses List
or by using your cli_wallet command:
vote_for_witness "YOURACCOUNT" "gtg" true true