Google's Bot Imposes a Large Impact on the Performance of SteemYY.com

In the last 24 hours, I have observed several alarms of the high CPU load average of steemyy.com. I checked the logs and found out there were a huge spikes of the bots from Google (in the User-Agent header, it says Googlebot).

I checked the logs, most visits are the Blockchain Explorer of STEEM where there are many many many pages i.e. each block has a page which will be seen unqiue to Google bots.

There used to be a setting in Google's Webmaster to slow down the crawler access, however this was hidden in the new Version of Google's Web Master.

Anyway, a few solutions:

1. Do nothing - the site experiences short downtime and will be back to normal once the bots are gone.
2. Tell Googlebots to slow down by sending 429 Too Many Requests (e.g. by implementing a Rate Limiter)
3. Buy a much powerful Server (e.g. Dedicated servers)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Thank you for reading ^^^^^^^^^^^^^^^

NEW! Following my Trail (Upvote or/and Downvote)

Follow me for topics of Algorithms, Blockchain and Cloud.
I am @justyy - a Steem Witness
https://steemyy.com

My contributions

• Video Downloader
• Steem Blockchain Tools
• Free Cryptos API
• VPS Database
• Computing Technology Blog
• A few useless tools
• And some other online software/tools
• Merge Files/Videos
• LOGO Turtle Programming Chrome Extension
• Teaching Kids Programming - Youtube Channel and All Contents

Steem/Swap to USDT Swap

I also made this Super Easy/Handy Service to Convert your STEEM or SBD to USDT (TRC-20)

Delegation Service

Voting Power Considered in Voting Schema and Important Update of Delegation Service!

• Delegate 1000 to justyy: Link
• Delegate 5000 to justyy: Link
• Delegate 10000 to justyy: Link

Support me

If you like my work, please:

• Buy Me a Coffee, Thanks!
• Become my Sponsor, Thanks!
• Voting for me:
  https://steemit.com/~witnesses type in justyy and click VOTE
  

1. Delegate SP: https://steemyy.com/sp-delegate-form/?delegatee=justyy
2. Vote @justyy as Witness: https://steemyy.com/witness-voting/?witness=justyy&action=approve
3. Set @justyy as Proxy: https://steemyy.com/witness-voting/?witness=justyy&action=proxy
   Alternatively, you can vote witness or set proxy here: https://steemit.com/~witnesses

---