Performance Degraded: Investigating the Nodes

Since yesterday, many services were impacted. As I am currently travelling, so may not have enough time to fix this properly. But I have investigated a bit and identified the root cause.

Broken RPC Node

Most of the services use a list of RPC nodes, and one of them malfunctioned and returned outdated data. I guess the RPC node got stucked.

I have restarted the services yesterday, but today, it got problem again. And I see that when it switched to that problematic node, the services broke.

Fail-over

There is a fail-over so that when a node is down or simply being rate-limited, it will be switched to next available node (Round-robin scheduling). However, there is currently no checks for the actual data returned. The "down" check is more like checking the HTTP status to see if it is 200.

I have removed the broken node from the list, and will closely monitor this.

TLDR;

When a RPC node returns outdated data but with a 200 success response, it will mess up the services. We need to have a better way to identify this (at a minimal cost).

Steem to the Moon🚀!

• You can swap the STEEM/SBD to SOL Solana via steem2sol!
• You can swap the STEEM/SBD to ETH Ethereum via steem2eth!
• You can swap the STEEM/SBD to Tether USDT (TRC-20) via steem2usdt!
• You can swap the STEEM/SBD to TRX (TRON) via steem2trx!
• You can swap the STEEM/SBD to BTS (BitShares) via steem2bts!
• Register a free STEEM account at SteemYY!
• Steem Block Explorer
• ChatGPT/Steem Integration: You can type !ask command to invoke ChatGPT
• Steem Witness Table and API
• Other Steem Tools