SteemDB.com - Migration complete and a few thoughts on it's future.

in steemdev •  7 years ago 

Well, it took longer than I anticipated, but SteemDB.com is slowly coming back to life.

The data migration I started a couple days ago completed earlier today and I'm bringing the services back online. If you still see the "DB Migration" message still (on every page), your DNS has not updated to the new server. The DNS should update in the coming hours and the site will then become available once more.

If you're very impatient, you can always access the site via it's IP address: http://144.76.176.150

New Server

The new server SteemDB is hosted on is now provided by @privex, and I've had great support from @someguy123 getting this all setup.

SteemDB went from a 96gb RAM server with 8 cores to a 256gb RAM server with 12 cores. I've optimized and upgraded all of the software to the latest versions, configured the filesystem for high IO throughput, and should have plenty of room to grow over the next year or so.

Current Issues and Status

A few notes about this new deployment and issues surrounding the migration:

  • The database sync is currently catching up, as I write this it was processing block 18636474 (2018-01-02). So currently the 9 most recent days of data are missing from some sections. This new server is processing these blocks at a rate of about 3-6 blocks/second. We are currently at a height of 18892572, which means there's about 250k blocks that need to be processed. At a rate of 3 blocks per second (to be conservative), it will catch up to the current block height and finish processing in about 24 hours.
  • The witness pages and other sources that pull data live are unaffected by the sync process, but pages like an accounts transfers/votes will be impacted by the sync.
  • The account history module is disabled for the next 24 hours, so no new snapshots of accounts will be created until the sync is completed.
  • The apps page is missing data and somewhat broken, and I will investigate what's happening here. It's most likely due to the inactive history module, which also creates snapshots of app usage.
  • The homepage is no longer cached, so it's somewhat slow. I'll work on caching this again in the future.
  • Any missed blocks by witnesses during this migration were not recorded unfortunately.

Where SteemDB goes from here

Over the past month I've been doing a lot of thinking about the various projects I maintain in regards to the Steem ecosystem, and one reoccurring thought I have is that I have too many active projects. I am still in the planning phases, but one of my goals in 2018 is to shed or offload some of this workload so I can focus on the projects that actually matter to me, as well as the ecosystem as a whole.

With that being said, SteemDB isn't one of the projects that I want to personally stop maintaining. For months now in casual conversations I've mentioned that I really want to rebuild SteemDB and create a v2 of my block explorer code, and I am hoping to find the time to do so this year. SteemDB was my 2nd Steem related project (created Aug 2016) and I've learned so much since it's creation.

In 2018, I would very much like to take the time to sit down and rebuild SteemDB into something better. Improve the technology, the scalability, it's responsiveness, and usefulness. These foundational tools (wallets, blockexplorers, etc) are crucial to on-boarding new users and developers alike, while providing external transparency into the operations happening on the blockchain.

Luckily, this new server has room to grow, and I can fit a v2 right alongside v1 until it's ready to replace it :)

Authors get paid when people like you upvote their post.
If you enjoyed what you read here, create your account today and start earning FREE STEEM!
Sort Order:  

Awesome, now lets get some content to the top thats has some artistic nature to it! Right now its all dominated by memes and blockchain news. Please check out my new grand canyon time lapse films if you have a moment. https://steemit.com/video/@cinemaninja/bmvu7-grand-canyon-a-4k-time-lapse-film-ninja-cut Thanks!

I couldn't agree more. Once we start having less of a crypto focus and more of a general focus... that's when Steemit will really start to take off.

Good luck taking this app to the next level.

Hi @Jesta,

I heard you during the meeting on Discord in December. I use this comment section as an opportunity to thank you for the work you are carrying out for the community. I discovered Steemit about 3 months ago, am now completely immersed into it and am pretty amazed about what is going on here!

And a lot of this originates from people like you.

So, warm thanks :-)

It's a pretty deep rabbit hole :)

Thanks!

@jesta my Dear friend Your Such Great posts Check My Blogs Did You like Support Me Thank you bro

yes its great.
thanks for nice words.
@qami

we're pioneers ;)

Best of luck for your data migration work, I hope you did it very well and in the future you will do better work.

That's a monster server!

Out of curiosity is the data stored on SSD drives?

Yup, all stored on SSDs, and the entire DB is kept loaded in RAM for even faster access.

Explains why you need that monstrous amount of ram lol kudos!

Sweeeeet!

hy

Checking it now, very cool!

so interesting!

  ·  7 years ago (edited)

Never knew this wesite and I know almost all the tools. Thanks man, I'll check it out!

Hey all!
I have put together some picks. They are low market cap coins that I believe have great potential for profit. Obviously there is risk and do your own diligence before buying. I am not a financial planner or advisor.
In no particular order:

  1. AirSwap AST (Binance)
  2. Lamden TAU (EtherDelta)
  3. Loopring LPC (Binance)
  4. aelf ELF (Binance)
  5. Luxcoin LUX (Cryptopia)

(Follow me on my new Steemit for daily picks, and have an excellent day!)

Okay I will check

Nice

your information is good

I bought AirSwap in Binance some weeks ago.Good call

good . thanks for sharing nice information.
@qami

Absolutely. All the analytics sites that are being built are a huge help to managing accounts.

You are right bro, the system needs more tools to gain control of our accounts.
@racoo

I'm learning about new tools as I don't know about more #steemit tools.

Yes, I've the same minings as you, shohana1

wow we both are learning, nice to meet you new friend

Nice

I also will check this out! Thank you 👍🏽
Upvoted 👍🏽👍🏽👍🏽

upvote

Thanks!

How did I not know about this, the website looks amazing man! Glad someone tipped me off to follow you, please keep improving the site, I think it's looking clean and professional already, and so much info, this is an awesome find for me.

Voted for you as witness, really appreciate the work!

true, totally agree with you! there needs to be more awareness of the site!

WHATAVAR YA DU PPL : HODL !

Nice

it's good!

Finally I can see how much my reputation SUCKS :P

better than mine. I'm not even worth 3 bucks. Way to depreciate one's organs, steemit.

Well, it took longer than I anticipated, but SteemDB.com is slowly coming back to life.

Glad to know @jesta. Thank you for your contribution! SteemDB is a very helpful service!

Sounds like things are progressing well. You always do the best work and i can tell you work hard. Thanks for everything you do here for all of us

wow, thats some upgrade ,from 8g to 25g, its nice to have steemdb.com back online, missed using it,a very resourceful steemit/steem data collection site

I love jesta creations. Thanks for the hard work you do. We need it. :)

I agree with you, we need it.

Thanks for the update, @jesta. Which projects are you thinking about offloading? I agree that a richly featured block explorer is vital for people to understand what's going on under the hood. I'm glad that's not the one you are considering dropping.

Thanks for the updated information and looking forward to SteemDB.

Glad to see it coming back as I know so many of us rely on it!

Great though on future.

Are there plans for dealing with the copy/paste reply bots that people are using to game this system? How about for disincentivizing people to post crap comments that contribute nothing? These kinds of things are going to be very bad for this platform in the long run if a stop isn't put to it early on.

While I agree with your sentiment, the question then becomes (as in the real world) who is to judge what is a bad comment or in searching for the users gaming the system? How do we find the middle ground where that type of system doesn't censor opinions that are unpopular or that deal with sensitive topics? The second this turns into a Twitter or Facebook with censoring users I believe this platform is heading in the wrong direction.

I agree, they are problems. There's a lot of talk happening on this subject at this point between the witnesses but no concrete answers as of yet.

SteemDB won't solve those problems though :D

Thanks @jesta for bringing back SteemDB.com online again...

Great, it would be super cool to have a chart area also!

The only place for steem to go from here is up!!!!!!!!

Say no more.

I will check this. Thanks for sharing

I have no idea about the #steem ecosystem. If it's growing up, that's good sign.

@alexKARKI

@jesta did you do all this? Wow, you must be so smart!..

you guys are doing a great job

@jesta Goodluck hopefully it is going to be fully functional soon!

sounds good, I remain hopeful of the future

100% UpGoat.

Amazing and very interessing post thank you for sharing
resteemed and shared in the steemit

I'm glad to hear that you will continue to support SteemDB. Thank you!

Great job man, big BRAVO!

This is great news! I regularly use steem.supply and steemd, but less familiar with SteemDB. Would love to hear from others what the most useful data on SteemDB is so that I can better take advantage of it!

Que buen recuento, tratare de volverlo a ver, para aprender mas .

Nice

This is the best post on Steemit. I gave you a vote now, ya heard?

I keep seeing this comment copied and pasted all over from different individuals. The exact same words are even on my last reply I recieved on a post. Is this a group ?

happy to join in steemit, successful greetings for steemit lovers,
continue to work
hopefully hope and dream of steemit users in this year can be achieved all..!

  ·  7 years ago Reveal Comment

Dont buy ripple its the banks...buy bitcoin

I've missed SteemDB, welcome back!

  ·  7 years ago (edited)

@jesta thank you for the SteemDB!

This are very good informations. thanks

A few days were in great trouble.I'm glad that the problem is partly healed. Thank you for the information.

  ·  7 years ago Reveal Comment

i am happy to see this post. since i joined i started using SteemDB.com and depended on the site. but i had some problems and as a new user thought i did some mistake and it didn't work for me. late i came to know there are some issues and finally they are getting rectified. good to know this. when we are used to one site conginuing on it is very conveninet for me.

finally it is back, everytime I went to visit this site, always got the error|

Cracking work you have done.. witness vote your way!!

Is this what you mentioned?

this is great and useful . thanks!

Glad to know @jesta. Thank you for your contribution! SteemDB is a very helpful service!<3 Resteem

I really have alot to research about this, because i honestly didn't understand more than half of this

wow! so youre the one who is running this thing. Man you rock!! this thing helps, the realtime data and stats. thanks for this.

I just started last week and I only had it for a day or two before it went down and I was wondering what was up with that. Thanks for the hard work.

wow..!!!! that's great

good

I'm not familiar with this site but I had a good omen when I checked it out

Hey that was really amazing man .. we need something like that to learn from ❤
Thanks buddy ❤

thanks @jesta, this article is very useful for me, thanks many of my brother

How did I not know about this, the website looks amazing man! Glad someone tipped me off to follow you, please keep improving the site, I think it's looking clean and professional already, and so much info, this is an awesome find for me.

Voted for you as witness, really appreciate the work!

How did I not know about this, the website looks amazing man! Glad someone tipped me off to follow you, please keep improving the site, I think it's looking clean and professional already, and so much info, this is an awesome find for me.

Voted for you as witness, really appreciate the work!

Didn't knew this part of the steem ecosystem. Nicely done, keep on doing the good work!!!

Wish you best of Luck

Nice 👍👍

hi to work well with your willingness to pay for your work

Thanks for this endeavor, I appreciate it.
The system needs more tools to gain control of our accounts.
@racoo

Wow the website looks amazing, i will check it out soon!!

All I can say is that the future looks bright and secure. I just hope the programmer breaks the code before they steal it. Hahaha!!!

amazing! now I don't have to click on page by page to check the value of steem vs btc.. thanks for this @jesta. helps alot!!

Really thannk you for this service!

Great feat! You inspire me greatly with this project! Keep up the good work!

This post has single-handedly dealt with issues of immense importance. It was so helpful especially to new people here'. Thanks @jesta

Do you have a discord group??

@jesta thanks

  ·  7 years ago (edited)

Thank you, will check it out. I'm already using your steemstats monitor and find it very useful. Just voted you for a witness, thanks for all your work to make stemmians lives easier

Wow, I'm quite excited that I belong to this great family of diverse opportunities

We cannot create an account unless we have steempower already? Thanks :)

great information sir. Hope you could follow me someday

It requires a lot of effort, thanks for the had work :)

all the tools is old add something new in the site.

nice..i am hopefull you will upvote my posts also

So thankful for the awesome information, waoooh this is rich. Thank you.

Great info and I will check it out. Voted you as witness.

@jesta what is happening with
https://golosdb.com ?
seems it is falling weeks behind

Probably the same issues that SteemDB has been suffering from over the past couple months.

Over the past few weeks I've made tweaks to golosdb, but the RPC nodes available to retrieve data from are also very slow on the golos end of things. It's still running... just slowly.

Yet another reason why I really want to start working on a new version of the explorer (so it better supports multiple chains).

thanks a lot for such information as I look for some answers which I get from here!

very good

Thank you for your contribution! SteemDB is a very helpful service! ;)

you have to be patient

follow me and I follow you @sahari

Excelente iniciativa @jesta
Como verás hace poco me inicié en Steemit y realmente estoy fascinada con todo lo positivo que se maneja en éste mundo de aportes.
Considero que vienes trabajando de una manera genial y te deseo muchos éxitos y más crecimiento...!!!

Thanks for that information sir. Hope you could follow me someday

I hear that we can earn steempower when comment? really?

Thanks for the update

Excellent! I was going to test with connecting apps I was developing only recently and appreciate your efforts, and monster server :D Are there any public projects that I may help out with? Cheers!

i ever seen this site...after all reading your blog i saw that and i kindly requesting to improve your site, i'll check out of there

Good post! .. I will follow you follow me too and we help each other ... I would appreciate a vote in my last post thanks
https://steemit.com/openmic/@misc.jocs/my-most-appreciated-possession

Thank you so much for all the hard work you have put into the site. It clearly shows. Right now I've been trying to tell my friends to migrate over here and leave the exploitive nature of so many other places. I feel really cherished on here.

Hey all, I'm sorry to be commenting on an unrelated article, but I am new to STEEMIT, and I would like to do livestreams a few times a week. How does one start this process ?? Any help is greatly appreciated. Thank you all.

I'm not sure I have a great answer for ya on how to accomplish that. I've seen people that write posts when they start live streaming, to direct them to other platforms where the live stream is happening, but I'm not sure how that'd work here. Sorry I'm not of much help in this regard!