SteemData now has new, multi-threaded blockchain indexers. This is an long-overdue update, bringing better scalability, performance (one order of magnitude increase) and consistency guarantees.
Further, SteemData has been re-synced from scratch to address past inconsistencies.
Syncing Status
- Operations (100%)
- Accounts (100%)
- Account Operations (99.9%)
- Posts and Comments (93%)
The database is currently @ manageable 150GB. You can see the status of the indexers and other basic stats here.
Real-Time Updates
The last open concern is state change propagation in near-real time.
For example, a single vote operation should trigger cascading updates of all affected Account, AccountOperations and Posts/Comments entries. This is because the vote affects the voters state (voting power), voters history, and the post/comment the vote was cast on.
As the Steem Blockchain usage grows, the amount of dependent state that needs to be updated with each block grows exponentially. Scaling SteemData infrastructure to support this naive model might be tricky in the future. I am currently evaluating 2 options: Using an indexer, which is more efficient and offers better consistency guarantees, vs using a distributed task queue which offers higher capacity (scalability). I hope to have some time this weekend to reach closure on the matter.
hey man,
pardon the off-topic comment…
I made a post today regarding a large-scale idea to advance Steem’s development, and it was recommended to share with the witnesses to help get this in front of the audience who’d be in the position to do something with/about it:
The $1 Billion Steem Development Fund: How Steemit Inc.'s Stake Could Be Best Allocated To Grow A Thriving Network Of Applications And Users...
would be cool if you could have a read, and IF you feel it’d be a great idea that’d serve the community, forward to anyone in particular you know who might be in a position of influence to advance the discussion.
either way, I thank you for your continued service to this community. 🙏
cheers,
Rok
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Thanks for giving your service to this community! I truly appreciate that you do this, and I'm happy to support you in any way I can. Even though I don't understand it at all I know MANY depend on it, and that ultimately enhances my own steeming experience. Thanks for doing what you do so that I can come here and play!
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Maybe this post can clarify you some questions. https://steemit.com/steem/@lightproject/recap-why-steemdata-is-so-important-for-steem
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Yes! Thanks for bringing that to my awareness, @lightproject! This community is so helpful and resourceful!!!
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Good job!
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Excuse me @furion, accounts Collection seems to be not 100% synched. Been checking "lastpost" timestamps and are not updated. Also found couple of account that are not indexed. For instance, cannot find in Account collection account "vcelier".
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Interesting...I'm looking into it.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Just for the record. Accounts that I found missing seem to be indexed now. Timestamps tho, seem not to be synched yet. Thanks a lot in advance for the efforts @furion.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Sounds great and great project!!!
I have just tried to make some Queries. if you get the account "cljg918" ( {name: "cljg918"} )
It says that it have 0 (zero) comments in "comment_count" but if I get comments by this author there is 700 and if I do the same query in steemDB it returns 1386? Do you know why?
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
HI, your web-api ( https://webapi.steemdata.com )
returns: 503 Service Temporarily Unavailable
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Thanks for this update...SteemData very useful great work..
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
The Charts app is very insightful as well!
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
The charts app will be restored next week. Also, the Notify app is due for overhaul, and I have 1 unreleased app pending from the Steemfest 2 hackaton.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
You are the best writer @furion
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
oh my god
your post is so amzing and really helped me
thank you very much
please help me to vote my post :) :)
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
good pos
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
I need all people post plenty
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
">The database is currently @ manageable 150GB. You can see the status of the indexers and other basic stats here.<"
Does database grows linear with growth of members? Say membership grows from 0.5 Millions ten times to 5 Millions, would database increase to 1.5 TB?
Would then every witness need to have 1.5 TB space reserved for Steemit? I' just curious how is DB scallability working, what happens when membership grows say 100 times.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
It all depends on the user activity
For example In 'Accounts' collection objects I see arrays like 'tags_usage' or 'transfer_history'
So 1 user using a lot of tags and doing a lot of transfers takes more space than 100 inactive users
I think the size of the DB will mainly depend on number of posts and comments no matter if they will be done by 1000 users or 10 very active ones
But I'm not @furion so he would need to validate this
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
This is an interesting read, quite interesting and good content. Thanks.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
NICE good to know thanks for the update bro!
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Great updates!
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Nice! Any Viewly update regarding release?
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Good to hear. Steady improvement is the key for long term success
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Great job
I still see missing posts hope the sync will finish soon
DB response times even for bigger queries is great
You either did a good job with Mongo perf tuning or have nice hardware for this :)
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
I am not in the "Accounts" list. All other i checked were inside. Any clue?
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Thank you for the update! :) Looking through a bit this seems interesting! :3
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
SteemData seems much fitter overall, just made a quick check and many TimeStamps are synching!
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Hi @furion I can see you good heart and hardwork to support steemians. That's why I upvoted you as one of the good witnesses. May you continue to support us! @jejes
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Good
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
When you repeat the same comment on multiple post you sound like a bot! If it walks like a bot, squawks like a bot, it may be flagged for being a bot!
Your Reputation Could be a Tasty Snack with the Wrong Comment!
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
How long to live mysql base? :)
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Congratulations you are a true writer @furion thanks for the information
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
still searching for the best steemit app to use.. the current one running on my android doesn't give me notifications and it always have a blank/white screen when i launch it.
I love what you are doing @furion .. i hope this update helps develop a better application for Steemit
Keep Steeming!!!
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Good post
My name is muhammad rizal
I'm a new member of steemit
Nice to meet you
Please followback and upvote my post https://steemit.com/picture/@riskiakbar/drawing-of-3-dimension-8c267d06ca597
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Thank you for sharing @furion ...
This is very helpful ....
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
voted and followed you :D
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Thank you @furion for your great support.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
hello@furion, i am @princessjoyesto from philippines. I just want to inform you that I voted you as my witness thank you in advance, have a great day ahead.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit