In the concert of Jeff Bernat in Seoul, South Korea
I have just joined the STEEMIT, but now I am planning a project based on WEB App for STEEMIT . In order to extract the latest and popular articles in real time from STEEMIT, I am applying various systematic designs.
The first thing I approached was steem.js, which is available as a Javascript library.
steem-js GIT
Steemit API Reference
Although the poor documentation was found, it was okay, Steamit is still a beta service, and just thanks for opening the API. I am trying to run methods without any parameters described. Surprisingly, I am very impressed that the same data as Steamit web is coming in real time through the API.
SteemJS Advantage: Various real-time data and statistical data just same as STEEMIT web can be received
In order to apply this directly to the web application, several RestfulAPI methods were implemented and tested. Unfortunately, It took 10 seconds or more per response. Moreover, the returned data was very heavy.
SteemJS Disadvantages: Response data is too large, and it makes slower response
For an example, try loading 30 posts from trending in the life category. It takes about 10.9 seconds, and the data size is about 1.7MB. Try loading 30 posts from new in the same category. It responds quickly only 2.4 seconds, and is less than 1/10 of the size of 111KB.
Looking at the data, the data returned by the API is really "everything." post, as well as the id, weight, percentage, reputation, rshare, and every voter who has voted for the post. Of course, this is good data, but the size of hundreds of voting data is making it heavier.
Unfortunately, I could not found a way to set the parameter to return only the information I want in api request. Because of this, it had to call the this heavy api multiple times in order to perform various operations in real time.
After some researches, I found a database service based on mssql called steemsql.
SteemSQL
It seemed to be a service to cache steem blockchaing data with MSSQL, but it seemed to contain more information available than I thought. The sync gap of the server is about 2300 seconds, which is about 30 minutes to 40 minutes.
SteemSQL Advantage: faster response speed and complex query operations are an advantage
There is definately no benefit to steem.js when using queries of the form "SELECT * FROM TABLE". "SELECT author, permlink FROM TABLE ", querying only the columns you want is faster and lighter.
The strongest points of RDB are join, nested query, and aggregate function. It is a great advantage because you can perform complex operations with only one query. The number of requests can be reduced, and this means the execution time can be optimized. For example, it has the advantage of being able to solve "find users who i am following and never vote for me" and "find user who gave the biggest contribution of the reward of posts".
I like it so much.
SteemSQL Disadvantages: Unfortunately, It is not free (1SBD per day, 10SBD per month)
I do not have 1 SBD in my wallet at this moment, so I have to pay the real cash to my friend for paying for the SteemSQL and subscribe. I have been hesitant to pay for it, but I feel that it is not too expensive in terms of the quality of service. I think that it will be stable and long-term operation because it is operated as a charged service.
If you are thinking about cooking STEEMIT data, please consider SteemSQL.