What the heck is custom JSON anyway?
For the non techie Steemians out there. Here is what Wikipedia is saying:
JavaScript Object Notation (JSON) is an open-standard file format or data interchange format that uses human-readable text to transmit data objects consisting of attribute–value pairs and array data types (or any other serializable value). It is a very common data format, with a diverse range of applications, such as serving as replacement for XML in AJAX systems.
JSON is a language-independent data format. It was derived from JavaScript, but many modern programming languages include code to generate and parse JSON-format data. The official Internet media type for JSON is application/json. JSON filenames use the extension .json.
Basically, it is a standardized data format that can be used by a different application, providing interoperability between different programming language. It is a very simple format as well. It goes something like this:
"account":"dalz",
"reputation":"66",
"posts":"268"
You get the idea. It can than become a bit more complex with nesting more data in, but that’s not the point here.
If you want, you can broadcast a custom json transaction and just store text. Maybe you have some important text to store. You will just need to remember the block and look at it in the blockexplorer.
The Steem blockchain has this ability to store json data. It has been a secondary thing for the blockchain, as the main purpose is posting, commenting and upvotes. The json data is not fully displayed by the steemit.com and the tribes frontends as posts. They may use some of it, but usually it just sit in the background, and one of its first usage was followers. Nowadays it is a totally different thing.
Share of Top 5 Avtivities on the Steem Blockchain
Here is a chart I complied from the @arcange stats.
It is based on daily data on a given day.
The above are the top five activities on the blockchain that happens each day. They consist more than 95% of the overall daily activities.
The json data is the blue color. In the last 11 months the share of the json activities on the Steem blockchain has grown from around 20% to 50%. In absolute numbers it has more than doubled from 200k custom json per day to more than 400k per day.
In this same period the number of posts and comment per day has went down form around 60k to 30k. A totally opposite pattern.
Distribution on the top 5 transactions March 28, 2019
Distribution on the top 5 transactions January 28, 2020
The voting was a top transaction before, but the json transaction have overtaken the lead.
Who is making the custom json transactions?
Users basically. But from which dapps. Here are few:
- Steem-Engine
- Tribes
- Steemmonsters
- Nextcolony
- New games
Yes, Steem-Engine, Tribes and Steemmonsters has been driving the increase in the custom json transactions.
The number of applications taking advantage of the custom json transactions and data stored on the Steem blockchain is growing. More and more apps are doing it. They just use the blockchain for it essential function to store data on an open public database that is immutable and protected.
JSON Transaction are now dominating the Steem blockchain
Looking at this numbers and the development activities it really opens up the discussion in which direction should be Steem heading. The social media blockchain? The upvote of the internet? I guess there will always be a space for that and to power up web application trough an upvote mechanism, but this other option has appeared and grow larger than the original intends.
I don’t know what exactly to think about this for the moment. How about you? Any comments?
All the best
@dalz
Posted via Steemleo
One thing that's missing from making
custom_jsona true killer feature is indexing. If you broadcast (write) acustom_json, that's all well and good. But how do you retrieve (read) it later?One way would be to buy a subscription to SteemSQL, but that's 40 SBD per month.
Another way would be to stream the blockchain from a certain block height (the genesis block of your app). The problem is, if you ever need to rebuild the data after a few years of uptime, you're going to have to replay all of those blocks again.
Steem Engine NFTs can be used for this instead. You create an NFT, then declare all of the properties your app needs. Then you create instances of the NFT (objects) and assign values to those properties. Then you can query them from the Steem Engine API.
If you goal is just pure data storage/retrieval, it might be a good solution. Plus, if you can figure out how to make the NFT instances useful as tradable assets, that's even better.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
I use the sql database :)
It is great but as you said a bit costly. Not much of an expert of the topic of indexing, but if it improves the data searching and retrieving that is great! Maybe we should look to implement this in some future hardfork, since jsons are now so wildly used and probably will grow even bigger.
The NFT options is something interesting, I was just exposed to NFTs with gerbers test game .... wasn't aware that you can do an unlimited objects with them. Looks like the NFT will open yet another door on the Steem blockchain.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
I wouldn’t say NFT objects are unlimited. The number of NFTs is limited by having to lock another token, so that’s basically a lateral cost. Also, after defining three properties, there’s a cost to add more. So there’s a width cost.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
In terms of having the blockchain itself do the index functionality, that’s a matter of scalability. If they define a dedicated plugin to index these ops, it will impact replay times, if that plugin is enabled on the node.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
The cost of data indexing doesn't go away just because a different bit of code is doing the database calls.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Correct, so instead of your app indexing just your app’s data, the plugin is indexing everything. I’d imagine that kind of solution would benefit from a setting to tell it at least which ID you’re interested in, so that if you’re running a private node, the impact is less than someone running a public node that indexes all IDs.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Sure, that seems like a reasonable approach. Or maybe a plug in that allows push or pull of relevant IDs to support indexing externally without needing to stream the entire blockchain.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Like a dedicated p2p network? Yeah. We basically get that with Steem Engine, only it doesn't rely on a C++ dev to implement. Not that there's anything wrong with that.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
I'm not sure what you mean by a dedicated p2p network. Steem Engine streams the whole blockchain right? I'm saying if all you want is a subset of data from a subset of custom_jsons, the right plug in would let you do that without the whole chain. The plug in would only need to be implemented once, then anyone (i.e. non-C++ devs) could use it. In fact Steem Engine could use it too.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Is there documentation somewhere on how NFTs on Steem Engine work?
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
You can go trough @aggroed posts on NFT where he is explaining things..
https://steemit.com/steem-engine/@aggroed/nfts-going-live
https://steemit.com/steem-engine/@aggroed/the-first-nfts-have-been-created
Also check Steem Engine discord for question...
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Thanks for the links.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
https://github.com/harpagon210/steemsmartcontracts-wiki/blob/master/NFT-Contracts.md
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Thank you!
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
I don't understand the problem with indexing. Why couldn't you store transaction id and block num into the application database and retrieve them fast? This is the way how taverngames works
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
I believe because the indexing is not decentralized. If you ever lose "the application database", you need to replay blocks since the application genesis to recover the index.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
That in itself wouldn't be that bad. At least you have a recovery process. It should also be possible to checkpoint the database to a backup periodically and then you only have to replay since the checkpoint.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
This is far more ideal.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
I don't see a problem here. Why would I lose application database?
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Yeah, I don't see the problem either. You got to be very sloppy to lose the entire database especially without backup checkpoints.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
It happens or what is more likely to happen is corruption where you have to rebuild from source.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Yes. @smooth has right. This is what I had on mine mind. You can always restore the backup.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Yes, if your app broadcasts the op, on your server, that’s a solution. But if a user broadcasts it from their browser, it starts to get tricky. Does your app then send the block number to you? Did they even use your app? Or did they use SteemConnect/Steem Keychain? Did they use a third party app that doesn’t know it needs to report the block number to you?
In a sense, the blockchain is the method of communication, so it’s redundant to also use another protocol to report the block number. Ideally, you just monitor the blockchain for new data or just some other indexing strategy, not both.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Users use my app and steemconnect as well. When they broadcast the transaction they get the transaction id and block num, then they send it to app endpoints.
Users don't use a third party apps.
If you said it's redundant to use also another protocol and blockchain is the method of communication why do you need SteemSQL then? It is also redundant.
Btw storing the data in the app application makes you independent, you don't need to count on someone's app like SteemSQL (what if they will disable it?)
You do the indexing on your side. I don't see there any disadvantages.
Also you don't need to stream all transactions from the blockchain only those that you really need.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
If you accept requests where your users report block numbers, you’re bypassing the need to index independently, which is a great solution. If your app is designed to only accept requests this way, you’re golden.
The problem with this method is if you miss a block number being reported from your users, you need some strategy to find the stragglers. Whatever process you use, assuming it isn’t a manual process, why don’t you always use that strategy instead?
Well, technically, if you’re streaming the blockchain, you are getting all transactions. You’re ignoring most of them, but that’s at least 500 MB of bandwidth, daily, last I checked.
Streaming works. But you need a strategy for knowing what the last processes block number was, so you can ensure you didn’t miss a block if your stream is interrupted.
By the way, if you have several apps all streaming independently, you should consider:
https://github.com/inertia186/meeseeker
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Yes you are right.
Btw my app works in two way. Sometimes accept the requests from users (during gaming) but I also need to stream the blockchain (for shop purchases, yes they are not tracked as you mentioned because steemconnect doesn't return with the trx id and block num).
I have the complex solution for streaming blocks, which also detects missed blocks based on previous block, also It retries to preprocess the block when some block is not yet available through the API.
I can agree that all the things depends on the application specification.
I'll take a look at this github repo! Thanks :)
I also invite you to checkout how it works in the https://taverngames.io :)
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
What happens if the user's client crashes in between the time they broadcast the transaction and the time they send the id and block number to the app?
What happens if there is a chain reorg and the transaction ends up in a different block?
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
For custom jsons it doesn't matter, user doesn't lose his STEEM only some RC amount so it can be retried.
I've never noticed the chain reorg during custom json processing. I've processed many of transactions and no issues reported.
Transfer transactions are streamed from blockchain in the background process. So I never lose them.
Come and checkout : https://taverngames.io
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Chain reorgs on Steem are usually rare, and usually only one or two blocks (so resolved within seconds), so you may not see them much. But sometimes they do still happen and sometimes take much longer to resolve, so you need some strategy for handling it.
If your app expects the user to retry or accept lost transactions occasionally, that may be a perfectly acceptable strategy.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Yes it is. Everything works as expected.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
I am using custom JSON for transactions in my game (still under development). All good except sending Tokens with Steem Keychain approval. I have problems with that and I can't find help in the community. With more documentation and support there would be many more projects.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Totally agreed on documentation ... It will be much easier for developers. Now is like asking around and figuring it out. But the knowledge is spreading and as more apps appear and people involved it will become easier overtime. What exactly is the issue you are having?
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
I'm at work now but my request is on #developers channel in Steem Engine Discord
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Thanks for ilthe info @dalz! Definitely not a techie but learning slowly through posts like these. I see "custom JSON" all the time in steemworld and knew I was making them by claiming SE tokens and things.
Posted via Steemleo
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Yep we are all making them, aware or not :)
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Thanks @crokkon ....I havent specificly for these stats.... but as I can rememeber from other stats I think the ssc id is somewhere around 40k to 50k daily transactions, meaning somewhere around 10% of the json tranasactions .... but take this number with doubt :) .... I will probably do a follow up breaking down the custom json
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
We use custom jsons to make all our games transparent by saving the game results to the blockchain and then read them on our server side :) We really like them!
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Yep ... they are great :)
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
An excellent post. This is terrific insight as to what is taking place. We are seeing a lot more activity that does not come from the blogging end of things. I knew Steem-Engine was growing in its daily activity. Splinterlands is growing and now we see a few more games going live. This will only push the activity higher.
Posted via Steemleo
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Thanks!
Looks like the trend is for dapps using custom jsons.... not sure how will smts impact this.
May do a follow up post with closer look and data who is making the json transactions.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Would be nice :)
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Hi, @dalz!
You just got a 14.16% upvote from SteemPlus!
To get higher upvotes, earn more SteemPlus Points (SPP). On your Steemit wallet, check your SPP balance and click on "How to earn SPP?" to find out all the ways to earn.
If you're not using SteemPlus yet, please check our last posts in here to see the many ways in which SteemPlus can improve your Steem experience on Steemit and Busy.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
This is great post..... I am also with Nextcolony gaming, so how can I use in this project. I want by the study this project and follow for more stuff.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
NextColony have great documentation
https://jarunik.github.io/nc-docs/
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
The JSON ability is great. It opens up the chain capabilities to more developers, allows for a variety use cases, and can be adapted to your specific needs without feeling like you're breaking the system or using a loophole.
It's cool to see more people taking advantage of them! I'd like to know the specific breakdown for each dapp & game that are using these :^)
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Yea this was mentioned already ... will do it soon :)
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit