Hello Steem, today I will be outdooring a new Steem utility I have been working on: App Stream.
During the development of Peer Query beta as a full fledged platform featuring content created only on it, we run into a challenge.
While we could index posts and comments made on our site, we could not index comments/replies made from other clients on posts created through Peer Query.
This problem applies not only to Peer Query, but also to all other Steem clients.
The challenge
Steem is great for content publishing, yet its open source data nature means that the same content can be served and interacted with from over several different interface.
This feature is good for open source and censorship resistance, however it poses a challenge for app/client owners who want to index all activities on posts made from their clients.
This is because while a post may be created on a one client, it would be edited, commented on, voted and resteemed from other clients.
Consider this challenge:
- Ama creates a query/post through PeerQuery.com
- John comments on it through raw Steem JS
- Jin replies to John's comment using Busy.org
- then the author Ama, edits the same post using Steemit.com
- Mary replies to Jin's reply using eSteem app
- countless more activities(resteem, votes, ...) happen on the same post from different clients
No matter how much indexing Peer Query does for activity on its site, we would definitely miss those made from other clients.
The only way to solve the problem is to stream all operation on the Steem blockchain and check if they match something in your DB or not.
App Stream
App Stream is designed for this challenging task, and is designed to be as modular and efficient as possible. App Stream is a standalone Node.js program that curates all posts as well as its comments, replies and even edits from whatever target you set in a MYSQL DB.
There are 3 types of target:
app
category
author
There are six Db engines:
post-by-app
post-by-author
post-by-category
post-comment-by-app
post-comment-by-author
post-comment-by-category
post-comment-reply-by-app
post-comment-reply-by-author
post-comment-reply-by-category
Countless other features and configuration instances, see Docs for more.
How to install
- Create empty DB, make sure MYSQL version is 5.7+ for
json
support - Create
.env
file in root folder and enter DB details. See sample indocs/sample.env
- Git clone
ssh:[email protected]:peerquery/app-stream.git
orhttps://github.com/peerquery/app-stream.git
- Install dependencies:
npm install
- For custom setup, edit:
/config/config.js
- Run system through setup manager:
node setup
- Use MYSQL Workbeanch/PHPMyAdmin to explore data in DB or visit
api.localhost/api
Note:
You do not need to create any tables for the DB. App Stream runs on:
- 2 tables with all kinds of fields: json, text, timestamp, varchar, auto-increment, ...
- 1 view of the each of the two tables above
- 3 complex conditional db indexing stored procedures each with 11 IN variables and 1 OUT variable
- 10 db api stored procedures with IN and OUT variables
Manual setup would be inefficient and difficult, and yet it has to be setup successfully before the main server is initialized.
The db_setup
module handles all this automatically for you, so you do not have to touch the DB at all. After all is done successfully it will automatically call the server.
Who can use App Stream?
Steem clients
Curating by app was App Stream's primary purpose, Steem clients would benefit the most from App Stream as they can use it to curate all posts made from their clients.
With the app_match
option, you could curate data from only a specific version of an app, eg: esteem/1.6.0
, esteem/1.5.1
, or esteem/whatever
.
With a separate App Stream server, clients could server all content directly from their DB instead of relying on fetching content directly from RPC node endpoints.
App/client owners now can index all posts, comments, replies as well as edits made to posts made from their client, no matter which thirdparty client the activity originated from.
Steem Curators
Curators would also greatly benefit from App Stream by using any of the category indexing engines:
post-by-category
post-comment-by-category
post-comment-reply-by-category
We're working on Curator, a revolutionary open source interface for curating content which perfectly integrates into App Stream. Stay tuned.
Steem blogs
I don't know if this is a thing yet, but with the post-comment-reply-by-author engine you can setup this system to curate all posts by an author, as well as the comments and replies by it.
While this is not recommended because Steem API allow this function through steem.api.getDiscussionsByAuthorBeforeDate()
, with App Stream's source_app
you can get specialized curation such as curating the author's post only if they are from a specific client such as Steemit.com, Busy.org, ...
Steem BI
With App Stream, you can curate all posts, comments and replies from any Steem author, category or app.
Such data would allow you to gain specialized data for specialized analysis.
Documentation
App Stream comes with a painfully long and detailed documentation. You do not need to read it all to get App Stream running, its only there to explain everything about App Stream with examples incase you need to tweak it or get lost getting something to work.
See the documentation on the Github repo or via this link: https://github.com/peerquery/app-stream/blob/master/README.md.
Hopefully the detailed Table of Contents would make it easier to navigate.
Cool addons
- Depth determiner for each post, comment and reply - will accurately add depth to a reply whose dept is 37!
- API documentation on API endpoint bases
- Subdomain support
See Docs for more info with examples.
Cool hacks
- Jailbreak to stream all Steem posts, comments and relies regardless of the source
- Host App Stream as a service and serve content via API
- Integrate with Hivemind for richer data interface
- Add a notifications feature
What's next
Curating only for the purposes of indexing in not enough. For clients such as Peer Query, we will need a full curation functionality.
I am already half way done with Curator, a powerful open source content curation interface for Steem. Curator will integrate with App Steam and allow a team(of multiple admins, moderator and curators) to curate content from Steem.
Curator comes with a full bot support which means all you have to do is supply the bot's account and posting key - it will periodically upvote all contents approved by the curators.
Like App Stream, Curator was initially being designed for Peer Query, however I realized that most of the content curators on Steem do so without a standalone app interface.
Then came the need to design Curator as a standalone app so other content curators(by tag, app or author) can also use it.
When Curator is done hopefully before this moth ends, Peer Query beta would be live; a complete move away from current query tag curation into a full platform with users, reputation, projects, curators and bots and more.
Acknowledgment
Thanks to @smooth and @howo for their recommendations on how to structure the db indexing functionality. Templates for the two streamer apps are by @almost-digital and @howo.
Thanks to you for reading, I hope you find App Stream of use.
Links
Github repo: https://github.com/peerquery/app-stream
Docs: https://github.com/peerquery/app-stream/blob/master/README.md
Thats a good idea! Ill have to check it out looks cool
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Are you planning to unleash a web version? A local machine installation is too much of a bother for me and most other people, I reckon. Besides, not everyone can do it even provided the instructions, lol,
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Thanks for asking - App Stream is a Node.js app that anyone can install anywhere: locally or on some cloud servers.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Let me put it this way, are you gonna host it somewhere? Like, make the official site for that?
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Maybe this tool will help many people especially app owner save a lot of time.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Hey @dzivenu
Thanks for contributing on Utopian.
We’re already looking forward to your next contribution!
Contributing on Utopian
Learn how to contribute on our website or by watching this tutorial on Youtube.
Want to chat? Join us on Discord https://discord.gg/h52nFrV.
Vote for Utopian Witness!
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
I really loved the amount of work you have put into the contribution.
I also liked the presentation of the post.
I would love to recommend you to use JSDoc or some auto documentator to make a simple webpage which makes this documentation more presentable and attractive as devs prefer to read
README
but some prefer to see a simple webpage as a documentation.I have staff picked this contribution because I believe its really high quality.
Thanks for providing such contributions.
Keep contributing.
Your contribution has been evaluated according to Utopian rules and guidelines, as well as a predefined set of questions pertaining to the category.
To view those questions and the relevant answers related to your post,Click here
Need help? Write a ticket on https://support.utopian.io/.
Chat with us on Discord.
[utopian-moderator]
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
thank you
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Introducing "App Stream": Curate and serve Steem posts/comments/replies by app/author/category has been resteemed by @resteemmuse
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
I agree with you apps are d best, but you know apps though can easily get deleted
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
who can delete the apps?
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
As in maybe a virus attack or something like a force factory set.
Though I still agree with you, irrespective of any short coming APPS is still the best choice so far
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Hi @dzivenu! You have received 1.8 SBD @tipU upvote from @cardboard !
Earn daily income on steem: @tipU distributes 100% profit and additional 60% curation rewards to all investors and allows to automatically reinvest selected part of your payout.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
thanks @cardboard
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Fanastic post friend
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
app steem is a great idea.... in app system there will be no posibility to hacking.....
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
wait for the readiness of the application, hopefully easier to help in interacting in steemit with this application, hopefully quickly resolved, so we will quickly use it
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
thanks for your interest, its already ready and you can use it right away.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Well, hope it helps me
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Congratulation dzivenu! Your post has appeared on the hot page after 57min with 43 votes.
Thanks to @souldelas.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit