Introducing utopian_data - Store public Utopian data on your local Mongodb

in utopian-io •  7 years ago  (edited)

utopian_data is a CLI application which fetches public data of Utopian into your Mongodb instance.

Supported collections:

Why

I have been working on a separate statistics application (private) for a while to analyze Utopian data better. However, using a REST API for statistics is not practical.

You need a fast system where you can make complex queries. Syncing Utopian posts' data into your system (you can set a crontab and sync it in every 5 minutes) is much better.

Example Mongodb aggreegation Queries


Total Contribution Count

db.posts.find({}).count()

mongoshell output

Top 3 moderators on blog category (moderation count)

db.posts.aggregate(
  [
    {"$match": {"json_metadata.type": "blog"}},
    {"$group": {_id: "$moderator", count: {$sum : 1}}},
    {"$sort": {"count": -1}},
    {"$limit": 3},
  ]
)

mongoshell output

Top 3 Contributors

db.posts.aggregate(
  [
    {"$group": {_id: "$author", count: {$sum : 1}}},
    {"$sort": {"count": -1}},
    {"$limit": 3},
  ]
)

mongoshell output

And more... You can pretty much filter/group everything you want once
you learn how Mongodb aggregation queries work.

Installation and Usage

You need to install and configure golang first. After that:

$ go get github.com/emre/utopian_data
$ cd $GOPATH
$ go install github.com/emre/utopian_data
$ cd `$GOPATH`/bin
$  utopian_data --mongodb_uri=localhost

Output

Console Output


Let me know if you encounter any issues.



Posted on Utopian.io - Rewarding Open Source Contributors

Authors get paid when people like you upvote their post.
If you enjoyed what you read here, create your account today and start earning FREE STEEM!
Sort Order:  

Thank you for the contribution. It has been approved.

You can contact us on Discord.
[utopian-moderator]

Hey @ms10398, I just gave you a tip for your hard work on moderation. Upvote this comment to support the utopian moderators and increase your future rewards!

something wrong with the posts ( api/posts) link.

Good for who is already friendly with those api, mongodb and golang :)
I just wanna see a list of graphic contributions, filtered by date. to see daily approval / rejection percentages and which mod intent to reject or accept etc. yet there are no end-user friendly options for it. Hope we'll get there at some point in the future.

Fixed! Thanks.

This is for developers/power users, not end users. For the end users, I have another app in progress.™

sneak peek



This one step I had to run like this:

$ ./utopian_data --mongodb_uri=localhost

As opposed to :

$ utopian_data --mongodb_uri=localhost

Nice article. I enjoyed it and will try the code. Like to see more Gophers on Steem :)

Hey @emrebeyler I am @utopian-io. I have just upvoted you!

Achievements

  • Seems like you contribute quite often. AMAZING!

Community-Driven Witness!

I am the first and only Steem Community-Driven Witness. Participate on Discord. Lets GROW TOGETHER!

mooncryption-utopian-witness-gif

Up-vote this comment to grow my power and help Open Source contributions like this one. Want to chat? Join me on Discord https://discord.gg/Pc8HG9x

If you like Mongodb then you should try Elasticsearch. You would be able to learn Elasticsearch pretty quickly. Mongodb is kinda harder to learn than Elasticsearch in my opinion. Thanks for the article.