Introducing AskSteem - A steem search engine

in steemit •  7 years ago  (edited)

Asksteem
Hello, Steemians!
Over the past month, I've been building a new search engine that indexes the steem blockchain. It's currently live at asksteem.com. The goal of AskSteem is to provide a reliable, powerful, and fast search engine that is optimized for steem. In this post, I'd like to cover some of the features that are available.

Query Syntax

There are many different ways that you can query the AskSteem index. I've created a video demonstrating each of them, but you may also read their descriptions and examples below.

Keyword/Phrase Search

Like many other search engines, you can search for general phrases and terms. AskSteem will try its best to find the document that is most relevant to your query based on our ranking algorithm.
Example Queries:
Tip: Click the example to go to that query on AskSteem
How to buy bitcoin
What is steem
Markdown tutorial

Exact Search

Putting a query into quotes requests that AskSteem only returns documents that have exactly that phrase in that order.
Example Queries:
"How to buy bitcoin"
"What is steem"
"Markdown tutorial"

Tag Search

AskSteem allows you to filter posts by tag.
Example Queries:
tags:life
tags:steemit

Author Search

You can filter posts by the author too.
Example Queries:
author:thekyle
author:abit
author:steemit

Creation Date Search

AskSteem provides a highly flexible and powerful date search tool for posts. You can search by exact date or by date range. Dates must be in the form of YYYY-MM-DD.
Example Queries:
Search for all posts posted on June 2, 2017
created:2017-06-02
Search for all posts posted between May 1, 2017, and May 31, 2017
created:[2017-05-01 TO 2017-05-31]

Search by Number of Votes/Comments

Similar to dates AskSteem has another set of robust tools that allow searches based on the number of upvotes or comments a post receives.
Example Queries:
Posts with 150 votes:
net_votes:150
Posts with between 100 and 150 votes
net_votes:[100 TO 150]
Posts with 50 comments:
children:50
Posts with between 40 and 50 comments:
children:[40 TO 50]
Posts with more than 50 comments:
children:>50
or less than 50:
children:<50
this also works with votes, less than or equal to 10 votes:
net_votes:<=10

Searches with Boosts

You can prioritize certain parts of your query with boosts. These are indicated by placing a ^n at the end of a term, where n is the power you want to boost that part of the query to.
Example Queries:
Give the term bitcoin a boost of two:
I really want posts to have the term bitcoin^2 in them.
Give the term mine a boost of two, and term steem a boost of three:
How to mine^2 steem^3

Inclusive/Exclusive Search

You can indicate whether you want documents to contain certain terms by placing a + or a - in front of the term.
Example Queries:
Find documents about mining but not bitcoin:
cryptocurrency +mining -bitcoin

Wildcard Search

You can use the wildcard expressions of ? for a single character, or * to match any number of characters.
Example Queries:
How to mine any cryptocurrency:
How to mine *

Boolean Search

AskSteem supports any combination of the previously mentioned search types in a single powerful query. This uses boolean values of AND, OR, and NOT, along with parenthesis to separate statements.
Example Queries:
Posts tagged with asksteem by @thekyle:
tags:asksteem AND author:thekyle
Posts with between 50 and 100 comments that have more than 500 upvotes and that are tagged with 'bitcoin' or have the term bitcoin in the document:
(bitcoin OR tags:bitcoin) AND (net_votes:>500 AND children:(>50 AND <100))
Posts created on June 2, 2017, with 100 or more upvotes but less than 10 comments:
created:2017-06-02 AND net_votes:>=100 AND children:<10

Developers


Because AskSteem integrates directly into the steem blockchain it can read metadata directly from posts and use that data when performing queries and displaying results. We encourage developers to add AskSteem compatible metadata to their posts so that we can show links to your application in our search results. The full documentation can be found at asksteem.com/developers, however, in this post I will summarize the most important tags.

TagDescriptionExample
domainThe domain name or web address that your application is hosted on.example.com
locatorThe path to reach the post on the domain relative to the root./CATEGORY/@AUTHOR/PERMLINK
protocolEither 'http' or 'https' if not provided then http will be used by defaulthttps

If none of the above metadata is provided then AskSteem will link to steemit.com for all posts by default, however, it is assumed that the platform creating the content will have the best interface for viewing it, so we would rather link there.

The domain and locator tags are required for custom linking to work, however, the protocol tag is optional and will default to http.

  • The domain tag should be the domain name that your web-based steem application is hosted on and is subdomain sensitive (so if your hosting on www subdomain then put that).
  • The locator should be the permalink to that particular post in your applications URL structure, also notice the leading forward slash, this is required.

The final URL that we point to will be generated by concatenating the domain and locator together with the protocol at the beginning which will be http unless otherwise specified.

Additionally, if you are building an application on the steem blockchain and need a search API please email us at [email protected], we are able to query custom metadata and make various other customizations to the ranking algorithm to support your use case.

Funding

The harsh reality is that search engines are expensive to run and that adding new features and improving performance are difficult if the basic funding needs of the project are not covered. AskSteem currently costs me about $100/month to run, and that number will, of course, continue to increase as steem grows and the index size increases.

Ideally, my goal is the have those costs covered through upvotes from the steem community and to use any extra money for adding new features and scaling the search infrastructure to meet demand. If this works then AskSteem will be the first search engine in the world to use a cryptocurrency based revenue model, instead of selling advertising.

Thank you for your time, and happy searching!

Authors get paid when people like you upvote their post.
If you enjoyed what you read here, create your account today and start earning FREE STEEM!
Sort Order:  

Happy to see this. I resteemed.

  ·  7 years ago (edited)

Thanks, I look forward to making it even better!

yay

How do your 'Resteem' a post?

Highly recommended tags for such post #steemdev and #steem-project

You almost missed out on 300$ of reward by not having this tag.

Well thanks for letting me know, I'll make sure to include those in future posts.

you can edit

Done!

This is something many were waiting for! Tried and it works well! It has to be developed, congrats for the project!

Yes thank you! I can now search then click back after reading an article and it will take me back to my search results. Small thing but needed! 😎

Glad you like it!

Looks great. We have the metadata, so searching by that should make life easier. Thanks

Hi @thekyle.

I'd like to tell you how happy I am for the lesson, not just on asksteem.com, but also on how to do a search and ferret out what might otherwise be obscure - but important - results. I've been doing a little research now and then on a couple of open source investigations, and just learning more about how searches operate will help me to research for my own topics. It will also help me better assist the activists and freedom-fighters who are busy helping us reclaim our democracy.

It's a pleasure to meet you. I'm @joanroberts, and I've been here for a whole week. Forgive me for not upvoting - my (literal) 2 cents is going to go to someone else, who is new, but who, like yourself, adds to the community. Trust me. When your vote is only worth 2 cents, when you see someone upvote you and you've got 4 cents, while it doesn't make you rich, it does make you smile.

Hi Joan thanks for your comment. I'm glad you're enjoying AskSteem and I hope you'll continue to use it for your investigations. Have a 23 cent upvote from me. :)

@thekyle, you just brought tears to my eyes. Thank you.

Great tool, thanks for building this. Is there a way to sort the search results? For examply by number of votes or comments? Or does anybody know another tool that does this?

Not currently through the Web UI, but that functionality is available in our developer API so it may come in the future.

Ah great, you have an API. Thanks for answering, I'll definitely look into that.

@thekyle please repost this for the benefit of those who haven't had a chance to see it. I wanted to resteem but that option is gone as the post is too old. Old posts can still be a valuable resource for us on Steemit.

I'm not going to repost because I don't want to put duplicates in my feed, however, feel free to resteem this newer post about user search functionality if you'd like to spread the word.

Great post! resteem! I hope one day it will have half of google traffic :)

Good work!!! Followed, up-voted and re-steemed the post for more visibility! I hope they can add it here on Steemit and replace the searcher we have now, it is really bad.

Yeah, I would definitely love to see some steem clients (including steemit.com) start using AskSteem as their search backend.

That would be perfect! :) Nice work man!

This is good. Got a question though: Is there a way to check who resteemed a post?

  ·  7 years ago (edited)

No, and sadly its not likely to be added in the future because resteems are not part of the blockchain and are kept in steemit.com's private database I have no way to index them.

Too bad :/ Thanks for the quick answer.

The goal of AskSteem is to provide a reliable, powerful, and fast search engine that is optimized for steem

Great effort @thekyle

Thanks!

You're welcome, @thekyle

This is really cool! I added you to my Tools Collection

Thanks! Looks like a cool list so I'll have to check out some of the other stuff on there.

I like everything but the name. Expect slight competition from SteemCrawler and SteemaVista, but then Steemhoo and Steeng, will come in too. Hopefully before all that you will be bought out by Stoogle.

Haha well, thanks. Glad you like the concept, and maybe the name is one of those things that grows on you over time. :)

cool stuff ...resteemed

Thank you for providing a Steem search other than Google!

Yacy might make your indexing efforts more distributed and cost effective.

Yacy is a great Distributed Search engine. You can host a node that only indexes Steem in collaboration with anyone else who wants to index steem and share results. Yacy runs on Desktops, laptops and servers. Linux, Windows and Mac.

http://yacy.net/en/index.html

There is also a built in interface for your site that works out of the box. Together we are strong. Take a look if you're interested.

I have a basic demo that indexed some of steem and reddit on my Desktop here. http://camb.hopto.org:8090/
(this demo won't be up much longer)

I will be moving it to a proper server soon and begin indexing only distributed networks.

Good luck!

Very interesting, I'll be looking into this. :)

Yacy has been an undervalued project for years. It works great. If you have any questions getting started, feel free to ask me and I will answer as best as I can.

Hope my upvote helps a bit! Really cool tool!

Well they say every little bit helps. :)

Thanks, it looks and works good!

This is an awesome tool! Thank you for taking the time to create this for the benefit of all Steemians. ReSteem, followed.

Glad you like it! :)

About time, lol GREAT job man ! This is just ... well, Great !!! Hope more folks will start to appreciate the value of this Search Engine, got to pay the bills !, Good luck ! Come on Folks, vote it UP !!

Read More, Reason More ... JTS

Amazing work @thekyle. Hopefully the Whales will upvote. Re-steeming.

Really glad you like it! And thanks for the resteem.

I was just thinking today I f*cking hate the search bar on Steemit. Glad someone else thought the same and did something about it. Resteemed. Hope you get the funding! Cheers mate!

  ·  7 years ago (edited)

Thanks, good to see I'm not the only one who saw this as a problem that needed solving.

Kyle, Thanks for a great too. As a trader in all markets, your search tool helped me find two other Steemians who also post market and trading data.

Glad you found the tool useful. :)

Thank you, resteemed and have a question:

Is it possible to combine 2 search tags like author and exact search in one search request?

  ·  7 years ago (edited)

Glad you like it and yes. You can type author:thekyle AND bitcoin to get posts created by @thekyle (me) about bitcoin. This works with any two types of search. For more info read the part about boolean search in the above post.

Thanks for answering!

Very cool! Good job. Upvoted.

Glad you like it. Thanks for the upvote, every little bit helps :)

Great job, a search engine is sorely needed!

Great, Very likely awesome thing! thanks for it, resteemed it !

Woow....genius, here to lend all the needed support

Awesome! I'm gonna need all the support I can get. :)

Solid!

Awesome.... Just upvoted and Resteemed...

nice work, though it would also be great if an author's comments could be searched as well!

Thanks, I haven't really had any plans for allowing users to search authors comments, but if people start making comments that might answer queries then perhaps in the future I will allow comment searching. :)

I just know that for me, I often find myself searching my past comments in reference to a new post or comment (or even in chat).

Just tried it and it works!!
God job.

I had a friend on here @kaptainkrayola who had a nice and effective search engine running for steemit called steemshovel.com until Feb 1st, 2017. He ended it up shutting it down for exactly the reason you indicated. No funding, and he was actually losing money by running it.

Yeah, I looked into SteemShovel before launching AskSteem. Technically in the backend both search engines use ElasticSearch for indexing the blockchain, and think SteemShovel may have also been hosted on AWS. Hopefully, my goal of funding the search engine through upvotes will work out, otherwise, there are some other ideas I have for getting funding.

Yeah my friend didn't post here enough or he likely could have funded it. He is a developer, but didn't really want to blog.

Funny enough it was him talking to me about Eth mining that caused me to stumble on steemit, before him. He was kicking himself and wondering how he missed it.

If you need to use the account @asksteem let me know :)
It's being used to ask questions , but maybe it can be used for people to query right inside steemit to pull up info from your search engine.

Yeah, I could build a bot that allows people to query the index from steemit directly. Are you available on steem chat?

Yeah, I could build a bot that allows people to query the index from steemit directly. Are you available on steem chat?

I am good sir. Steemitqa under steemit.chat.

Yes, it would allow people to query the index from steemit directly and come back with the results.

This seems like a interesting and helpful resource, thanks for sharing.

It would be really nice to have a search engine dedicated to steemit! Waiting for the tool to go live.

It is live! https://www.asksteem.com :)

Boolean + tag search + everything else = Winner! We have been needing something like this, and perhaps it can be implemented on the blockchain itself in a future hard fork? Got to replace the current search bar with this!

It unlikely that you could include a full search engine inside the of the blockchain and make it as fast as AskSteem, however its possible in the future that they could include the ability to natively sort posts by tag (I know I was surprised the blockchain didn't have that already too)

Out f'n standing!

This looks great. I'm sure it will be way better than current searching methods. Following for new updates. ^^

Genius!

Nice job! Thanks so much for this!

oh hell yeah.
we've needed that for a long time.
you got my vote.

Resteemed

This is fantastic

I can already think of so many ways AskSteemit will help Steemians. No, not just newbs who are starting.

I'm saying this tool will be especially useless, ESPECIALLY, to steem veterans who wanna up their game.

I'm Glad this initial post has help covered at least 3 months cost and I'm sure the community will be happy to support you all the way.

Your tool will be part and parcel of many of us in our Steemit life.

Thanks for "coming out of nowhere" with this amazing engine. I can say your tune and effort spent will be worth it.

Awesome!!

Are you using ElasticSearch?

Yes, we are.

Sounds like a great idea. Bookmarked and upvoted!👌

Hello, i just got here..
But..

If you have time to check my blog,
Maybe you can help with my proposal..
And say something about it.. My pleasure

Good job @thekyle! Do you use LUIS (Microsoft services) or Amazon services to implement this cognitive search?

We use ElasticSearch hosted through AWS, with our own custom algorithm on top.

Interesting approach, I also use ELK on my daily job but I didn't expect to do that only with ELK. Congratz! :)

Thanks, we I'm hoping to also include some type of natural language processing in the future to help the search engine better understand posts.

Yes this will be the best option. I have played a little with language processing and image recognition services from LUIS and I had a good experience. :)

Very cool concept! I'm still wrapping my head around exactly what we can build with the steem framework.

I'm all for anything that takes advertising and personal data mining out of something as common as a search engine!

Nice article!By the way,I have upvoted your post,can you upvote my new post? Thanks

upvoted and followed.I like this and I can actually understand it! Ty!

Great idea!

Very impressive! Nice work.

There's a definite need for this. Steemit's google-integrated search is almost the only time I use google search and I prefer to never use it at all.

Wow, that is called great effort! @thekyle

Resteemed, upvoted and bookmarked. Its like watching Google being developed for the first time. Excellent work

great post i vote and resteem

asksteem.com is still open for an account.
I'll send you one sbd if you register it, and two more if you will give me the code that follows net_votes:<20 that limits for 6 days of age and less than a dollar payout.

  ·  7 years ago (edited)

Not sure what you mean by asksteem.com is still open for an account, but the code to filter posts less than 20 votes and 6 days of age would be as follows: net_votes:<20 AND created:[2017-06-05 TO 2017-06-11]The ability to search by payout is coming in the future, also notice that you would need to adjust the timestamp based on what day you are performing the search. Thanks.

Having that account will give a central place to send questions?
Instead of your personal account?
Make it easier to sell?

I don't know, it's your's.

Thats true, well thanks for letting me know :)

Thanks for providing a solution for request I have had for over a year!

I'm getting a server error? 500?

Hmm... Your right let me check into this.

Can I get the code to limit payouts to less than a dollar?

When you got it.

I've updated my previous post. Here is the full working code: net_votes:<20 AND created:[2017-06-05 TO 2017-06-11] AND pending_payout_value:<1 There is currently no way to search by total payout (coming in future) however, I've added in my example a way to search by pending payout which will work as long as the date range is less than 7 days.

I'm getting that server error, again?

Edit, and now i'm not,...

Well done, this is much needed to find stuff in steemit

Awesome post, app & tool dear @thekyle. To make the things on steemit flow as they should. With absolute order and harmony!!
Kickass search engine mate. A much succulent delight to be used.
A plain inspiration for my next post. Especially now, that I'm finally able to order and organize My original crazy stuff as God commands. Cheers!! }:)

¡Upvoted & Resteemed!

Glad you like it! :)

Oh! Much more than to like it my dear developer. I'm just beginning to squeeze the real juice out of your excellent creation.
Keep Up with the good work! :)

Congratulations on a fantastic project, I have voted and resteemed

  ·  7 years ago (edited)

Have been looking for sth. like that! Good Job. Hope this will be implemented in steemit.com! Resteemed.

Thanks.

Wow the syntax options are great! It will be nice if you integrate these into the user interface. For instance a quick button to sort the results by the most recent post would be helpful.

  ·  7 years ago (edited)

Thanks for the feedback, I've considered adding shortcuts but I don't want to add any clutter to the UI so it may take some time to figure something out.

Great tool, thanks for the effort. This post paid really nice too, right on.
Sorry I found it too late, but now following
Cheers

Thanks, glad you like the tool.

What a cool idea! Just the other day I saw a comment from someone complaining about how crude Steemit's built-in search is. I wished I could point them to some nifty tool that improves the experience. Now I can! Going to go try this out right away... you sir have earned another follower!

Glad you like it! There is some room for improvement but it definitely works better than generic web search engines like Google (which is what steemit currently uses) which are not optimized for searching blockchains. Thanks for the follow, and make sure to spread the word if you see anyone else looking for an alternative to the default search. :)

I definitely will. Spent some time playing around with it and it seems to work pretty well so far.

Great job !!!

Thanks for this!
I've just joined here, made a few posts and comments and tried asksteem, and it can't find myself.

Is it normal?

I was willing to translate your guides to my language to help others but need to make sure it works first.

Thanks again

Hi there,
As you can see here it has indexed all of your posts. It may take a few days for your user account to get indexed.

Thanks i though something like that was happening

I've waited since 2016 for this. There's a lot of great old articles that are hard to find because Steem is focused on new content. I wish authors could get paid for old articles that might have been low rated originally, yet later were rediscovered for very useful content.

I've been searching for something like this, I think it should be steems default search engine.

Great tool, thanks!

I like it. Works much smoother than the search on Steemit.com and has some nice features.

Resteemed!

Thanks for the resteem, I'm going to keep improving it, but I'm glad you like it. It should work better than Steemit's default search because it interfaces directly with the blockchain rather than using a HTML proxy layer (like the current steemit search)

Yes the google search function is a bit primitive so this is a great step up :)

  ·  7 years ago (edited)

Why 10/11 months old posts are reported with a today research as follow?
"@oneshot" linux created:[2018-01-01 TO 2018-02-28]

new.jpg

You need to separate the parts of your search query with boolean operators. The correct query would be: linux AND author:"oneshot" AND created:[2018-01-01 TO 2018-02-28]

When searching posts by keyword is possible to obtain a given chronological order? From newer to older and vice-versa?

It is possible to do that via our API but not through the web interface.
https://api.asksteem.com/search?q=asksteem&sort_by=created&order=desc

@thekyle
I've just been playing with AskSteem and I love it. I'm very impressed and I'm certainly going to adopt it as my goto source for steem based content.

I do have a couple of questions/suggestions that I would like to ask if that's ok...

  1. As content on the steem blockchain is being published constantly, I was wondering if there is any way to search for new content using a time variable? The meta of every post gives an indicator of when the post was published if, within the last 24hrs, this is a min-by-min indicator. It would be incredibly powerful if I could run a time-based search or a search for content published 25mins ago. This would allow users to benefit from the 30min voting.

  2. Is there any information on the SEO of your search engine. For example, if I searched the tag, "introduceyourself" how would I get my post to be listed first?

EDIT: Btw, I seem to be getting a server error (500) when I click on dashboard.

You're the man! Thank you very much for your great work! It's really fast :) Would be nice to have the filter possibilities in a html form. Maybe with a link to an "advanced search" or so... I'll resteem ;)

Thanks, I've been planning to create an advanced search page since I launched AskSteem so that is definitely on the to-do list.

Perfect! Do you have access to the number of views or are they tracked by Steemit only? If that would be possible, you could add this for searching "lost content", which didn't get many views (for whatever reason), but nevertheless could be interesting content for some of us...