Steem Data Services (SDS) / Upcoming Update Info / Version 0.1.7a

in sds •  3 years ago  (edited)


Version Details


SDS will be updated to version 0.1.7a within the next few days. Currently re-indexing of all blocks is already running at the sds0 endpoint (developer requests going to sds0 are automatically proxied to sds1 in the meantime).

Here are some of the most important new features and changes in this version:

  • Field word_count in all feed lists
    This will allow us to get the word count for each post directly from the result without the need of rendering the Markdown and stripping Markup tags from the body. It is not perfect yet, but will be optimized over the coming weeks.

  • Feeds by hot/trending score without whale votes
    The newly added trending score algorithm allows to get hot/trending posts without whale voters. It will enable us to retrieve lists of trending posts with almost all 'up-boted' posts removed.

  • Feeds by interaction score
    The new interaction score algorithm allows to get trending posts by user interaction. Instead at the voted rshares (payout) of each post, it only looks at author and reader interaction (more details will follow in an extra post at some point).
     
    The algorithm will be optimized over time (@realrobinhood will contain a small overview in his coming daily stats reports, where the results can be discussed to further improve the whole thing).

  • Field image_url deprecated
    The field image_url in the feeds_api's results will be removed (returned empty for now and removed completely in one of the next updates), because it can't be reliably filled on the server side. For example, getting preview images for videos should always be done by the client, because it depends on the used device's screen size which resolution should be loaded (mobile devices should load a smaller preview).
     
    Some video platforms (like vimeo) don't even allow loading preview images without making an extra request to their API. Additionally, not all front-ends will fill the image field in json_metadata correctly, so it does not make sense to have the logic implemented on two ends. The viewer app needs to be able to get the image from the body anyway. Therefore I will remove this redundant field from the API.

  • Improved getSteemProps API method
    As many developers requested to be able to retrieve a bundle of most important Steem properties with one request, I added the following API method (ready to be used on sds0, sds1 will be updated in a day or so): https://sds0.steemworld.org/steem_requests_api/getSteemProps
     
    The result also contains a field steem_per_share that provides the current STEEM per VEST value. Currently the data is updated every minute, but I will change this to be almost in sync with the blockchain. I also added a few more fields like the current market ticker data. If you should miss any property in there, just let me know ;)

Status


  • [x] Re-index all blocks at sds0 endpoint
  • [x] Copy parsed data to sds1 endpoint
  • [x] Copy parsed data to main sds endpoint
  • [x] Replace second steemd_ah instance with new lite (500 GB) version
  • [x] Make use of the new getCommunitiesBySubscriber method and show the user's subscribed communities in the Dashboard on SteemWorld

I wish you all a super inspiring week! Steem on ~ 🚀


If you like what I do and you want me to be your Steem witness,
please vote for @steemchiller on steemit.com/~witnesses or steemworld.org/witnesses.


Authors get paid when people like you upvote their post.
If you enjoyed what you read here, create your account today and start earning FREE STEEM!
Sort Order:  

thank you friend, this new update is amazing in my opinion this is very good for the future on the steemit platform in the future. Thank you very much for your hard work sir and thank you also for providing a platform to make this program a success.

and I try and also hope that all of you need your help in the extraordinary hard work so far and I also want to know more in this steemit one of which is how I get good steem to do power ups and we need it I hope always happy to meet you you sir.

good thing steemworld has helped us a lot to make things easier.

  • thank you very much sir @steemchiller hope you are always in our hearts. 🥰

I am always excited to see your update Sir @steemchiller and trust me, this is one of the best. This new update is so great and interface-friendly. Thank you for all your great Jobs, Sir, and thank you for contributing to the success of this platform.

I hope to offer any little assistance wherever is needed someday sir.Thanks for your hard work.

Muchas gracias por la información, excelente trabajo actualizado por la calidad. Saludos!

Hello @steemchiller this is the account that my daughter used, I had to borrow it to write to you because due to a silly mistake on my part I accidentally clicked on a link, I thought my account could be hacked and I wanted to change the password, but in the process I didn't copy the new generated password and I accidentally hit accept. The key was changed and I had not copied that key. Is there any way to recover my account?

Could you please help me, I beg you, I want to recover my account.

Last night one of the guys who is with me from the curation gave me your Discord contact and I sent you the friend request, if you want we can communicate by that means. Thank you

@daytona475

Hi, please start the account recovery for @daytona475 on https://steemitwallet.com/recover_account_step_1 with the previous password (the one you still have) and the email address that was used when signing up the account on steemit.com .

You have 30 days (from the moment you changed the password) until the account can no longer be recovered. In case you have any questions later on, just let me know.

I did what you told me, yes, I have the mail. I expect a prompt response to recover my account... Thank you very much for your help! God bless you!

I placed the email with which I registered and I still haven't received the email, is it normal for it to take so long?

Normally, the email will be sent out automatically (only when the email address matches the one from the registration process). Let's wait a bit and see if it will arrive later today. In case it should not arrive, I will contact the Steemit team to start the recovery manually.

Wait until now to check my mail and it still hasn't arrived, I even checked my spam folder and nothing. Before recovering my account I made sure to put the correct email, I checked and I have all the emails that have come to me from Steemit since I registered. So I have no doubt that I put the registration email.

The following image was the email I received when I registered

image.png

Thank you!!

Hello, I still haven't received the email to recover the account :(

I reported your case to the team, so you should receive the email within 48 hours. You don't need to worry. As long as you have the old account password, we can recovery your account 100%.

Thank you very much for your valuable help, I finally have my account again. I am very grateful!

God bless you, have a happy day!!

Great work. Many thanks to your effort.

yes Mr @steemchiller always release information that is very important for us to know the Steem Data Service. thank you very much for providing more information regarding Steem Data Services (SDS) / Update Notice / Version 0.1.7b .

Thanks a lot for the update information, Mr. @steemchiller
Great work!
Steem World has helped us many things.
We support you 100%
🥳🥳🥳

Hi @steemchiller,

I have been working on the deep learning curation bot, and did not like the first version I created (it was alright at predicting article values, but not great). So I've gone back to add more complex features, and this time I'm using historic data on the author and the top/median curators. One problem I've run into is that it's really slow due to rpc errors. I've realized a way I could avoid the rpc errors is by setting up my own private node, but I don't know how much storage that takes, and wanted to ask you since you're a witness and run your own node. Do you have any advice on setting up a node if it is possible for my computer (if not perhaps I could invest in cloud storage if it's not extremely expensive)?

Thanks for your help!
@cmp2020

Hi, the requirements for running an own node depend on the APIs you want to use. In general, you won't get far with a 500 GB disk anymore and for most common cases I would recommend at least a disk with 1 TB SSD space. The amount of available RAM is not so important (32 GB would be good, but even 16 GB would currently work without any issues).

If you also plan to additionally run something like a Hivemind node (in case you are making use of the Bridge API methods), which receives the blocks from a regular full node, you will need even more disk space, of course.

So, the first thing to do would be to find out which APIs your current (and maybe coming) projects will require to be realized. Basically, it's always more fun to work with a fast SSD (for best experience even NVMe). In case you should decide to run an own node, I recommend to use WebSocket connections instead of normal HTTP requests, because this will perform way better when running many requests in a row ;)

Here are some links that might help you to get started faster:
https://files.steem.fans
https://github.com/steemfans/steem-docker-ex

  ·  2 years ago (edited)

Thanks for your response! Unfortunately, my computer has a maximum of 16 GB of RAM so I think I will have to figure out a different way to do this until I have a proof of concept to show that it would be worthwhile to invest in either cloud computing or a computer with more ram. I did invest in 2 TB of external storage today though. I would appreciate any recommendations on how to download data faster with what I have. I am considering just cutting the historic data for now and working with the other features I've added.

Thanks again!

Edit

We decided to try setting up a node on our extra computer (which we hardly use) with 16gb ram and the external storage I bought. So far we haven't run into any problems so we might succeed at setting up the node. My dad and I just didn't think it would work because we're running it on WSL, but so far it seems to be working.

Yet another edit

We got it working, and have started syncing the blockchain. Thanks for your help! I hope to post an update on the ai soon enough ;)

Hey greetings @steemchiller gorgeous guy i have a problem so that i am discussing with kindly help me .
Dear whenever i tag any user than he didn't get my tagged notification for example i am tagging a my friend @shafqathussain but sadly to say he didn't get my notification what is the cause kindly i shall be very thankful to you look at it
Screenshot_2022_0327_150521.png

Hi, that seems to be related to steemit.com . When I switch to @shafqathussain on SteemWorld (https://steemworld.org/@shafqathussain), I see this in the Mentions section:

Screenshot at 2022-03-27 18-47-54.png

thank you for give info that thank you sir have a nice day Greetings...😃

This is an issue on steemit. There is a limit on the total number of people who get notified when you mention them in a post/comment. If you have mentioned more than 10 users in a single post/comment, only the first 10 will receive the notification. All the others are ignored unfortunately.

Yes it is, but i mentioned only user and no notification receive to him

Danke für die ganze Arbeit!
Habe mich immer gefragt, wie Leute hier einen %-Vote verteilen, sehe gerade das man per Manual-Vote das ganze auf SteemWorld realisieren kann!
Werde ich auf jeden Fall nutzen mit ansteigender Voting-Power! :)

Your post is manually rewarded by the
World of Xpilar Community Curation Trail

STEEM AUTO OPERATED AND MAINTAINED BY XPILAR TEAM
https://steemit.com/~witnesses vote xpilar.witness

Thanks a lot for the update information, Mr. @steemchiller
Great work! Thank you for updating

Buen trabajo sr @steemchiller se nota que usted es una persona que se preocupa para que las cosa marchen como deber ser, todo actualizado. Un placer que dios siempre lo ilumine un cordial saludo desde la isla de Margarita-Venezuela humildemente estoy a la orden. Gracia por el apoyo que me ha brindando.

Danke für die Infos!
word_count finde ich gut, da kann ich mir das Bearbeiten/Splitten des body sparen :-)
Die neuen Scorings sind konsequent. Bin gespannt auf die Details. Wird dies auch auf die "Trending"-Posts auf steemit.com Auswirkung haben?

Deine Entscheidung zum image_url bedauere ich, kann sie aber mit den genannten Gründen nachvollziehen. In Bezug auf die Videos war mir das so nicht bewusst, und ist mir so auch noch nicht untergekommen . Schade nur, dass auch die Metadaten da nicht zuverlässig nutzbar sind...


Thanks for the info!
I think word_count is nice, so I can skip the editing/splitting of the body :-)
The new scorings are consistent. I'm curious to read the details. Will this also have an effect on the "trending" posts on steemit.com?

I regret your decision to deprecate image_url, but can understand it with the reasons given. I wasn't aware that it was difficult with videos. I haven't come across it like this before. It's just a pity that the metadata can't be used reliably too...

Wird dies auch auf die "Trending"-Posts auf steemit.com Auswirkung haben?

Zunächst nicht, da steemit.com noch Hivemind verwendet und dort noch nichts in der Richtung implementiert wurde, aber ein ähnlicher Punkt ist in der noch entstehenden Steemit-Roadmap enthalten (Trending page without bot-votes).

Deine Entscheidung zum image_url bedauere ich, kann sie aber mit den genannten Gründen nachvollziehen.

Das Problem mit den nicht richtig gefüllten Metadaten betrifft nur wenige Fälle, aber ich bin mir ziemlich sicher, dass es in Zukunft immer mal wieder auftreten wird. Die über steemit.com erstellten Posts setzen die Daten auf jeden Fall korrekt.

Man muss immer bedenken, dass auch viele Posts über Bots erstellt werden (z.B. Statistiken) und dann kommt es auf den jeweiligen Entwickler an (da solche Posts auf steemit.com trotzdem richtig mit Bild angezeigt werden, merkt man das nicht sofort).

Wenn ich das Feld rausnehme, kann ich mir damit auch kommende Fehlerberichte von Entwicklern ersparen. Im Front-End würde ich es erstmal so lösen, dass die Logik für manuelles Ermitteln des Bildes aus dem body nur aufgerufen wird, wenn json_metadata.image nicht gefüllt (oder kein Array mit URL-ähnlichen Inhalten) ist.

Würden es alle Video-Plattformen so wie Youtube lösen, hätte ich die Logik dringelassen, aber während des Parsens der Blöcke Anfragen an verschiedene Server zu stellen, geht mir dann doch etwas zu weit.

Interesting and great news, some new tools that improve the platform.

Hi. If I have a witness node, do I have to put a full node or seed node, and how can I do that.

Hi, a witness should run a seed node at least, as this makes the network more reliable/stable. See this conversation for additional information: https://steemit.com/sds/@boylikegirl.wit/r9ugw7

Hi. Can I ask you a few questions?

  1. The manual says that the seed node has an open address and the witness node must be closed, to avoid hacking. This suggests that there should be two nodes, the seed node and the witness node.
  2. I understood from your comment that the witness node can include the seed node as well. In that case, the node can be one?
  3. How can I add a couple of keys to be able to switch between the nodes?

This suggests that there should be two nodes, the seed node and the witness node

Exactly, the IP of the server that runs your main witness node should not be known to anyone. The seed node can include a witness node, but of course, this should only be used as backup node (for example, when you install security updates on your main server).

How can I add a couple of keys to be able to switch between the nodes?

This can be done with the 'cli_wallet' app that's also included when downloading the source code of 'steemd'. Same as 'steemd' it needs to be compiled before you can use it. So, you would compile and run 'cli_wallet' on both servers and create a new random key-pair on each of them. The resulting private signing key then needs to be added in the 'config.ini'. The corresponding public signing keys can then be used to switch between the nodes from outside the server (can also be done with SteemWorld).

Thank you very much. I now have a better understanding of how it works. The only question that remains is about the public key node, what is it for, and is it possible to combine the seed node with the public key node

What do you mean by public key node? A public key always is derived from a private key. So, same as for posting, you have a private signing key (non-public) that is used for signing the blocks and the public signing key that is derived from the private one.

When broadcasting the witness_update operation, you set your current public signing key in the field 'block_signing_key'.

Thank you very much

Once again, thank you very much for your help. I was able to start a seed node thanks to your advice. How can I make my site be visible to everyone on steemworld.org? When I write the site address in https://steemworld.org/seed-nodes-check and do a check, it passes. Is there anything else I need to do?

Just tell me the address of your seed server and I will add it in there ;)

  ·  2 years ago (edited)

Hi @steemchiller
Is the seed node configured like this?
plugin = witness condenser_api network_broadcast_api account_by_key database_api

How to test?

Hi, I think your config should work. By adding the witness plugin you will be able to use that node also as backup witness node (I did the same).

I would create a new signing key pair and add it to the config, so you can easily switch to that one by just changing your current signing key (can also be done in SteemWorld -> Dashboard -> Witness Details), in case you need to update your main witness server (for example, when installing important security updates for your system).

You need to open the P2P seed port (default is 2001). This tool can be used to check if a socket connection can be established:
https://steemworld.org/seed-nodes-check

@etainclub recently created a tutorial on how to create a seed node. This might also be helpful:
https://steemit.com/hive-101145/@etainclub/tutorials-seed-node

  ·  2 years ago 

Thank you! I use this tool https://steemworld.org/seed-nodes-check, my seed node is working.

Don't we also need to enable the p2p plugin? Wouldn't that help distribute the load amongst the seeds?

The seed node would not work without the p2p plugin, but the plugin will be loaded automatically, because it's a requirement for the network_broadcast_api.

From the 'appbase/README.md':

All plugins complete the Initialize step before any plugin enters the Startup step. Any dependent plugin specified
by APPBASE_PLUGIN_REQUIRES will be Initialized or Started prior to the plugin being Initialized or Started.

network_broadcast_api_plugin.hpp:

APPBASE_PLUGIN_REQUIRES(
   (steem::plugins::json_rpc::json_rpc_plugin)
   (steem::plugins::rc::rc_plugin)
   (steem::plugins::chain::chain_plugin)
   (steem::plugins::p2p::p2p_plugin)
)

Ah... I see.. Then I've got it enabled twice on mine.....

It's not really a problem to have it twice, as it will still be loaded only once.

This is very interesting!

it's great news. thank you. Steemchiller.

Moin @steemchiller,

ich hoffe es geht Dir gut.

Mal eine Frage - weißt Du wo es die Keychain Addon für steemit für den Chromebrowser noch gibt?

https://steemit.com/deutsch/@indextrader24/wo-kriege-ich-das-steemkeychain-add-on-fuer-chrome

Die Datei ist ja nur 3.5 MB groß aber wird bei google play store nicht mehr angeboten. Für Firefox gibt es die Add On offenbar noch im Netz.

Beste Grüße und vielen Dank vor ab.

!invest_vote

Hallo Steemchiller ich versuche in SteemWold die copy vote aber als du das tool neu implantiert hattest ging es aber seit langer zeit versuche ich es immer wieder ohne erfolg
Ich bin mit meinem Postingkey angemeldet und wollte chriddis votes kopieren
Was mache ich wohl falsch?
VgA

Moin, ich kann dir leider nicht sagen, was du falsch machst. Es funktioniert bei mir ohne Probleme. Normalerweise müsste eine Fehlermeldung erscheinen, wenn etwas beim Voten schiefgelaufen ist. Sollten die Posts gar nicht erst erscheinen, liegt es wahrscheinlich an den im Tool gesetzten Filtern.

Die filter habe ich von 0-100 und o-1000 komisch
Danke für deine schnelle Antwort
VgA

Wahnsinn genau so habe ich es versucht das geht schon seit Monaten nicht
Nochmal danke ich verstehe es nicht

BA55FC9C-D17F-4C6F-AB87-E52DB0CCF230.png

Kannst du mal prüfen, zu welchem Endpunkt (Node) du verbunden bist (Dashboard -> Settings -> Steem Node)?

AA36E1F6-418D-4DCB-B547-585BEABA7A82.png

Dann gehen mir so langsam die Ideen aus. Vielleicht ein Problem in dem verwendeten Browser... Mal einen anderen probiert?

Aha, oho , ja .

Hi
Sorry I suppose you have been asked numerous times.
I would like to see what possibilities there are to integrate some coding. I am a rusted DotNet developer. Do you perhaps have some sample code or reference where I can start.
Regards

Hi, I don't know much about .NET, but maybe @starlord28 can help you getting started...

Thank you. Need to make curation a bit easier for my wife😂

Hey. Sure. I could share an existing codebase of mine with you. But that's already large and confusing at times. So, instead, I will share a sample repo with you in sometime or early in the morning tomorrow. Just let me know with what exactly you need help regarding..... Do you have any confusion with accessing the API response through code or something else? This will help me give you the most simplest example for your problem.

Hi. Thanks for the reply.
There is a number of apps currently available but they are quit confusing, well more for my wife that is using them to curate.
All u want to do is to create a screen where she can see whats she needs. No posting of data, well defantly not in this phase.
I have manage to query the json api functions and convert the json into classes. Now i need to create the display interface. Would like to do in a Blazor app for other to use as well.
At present I am still trying to find out exactly how this works.
Any simple examples will help.
Thank you

Great. I'll create a basic application to translate the API response into something that you can see on the screen soon and share the link here :)

Thank you 🙏🏻

https://github.com/starlord2826/DemoSDS

Here is a very simple app to fetch a few random properties from the accounts API of SDS. You could add more properties to the DTOModel and map the respective properties from the API response and display the ones that you feel are needed.

I am no UI expert. So, the UI here is very basic. Feel free to make it fancy and create a PR if you feel more people will benefit from the changes that you make.

Do let me know if you face any issues with getting started. I could either help you out over discord or maybe even make a small post explaining how the flow works.

Hi
Do you perhaps know how to convert the date fields. It is displayed as an Int
Thanks

Yup. You could use something like this to do that ((DateTimeOffset)input.fromDate).ToUnixTimeSeconds();

Thank you so much. Need to install Vs 2022. Still have 2019. Will update you soon.

Thanks works very nice. Would be nice if you could remeber steem usernames.
Also no email address or contact details of you in the readme file:)

Haha. Well, one can always contact me here on Steem or even on discord. Plus I don't check my inbox that often. The readme is basically just simple instructions meant to get developers started quickly. So, don't really see the need for an email there😉

hello @steemchiller, I need your help with something, my discord address is Ahmt#3269, if you can contact me, I would like to talk to you.

wow amazing ,Interesting and great news, some new tools that improve the platform.

Best work

Great work

Thanks a lot for the update information, Mr. @steemchiller
Great work! Thank you for updating

Good morning @steemchiller, this is to ask for your help. My brother's account @edgarf1979 was stolen, the procedure for recovering stolen accounts has already been done, but still no response. I sent an email to Steeemit support, and they said that the request is in process since March 24 when my brother's account was stolen.

I would appreciate if you could help my brother to recover his account.

Thank you very much and sorry for the inconvenience.

Hi, I reported your case to the team. In many cases it has to do with using a wrong email address. I will tell you when I have any news on this ;)

Thank you very much for your prompt response, and thank you very much for the help you are giving my brother.

Good afternoon @steemchiller, greetings and I hope you are very well! sorry for the inconvenience that you have had response from the account of my brother @edgarf1979?

I didn't know this case is still open. As far as I know, the request was approved and he should have received an email with further instructions. Did he check the correct email account (the one he used when signing up on steemit)?

Good morning @steemchiller, the images that I will place below are: the first one when my brother created his account in Steemit with his email [email protected] and the second one the times that he wrote to the steemit support email and it tells him that this request is in process.

imagen.png


imagen.png


Thank you very much in advance for the support you can give to my brother.

Hi, I can't help you when I don't know what's the problem. Your brother should have received an email with the instructions from the steemit team. Those instructions need to be followed by your brother to recover the account.

So far I have not received this email

It looks like he didn't use his signup email address when starting the request. Which email address did you use when you started the account recovery? The request will automatically be approved, if the used mail account is the correct signup mail account.

In any case, as I contacted the team, they approved the request (even if the used email address was not correct). This was 5 days ago. You should have received an email in the email account that you entered when starting the request.

Good morning @steemchiller, if you put the email with which you registered in steemit.

Thank you

Thanks a lot update information @steemchiller
Great work! Thanks for sharing so many quality post. Wish your best success by @steemchiller🤗❤️ sharing upvote resteem,,,