project: bullfrog-system | Mine STEEM with ease

in steemit •  8 years ago  (edited)

bullfrog-system
STEEM System Configuration Tools

@kilrathi

The bullfrog-system project creates a clean, complete, and standardized installation of the STEEM Daemon suite on Ubuntu 16. It provides all the tools necessary out-of-the-box to mine, witness, and bot against the STEEM blockchain.

GitHub : bullfrog-system

Background

This is the #mining system that I currently use with @bullfrog et al. I keep roughly 40 accounts to handle the miner witness queue. I mine strategically. When I do mine, I maintain a hashrate of roughly 5.7 mhps. If I'm feeling froggy that day, maybe 11.4 mhps.

I do not maintain mining hardware. I do this all with VMs via Amazon's EC2 service.

This system will be a pre-requisite for the NodeJS bot framework that I'm building here: GitHub: bullfrog. I will announce its release on #steemit once it's ready.

Installation

The target system must be a freshly installed Ubuntu Server 16.04 OS. Only "OEM" 16.04 systems are supported.

Installation is as simple as one command:

curl https://raw.githubusercontent.com/roylaurie/bullfrog-system/master/install-frog.bash | bash

Once the installation has completed:

  • Add your miner accounts to the config files found in /var/local/steemd/configs.
  • Use ~/bin/config-steemd.bash to select a role (miner, witness, synconly).
  • Reboot.

You will know that the replay & sync has completed when you see the bottom two lines as shown.

What it does

The installation script configures the system as such:

  • A new user, 'frog' will be created which you will use for all steem maintenance tasks.
  • steemd and cli_wallet -d will be installed as system services, out of /var/local/
  • Users 'steemd' and 'steemwalletd' will be created to run steemd and cli_wallet -d respectively.
  • Management tools will be installed in /home/frog/bin
  • steemd will automatically start on boot.
  • The latest blockchain snapshot will be downloaded (hosted by @steemychicken1).
  • steemd will immediately begin running --replay and sync after install.

Running the steemd service

To start/stop/monitor the steemd service:

sudo service steemd start
sudo service steemd stop
sudo service steemd status

To enable/disable steemd starting on boot:

sudo systemctl enable steemd.service
sudo systemctl disable steemd.service

To follow the log:

sudo journalctl -u steemd.service -f

Running the steemwalletd service

To start/stop/monitor the steemd service:

sudo service steemwalletd start
sudo service steemwalletd stop
sudo service steemwalletd status

To enable/disable steemd starting on boot:

sudo systemctl enable steemd.service
sudo systemctl disable steemd.service

To follow the log:

sudo journalctl -u steemwalletd.service -f

Management tools

bin/config-steemd.bash

Switches between pre-existing config.ini files. Default names available: miner, witness, and synconly.

bin/replay-steemd.bash

Switches into synconly mode and runs steemd --replay

bin/recompile-steemd.bash

Re-clones the steemit source-code, recompiles, and re-installs.

Configuring mining accounts

To generate accounts with your wallet:

cd ~/wallet/steem
cli_wallet
>>>> suggest_brain_key

The steem wallet will generate a new random private key for you to use:

suggest_brain_key
{
"brain_priv_key": "SUEDE TETRIC FUZZILY BLICK AMANDIN NUMBING BICYCLE BARPOST OUTFLY ...",
"wif_priv_key": "5J8EBoJv42M6Eg2wX9ApwsZQsAoTAEwKfqAqjjZYX3UmK3Pa9XJ",
"pub_key": "STM7NZ9DQXaSRDH9z4MUHkx31axQhngUdwgJ3F5udNQN8aYBCDJy2"
}

Copy the wif_priv_key, that's your miner's private key.
Choose an account name that isn't taken on #steemit. Check the website to make sure.
Edit /var/local/steemd/configs/config.miner.ini. Find the area that looks like:

# name of miner and its private key (e.g. ["account","WIF PRIVATE KEY"] )
# BEGIN MINERS

#miner = ["MINER-NAME","MINER-WIF-KEY"]

# END MINERS

Uncomment the miner = line and fill it in with your account information. It should look like:

miner = ["etherqueen", "5J8EBoJv42M6Eg2wX9ApwsZQsAoTAEwKfqAqjj ..."]

Now edit the /var/local/steemd/configs/config.witness.ini file.
Repeat the exact same steps as above for the miner = line.
Find the area that looks like:

# name of witness controlled by this node (e.g. initwitness )
# BEGIN WITNESSES

#witness = "MINER-NAME"

# END WITNESSES

Uncomment the witness = line and fill it in with your account information. It should look like:

witness = "etherqueen"

Repeat these steps for each additional account that you wish to use. Remember to save the wif_keys.

How to handle a Hard Fork

First, SSH into your machine with your login user. Then ...

sudo su frog # maintenance account
cd # /home/frog
sudo service steemd stop
sudo ~/bin/recompile-steemd.bash # will install as well
sudo ~/bin/replay-steemd.bash # will run replay in synconly mode until killed

You'll see the steemd log as it replays the database then syncs. Once sync'd completely, CTL-C to kill steemd. Then ...

sudo ~/bin/config-steemd.bash miner
sudo service steemd start

Multiple mining accounts

Once your miner finds a POW (proof-of-work), it gets placed on the bottom of the miner-witness queue seen at steemd.com. While on that queue that miner account is on timeout; it cannot be used to mine. So, you'll need at least two accounts. Preferably, the same number accounts as the total number of POWs that you find on average in a 3 hour window.

Once your miner reaches the top of the queue, it's allowed to act as a Witness once and generates 1 block (or misses its turn) for which you receive 1 STEEM in the form of VESTS.

After that, the miner account is available to use again.

The next miner on the queue will then generate the next block, and so on.

Multiple mining machines

You can run multiple machines with the same pool of mining accounts, with one very important caveat:

Thou shalt not double-witness.

You must have one and exactly one machine that includes the witness = configuration. All other machines cannot include witness configs, and must be purely miner = configs.

You can use the ~/bin/config-steemd.bash <miner/witness/synconly> command to switch between the two.

If you get caught double-witnessing, you will be flayed, beaten, and impaled. No big deal, really.

Changelog

versiondatechanges
v0.0.02016-07-21
  • Alpha release
v1.0.0tbd

License (MIT)

Copyright (c) 2016 Roy Laurie

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Authors get paid when people like you upvote their post.
If you enjoyed what you read here, create your account today and start earning FREE STEEM!
Sort Order:  

This is very cool!

I'm all set up but not mining.

I edited the 2 ini files as instructed. I am synch'ed to the latest block.

I tried to do sudo ~/bin/config-steemd.bash miner, which I assume will put me in miner mode, but it is asking for the frog password and I have no idea what to enter. I tried the password I created to get into the ssh via putty, but that did not work. So I am out of ideas! Any thoughts on how I can move forward?

  ·  8 years ago (edited)

My apologies. At the end of the install script, it gives at list of things you need to do, but I didn't include one of them in the docs and of course now I can't edit.

=== Installation complete. Starting replay. Once complete:
* CTL-C to kill steemd.
* sudo passwd frog
* Configure files in /var/local/steemd/configs.
* sudo /home/frog/bin/frog/config-steemd.bash
* sudo service steemd start

User 'frog' is created without a passwd to begin with. The first thing you need to do after installation is set the password so you can use sudo with that account.

Thanks for your help!

I have now created a password for frog, after doing ctl-c, using sudo passwd frog
Then I changed to the frog user, using sudo su frog
I went to the directory with config-steemd.bash, using cd ~/bin
I did ls, and I see the file config-steemd.bash in green. If I run it, it says "command not found" both with and without sudo as a prefix. config.ini is in red and the other 3 are in green, when I do ls.

frog@xxx:~/bin$ ls
config.ini config-steemd.bash recompile-steem.bash replay-steemd.bash
frog@xxx:~/bin$ sudo config-steemd.bash miner
sudo: config-steemd.bash: command not found
frog@xxx:~/bin$ config-steemd.bash miner
config-steemd.bash: command not found

Not sure how to proceed. Seems strange to get "command not found" when it is obviously there, but then again, it may be something obvious as I am very rusty at linux!

Thanks for any light you may be able to shine on this!

  ·  8 years ago (edited)

Similar to Windows, in 'Nix, if a command is not in your $PATH (run "echo $PATH" to see), it will not "find" it. You have to specify a command's path manually if that's the case.

I didn't set up '/home/frog/bin' as a default path as a security precaution for now. You have to include the full path when calling the command:

sudo /home/frog/bin/config-steemd.bash miner Or use ~/, which represents $HOME (/home/frog):
sudo ~/bin/config-steemd.bash miner
Or even:
sudo $HOME/config-steemd.bash miner

Thanks again for your help! I am making some progress I think! This is what happened this time:

frog@xxx:/home/xxx$ /home/frog/bin/config-steemd.bash miner
realpath: /var/local/steemd/configs/config.miner.ini: Permission denied
/home/frog/bin/config-steemd.bash: line 14: cd: /var/local/steemd/witness_node_data_dir: Permission denied
ln: failed to create symbolic link 'config.ini': Permission denied
chown: cannot access 'config.ini': No such file or directory

realpath: /var/local/steemd/configs/config.miner.ini: Permission denied

You have to use sudo (Super User Do) on that one. That script is trying to alter a system file so it needs administrator privileges when running, thus the "Permission denied".

sudo /home/frog/bin/config-steemd.bash miner

You're getting there! Keep hacking at it.

  ·  8 years ago (edited)

I've added piston.rocks support. It will automatically be installed and available at /home/frog/bin/py/piston.

Also, I've moved the bullfrog-system bin files to /home/frog/bin/frog so that other projects can be added directly to bin without affecting the bullfrog-system project's git status.

Finally, I added a custom MOTD .

The alpha (v0) is now in code-freeze while people try it out. I'll cut v1 a week from now and announce it on #steemit. The master branch will be recreated as a release branch.

  ·  8 years ago (edited)

As we all know, killing steemd while it's mining always corrupts the blockchain db. Always. .... Always. Did you hear that devs?! :P

That's what I use the synconly mode for. You can kill steemd in synconly mode and 90% of the time it won't corrupt. You can use this to prep VM snapshots so that the db is clean and ready to go on boot.

I see. Is mining complicated? I'd like to try but i'm not sure I'll be able to.. Can you share a link in your post for mining steem with this. I'm sure many people new to crypto might find it useful to get more information on using your tool to mine Steem.

Mining is complicated, which is why I built this.

This sets everything up for you, then you just add the accounts to the config files, run the config command to tell steemd its role (witness, miner, synconly), then reboot. Done.

Thank you so much!

I added a link that details how to configure everything. I'll add another story on how to use EC2 shortly.

I will follow through this article and try this out. Thanks again. :)

I'd upvote this several times if I could.

i'll do it right now, thanks @kilrathi

How'd it go?

I have not yet had time to do, I own all this weekend, maybe I ask you something to you dude 8]

Sure, I'll be around.

im looking for a solution to mine independent of pc, ubuntu, os. Something like a Cubieboard with chip to do hashing and block erupters to add to processing....or do I have it all wrong. What I currently have is CCMiner loaded onto Cubieboard running the mining program and each block erupter is 8gh processing power. This only mines, nothing else, Cubieboard chip is dedicated to CCMINER

Can you add automatic download of blockchain from http://www.steemitup.eu/ please?
cd /var/local/steemd/
wget http://www.steemitup.eu/witness_node_data_dir.tar.gz
tar -xvf witness_node_data_dir.tar.gz

Yes! I've been looking for a good link all day for that very purpose.

Thanks, procto. I'll update the install script to grab that and I think I'll add a standalone script to import it at any time as well.

recompile-steem.bash
missing line
rm -fr /home/frog/project/steemit

Guide says bin/recompile-steemd.bash
file says
bin/recompile-steem.bash

Nice to help out

echo -e "${GRN}|=== Configuring steemd for 'synconly' ... |${NC}"
sudo /home/frog/bin/config-steemd.bash synconly
is obsolete now

Fixed and tested.

  ·  8 years ago (edited)

Good catch. bin/replay-steemd and recompile-steemd do that on their own now ...

also
echo -e "${GRN}| * Configure files in /usr/local/var/lib/steemd/configs. |${NC}"
should be
echo -e "${GRN}| * Configure files in /var/local/steemd/configs/. |${NC}"

Thanks! Fixed.

Testing the download now ...

Done. Implemented & tested. Thanks, again.

I've added a changelog section so that people can follow releases as well as future enhancements.

i've tried to but it seems that i have no skills in this ) is it possible to make a video tutorial on this article of yours?

I intend to eventually.

That ... looks like a Linux problem. Is that in a VM or directly on hardware?

vm 12 workstation player. pfunk's guide finally helped me out, but I'm curious one about your miner as well)

  ·  8 years ago (edited)

Yeah, I used @pfunk's guides to get started. They're awesome.

My setup is basically just the result of trying out everyone's guides. :)

Can't thank you enough for this. I have been scouring steemit for ways to create new accounts. I knew there was an easier way especially since new accounts are by invitation only.

  ·  8 years ago (edited)

hehe I'm not sure if it's easier.

The catch there is that you can't login to steemit.com with a WIF key, so you're going to be limited to using @xeroc 's piston.web

Also, you have to mine a POW before it creates the account, so technically you're paying for the account whereas an invitation is not only free but you get hooked up with some faucet money.

Ok. I need to provide a disclaimer before I ask this question. This is my first attempt at mining any coin.

With that said, my question is concerning WIF keys and user accounts. As I understand, once I create the WIF key, I need to pair it to a username. So in my config file would it look something like
["newusrnm","newWIF1"]
["newusrnm2",newWIF2"]
etc, etc?

Also, if the blockchain is running and I decide to add a new username, am I able to adjust it on the fly or do I need to stop the mining process, adjust the config file, and then restart?

Thanks in advance!

Correct on the first question, that's how your config would look.

You need to restart the steemd service for the changes to go into effect.

What size of instance do you use? I'm trying on a free instance which is 1GB - 1 CPU. My guess is this is not enough horse power? What would you recommend?

will try it tomorrow 'cause there's no luck with windows. appreciate ur work.

do i really need freshly installed ubunu version? and Why?

Thank you! I'm sitting here just got my rig mining steem two days before the hack since then I haven't been able to get Ubuntu running. Pain in the ass, I'm gonna try this and see if I can get it going. If it works you deserve my upvote and much more lol.

  ·  8 years ago (edited)

Let me know if you have any trouble either on here or on the official #mining chat channel; https://steem.slack.com/messages/mining/

P.S. I'll add a screenshot of my mining results tomorrow morning.

how do you deal with forced shutdowns and the reindexing that follows???

That's a huge annoyance.

After a shutdown, I config-steemd and restart it in "synconly", which doesn't include any miners and usually doesn't corrupt the DB when killed. Then, after it's has re-indexed and re-synced, I shutdown again and my DB is rebuilt and ready to go.

Here is an Archive of Cryptocurrency App building Code on Github for anyone creating a Steemit app
https://steemit.com/steem/@marsresident/github-cryptocurrency-app-creation-archive

a is abc