Setting up the project

Hey folks,

So I've wanted to do a blog for some time and have finally landed on the topic for it. We're going to build some tools that can be used to setup some machine learning for crypto trading. I'm going to do it all in NodeJs. This project is probably going to span multiple months as it's my pet project and I have a lot of full time obligations. But let's just get right into it.

If you want to follow along feel free. You can install nodejs from the download links on their site https://nodejs.org/en/download/ if you don't have it already.

Logging

I'm imagining building this thing in stages. We'll need some tools like a data importer, data processor, data loader, the machine learner, possibly a trade bot etc... This is all very fluid right now and we'll make some mistakes along the way while we nail it down. But one thing is for sure. We're going to want some flexible logging setup from the beginning. It's really going to go a long way in debugging. I also don't want to be tied down to logging things directly to the console or a file or any other medium. I'd like the ability to change it on the fly so I'm going to use an npm package called winston.

First I'm going to create a new folder:
mkdir cryptoML

Then lets initiate a node package:
npm init --y
The --y flag will tell node to just use the default values for the package.json. We can change them later if we want.

Next let's install the logging package. I like to use yarn personally but npm is fine also:
yarn add winston or npm install winston

Let's also go ahead and install a package to read from a .env file. We can use this to store sensitive data and user specific settings. I'm going to use yarn add for the commands but feel free to use npm install if you prefer or don't have yarn going forward. Yarn has some performance and security enhancements so it's my preference at this time.
yarn add dotenv-safe

Cool now if we take a look at the dotenv-safe documentation we see

Identical to dotenv, but ensures that all necessary environment variables are defined after reading from .env. These needed variables are read from .env.example, which should be committed along with your project.

So we need to create a .env and a .env.example. Create those in your root directory and let's start with just adding a LOG_LEVEL variable to them. Open up your favorite editor, I'm using vscode personally. Add the line:
LOG_LEVEL=info
to both.

With that set let's create our root project file. You can name it whatever you want, I'm naming mine main.js. Inside I'm going to start with pulling in the .env file and just test that things are working. So open it up and at the top let's write:

require('dotenv-safe').config();
console.log(process.env.LOG_LEVEL);

If everything is linked up correctly we should be able to run the command:
node main
In our terminal and get an output of:
info

Okay with that done let's delete the line beginning with console.log since we don't want to use that for logging like I mentioned earlier we're going to use our logging library. Let's setup a folder called logger in our root directory and create a file called logger.js inside. Let's make sure to require it right away in our main.js. So it should look like:

require('dotenv-safe').config();
const Logger = require('./logger/logger');

Now let's open up the logger/logger.js file. In there let's write require some different things from the winston logging library.

const { createLogger, format, transports } = require('winston');
const { combine, timestamp, printf, colorize } = format;

This is probably a decent time to mention this isn't exactly a beginners tutorial for JavaScript but I will make note of some things along the way. If you're not familiar with the above syntax we're using the require keyword to load the module winston for starters. Then we're using object destructuring destructuring to assign different pieces to different variables for simpler syntax.

Alright now that we have our library loaded let's write the logger code. For now I just want to be able to log to the console but by using the winston library (there are others like bunyan, chalk, loglevel, etc...) we can easily add or change transports down the line. Here's what our logger looks like.

const outputFormat = printf(info => `${info.timestamp} ${info.level}: ${info.message}`);

const Logger = createLogger({
  level: process.env.LOG_LEVEL || 'info',
  transports: [
    new transports.Console({ 
      format: combine(
        timestamp({format: 'YY-MM-DD HH:mm:ss'}),
        colorize(),
        outputFormat
      )
    })
  ]
});

module.exports = Logger;

The first line is defining an output format that I'm thinking we'll use throughout the app for both console and file logging. In the printf function the info comes from the log call. So info.timestamp is the timestamp from the log object and we can get other data from it like the level and message. So this line is saying we want it to look like:
timestamp info: some kind of informative message

Next we create the Logger with createLogger. Right away we assign our LOG_LEVEL env variable the logger's level property and default it to info if one isn't defined. Using dotenv-safe should ensure one is defined, but I generally like to be defensive with my code.

Next we define a new transport. Right from the winston docs:

In winston a transport is essentially a storage device for your logs. Each instance of a winston logger can have multiple transports configured at different levels. For example, one may want error logs to be stored in a persistent remote location (like a database), but all logs output to the console or a local file.

Right now we're just using the console which we get from the transports that we imported. Next we use the format combine function to put our formats together. He're we're formatting the timestamp and adding colorize for aesthetics. You can read about it more at https://github.com/winstonjs/winston.

Last we use module.exports to export our Logger object.

Aaand done. We have our initial logger setup. Let's test it. In the main.js let's try to log some things. We can do this with the Winston Logger.log function like Logger.log({level: 'info', message: 'message'}) or through the convenience functions like Logger.info('message'); So let's below the initial required libraries:

Logger.debug('This is a debug message');
Logger.info('This is an info message');
Logger.error('This is an error message');

Then run it again via:
node main
Your output should look something like:

So because our logger's level is set at info it will not log messages of a lower priority like debug. Let's see what happens when we just change our LOG_LEVEL in the .env to debug and run it again. We should get an output that looks like:

Sweet. We setup the logging for our app. Next I think we're going to tackle importing some data.