[KnackSteem API] - More friendly endpoint and moderation tools

Repository & Pull Request

https://github.com/knacksteem/knacksteem-api
https://github.com/knacksteem/knacksteem-api/pull/7
https://github.com/knacksteem/knacksteem-api/pull/8

What is KnackSteem?

"Do you have any talent? If yes! then KnackSteem is for you."
"Rewards people with talents, it can be any talent, anything you know how to do best is highly welcome on the platform. "
Source: Discord Channel :D

Changes made

Refactor the is moderator and is supervisor middleware

As suggested by @amosbastian, I have refactored both middlewares and merge them into a single function. This function will take the role to check against as a parameter and can be used to check any role. So, duplicated code is not there anymore. The function is basically the same but instead of having a fixed role, the role is passed from the file which is using it.

Thanks, @amosbastian.

Related code:

/**
 * Check the role of the current user
 * @author Jayser Mendez.
 * @public
 */
// eslint-disable-next-line
const checkRole = (role) => {
  return async (req, res, next) => {
    try {
      // Try to find the username in database.
      const user = await User.findOne({ username: res.locals.username });

      // If the user is found, check if the user indicated role.
      if (user) {
        if (user.roles.indexOf(role) > -1) {
          // Pass to the next middleware if the indicated role is present in this user.
          return next();
        }

        // If the user has not the indicated role, return an error in the middleware
        return next({
          status: httpStatus.UNAUTHORIZED,
          message: 'Unauthorized access',
        });
      }

      // If not user is found, let the client know that this user does not exist
      return next({
        status: httpStatus.UNAUTHORIZED,
        message: 'This username does not exist in our records. Unauthorized access.',
      });

      // Catch any possible error.
    } catch (err) {
      // Catch errors here.
      return next({
        status: httpStatus.INTERNAL_SERVER_ERROR,
        message: 'Opps! Something is wrong in our server. Please report it to the administrator.',
        error: err,
      });
    }
  };
};

So basically, you can refer to any role as:

• checkRole('supervisor')
• checkRole('moderator')

and any other role in any route.

Middleware to check if a user is banned

When a user is banned, this user should be not allowed to perform actions such as making a new post. Since the posts feed is loaded from our database and merged into Steem API to complete the missing data, posts made outside the platform will not be shown in the plaform. So, if a banned user try to make a post outside our frontend, it will not be shown. This middleware will basically check the isBanned field and bannedUntil field of the User schema. The isBanned field will tell if the user is banned or not and the bannedUntil will tell until when is this user banned. When the bannedUntil is less than the current time, the ban will be released and will allow the user to perform actions as before.

Related code:

/**
 * Check if the current user is banned
 * @author Jayser Mendez.
 * @public
 */
const isBanned = async (req, res, next) => {
  try {
    // grab the user object from the last middleware which was store in the locals.
    const { user } = res.locals;

    // If the user is found, check if the user is banned and ban is not expired
    if (user) {
      if (user.isBanned === true && Date.now() < user.bannedUntil) {
        // Deny access if the user is banned
        return next({
          status: httpStatus.UNAUTHORIZED,
          message: 'Unauthorized access. You have been banned!',
        });
      }

      // If the user is not banned, move to the next middleware
      return next();
    }

    // If not user is found, let the client know that this user does not exist
    return next({
      status: httpStatus.UNAUTHORIZED,
      message: 'This username does not exist in our records. Unauthorized access.',
    });

  // Catch any possible error.
  } catch (err) {
    // Catch errors here.
    return next({
      status: httpStatus.INTERNAL_SERVER_ERROR,
      message: 'Opps! Something is wrong in our server. Please report it to the administrator.',
      error: err,
    });
  }
};

Function to ban users (available for supervisors only)

Since only supervisors will be able to ban users, this function will excecute if and only if the checkRole middleware returns true. Otherwise, access will be rejected. This function will modify 4 fields in the User object which are: isBanned, bannedBy, banReason, bannedUntil. isBanned will tell whether or not the user is banned. bannedBy will tell who banned this user. banReason will tell the reason of this ban. And bannedUntil will determine when this ban expires (for lifetime ban, just set a really high date).

Related code:

/**
 * Method to ban a user (Only for supervisors)
 * @public
 * @author Jayser Mendez
 */
exports.banUser = async (req, res, next) => {
  try {
    // Grab the user data and ban data from body
    const { username, bannedUntil, banReason } = req.body;

    // Grab the supervisor username from the locals
    const supervisor = res.locals.username;

    // Update the post with the new ban data
    const user = await User.findOneAndUpdate(
      { username },
      {
        isBanned: true,
        bannedBy: supervisor,
        banReason,
        bannedUntil,
      },
    );

    // If the user was banned correctly, send the message to the client.
    if (user) {
      return res.send({
        status: 200,
        message: 'User was banned correctly.',
      });
    }

    // If the user is not found, tell the client.
    return res.send({
      status: httpStatus.NOT_FOUND,
      message: 'User is not found.',
    });

    // Catch any error
  } catch (err) {
    return next(err);
  }
};

Make the v1/posts endpoint more friendly by accepting more params.

Before, I had different endpoints for different cases: byAuthor, byCategory, all posts, etc. This can be reduced to only 1 endpoint which can accept all these cases (thanks for a team member for this suggestion :D). Hence, I decided to refactor this function. First thing to take in mind was to build the correct query depending on the parameters given. Since parameters can be combined, I had to think a way to find them and perform the query. Hence, I ended up with the following function to build the query:

/**
 * Method to construct query based on parameters from the URL
 * @param {Object} req: url params
 * @private
 * @returns a ternary operator with the query for MongoDB
 * @author Jayser Mendez.
 */
const constructQuery = (req) => {
  /**
   * Ternary conditions to decide which query to load.
   */

  // Query to get posts by author
  const authorCondition = (req.query.author);
  const authorQuery = { author: req.query.author };

  // Query to get posts by category
  const categoryCondition = (req.query.category);
  const categoryQuery = { category: req.query.category };

  // Query for full text search using title and description fields.
  const searchCondition = (req.query.search);
  const searchQuery = {
    $or: [{ title: { $regex: req.query.search, $options: 'i' } },
      { permlink: { $regex: req.query.search, $options: 'i' } }],
  };

  // Query with all the options together
  const allConditions = (authorCondition && categoryCondition && searchCondition);
  const allQuery = {
    author: req.query.author,
    category: req.query.category,
    $or: [{ category: { $regex: req.query.search, $options: 'i' } },
      { permlink: { $regex: req.query.search, $options: 'i' } }],
  };

  /**
   * If the author,category, and search are present in the query, query by author and category
   * using full text search.
   * Else if the author is present in the query, query the posts by author.
   * Else if the category is present in the query, query the posts by category.
   * Else if the search is present in the query, do a full text search
   * Else, query all posts
   */
  // eslint-disable-next-line
  return allConditions ? allQuery : (authorCondition ? authorQuery : (categoryCondition ? categoryQuery : (searchCondition ? searchQuery : {})));
};

Since it is not a very elegant solution, it worked as a charm and saved me a lot of lines of code by using if, else if, else if, else. This function takes the query params of the url and will decide which query will return. Then, the main function will receive this query and will perform it:

/**
 * Get posts from database based on criteria and sorting.
 * @param {Object} req: url params
 * @param {Function} res: Express.js response callback
 * @param {Function} next: Express.js middleware callback
 * @author Jayser Mendez
 * @public
 */
exports.getPosts = async (req, res, next) => {
  try {
    // Query the posts from database in a descending order.
    const { username } = req.query;
    let { limit, skip } = req.query;
    const sort = { createdAt: -1 };

    limit = parseInt(limit, 10);
    skip = parseInt(skip, 10);

    /**
     * Construct the query based on the parameters given.
     */
    const query = constructQuery(req);

    // Query the posts from database given the query.
    const postsList = await Post.find(query).sort(sort).limit(limit || 25).skip(skip || 0);

    // Declare an array to hold the URLS to do the http GET call.
    const urls = [];

    // Iterate over the results from the database to generate the urls.
    postsList.forEach((post) => {
      urls.push(`https://api.steemjs.com/get_content?author=${post.author}&permlink=${post.permlink}`);
    });

    // Do all the http calls and grab the results at the end. it will do 15 parallel calls.
    async.mapLimit(urls, 15, async (url) => {
      // Fetch the http GET call results
      const response = await request({ url, json: true });

      let isVoted = false;

      // Check if there is a user provided in the params.
      // If so, determine if this user has voted the post.
      if (username) {
        isVoted = helper.isVoted(response.active_votes, username);
      }

      // Parse only the fields needed.
      // TODO: Determine what fields we need
      return {
        title: response.title,
        description: response.body,
        category: response.category,
        isVoted,
      };

    // Grab results or catch errors
    }, (err, results) => {
      // If there is any error, send it to the client.
      if (err) return next(err);

      // Send the results to the client in a formatted JSON.
      res.send({
        results,
        count: results.length,
      });

      return true;
    });

    return true;

  // Catch any possible error
  } catch (err) {
    return err;
  }
};

So, here are some of the available combinations:

• To get posts by category: v1/posts?category=test
• To get posts by author: v1/posts?author=jaysermendez
• To get posts by category and author: v1/posts?category=test&author=jaysermendez
• To perform a search: v1/posts?search=test
• To perform a search by category: v1/posts?search=test&category=test
• To get posts and see if you have voted them: v1/posts?username=jaysermendez
• To set a limit on how many posts to return: v1/posts?limit=5
• To tell the server to skip a n quantity of posts: v1/posts?skip=5

Prevent moderators to moderate a post more than once

Once you moderated a post, that's it! You cannot moderate it again unless you get permission from a supervisor (if reasonable reasons are given). To achieve this, I just created a middleware to check if the post is already moderated. If so, the server will tell the client that this post has been moderated by X moderator and will ask to contact a supervisor to moderate it again.

Related code:

const httpStatus = require('http-status');
const Post = require('../models/post.model');

/**
 * Check if a post is already moderated
 * @author Jayser Mendez.
 * @public
 */
const isPostModerated = async (req, res, next) => {
  try {
    // Grab the permlink from the post request
    const { permlink } = req.body;

    // Ask database if this moderator has moderated this post before
    const isModerated = await Post.findOne({ permlink });

    // If this post is already moderated, prevent the moderator to moderate it again.
    if (isModerated.moderation.moderated === true) {
      return next({
        status: httpStatus.UNAUTHORIZED,
        message: `This post is already moderated by ${isModerated.moderation.moderatedBy}. Contact a supervisor to change its status.`,
      });
    }

    // If the user is not banned, move to the next middleware
    return next();

  // Catch any possible error.
  } catch (err) {
    // Catch errors here.
    return next({
      status: httpStatus.INTERNAL_SERVER_ERROR,
      message: 'Opps! Something is wrong in our server. Please report it to the administrator.',
      error: err,
    });
  }
};

module.exports = isPostModerated;

Example trying to moderate again a moderated post:

Merge all moderation tools

Before, I was having separated endpoints for each moderation tool. It was fine but once I see how nice the posts endpoint worked witch only 1 endpoint, I decided to refactor this as well. Before I was having the following:

• To moderate a post: POST v1/moderators/moderate
• To ban a user: POST v1/supervisors/ban

Now, I have:

• To moderate a post: POST v1/moderation/moderate
• To ban a user: POST v1/moderation/ban

which is more consistent.

Add master user when the server runs its first time

The first time that the server runs, there should be a master user defined able to create more users from a frontend. For this, I created an environment variable entry to load the user from tne envs. Then, I have the following function to create the user:

/**
 * Method to insert the master user in the database.
 * @param {String} username: Username of the master user
 * @private
 * @author Jayser Mendez.
 */
const createMasterUser = async (username) => {
  try {
    // Create a new user object with the required data.
    const newUser = new User({
      username,
      roles: ['supervisor', 'moderator', 'contributor'],
    });

    // Insert the new username in database.
    return await User.create(newUser);

  // Return just false if any error
  } catch (err) {
    return false;
  }
};

Which is used when node connects to mongo:

/**
  * Connect to mongo db
  *
  * @returns {object} Mongoose connection
  * @public
  */
exports.connect = () => {
  mongoose.connect(mongo.uri, { keepAlive: 1 }, () => {
    // Check if the user count is 0. If so, declare the master user.
    User.count((err, count) => {
      if (!err && count === 0) {
        createMasterUser(config.master_user);
      }
    });
  });
  return mongoose.connection;
};

Basically, it will check if the user collection is empty. If so, it will add the master user. Next time the server runs, this will not happen since the user collection is not empty.

Refactor single post endpoint

The single post endpoint was done and was working correctly. However, it was not following the code style of the other files (in other words, the linter warnings/errors were ignored). I decided to refactor this function to also be async and prevent the callback of lightRPC.

Related code:

/**
 * Method to get a single post from Steem Blockchain
 * @param {*} req
 * @param {*} res
 * @author Huseyin Terkir (hsynterkr)
 * @returns an object with the post from Steem Blockchain
 * @public
 */
exports.getSinglePost = async (req, res, next) => {
  try {
    // Grab the parameters from the param
    const { author, permlink } = req.params;
    const { username } = req.query;

    // Call the get_content RPC method of Steem API to grab post data
    const post = await client.sendAsync('get_content', [author, permlink]);

    // If there are not results from this post, let the client know.
    if (!post.author || !post.permlink) {
      return next({
        status: httpStatus.NOT_FOUND,
        message: 'This post cannot be found in our records',
      });
    }

    // Parse the JSON metadata of the post since it is not parsed by default.
    post.json_metadata = JSON.parse(post.json_metadata);

    // Get body image of the post.
    // eslint-disable-next-line
    post.image = post.json_metadata.image[0];

    // Use steem formatter to format reputation
    post.author_reputation = steem.formatter.reputation(post.author_reputation);

    // Calculate total payout for vote values
    const totalPayout = parseFloat(post.pending_payout_value) +
                        parseFloat(post.total_payout_value) +
                        parseFloat(post.curator_payout_value);

    // Get the votes weight as percentage.
    // eslint-disable-next-line
    for (let i in post.beneficiaries) {
      // eslint-disable-next-line
      post.beneficiaries[i].weight = (post.beneficiaries[i].weight) / 100;
    }

    // Calculate recent voteRshares and ratio values.
    const voteRshares = post.active_votes.reduce((a, b) => a + parseFloat(b.rshares), 0);
    const ratio = totalPayout / voteRshares;

    // Calculate exact values of votes
    // eslint-disable-next-line
    for (let i in post.active_votes) {
      post.active_votes[i].value = (post.active_votes[i].rshares * ratio).toFixed(2);
      post.active_votes[i].reputation = steem.formatter.reputation(post.active_votes[i].reputation);
      // eslint-disable-next-line
      post.active_votes[i].percent = post.active_votes[i].percent / 100;
      post.active_votes[i].profile_image = `https://steemitimages.com/u/${post.active_votes[i].voter}/avatar/small`;
    }

    // Sort votes by vote value
    const activeVotes = post.active_votes.slice(0);
    activeVotes.sort((a, b) => b.value - a.value);

    let isVoted = false;

    // Check if the post is voted by the provided user
    if (username) {
      isVoted = helper.isVoted(post.active_votes, username);
    }

    // eslint-disable-next-line
    post['isVoted'] = isVoted;

    // Send the results to the client
    return res.send(post);

  // Catch any possible error.
  } catch (err) {
    // Catch errors here.
    return next({
      status: httpStatus.INTERNAL_SERVER_ERROR,
      message: 'Opps! Something is wrong in our server. Please report it to the administrator.',
      error: err,
    });
  }
};

TODO: Keep splitting this method since some functions will be used in the posts endpoint.

Validate fields for endpoints

Fields' validation is really important. For instance, if you forget a field, the server will take it as undefined and will try to perform the task anyways. We don't want that. Thus, I create the validation for the following endpoints:

• Moderation: moderate
• Moderation: ban

Related code:

const Joi = require('joi');

module.exports = {
  // POST /v1/moderation/ban
  ban: {
    body: {
      access_token: Joi.string().min(6).max(512).required(),
      username: Joi.string().required(),
      bannedUntil: Joi.number().required(),
      banReason: Joi.string().required(),
    },
  },
  // POST /v1/moderation/moderate
  moderate: {
    body: {
      access_token: Joi.string().min(6).max(512).required(),
      permlink: Joi.string().required(),
      approved: Joi.boolean().required(),
    },
  },
};

If any of the field is missing, the server will let the client know that there is a missing field in the request:

What is next?

• Allow moderators to reserve a post to moderate so others moderators will not be able to moderate the same post (reservation will have an experiration time of 1 hour. After one hour, if the post is not moderated, other moderator will be able to reserve this post)
• Create stats endpoints to see how much posts, users, etc. we have.
• Add more filters to the posts endpoint to see only moderated posts, unmoderated posts, posts being moderated by a X moderator, and so on.