Introduction

Dtube is a video plateform based on the steem blockchain that store videos on IPFS. Because most of users doesn't use ipfs. Dtube is a central point that will store content on IPFS node.

Dtube community support is a bot that will "listen" for new dtube content for a specific steem tag and store the video on a local IPFS node. Plus it stores metadata about dtube content like title (steem post) / video size / permlink / author. The main goal is to persist dtube content of your favorite community on IPFS.

DCS will retrieve 480p videos. if not available, it will retrieve the source file.

Pull Request

New Features

Enhance remove action

In order to make DCS more usable, I've added several functions to improve the management of ipfs repository .

Remove content is the key and I've added the following options :

• Remove all content for a specific author
• Remove all content older than a specific date

Remove by author

Remove by date

All theses fonctions are located in del.js.

The file is divided into 3 parts :

• main function (exportable) : Used to catch user input, call filter function, call remove function
• filter functions : Used to extract metadata (stored un JFS DB) by author, date, ipfs hash (pinset)
• remove function : Remove all contents passed in argument (metada extracted by a 'filter' function.

This structure will facilitate the addition of new features

More granularity

I'm running this bot to persist #fr content and after a while, I realized that it was not always relevant to persist some contents (eg : streaming).
On the other hand, I wanted to persist some authors that did not use the #fr tag (used by francophone).

I've created two array in config.json in order to blacklist or whitelist some authors (regardless the tag).

These lists are used in index.js which has for function to detect the contents to be persisted on the ipfs node (when using dcs has a daemon).

More specifcally, the following condition is used to detect the relevant content.

In a human readable way. A content is pinned only if :

Dtube content has the proper tag (configured in config.json) AND the author is not blackisted

OR

The author is whitelisted.

Bug fix

Duplicate entries in JFS database

JFS is not properly documented and it appears that some cache are used. This cache cause some duplicated entries in JFS DB (not in ipfs node because this is not possible to have duplicate entries).

I have fixed this behavior with this commit. In index.js, I create db object each time streamOperation() return a transaction. The workaround avoid to use unwanted cache.

Check if enough free space in the ipfs repository

According to the ipfs documentation, 'storageMax' (in ipfs config) is not used to prevent pinning new content if not enough free space in ipfs repository.

StorageMax A soft upper limit for the size of the ipfs repository's datastore. With StorageGCWatermark, is used to calculate whether to trigger a gc run (only if --enable-gc flag is set).

I've modified this behavior in order to use 'StorageMax' has a hard limit.
The workaround has been pushed in commit 235f869 (add.js - add content manually and bb08e3c (index.js - add content automatcally).

File size has determined by summing the size of all blocks

ipfs.ls(input.pinset, function(err2,parts) {

parts.forEach(function(part) {

size += part.size;

});

Repository usage can be retrieved with ipfs.repo.stat function.

Complete code :

• Index.js : https://github.com/evildido/dtube-community-support/blob/b69d3f67319ba95632dea84a40a05f8fd832e717/index.js#L126
• add.js : https://github.com/evildido/dtube-community-support/blob/b69d3f67319ba95632dea84a40a05f8fd832e717/actions/add.js#L85

Next step

streamOperation function provided by steem-js has some limitations and not very resilient. After talking with @fabien (@busy team), it seems that lightrpc library is better to manage pinning process even when full node api is unstable.
I'll look on it

In order to pin relevant content, I want to add the possibility to discard contents based on specific #tag (like #streaming).

---

Posted on Utopian.io - Rewarding Open Source Contributors

---