My Idea for a distributed filesystem built on the blockchain

Architecture (C#)

• HashChain Client made up of:
  HashChain Client Application
  HashChain Library
  HashFS Library
  WCF Library
  .NET Library

• BlockChain Server is:
  BlockChain Server Application
  HashChain Library
  HashFS Library
  WCF Library
  .NET Library

• HashFS Server is:
  HashFS Server Application
  HashFS Library
  WCF Library
  .NET Library

hashchain version 4 is

• a block chain that sits on top of hashfs
• a block chain inspired from bitcoin + ethereum
• a block chain that publishes blocks via the hashfs
• don't need any additional networking

hashfs version 4 is

• a hashtag based resilient distributed file system for the web
• users register at hashfs.io
• users can vote a hashtag to become a system hashtag @ hashfs.io
• allows users to search for files using '3 column' auto association
• acts as the central system hashtag server (updated every hour)
• acts as the central client application binaries distribution
• system hashtag urls are published on hashfs.io
• system hashtags served as a JSON file using:
  http://hashfs.io/?file=SYSTEM,HASHTAGS
  http://hashfs.io/?file=SYSTEMHASHTAGS
  http://hashfs.io/?file=ABC457439347893

hashuser is:

• a username
• a storage contributor
• a single index hashblock (index) - this is the entry point into local cache
• a collection of hashblocks (data or index) representing the local cache
• a block server
• an index server

There are three types of hashblocks (file, index and data)

The File block acts like the genesis block of a blockchain

The User can delete file blocks they created

Blocks with no parent hash get automatically deleted from the large index

The User must contribute at least Nx the storage capacity of the hashblocks
related to their username (assume that N is 2 for this version)

The User must include at least all file blocks they have created in their local cache

hashblock (file) is:

• flag indicating its a file block
• the hashtag #username
• collection of sorted system hashtags ['8' tags, '256' length]
• collection of sorted user hashtags ['8' tags, '256' length]
• the file extension (really just for windows apps)
• hash of the index block that points to index or data blocks
• overall hash of the final data
• block hash

hashblock (index) is:

• flag indicating its an index block
• list of hashes for required hashblocks (index or data)
• parent hash
• block hash

hashblock (data) is:

• flag indicating its a data block
• binary data
• parent hash
• block hash

In order to improve resilience, a node needs to find another random node, and
exchange blocks. To do this we need nodes to cooperate:

• each node contains a list of (IP,credit) of other nodes
• each node communicates directly with other nodes
• each node merges the list of (IP,median credit) from other nodes
• each node suggests blocks it would like other node to store for it
• each node randomly selects a suggested block and sends a request for it
• each node is responsible for tracking which nodes it has provided blocks to
• each node aims to get 'N' remote copies of all its blocks
• each node can ask a remote node provide a block as proof of storage
  • challenge could be a request for the block once per 5 minutes
  • credit = challenges correct over last 24 hour period (max = 24*12)
• each node gives credit to other nodes for requesting/storing their blocks

Recent node IPs

• when hashfs client starts up, it notifies hashfs.io of its IP address
• hashfs.io maintains a table of IPs of nodes connected in the last 5 minutes
• a node should seek this recent list if it has trouble finding cooperative nodes
• a client should seek this list at most every 5 minutes

Example: Block Distribution for 3 nodes, Single file with 9 blocks

• Aim is every block in local block tracker has at least two IP entries
• Assume that redundancy is set to '2'
• This implies we should have 2 copies of the block between the nodes
• A block should be assigned to multiple nodes for redundancy

Green Path Create File Scenario is:

• All users have no blocks

• User 1 starts client

• User 1 IP is added to recent IP list

• User 2 starts client

• User 2 IP is added to recent IP list

• User 3 starts client

• User 3 IP is added to recent IP list

• User 1 retrieves recent IP list

• User 2 retrieves recent IP list

• User 3 retrieves recent IP list

• User 1 creates a file with 9 blocks (1 index block, 8 data blocks)

• User 1 send the hashblock (file) to hashfs.io to support search

• User 1 updates block tracking (B1=>local)(B2=>local)...(B9=>local)

• User 1 generates list of blocks with N < 2 (i.e. blocks 1-9)

• User 1 suggests block 1,2,3,4,5 to User 2

• User 1 suggests block 6,7,8,9 to User 3

• User 2 requests block 1,2,3,4,5; User 1 provides blocks 1,2,3,4,5

• User 1 sets User 2 score to +5

• User 1 reports list (User2 IP, +5)(User3 IP, +0)

• User 1 updates block tracking (B1=>local,User2)(B2=>local,User2) etc.

• User 1 generates list of blocks with N < 2 (i.e. blocks 6-9)

• User 1 suggests block 6,7 to User 2

• User 1 suggests block 8,9 to User 3

• User 2 requests block 6,7; User 1 provides blocks 6,7

• User 1 sets User 2 score to +7

• User 1 reports list (User2 IP, +7)(User3 IP, +0)

• User 1 updates block tracking (B1=>local,User2)...(B7=>local,User2)

(etc)

Green Path Search File Scenario is:

• User 4 performs a search on hashfs.io/?search=H264
• User 4 gets a collection of urls to block (file) data:
  http://hashfs.io/?file=ABEEFF0057ABEF005481
  http://hashfs.io/?file=ABEEFF0057ABEF005482
  http://hashfs.io/?file=ABEEFF0057ABEF005483

Example: HashFile urls for GOT S06E01 H264 HD EFY
http://hashfs.io/?file=EFY,H264,HD,S06E01,GOT => hashblock (file)
http://hashfs.io/?file=EFYH264HDS06E01GOT => hashblock (file)
http://hashfs.io/?file=ABEEFF0057ABEF005483 => hashblock (file)
http://hashfs.io/?search=EFY,H264,HD,S06E01,GOT => list (hashblock (file))
http://hashfs.io/?search=EFYH264HDS06E01GOT => list (hashblock (file))
http://hashfs.io/?search=ABEEFF0057ABEF005483 => list (hashblock (file))

Example: File stored on file system
http://hashfs.io/?file=EFY,H264,HD,S06E01,GOT => EFY,H264,HD,S06E01,GOT.mp4
http://hashfs.io/?file=EFYH264HDS06E01GOT => EFYH264HDS06E01GOT.mp4
http://hashfs.io/?file=ABEEFF0057ABEF005483 => ABEEFF0057ABEF005483.mp4

Example: How to create a new file using the windows client

1. The client app downloads the system hashtags in the background
2. User drags a file on, or selects 'add file'
3. The client app guesses system hashtags for the file
4. The client app uses the filename as the user hashtag
5. The client app extracts the extension from the file extension
6. The client app fills out the hashfile header including block checksums
7. The client app adds the file data to the local file cache

Example: How to request an existing file from windows client

1. User requests a file using the URL format
2. Check local file cache for the complete filehash (using the cryptohash)
3. Hit hashfs.io to get a hashnodes list if local hashnodes cache is old
4. Retrieve index information from index servers if local index cache is old
5. Connect to N block servers
6. Request blocks not in local block cache randomly from servers
7. Store blocks in local block cache
8. File is create in local file cache when all blocks have been read

Example: Use Cases

• stream video from video files

• locate a specific file via a html url (?using cryptohash)

• locate all user files via a hashtag

• locate a specific file via a crypto hash

• use a system hashtag to locate video or image based on quality

• SEARCH the filesystem using chrome via hashfs.io

  • can search using #tags only
  • a single search gets allocated to a 3 column search:
    • exact match on username
    • exact match on system #hashtags
    • otherwise an edit distance match on all hashtags

• automatically sync files from a particular user #tag

• automatically sync files from a collection of hashtags

• access hashfs files via hashfs.js

hashnodes is:

• formatted in JSON
• a collection of IP addresses of block servers
• a collection of expected checksums of all the blocks
• the hashfile header information