On the 17th October in a block #22,040,639 atticlabeosb signed a transaction spending 4620 EOS from an address blacklisted by ECAF. Realizing the responsibility of being an EOS Block Producer, Attic Lab conducted an internal investigation into this issue. The present report outlines its results.
Background
The theft victim claimed that his/her EOS coins got lost on the 9th of October. To obtain funds back he/she turned to ECAF and started a dispute which was registered under the number #ECAF00000487. The Emergency Arbitrator, who was appointed to review the case, issued the Order of Emergency Protection on the 13th of October and disseminated it to the Block Producers through the ECAF website and the blacklist plugin.
With this order, all EOS Block Producers were obliged to refuse processing transactions from the accounts guzdanrugene and earthsop1sys. Despite the precautionary measures taken by ECAF, in the following five days, two cases of spending from the blacklisted accounts took place.
- The first one happened on the 15th October in a block #21,710,433. The block contained two transactions spending from a blacklisted account gyzdmmjsgige:
- The first transaction with an id c19016947a017827cd02340ddb9e52cd6863b9373757567df3f1b0391a916bab transferred 6670 EOS to kucoindoteos account.
- The second one has an id 5b830f2e9f65c29f8abc79a7e245197ae24a3d1a0c80e638163bf812d0defadd. It self-staked 5 EOS to NET and CPU cumulatively.
- The second case took place on the 17th October. Attic Lab signed a block #22,040,639 which contained a transaction with an id 91289921281f19d7a3aea7503dd67b0284e0a724f63d2f3fe9d6dbfbacd9cac1 spending 4620 EOS from guzdanrugene account to eosblackdrop.
Note, the gyzdmmjsgige account has nothing to do with the case #ECAF00000487 and the Order of Emergency Protection as of 13th October. It is subject to a different dispute, which was registered under the number #ECAF00000494. The Order of Emergency Protection for the case #ECAF00000494 was issued on the 12th of October.
Investigation
To maximize uptime and ensure fault tolerance, Attic Lab runs a cluster of block producing nodes. On the 12th of October in the morning, Attic Lab team decided to add a new node to the cluster. To set it up, a configuration file (config.ini) of the existing nodes was used. Note, that blacklisted addresses are usually stored in the config.ini file. As of the morning of the 12th of October, the file was entirely up to date.
At the end of the day (2018-10-12 21:40 UTC) ECAF issued the Order of Emergency Protection relating to the case #ECAF00000494, which outdated config.ini of the new node. 2 hours and 30 minutes later another Order of Emergency Protection, relating to the case #ECAF00000487, was issued (2018-10-13 00:10 UTC). These two orders obliged Block Producers to refuse “to process transactions of any kind for the affected EOS account names and/or public keys, pending further review of the case by an Arbitrator”.
Approximately 6 hours after (2018-10-13 06:00 UTC), three accounts were appended to the blacklists of the existing nodes: guzdanrugene, earthsop1sys and gyzdmmjsgige. By that time the new node was still synchronizing, and its config.ini file was not revised.
The new node with an outdated config.ini completed its synchronization on the 14th of October. However, it was not until 17th October when it commenced its functioning as a part of the cluster. At the same day it signed the transaction 91289921281f19d7a3aea7503dd67b0284e0a724f63d2f3fe9d6dbfbacd9cac1 spending 4620 EOS from guzdanrugene account to eosblackdrop.
Action Taken
On the 17th of October, immediately after the accident took place, the node was switched off and excluded from the cluster.
To prevent such situations in the future, Attic Lab wrote and launched a script. It utilizes blacklist plugin by EOSLaoMao. On node start, the script extracts the hash of the most recent blacklist from the EOS blockchain and compares it with the local one. If these do not match, the node does not start.
Outcomes
Realizing its role as a Block Producer, Attic Lab understands its high responsibility to EOS community. The internal investigation revealed that Attic Lab is fully accountable for signing the transaction 91289921281f19d7a3aea7503dd67b0284e0a724f63d2f3fe9d6dbfbacd9cac1 that spent from the blacklisted address guzdanrugene.
Nonetheless, Attic Lab never broke ECAF orders deliberately. The transaction was signed due to a mistake of a systematic nature. The way blacklists currently work, leaves ample room for attacks as the process of their update is not automated and, thus, not instant.
Refunding guzdanrugene account in full is a very serious precedent. Our decision will impact all present and future Block Producers and might turn them into targets of swindlers. In view of this, we want to ask EOS community to start a discussion regarding the extent to which Block Producers should be held liable for losses caused by their actions. Feel free to join the discussion on Reddit.