The year was 2013, Eric Snowden was in the throes of exposing the National Security Agency(NSA) with over 1.7million documents stolen, and the New York Times had just published a damning report confirming the allegations. The news publication claimed they had documents in their possession, never to be released publicly, that "appeared to confirm" the allegations were real. Amongst the many revelations, the rumors a backdoor had been intentionally created and used for years by the NSA, as part of what they called the Bullrun decryption program. The program, a government program with a $250 million yearly budget, was created to compromise software and hardware, by creating backdoors in encryption software and essentially create faulty measures to facilitate the agency's surveillance. One of the major cryptographic algorithms The NSA and NIST(National Institute of Standards and Technology )worked together in secret to compromise, was the Dual_EC_DRBG standard cryptographic hashing algorithm. OpenSSL, Microsoft, and Cisco, blackberry and others all had the libraries which used that algorithm and millions were unknowingly affected.
Eric Snowden confirmed what many in the cryptography community were saying for years
Amongst the mounting media reports, there was also an exposé by Reuters in 2004 with full details detailing how the NSA paid security firm (RSA Security) $10million to compromise the algorithm as the default in their cryptography library. RSA Security went on to become the most prominent distributor of the faulty algorithm. There were denials for years by the CEO and other top level staff, even after the Snowden exposé began.
Unknown to the wider public, this very specific algorithm, Dual_EC_DRBG standard was known and publicly criticized by the cryptography community before the algorithm ever became part of a formal standard that was endorsed by the ANSI, ISO, and by the National Institute of Standards and Technology (NIST. All were ignored).
Approximately a decade later in October of 2012, after much scrutiny and stronger attacks that weakened the popular hashing algorithm, SHA1’s security, the community worried that next step up in algorithms, SHA2 was a ticking clock, and it certainly didn't help that all trust was lost in the agency(NIST)and the process that created it.
A simple visualization of what hashing is at a basic level
On October 31, 2008, the NIST decided to upgrade their standards and put in motion a publicly held process to improve the algorithms with a highly vetted public competition containing stringent standards. 64 applicants applied world wide, and five years later, a team composed of 1 Italian and 3 Belgians were chosen as the winners with their Keecak256 algorithm(pronounced 'kechak'.
This was approximately around the same time Ethereum was developing as an organization, and the Organization decided to choose the more secure and stable Keecak256. It can be confusing as one starts learning more about Ethereum, because SHA3 and Keecak are used interchangeably, however the actual SHA3 came out a few years later, with a few modifications…. Keecak256, is now considered a subset of SHA3, and although SHA3 has the same sponge construction it has been modified to be a little speedier. This is why some people use the names interchangeably. SHA3 is a standard of secure Hash Algorithms family that SHA1 and SHA2 came from. Even though there was a later modification, Ethereum did not follow suit and adopt the FIPS-202 based standard SHA3, which was finalized in August 2015. Before SHA3 was released in 2015, more controversy emerged because doubters claimed that the NIST was trying, to change certain Keecak functions after it was already reviewed and released, without those new adjustments included as parameters in the previous competition....There are claims these new modifications are legitimate changes but because of the history of the Snowden revelations that the NSA and NIST were colluding together but because the outcry was massives, so the NIST acquiesced to the original Keecak choices, and just released a newer optional version that they called SHA3.
So why should you care, some of you newer folks may be asking?
You should care because Keecak256, is an integral piece to Ethereum’s platform. It is useful for many purposes in Ethereum. This tool used to encrypt and decrypt data is useful for digital signatures , message authentication, and more.
According to Wikipedia, ‘encryption is the process of encoding a message or information in such a way that only authorized parties can access it and those who are not authorized cannot.’ It should be known that in spite of everything, even though SHA2 has still not been compromised, there is the history that SHA1 and SHA2 both, were designed behind closed doors with no public involvement of the cryptographic community or any type of design assessment. If you were to refer to the standard that specifies SHA-2, the 2015 update of FIPS 180, according to the cryptographic community, there is no clear design rationale. With various posed threats around many with nefarious intentions, this is why Keecak256 was deemed transparent, open, the most reliable and secure by the Ethereum organization. In the Keecek's team's words, “Keccak256 is versatile cryptographic hash function that relies on the sponge construction.
Not this kind of sponge construction
It is used for authentication, pseudo-random number generation, and encryption.
Essentially the way the algorithm works on a very rudimentary level, is it has a cycle of an input being hashed/'processed' until all the blocks are used up(absorbed) and then once this is done and the data is 'absorbed', then when there is an output of data, '
The process is repeated until the desired number of output bits are produced. If the output length is not the desired number of bits it is truncated.' The technical term for this process of the output is called 'squeezed. There's so much to learn in this field, and this is a brief history of how things came to be. It will be really interesting to see where things go in the future.
As I'm learning more about blockchain development, It’s important for me to question how things came about and why, so I can gauge exactly why something is being used, how, and if there is a better alternative....I do this stuff not only because it is it all so fascinating/interesting to me, but I feel like this will make me be a better engineer in in the space. I thought I should share that as I was researching this article, I came across a name that was mentioned quite a bit. His name is Bruce Schneier and he is one of the greatest cryptographists. I think everyone who is a blockchain engineer should be aware of him and what he does. I've included his website as well as the Keecek team's website below if you'd like to dig some more.
Current Keecak team can be found at :
https://keccak.team