Access to fast computation and big data and blockchain should be a human right.
I was listening to a talk by Andrew Ng 5 days ago. The talk was about AI and its impact on society. He was speaking to a packed room at Stanford University. A person from the audience asked Andrew, what is the competitive advantages that Baidu enjoys in Machine Learning that enables it to be a leader in its space? His answer to the question sent a ripple through my spine. It increased my blood pressure to a dangerous level.
Ok, It wasn’t that dramatic or traumatic, but it was enough to elicit an immediate response. It forced me to write this story. So who is Andrew Ng and why should you care? He’s a big, BIG deal. Over 1 million machine learners from over 90 countries have taken his machine learning courses, They respect him as an expert in the AI community. Professor Ng is held in high esteem by his fellow data scientists in the United State, China, and Europe. He is a genuine celebrity, if there was ever one, in machine learning.
Ng is currently the Lead Artificial Intelligence Scientist at Baidu the Chinese search engine giant, before that he was the lead scientist on the Google Brain Project.
Here is how he answered the question.
There are only 2 competitive advantages that big companies like Baidu have over their smaller rivals: They have very large datasets and they have many high-performance Computers. If we follow his response to its logical conclusion, it is safe to assume that the size of the dataset a company has is a function of its position in its industry. Zero small company has big data set lying around in its large data center. That would be illogical. Smaller companies have no Giant data silos or Hadoop cluster hanging around. Before you go any further ask yourself this question:
Whose Data Is It Anyway?
We will revisit this question in an upcoming paragraph, but first, let’s continue to examine Andrew’s presentation.
Ng seemed assured that Baidu and other data hoarders will continue to monopolized, am sorry, I meant dominate their industry, because not only do they have lots of GPU’s, they also possess the 2 complementary assets namely: hundreds of smart data scientists, and equally brilliant high performing computers dev OPS experts.
I am not convinced that it is sufficient to stagnate the ingenuity of the upstarts. Law of replenishment states people working at coffices (coffee shops used as offices by entrepreneurs) will replace, supplant or disrupt the incumbents in many areas. It’s not a matter of if it will happen; it’s matter of when it will happen. So I am not worried.
Ng’s tone suggested that by not owning Big-data it presents an insurmountable barrier to entry for the upstarts who lacks these resources.
He further went on to say that the skills that you need to design such high bandwidth distributed computer architecture and mastery in machine learning are mutually exclusive. These skills are too specialized for any one or small group to have mastered both of them. He went on to say …
The machine learning experts do not have the dev ops focused expertise to create and manage HPC in a distributed environment, on the other hand, the dev-ops experts know little about the cleaning, the augmenting and or the feature engineering of the big dataset to make it useful for training. Why is this important?
Andrew Ng is to machine learning what Richard Feynman was to physics.
When Andrew Ng speaks, people listen. I listened to Professor Ng for 8 consecutive weeks while taken his machine learning course. It was given by Stanford University (a MOOC). He was the first to expose me to the mysterious world of machine learning. He is great a teacher. Professor Ng is to machine learning what Richard Feynman was to physics. They both transformed complicated subjects into simpler ones.
He de-mystifies machine learning for more than 100 thousand students around the world. His Youtube video lectures have 100’s of thousands of views. He continues to inspire and recruit scores of newcomers to AI from other fields through his classes, presentations, and videos. He’s a visionary in our field and in education. If that was enough, he is also a co-founder of the wildly popular MOOC website, Coursera.
That’s why what he said next was very troubling to me. So much so that his comments compel me to write this story. You will discover why I was concern as you read more, but first here is what he said;
If you are serious about machine learning, you should go to work for a Big company, especially one that possesses both HPC (high-performance computers) and a terabyte of terabytes of data. That will expose you to the cutting edge machine learning methods that are happening now.
While there was some truth to what he said, I disagree with the implications. I believe that cutting edge machine learning is going on daily at coffee shops, in basements, and garages around the world. A big company is not the only place where innovations are happening. Bureaucracy is guilty of the murder of a lot of innovative ideas that occur inside a large enterprise.
A big corporation is not always the best place for creative work. Most creative people who are still working for big companies would escape to freedom, but the can't. They are strapped by the paycheque. There is room for the shiny tools and data silos, but having too many tools can lead to paralysis of the creative mind. Having too many tools restricts your freedom and deny you the creative burst and insight that constraint give. The right amount of constraint can unleash flashes of insights resulting in breakthroughs that dazzles. The ability to do more with less is the vitamin that nourishes our creative souls.
Some Companies Are Too Big To Succeed. That fact escapes them.
They can’t escape from the self-created prison. The bureaucratic cage that they built to house the creative thought is keeping them locked in solitary confinement. The conservative policies of their boards of directors act as the prison guard.
For innovation to happen, it takes a group of people who are too stupid to know what is impossible.
To innovate in machine learning,
It takes people who will learn the rules like scientists so they can break them like artists. For an artist. Some big corporation is torture chamber to the artist. A startup is the heaven the liberated mind dreaming of.
Big companies are reasonable and level heading. It’s the unreasonable 4 entrepreneurs who set out to change the world, not too surprising sometimes they do change the world: Google.
Upstarts create breakthroughs. Big companies innovate incrementally. You can count on one hand the number of big companies that continue to innovate after their IPO. Google, Microsoft and of course Apple are great examples. These are unicorns; they are the outliers. If you are entrepreneurial, you probably are not going to work for any company, big or small. So Ng's advice about you going to work for the man is a bad idea. It just won't work for you.It’s just not a viable option.
Yes, I did say I listened to Andrew Ng.
But I didn’t say I agree with all his views. In fairness, he telling the truth but only for the unicorn. Since finding a job at a big company is not the dream most entrepreneurs aspire to, what is the solution then. we have to find a better way. One that doesn’t include bulldozing the unruly, the stubborn or the crazy to go work for IBM, Google or Apple. They are all Great places to work, just not for the founders, not the one who wish to make a dent in the universe and certainly not for the ones who want to make the world a better place. Getting on track,
The mantra of deep-learning is mooore dataaa, mooore data, fasterrr GPUU’s. Sounds like a line from a rap song?
There is theoretical and empirical evidence suggesting the more data lead to better results. This is true. Deep-wide and convolutional networks need a lot of data if don't consider pre-train weights and biases.
But should more data always be better to training our model? More GPU’s is not the only thing that improves the training time of our models. Graphlab, for example, creates very fast computation using outer of core processing, proving that there is a lot we can do with algorithms to speed up training.
I read a story here on Medium, where an Adobe machine learning expert in marketing bragging that they have the biggest dataset on creatives due to the Adobe’s near monopoly on software that used by photographers, illustrator, print and web designer. Adobe creates Photoshop, Illustrator and their creative cloud. That is their base cloud platform. Your data hoarding ability should not give you a competitive advantage. It shouldn’t be a startup barrier to compete or barrier to entry. If data hoarding is the only advantage they got, it is not a sustainable advantage.
If big data and possession of high-performance computers are the oxygen, then small startups would die of suffocation. We should fight against the centralization of machine learning if you are a ML soldier. If you are machine learning pacifist, you can work toward decentralization of machine learning. I am not saying that all big company is conspiring against the smaller startups, starving co-founders of data life lifeline saying we should do what can now to prevent that happen in the near future.
Before we start she Witch hunt against all big companies, it’s Important to remember that without forward thinking bigger companies like Google, for example, we wouldn’t have got this far. Having said that it's not it’s not unreasonable to think that company that was fair when everything going great can become data hoarding evil if its dominance is threatened by 3 founders in a coffee shop. This begs the question, could block chain help?
Our goal is to prevent future evil not punish past good deed.
Some companies’ business models are forcing them to empower the entrepreneurs and to play nice with smaller startup siblings. Google, Microsoft, and Amazon, for example, are outsourcing a lot of awesome tools to the community: Think Tensorflow from Google, it ‘san amazing tool for distributed neural network at scale. Works great with Google Cloud. We can rent a server from the cloud, but as a thought experiment, what do think would happen to the cost if a startup that is using Amazon architecture to compete with Amazon? Do you think the cost would price go or down? Seriously, would you lend your enemy your weapon to fight you?
One Voice Over IP startup that was using a big Canada’s network to compete directly with the big Canada company phone service. Voice Over IP accused the big Canada of sabotaging its packets, causing bad customer’s experience to their users. It ended in the courts. If I am not mistaken the Voice Over IP was victorious in that case. It goes to show that business will get evil if its dominance or revenue started to decrease. It's not personal. it’s business. They have a legal duty to their stakeholders to maximized returns. The extreme case of this is called externality
OpenAI is purported to be working towards decentralization of AI. I say purported because I don’t know how open OpenAI is. I don’t know is funding it and for what purpose. Is there an open ledge similar to the block chain? I don’t know. My guess is they are working more on the education side of things. My concern is more on the computation and data aspect.
Just as a side note: I mention about the wish to avoid GPU enslavement. Google has is called TPU which is a piece of hardware to train Tensorflow models. How affordable is it? It is not cheap.
At this stage you might be saying, Hey Roy, you list a lot of the problems. You told us about the tyranny and potential evils of computation divide and data centralization BUT up to this point, you have not said a word about any possible solutions. You are right. Let’s get to some the possible solutions. Put a brake on that for a moment, Let's think about this important question: what would you like to see happen in the next 5 -10 years for machine learning startup with say 4 cofounders? Here is what I would like to see happen.
In an ideal world, all startups should have at least a chance to compete with any others business with a non-zero probability of achieving success. .Startup should maintain its nibble-ness flat hierarchy and zero bureaucratic chains. Startups should not be held back by the lack of terabytes of data. They should not be torture be the deficiency of high price GPU’s and restricted by lack of High-Performance Computers.
The data that seems to be giving the Giants their edge. Where did they get it? Whose Data Is It Anyway? I argue that the data is ours. We create it when we surf. We created it when we click the like button in social media. We create it when we blog, and we create it when we fail to refuse to be tracked. Logical it follows that we are the rightful owners of big data? Who else thinking that. The people who’s championing distributed apps tend to think so and so do I. Does anyone knows how much user generated data is worth? Let me guess: Billions. Am I right?
Did we knowingly relinquish ownership in the fine print? How can we reclaim our online presence. If others are allowed to monetizing your surfing habits, your preferences could ask for some that money jut for being the creator of the online you. Those questions weren’t meant to be rhetorical. They required answers. You should answer them with the seriousness they command. Switching gears,
There might be a technology that is not as sexy as Machine learning but it has the potential to revolutionize even data science. We only hear how machine learning is going to transform or disrupt X, we almost never heard how Y will revolutionize or improve machine learning. I am taking about block chain. Blockchain will not disrupt machine learning because it can’t. Machine learning and block-chain compliment each other. One is basketball
star Lebron James and the other is a track star Usain Bolt.
They are the 2 most game-changing technologies since the advent of electricity: machine learning and blockchain. Our challenge is to create …
A network architecture based on the block chain. One that mimics the proof of work to train machine learning models. But instead of the nodes verifying transaction, the node are helping to train machine learning models. This would unleash the power of the peer-to-peer network for human kind. With a distributed peer-to-peer network to train ML models like , it would democratized machine learning which is a good thing.
We have seen the power of peer-to-peer network with Napster and we witnessing the power once again with bitcoin. The bitcoin rattling the centralized power structures of finance and remittance. The internet is peer-to-peer in nature. The good thing about what I am saying is this: I have no idea what is impossible. So I am not constrain by my what I know for sure, that ain’t so. I am more focusing on what I would like to see happen.
Machine learning should aim to be as permission-less as the block chain network. Let say, I wanted to send $100 in Bitcoin to a friend in Jamaica, I no longer need an intermediary such as a bank or Western Union to so. I just open my command line open 21.co send the money to his public address. He will get his money within minutes not days. I don’t have to fill out realms of paper work and bilk 10–15% of the money I am sending for fees. The receiver doesn’t have to wait 5 days to get his money.
Imagine we can train a neural network without having any need for an Amazon or a Google. Imagine you could, in a permission free environment away from the cloud platform police. Let’s aim to make training machine learning model as simple as sending a 100 dollars to a bitcoin address.
Why can’t we build on top of block chain or build something like it to automate the training and with compensation feature for trainers. What would training compensation be? It could be small amount of equity that you could trade with others for money or trade for other crypto-currencies like bitcoin. The shape it might take is a subject for the another story, but you get the idea. We should adapt and adopt what is working to work for us.
Pre-trained weights and biases are a very exciting development in machine learning world now. Bigger companies are training model at scale and then they will release the weights and bias to community. This is great because it allows the startup founders or researchers to do amazing thing with little data. This particular important in computer vision and other deep -learning task. All is not well, we are just seeing the weights and biases based on points of view of one architecture , one optimization and one use case. You are not seeing the actual data. You are deprive of access of the data that generate the the weights and biases. I said it before and I will repeat it for it’s full effect here. This post is to guard against future evil. It’s not aiming to ignore good of the past. We must aim for the stars if you miss you might land on the moon. Pre-train model is give from the data gods but not yet the key to the heavens. Nirvana is the destination.
Holy grail of machine learning for me is to be able to train a deep-neural network with a small data set, while maintaining an high accuracy score on the validation set. In other words, let’s negates the need for bigdata. That would be the ultimate levelling of the playing field for all involved.That would several good the thing. For example,
Creativity, colaboration and innovation would return as the competitive edge. A company’s size should not be the only prerequisite for future success. Another promising move in right direct as far using less data to train model is interactive training.
researcher at Stanford University are using interactive training in NLU (Natural Language Understanding). It has scalability issues. Interactive Learning allows for a human to train the computer to accomplish a task that is difficult for computer do by itself: Let say the computer is trying learn to understand what number of english sentences mean. The computer make a guess. The human nudge the computer in the right direction. Like re-enforement learning, but the human is re-enformencing the rules in a tight feedback loop with the computer.
Oh no, Natural Language Understanding is not a close subject, despite what you hear from the expert sport commentator on FOX. We are progressing in shallow understanding of unstructured text using word embedding, LSTM and GRU. We getting remarkable results extract sentiments, and named entity recognition. Meaning giving some text, we can figure what word is a place, people, or business. We can also predict the next word in a sequence with some semblance of awareness of context.
If you plot result of some models using SNA, a dimensionality reduction algorithm. You will see stunning consistency in the words and their synonyms. Some words will line up in predictable pattern. As great as that sounds, we are still far off from having the computer really understanding the text and from its contextual environment. Keeping with the spirit of this post, that is to train with less data is better. They are not ideal. For you to do the things I mentioned above, you still have to multiple 500 X 500 matrix. All this for shallow understanding of natural languages.
Interactive training take a lot less data but it the requires a lot of people inputs. You need people to nudge the computer in the right direct when it makes a mistake. It looks very promising.
In a computer vision course taught by the great Andrej Karpathy at Stanford (Youtube), I discovered that Data augmentation is a thing. It’s another eye opening technique. For many readers of this story , you might be veteran of the industry. I am not. I was excited to know that we could create data out of thin air. To me, it is like magic. Kind a like fractional reserve bank and bank notes to create money from the ether. Puff!
You can translate, rotate or transform an image in a myriad of ways to simulate an increase the size of your data set. Off course the data hoarders and GPU Mafia can imply these same trick. Nevertheless these techniques are levelling the playing field. It’s like giving the kid with glasses pair of contacts lens so she can get 5 minutes of playing time. There is no guarantee that she going to become a starter but at least she getting some playing time. Notice that all the techniques have one common thread.
Each of these ideas is slowly prying us from the claws of the data hoarding monsters and from distortion of the GPU mafias. Like clean water, clean air and clean oceans, AI belongs to all of us. Big data is a common goods. It doesn’t only belong to big and powerful corporations. Once you know, you are responsible. If you are the first to uncover a potential problem, that makes you the one most qualify to start looking for a solution. Let’s get active, let’s get doing.
Support the struggles against centralization of machine learning. You can do this with your comments, you can do with your engagements, and you can this by sharing your ideas. This story is too import for one person to write, you can help by adding your sentence, by adding your paragraph to a chapters to this story. This isn’t my story. It isn’t your story. It isn’t their story. It’s our story.
Let’s write it together.
Did you enjoy this post?
Recommend it, by clicking the hearth ♡ icon below.
Twitter Roydell Clarke
Public contact: www.21.co/roydell2b
Twitter: @roydell2