When it was announced not too long ago, GPT-3 set a new standard for the size of an AI model at a whopping 175 billion parameters, and an estimated cost of at least $4.6 million. Ever since the first closed beta examples started showing up, it has been blowing people’s minds and causing everyone to think about what is next.

One of the best things about programming (and open source in particular) is that everyone can play and have access to everything they need to build cool stuff. If I want to create an AI, I can download Tensorflow and convince myself that I can build the best AI in the world! Sure, I probably can’t, but that is besides the point because just the opportunity provides me with inspiration and the hope that there a global community of people working on this really cool problem.

Now that it has been shown that investing tens of millions of dollars in raw AI compute power makes a significant difference in the quality of the AI, it changes the game a bit. Instead of thinking that the next big AI breakthrough will come from a small software tweak, you have to be a little bit more convinced that it is more likely to come from brute force application of existing methods.

This brings me to the disheartening conclusion that you may just need millions of dollars to play this game. I have always been of the mind, however, that if the game isn’t fair, you should change it. If I’m not going to make an impact with modest resources and some software smarts, how can I make an impact?

My thinking so far is that it is expensive because we are leveraging compute architectures that are barely analogous to the neural networks we are trying to run. We are doing this on a single layer of photo etched silicon circuits designed to operate in a much more complex way than the task of gradient decent requires. I think in this case, AI hardware has the ability to take the cost of training this large scale hundred billion parameter network down orders of magnitude.

This conclusion gives me pause when thinking about my lingering plans to pursue a PhD in AI, but as always, I will continue to contemplate the idea.