Highlights of this year report on the AI industry and its trends
Artificial Intelligence (AI) is one of the hottest topics today. Recent advances literally talk for themselves — say hi to GPT-3, and it will greet you back. AI-discovered pharmaceutics is around the corner. Companies are hiring more Ph. Ds than ever while policy-makers are trying to make sense of this year tech with centuries-old laws. Exciting times for researchers and investors alike, maybe not so much for politicians and lawyers.
This year, for the third time, Nathan Benaich and Ian Hogarth teamed up to compile a state-of-affairs report on AI, covering the latest research, industry, talents, and policy news. Last but not least, the authors lay their predictions for 2021. No work about AI is done without some guesswork, right?
You can read the full report here.
In this article, I summarize the report’s main topics and findings, followed by my own opinions on the matter. I also invite you, the readers, to comment on your thoughts about this year’s AI and your predictions for 2021.
Report’s Summary
Research (Slides 10-62)
The report starts on the technical side: only 15% of AI papers have open-sourced their code, PyTorch has most of the “research market share,” and several university groups reached the billion parameters mark. Moreover, it remarks on the economic and environmental costs of massive models. Even though the hardware is improving, deep learning costs grow exponentially. Current SOTA models cost millions of dollars to train, let alone tune.
Applications-wise, Natural Language Processing (NLP) drew most of the attention this year. Beyond NLP, AI is also powering an on-going paper boom within the biology and medicine communities. Besides these two areas, Graph Neural Networks (GNNs) and Reinforcement Learning (RL) have also had their share of breakthroughs this year.
Of course, COVID-19 has also had its mark on the AI community, with efforts dedicated to almost all aspects of the disease.
Talent (Slides 63-81)
While the number of papers keeps growing, the number of professors leaving academia for big tech companies keeps growing as well — and universities are taking the blow. To fight back, universities are focusing on dedicated AI institutes and funding initiatives.
Even larger is the international brain drain. This year showed many scientists relocating from Asia to the US for their studies, and most of them remain in the US after graduation. America’s dependency on foreign talent is blatant. 70% of AI researchers working in the US are not US-trained. This translates to publication results. Chinese researchers account for about 29% of oral presentations on NeurIPS (acceptance rate is 0.5%).
Despite COVID, AI Talent demand continues high, and enrollment in AI courses keeps growing.
Industry (Slides 82-129)
The biggest highlight is the AI-based pharmaceutics. We are nearing the point where AI-found drugs will enter the market. This goes hand-to-hand with the biology/medicine papers boom. A point of contempt, however, is regulation and procedure. Current approval methods were not designed for AI-found or AI-led products nor constant improvement workflows.
The Autonomous Vehicles (AV) industry faces similar issues. Legislation on driver-less cars is lagging more than the AVs themselves — even though billions are being poured in. Part of the money is going to in-house hardware, especially custom LiDAR technology, and the other to the AV stack itself, which remains largely hand-engineered.
Meanwhile, the compute demand sparked new compute platform providers and specialized AI hardware, such as Graphcore’s M2000, Nvidia’s DGX-A100, and Google’s TPUv4. Conjointly, work on improved ML infrastructure and operations is also skyrocketing.
Slides 113 to 129 are dedicated to industry success stories.
Politics (Slides 130–170)
This year was marked by ethical issues of AI going mainstream, including, but not limited to, gender/race bias, police and military use, face recognition, surveillance, and deep fakes. In particular, the military’s interest in AI tech is alarming but far from being unexpected.
Conferences, such as NeurIPS, ICLR, and Google, have adopted new ethic codes, and some companies are leaning towards fairness and privacy ideals. However, there is still a long road ahead toward real change. Chip production and IP ownership seem of much greater concern to governments.
The buzz word in politics is AI nationalism: countries investing in being AI leaders and national-wide AI policies — a sovereignty concern.
Predictions (Slide 172)
Benaich and Hogarth end the report with their predictions for 2021. They are as follows (Slide 172):
1) The race to build larger language models continues, and we see the first 10 trillion parameter model.
2) Attention-based neural networks move from NLP to computer vision in achieving state of the art results.
3) A major corporate AI lab shuts down as its parent company changes strategy.
4) In response to US DoD activity and investment in US-based military AI startups, a wave of Chinese and European defense-focused AI startups collectively raise over $100M in the next 12 months.
5) One of the leading AI-first drug discovery startups (e.g., Recursion, Exscientia) either IPOs or is acquired for over $1B.
6) DeepMind makes a major breakthrough in structural biology and drug discovery beyond AlphaFold.
7) Facebook makes a major breakthrough in augmented and virtual reality with 3D computer vision.
8) NVIDIA does not end up completing its acquisition of Arm.
While these predictions are meant for next year, some of them are already a reality. Regarding (1), Microsoft announced their DeepSpeed library is already capable of “trillion parameter models.” While none was released so far, the road is clear for a 10 trillion model to emerge. Regarding (2), An Image is Worth 16×16 Words strides in this direction.
Regarding (6), beyond AlphaFold, we have… AlphaFold 2! Its recent release promises a similar, or greater, impact on biology than AlexNet had on computer vision back in 2012. Current media coverage seems to agree. I highly believe the authors will mark this one as correct in 2021.
Remarks on the Report
In the following, I add my own views to the report findings and some tie-ins with recent events, following slide order. Bear in mind the report was released in October. Since then, many things have happened.
Research (Slides 10–62)
- Only 15% of AI papers released their code (Slide 11): I wonder the percentage rate of other computer science fields. Besides, not all codes matter the same. A new architecture made of pre-existing components is less important, code-wise, than a brand new implementation. A novel loss or optimizer function can be as short as an embedded snippet.
All in all, I agree AI is not as open as we think but is still pretty open relative to other areas of computer science - PyTorch will surpass TensorFlow on industry use (Slide 13, 14): While I believe this to be true, the data is misleading. Only 30% of papers state their framework. Many might still be TensorFlow-bound. Besides, I find it odd that no Keras data is shown (Slide 14).
- The AI Race is incredibly resource-intensive (Slides 16–24): Recently, Timnit Gebru was fired from Google for a paper draft outlining the monetary and ecological costs of training large language models. According to her paper, a 0.2bi parameters Transformer trained on NAS costs about 1 million USD. GPT-3 boasts 175bi. The math doesn’t look any good for the planet.
- This arms race won’t lead us anywhere (Slides 16–24): In my opinion, the race for NLP breakthroughs leads to no real breakthroughs at all.
GPT-3 is pretty much GPT-2 on steroids. Given Microsoft’s DeepSpeed mentioned above, we shall continue seeing bloated models trending on media and no meaningful results on understanding from such efforts. - Universities can’t keep up, or can they? (Slides 22): There is no way any AI department can keep up with big tech. Universities need to play a different game. Small models research might bring as much performance at logarithmic costs. However, currently, companies are the ones leading research on efficient learning. For example, MobileNet / EfficientDet are Google’s, ShuffleNet is from Face++.
- Transformers are Remarkable (Slide 29): These models are based on the Attention mechanism, which is notoriously power and resources hungry, as it is N² given a sequence of N elements. Efficient attention is a hot topic, but no solution has been declared a winner yet. Most of the AI costs mentioned above can be traced back to this mechanism.
- Biology is experiencing its “AI moment” (Slide 30): Rightfully so. With AlphaFold 2, we might see big breakthroughs in biology in this decade, just as we saw with AlexNet and Computer Vision back in the 2010s.
- AI-based screening mammography (Slide 34): This is a highly controversial article. It claims super-human performance but lacks interpretability and no code or dataset was released so far for third party inspection and reproduction.
This highly-publicized article triggered a heated response from researchers worldwide, which cosigned “The importance of transparency and reproducibility in artificial intelligence research.”
We, as a community, must try to break AI from being a silly accuracy competition. How can a physician trust black box algorithms?
Talent (Slides 63–81)
- The Great Brain Drain (Slide 64): While this is about AI and 2020, I can’t help but mention how US-centric this view is. All the mentioned universities are US-based. Brains move all the time, especially from developing countries to richer ones. It just so happens that universities are “the poor countries” and companies are the “rich countries” this time.
- Departures correlate with reduced entrepreneurship (Slide 66): The connection, in my opinion, is poor. The market is just saturated with companies and lacks talents—a poor environment for more companies. Slide 73 agrees, as most Ph. Ds are foreign, and foreigners are more likely to join a large company than to start their own.
- Chinese-educated researchers contributions at NeurIPS (Slide 70): Correlates with China’s plan to be an AI leader.
- The majority of top AI researchers working in the US were not trained in America (Slide 71-75): These slides point to a simple fact: the US is highly dependent on foreign talent. Most students arrive for a Ph. D and stay for work on tech companies. Xenophobic laws are not in the USA’s favor. Nonetheless…
- Trump worked against the US (Slide 76): With his attempt to lock immigrants out of the US, Trump got nothing, but certainly raised awareness of America’s dependency on foreign talent.
Other countries looking for AI superiority might seize such opportunities to lure talents to their universities.
Industry (Slides 82–129)
- AI-first drug discoveries (Slides 83–92): Apparently, pharmaceutics’ investment is likely to reap benefits faster than all the money spent on autonomous vehicles. Besides, better drugs and increased disease coverage is likely to be more beneficial to humanity than self-driving cars.
- AVs are in their infancy (Slides 93-96): Legislation is premature at best and far from worldwide. If a flawless AV was released today, it would be forbidden almost everywhere or require a driver anyway.
- When even a billion dollars isn’t enough (Slides 97–106): And more will be poured in, and it will still not be enough. AVs are a time problem, not a money problem. AI is not mature enough, nor are our laws ready for it. Current vision research ignores that our world is continuous. We don’t need to detect signposts from a single image. We need to aggregate the results of multiple frames better. IMHO, companies are just dumping money on LiDAR and dead-end supervised problems.
- Compute Advances (Slides 107–111): Novel hardware is always nice. Yet, I wonder if the rest of the stack will keep up. A problem with large datasets + large compute is getting the next batch ready on time. The faster (and larger) the compute, the harder it is to fetch training data at the required pace (+ data augmentation).
Politics (Slides 130–170)
- Ethical Risks (Slide 131): This requires special attention. The recent firing of Timnit Gebru highlights how wrong the entire industry is in its treatment of ethics. Demanding big AI tech to lead AI ethics research is like asking oil companies to lead the fight against global warming.
Her firing shows that companies will play their part as long as it doesn’t harm their business model. It shouldn’t be any surprise.
Given how tied US universities are to corporate funding, it is hard to expect them to pick up the fight as well. - Face Recognition is a Major Issue (Slides 132-140): Current laws were designed from humans to humans. How can it generalize to a system that can potentially identify all individuals in a crowd? Are we entitled to anonymity, the right of not being identified? To which extent? Should companies be blocked but law-enforcement allowed?
In a sense, there is a parallel to superheroes. How would our laws apply to Superman or The Flash? Can we really expect average-people laws to be equally applicable to super-human capacities? - What about Speech and Text? (Slides 132–140): Face recognition concerns our presence, but what about everything we say on our phones? All we say can be processed, monitored, and misinterpreted. Passing laws on faces and ignoring other media would be ignoring the elephant in the room: everything we do is monitored.
- AI Nationalism (Slides 161–167): Most developed nations woke up to AI and its existential threat. AI supremacy can easily translate into military and economic dominance and affect sovereignty.
China is clearly ahead, as it has been working towards AI leadership for quite some time and is investing deeply in talent. As mentioned above, in my opinion, it is a matter of time for China to lure foreign talent out of the US. The same goes for India.
Artificial Intelligence is at its highest point ever. Results have never been so good. There have never been so many papers. The Alphas*, the *Formers, the muppets. It is quite easy to drown yourself in ablation studies and lose sight of what else there is to be seen. Besides accurate, models ought to be economical, inclusive, and interpretable. Sacrificing these properties for accuracy’s sake alone is narcissistic. It helps no one but the authors.
High accuracy is the first step, not the last.
On top of that, AI is not a natural resource. It does not belong to a nation, to a land — neither does computing. The race for AI supremacy is unlike other geopolitical struggles. It is an intellectual race powered by computing hardware and the brains to operate it — and talent flows.
Thanks for reading
