On LLMs Reasoning
Everyone and their uncle seems to have an opinion on the recent paper by Apple researchers on whether LLMs can reason. So here is mine. It involves Bayes Theorem.
I have offered my opinion already, in brief. This is more of an exposition on that.
What has surprised me over the last day or two, when this paper came out, is that people on all sides are acting as if it was something absolutely world changing. Having been on social media (both public (Twitter) and private (WhatsApp groups) ), I have seen reactions of all sides.
Before I start putting out all the reactions (including my own), it makes sense to summarise what this paper says. To summarize (this is my interpretation),
LLMs, even the so-called “reasoning” models, cannot actually reason. Instead, they simply interpolate from their existing knowledge base. And the reason they appear intelligent is that they have a lot of knowledge (vast training datasets).
In fact, when you give problems that are far outside the training set, the performance of LLMs “collapses to 0”.
Other People’s Reactions
Some people have reacted with outrage, that these researchers have dared to call LLMs unintelligent.
Some others have launched ad hominem attacks, alleging, for example, that one of the authors of this paper was fired by Google for “being too loudly woke”, and is inherently biased against AI.
Then there are those who have chosen Apple as their ad hominem target, saying that the company hasn’t done much of note when it comes to AI, and so it “has no right to comment about it”.
People have found flaws in the paper’s methodology.
There are people who have weighed in on this to make their favourite points about “AGI”. And then there are people who have said that irrespective of this paper, LLMs still outperform humans on a lot of cognitive tasks and so they are great, and cnanot be called unintelligent.
Then of course there are people who have said that they have always known this (that LLMs are not intelligent), and this is just further proof. For example, there is Cricinfo founder (and Columbia CS prof) Vishal Misra, who writes:
Current LLMs are magnificent interpolators. They can simulate reasoning when it lies within the support of their training. But they cannot extend that reasoning arbitrarily. The boundary of what they can do is the deductive closure of their priors.
Then, of course, there is Prof. Rao (Subbarao Kambhampati) of ASU who has always maintained that LLMs cannot reason (he’s simply posted a screenshot of India Today).
My opinion
Ok, enough of what others had to say about this. This is my blog and you hear me out.
I think a big mountain is being made out of a molehill here.
In my “humble” opinion, there is no real new information in this paper, just a formalisation (in a way academics can do) of what people already knew.
This is all I believe on this:
AGI is a laughable concept. Something that has been concocted by people who have read too much science fiction and too little linear algebra (and by OpenAI in an attempt to try and kick Microsoft off its cap table).
LLMs are incredibly useful tools. We have barely scratched the surface of what we can achieve using LLMs. The productivity gains will be insane.
LLMs are ultimately linear algebra. Vishal Misra puts this well in his blog, actually. They are just linear algebra on such large matrices that the results sometimes astound us (and sometimes frustrate us to an equal degree).
And they are so complex that we fully don’t understand how they work. And that makes us think of them as “magic”, which they are absolutely not.
LLMs will continue to get better. Yes, people can talk about all kinds of “scaling walls”, but we have intelligent enough people working on LLM research that they will squeeze out more performance out of them. And this will mean LLMs will get used more and more. This virtuous cycle will continue. But LLMs don’t need to be intelligent for this to happen.
And in the light of all these prior beliefs, this paper changes absolutely nothing. Whatever I believed to be true before I had come across this paper,
Ravikiran Rao has this thing that he tweets at the end of every election results - these elections prove that we need to do more of whatever I have strongly believed in, and less of whatever I’ve opposed.
This Apple paper is like an election in that sense - the way people have interpreted it has been strongly introduced by their priors.
Bayes Theorem everywhere
People sometimes dismiss the Bayes Theorem as a purely mathematical concept, and thus underestimate it. However, it is far more profound than that, having implications in all areas of life.
For example, I had written this long ago in Mint, about how Bayes theorem holds the key to online flamewars.
This, perhaps, helps explain the people with unwavering opinions. If your prior belief is extreme, there is no evidence that can make you change your mind. Even if your prior belief is tilted heavily towards one side, it takes overwhelmingly strong evidence for you to move your position significantly. And that is what seems to be happening in the discourse on social media.
This dependence on prior probabilities is what makes Bayes’ Theorem controversial and resulted in a wait of a couple of centuries before the theorem was accepted. Even now, debates between ‘frequentists’ and ‘Bayesians’ abound, and they are unlikely to go anywhere due to the reason we saw above—extremely strong priors.
This extends to AI as well, and what people have believed about it. If you have always believed that “AGI is around the corner”, this paper suddenly challenges these beliefs, and you are likely to dismiss the paper, launch ad hominem attacks, question the methodology etc.
If you have believed that AI is a fad, this paper does well in terms of confirming your beliefs, and you can now make a more public show of your beliefs.
And then there are people like me who have the perfect set of priors that this new evidence does absolutely nothing to it!
I guess I’m an eigenvector of this paper?
Yes, with unit Eigen value.