Apple published a new paper provocatively titled “The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity.” I’ve seen a lot of hot takes, ranging in character from, “this research is nothing but sour grapes from Apple,” to “this is a deathblow for LLMs.” My hot take? Neither of those are true.

In brief, the paper challenges “reasoning models” including Claude Sonnet 3.7 with Extended Thinking and GPT-o3-mini to solve puzzles with increasing complexity. One such puzzle is the well-known Towers of Hanoi, where you must move a set of disks from a starting peg to a final peg without ever stacking a larger disk onto a smaller disk. Here’s an online version of the puzzle, if you care to play.

All of the reasoning models seem to utterly collapse at solving these problems after a certain level of complexity. For Towers of Hanoi, the “reasoning” models start failing when they have to move 5 or 6 disks, but solve the puzzle fairly reliably before that. There are some other interesting findings, but that’s the main thrust.

Here are my 10 takes of moderate temperature:

1.) “Thinking” and “reasoning” are definitely misnomers in these models. The so-called “reasoning” approach is to iteratively prompt the LLM by using the LLM itself to generate interim prompts, provide itself with feedback, adjust those prompts based on the feedback, and decide when to stop. But crucially, all of the final output could have been produced with a single prompt if only you knew it ahead of time. Therefore, it is much better to think of test time compute as a search of the underlying LLM’s latent space, not as “reasoning” or “thinking.”

2.) Point 1 is pretty widely accepted in the research community, but the marketing language used to sell LLM products does not reflect this reality. I am once again begging researchers, executives, and marketing folks alike to stop using vague, misleading, bombastic, and anthropomorphizing language to describe these systems.

3.) Accusations that Apple has “sour grapes” are kinda fair. They are way behind the curve on LLMs, and seem to want to keep betting against the technology. Notably Apple did not tout Apple Intelligence much this week at WWDC. But the paper also shows real limitations and weaknesses with using test time compute. It’s more than just sour grapes.

4.) Could this just be the interpolation vs extrapolation problem and target leakage? Machine learning models are notorious for failing to generalize. That is, they are good at making predictions within the distribution of their training data and bad at making predictions outside of it. To support this argument the authors evaluate performance on two older math benchmarks Math-500 and AIME24 and one newer math benchmark AIME25. They write:

5.) Future and current models will benefit substantially from “tool use” and code execution. The industry is already moving in this direction and the first attempt to standardize a protocol — Anthropic’s MCP — is gaining steam. In this paradigm instead of relying on the mechanics of next token prediction to perform the steps of these puzzles, the LLM will generate and then execute an algorithm, which is also what a human would probably do to solve these puzzles at scale.

State of the art models can already produce working algorithms that solve the Towers of Hanoi problem, in part because this is a very popular introductory CS problem with limitless solutions found online and in textbooks. The models might struggle to generate novel algorithms, but I still think this approach would work better than current “reasoning models” approach for the problems tested by Apple.

6.) Relatedly, it was interesting — but not surprising to me — that the model would not reliably follow the steps of an algorithm provided by the authors in the prompt. Models aren’t trained to do that. Long running examples of following an algorithm are unlikely to be well-represented in training sets. It would actually be quite impressive and interesting emergent behavior if the models reliably followed the steps of novel algorithms provided at inference time, especially over long runs of the algorithm.

This is probably just the extrapolation problem again. There just isn’t that much text showing examples of really long running algorithmic steps. Here are the authors again:

7.) I suspect this problem could be easily “papered over” if the training data included several such long running “step by step” outputs from an algorithm. Even so, that wouldn’t solve the main contention that the models aren’t “reasoning” under any prior formal definition of the term — a conclusion I agree with. That said, it’s possible new methods will be developed to overcome the issue more robustly (see point 5).

Those declaring that this is a deathblow for LLMs are forgetting that this is just how progress gets made: you build something, you identify limitations, you overcome them, repeat. That said, those pretending this isn’t a significant limitation and that it should have no meaningful impact on LLM research are equally deluded.

It’s not a foregone conclusion that the problem can be solved, but a lot of smart people are going to spend billions of dollars trying. So I won’t count them out just yet.

8.) I think the path to some kind of recursively self improving “AGI” is still unknown, and I think this paper does actually provide some more evidence for this claim. A machine that had significant formal reasoning capability and the enormous amounts of computational power available to the tested models would do a much better job on these puzzles even at higher complexity. To me, neither scaling up the models nor searching the latent space of a pre-trained transformer model is likely to yield such a system. New methods will be needed for that.

9.) Sometimes these next token predictors are right for the wrong reasons. For example, in Apple’s paper they find that the examined the so-called reasoning traces — the intermediate text produced by the model during the test time compute process — and found that, “For simpler problems, reasoning models often find the correct solution early in their thinking but then continue exploring incorrect solutions.”

This is consistent with other research. Anthropic’s own mechanistic interpretability research suggests that LLMs “reasoning traces” are sometimes inconsistent with the actual internal mechanisms at play or contradictory with the final output produced. That same paper confirms other research that found, “LLMs perform arithmetic using neither robust algorithms nor memorization; rather, they rely on a “bag of heuristics.”

It’s hard to know how often this is happening, and other research shows that the internal mechanisms in LLMs probably do sometimes encode meaningful and accurate “world models.” If anything, this demonstrates just how much more work needs to be done on interpretability.

10.) The above notwithstanding, the current slate of models can do some pretty useful things. I think one thing some AI detractors get wrong is an unwillingness to acknowledge that the current slate of models are doing useful things already, and instead use every new critique as a way to say, “see this stuff is total garbage that will never be useful!”

People are using current models to brainstorm, write code, argue, persuade, draft corporate copy, and more. Some of these uses might not be wise. Some of them — like the heavily automated spam/scam ecosystem — are frankly quite horrible. We do ourselves a disservice and damage our credibility when we don’t acknowledge that something like 91% of developers are using some kind of code generation tools, and that they mostly like using them.

There is clearly some “there” there with LLMs, but I am continually more persuaded by the notion that AI will progress as “normal” technology, and won’t be something that upends the entire economy in the next two years.