EP09 - The Thinking Process of LLMs. With Sara Marjanovic
On other platforms: Web, Apple Podcast, YouTube.
I had the chance to chat with Sara Marjanovic, PhD student at University of Copenhagen, about the thinking process of LLMs.
Deepseek R1 has been the first open model with a visible thinking trace, and this opened the doors to new ways to evaluate and research LLMs. It made possible to benchmark thinking vs non-thinking models, compare different reasoning processes, look at traces to see what the reasoning process looks like, and find potential flaws or research direction to improve the effectiveness, as well as see how it influences the behaviour of the model.
What's interesting about looking at the thinking process? Few things stood out to me from the conversation with Sara:
Overthinking. Usually the agent defines the problem, find an answer and then verify again the thinking before wrapping up the reasoning process and provide the answer to the user. Sometimes, the agent enters in a loop and keeps repeating the same sentence, without being able to get out of this verification cycle. Sara and her team defined it as "rumination", because the LLM keeps thinking about the same exact thing and cannot really move on. Quite fun!
Context adherence. When giving to the agent a context that is clearly wrong, it may happen that the agent try to stick to it, even though it "knows" that it is wrong. This is visible from the thinking trajectory: the agent is clearly aware that the context is wrong, but it just follows it to "please" the user.
Unfaithfulness to thinking. When asked to create more and more complex ASCII images, the model just created them one after the other, without iterating on a draft to improve it. But looking at the thinking traces it is possible to see the model thinking about this kind of iterations for refinement. This means that the agent was not following its own thinking process.
Interesting? Then go ahead and listen to the episode for the full chat. And if you really want to dig deeper, have a look at the paper "DeepSeek-R1 Thoughtology: Let’s think about LLM reasoning".
That's it, see you at the next episode!