{"id":3259,"date":"2026-01-16T14:20:27","date_gmt":"2026-01-16T14:20:27","guid":{"rendered":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/2026\/01\/16\/making-very-small-llms-smarter\/"},"modified":"2026-01-16T14:20:27","modified_gmt":"2026-01-16T14:20:27","slug":"making-very-small-llms-smarter","status":"publish","type":"post","link":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/2026\/01\/16\/making-very-small-llms-smarter\/","title":{"rendered":"Making (Very) Small LLMs Smarter"},"content":{"rendered":"

Hello, I\u2019m Philippe, and I am a Principal Solutions Architect helping customers with their usage of Docker. I started getting seriously interested in generative AI about two years ago. What interests me most is the ability to run language models (LLMs) directly on my laptop (For work, I have a MacBook Pro M2 max, but on a more personal level, I run LLMs on my personal MacBook Air M4 and on Raspberry Pis \u2013 yes, it\u2019s possible, but I\u2019ll talk about that another time).<\/p>\n

Let\u2019s be clear, reproducing a Claude AI Desktop or Chat GPT on a laptop with small language models is not possible. Especially since I limit myself to models that have between 0.5 and 7 billion parameters. But I find it an interesting challenge to see how far we can go with these small models. So, can we do really useful things with small LLMs? The answer is yes, but you need to be creative and put in a bit of effort.<\/p>\n

I\u2019m going to take a concrete use case, related to development (but in the future I\u2019ll propose \u201cless technical\u201d use cases).<\/p>\n

(Specific) Use Case: Code Writing Assistance<\/h2>\n

I need help writing code<\/h3>\n

Currently, I\u2019m working in my free time on an open-source project, which is a Golang library for quickly developing small generative AI agents. It\u2019s both to get my hands dirty with Golang and prepare tools for other projects. This project is called Nova<\/strong>; there\u2019s nothing secret about it, you can find it here<\/a>.<\/p>\n

If I use Claude AI<\/strong> and ask it to help me write code with Nova<\/strong>: \u201cI need a code snippet of a Golang Nova Chat agent using a stream completion.\u201d<\/em><\/p>\n

\n \"tiny\n <\/div>\n

The response will be quite disappointing, because Claude<\/strong> doesn\u2019t know Nova<\/strong> (which is normal, it\u2019s a recent project). But Claude doesn\u2019t want to disappoint me and will still propose something which has nothing to do with my project.<\/p>\n

And it will be the same with Gemini<\/strong>.<\/p>\n

\n \"tiny\n <\/div>\n

So, you\u2019ll tell me, give the \u201csource code of your repository to feed\u201d to Claude AI or Gemini. OK, but imagine the following situation: I don\u2019t have access to these services, for various reasons. Some of these reasons could be confidentiality, the fact that I\u2019m on a project where we don\u2019t have the right to use the internet, for example. That already disqualifies Claude AI and Gemini. How can I get help writing code with a small local LLM? So as you guessed, with a local LLM. And moreover, a \u201cvery small\u201d LLM.<\/p>\n

Choosing a language model<\/h2>\n

When you develop a solution based on generative AI, the choice of language model(s) is crucial. And you\u2019ll have to do a lot of technology watching, research, and testing to find the model that best fits your use case. And know that this is non-negligible work.<\/p>\n

For this article (and also because I use it), I\u2019m going to use hf.co\/qwen\/qwen2.5-coder-3b-instruct-gguf:q4_k_m, which you can find here<\/a>. It\u2019s a 3 billion parameter language model, optimized for code generation. You can install it with Docker Model Runner with the following command:<\/p>\n

\n
\ndocker model pull hf.co\/Qwen\/Qwen2.5-Coder-3B-Instruct-GGUF:Q4_K_M\n<\/pre>\n<\/div>\n
And to start chatting with the model, you can use the following command:<\/p>\n
\n
\ndocker model run hf.co\/qwen\/qwen2.5-coder-3b-instruct-gguf:q4_k_m\n<\/pre>\n<\/div>\n
Or use Docker Desktop:<\/p>\n
\n                \"tiny\n        <\/div>\n
So, of course, as you can see in the illustration above, this little \u201cQwen Coder<\/strong>\u201d doesn\u2019t know my Nova<\/strong> library either. But we\u2019re going to fix that.<\/p>\n
Feeding the model with specific information<\/h2>\n
For my project, I have a markdown file in which I save the code snippets I use to develop examples with Nova<\/strong>. You can find it here<\/a>. For now, there\u2019s little content, but it will be enough to prove and illustrate my point.<\/p>\n
So I could add the entire content of this file to a user prompt that I would give to the model. But that will be ineffective. Indeed, small models have a relatively small context window. But even if my \u201cQwen Coder<\/strong>\u201d was capable of ingesting all the content of my markdown file, it would have trouble focusing on my request and on what it should do with this information. So,<\/p>\n