{"id":3426,"date":"2026-02-13T14:10:40","date_gmt":"2026-02-13T14:10:40","guid":{"rendered":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/2026\/02\/13\/how-to-solve-the-context-size-issues-with-context-packing-with-docker-model-runner-and-agentic-compose\/"},"modified":"2026-02-13T14:10:40","modified_gmt":"2026-02-13T14:10:40","slug":"how-to-solve-the-context-size-issues-with-context-packing-with-docker-model-runner-and-agentic-compose","status":"publish","type":"post","link":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/2026\/02\/13\/how-to-solve-the-context-size-issues-with-context-packing-with-docker-model-runner-and-agentic-compose\/","title":{"rendered":"How to solve the context size issues with context packing with Docker Model Runner and Agentic Compose"},"content":{"rendered":"

If you\u2019ve worked with local language models, you\u2019ve probably run into the context window limit, especially when using smaller models on less powerful machines. While it\u2019s an unavoidable constraint, techniques like context packing make it surprisingly manageable.<\/p>\n

Hello, I\u2019m Philippe, and I am a Principal Solutions Architect helping customers with their usage of Docker.\u00a0 In my previous blog post<\/a>, I wrote about how to make a very small model useful by using RAG. I had limited the message history to 2 to keep the context length short.<\/p>\n

But in some cases, you\u2019ll need to keep more messages in your history. For example, a long conversation to generate code:<\/p>\n

\n
\n- generate an http server server in golang\n- add a human structure and a list of humans\n- add a handler to add a human to the list\n- add a handler to list all humans\n- add a handler to get a human by id\n- etc...\n\n<\/pre>\n<\/div>\n
Let\u2019s imagine we have a conversation for which we want to keep 10 messages in the history. Moreover, we\u2019re using a very verbose model (which a lot of tokens), so we\u2019ll quickly encounter this type of error:<\/p>\n
\n
\nerror: {\n    code: 400,\n    message: 'request (8860 tokens) exceeds the available context size (8192 tokens), try increasing it',\n    type: 'exceed_context_size_error',\n    n_prompt_tokens: 8860,\n    n_ctx: 8192\n  },\n  code: 400,\n  param: undefined,\n  type: 'exceed_context_size_error'\n}\n\n<\/pre>\n<\/div>\n
What happened?<\/p>\n
Understanding context windows and their limits in local LLMs<\/h2>\n
Our LLM has a context window, which has a limited size. This means that if the conversation becomes too long\u2026 It will bug out.<\/p>\n
This window is the total number of tokens<\/strong> the model can process at once, like a short-term working memory.\u00a0 Read this IBM article<\/a> for a deep dive on context window<\/p>\n
\n
In our example in the code snippet above, this size was set to 8192 tokens for LLM engines that power local LLM, like Docker Model Runner, Ollama, Llamacpp, \u2026<\/em><\/p>\n<\/div>\n
This window includes everything: system prompt<\/strong>, user message<\/strong>, history<\/strong>, injected documents,<\/strong> and the generated response<\/strong>. Refer to this Redis post<\/a> for more info.\u00a0<\/p>\n
Example<\/strong>: if the model has 32k context, the sum (input + history + generated output) must remain \u2264 32k tokens. Learn more here<\/a>.\u00a0\u00a0<\/p>\n
It\u2019s possible to change the default context size (up or down) in the compose.yml file:<\/p>\n
\n
\nmodels:\n  chat-model:\n    model: hf.co\/qwen\/qwen2.5-coder-3b-instruct-gguf:q4_k_m\n    # Increased context size for better handling of larger inputs\n    context_size: 16384\n\n<\/pre>\n<\/div>\n
\n
You can also do this with Docker with the following command: docker model configure \u2013context-size 8192 ai\/qwen2.5-coder `<\/em><\/p>\n<\/div>\n
And so we solve the problem, but only part of the problem<\/strong>. Indeed, it\u2019s not guaranteed that your model supports a larger context size (like 16384), and even if it does, it can very quickly degrade the model\u2019s performance.<\/p>\n
Thus, with hf.co\/qwen\/qwen2.5-coder-3b-instruct-gguf:q4_k_m, when the number of tokens in the context approaches 16384 tokens, generation can become (much) slower<\/strong> (at least on my machine). Again, this will depend on the model\u2019s capacity (read its documentation). And remember, the smaller the model, the harder it will be to handle a large context and stay focused.<\/p>\n
\n
Tips:<\/em><\/strong> always provide an option (a \/clear command for example) in your application to empty the message list, or to reduce it. Automatic or manual. Keep the initial system instructions though.<\/em><\/p>\n<\/div>\n
So we\u2019re at an impasse. How can we go further with our small models?<\/p>\n
Well, there is still a solution, which is called context packing<\/strong>.<\/p>\n
Using context packing to fit more information into limited context windows<\/h2>\n
We can\u2019t indefinitely increase the context size. To still manage to fit more information in the context, we can use a technique called \u201ccontext packing\u201d<\/strong>, which consists of having the model itself summarize previous messages (or entrust the task to another model), and replace the history<\/strong> with this summary and thus free up space in the context.<\/p>\n
So we decide that from a certain token limit, we\u2019ll have the history of previous messages summarized, and replace this history with the generated summary.<\/strong><\/p>\n
I\u2019ve therefore modified my example to add a context packing<\/strong> step. For the exercise, I decided to use another model to do the summarization.<\/p>\n
Modification of the compose.yml file<\/h3>\n
I added a new model in the compose.yml file: ai\/qwen2.5:1.5B-F16<\/p>\n
\n
\nmodels:\n  chat-model:\n    model: hf.co\/qwen\/qwen2.5-coder-3b-instruct-gguf:q4_k_m\n\n  embedding-model:\n    model: ai\/embeddinggemma:latest\n\n  context-packing-model:\n    model: ai\/qwen2.5:1.5B-F16\n\n<\/pre>\n<\/div>\n
Then:<\/p>\n