{"id":2549,"date":"2025-10-02T12:13:26","date_gmt":"2025-10-02T12:13:26","guid":{"rendered":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/2025\/10\/02\/fine-tuning-local-models-with-docker-offload-and-unsloth\/"},"modified":"2025-10-02T12:13:26","modified_gmt":"2025-10-02T12:13:26","slug":"fine-tuning-local-models-with-docker-offload-and-unsloth","status":"publish","type":"post","link":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/2025\/10\/02\/fine-tuning-local-models-with-docker-offload-and-unsloth\/","title":{"rendered":"Fine-Tuning Local Models with Docker Offload and Unsloth"},"content":{"rendered":"

I\u2019ve been experimenting with local models for a while now, and the progress in making them accessible has been exciting. Initial experiences are often fantastic, many models, like Gemma 3 270M<\/strong>, are lightweight enough to run on common hardware. This potential for broad deployment is a major draw.<\/p>\n

However, as I\u2019ve tried to build meaningful, specialized applications with these smaller models, I\u2019ve consistently encountered challenges in achieving the necessary performance for complex tasks. For instance, in a recent experiment testing the tool-calling efficiency of various models<\/a>, we observed that many local models (and even several remote ones) struggled to meet the required performance benchmarks. This realization prompted a shift in my strategy.<\/p>\n

I\u2019ve come to appreciate that simply relying on small, general-purpose models is often insufficient for achieving truly effective results on specific, demanding tasks. Even larger models can require significant effort to reach acceptable levels of performance and efficiency.<\/p>\n

And yet, the potential of local models is too compelling to set aside. The advantages are significant:<\/p>\n

Privacy<\/p>\n

Offline capabilities<\/p>\n

No token usage costs<\/p>\n

No more \u201coverloaded\u201d error messages<\/p>\n

So I started looking for alternatives, and that\u2019s when I came across Unsloth<\/a>, a project designed to make fine-tuning models much faster and more accessible. Its growing popularity (star history<\/a>) made me curious enough to give it a try.<\/p>\n

In this post, I\u2019ll walk you through fine-tuning a sub-1GB model to redact sensitive info without breaking your Python setup. With Docker Offload<\/a> and Unsloth, you can go from a baseline model to a portable, shareable GGUF artifact on Docker Hub in less than 30 minutes. In part 2 of this post, I will share the detailed steps of fine-tuning the model.\u00a0<\/p>\n

Challenges of fine-tuning models<\/h2>\n

Setting up the right environment to fine-tune models can be\u2026 painful. It\u2019s fragile, error-prone, and honestly a little scary at times. I always seem to break my Python environment one way or another, and I lose hours just wrestling with dependencies and runtime versions before I can even start training.<\/p>\n

Fortunately, the folks at Unsloth<\/strong> solved this with a ready-to-use Docker image<\/strong><\/a>. Instead of wasting time (and patience) setting everything up, I can just run a container and get started immediately.<\/p>\n

Of course, there\u2019s still the hardware requirement. I work on a MacBook Pro, and Unsloth doesn\u2019t support MacBooks natively, so normally, that would be a deal-breaker.<\/p>\n

But here\u2019s where Docker Offload<\/strong> comes in. With Offload, I can spin up GPU-backed resources in the cloud<\/a> and tap into NVIDIA acceleration, all while keeping my local workflow. That means I now have everything I need to fine-tune models, without fighting my laptop.<\/p>\n

Let\u2019s go for it.<\/p>\n

How to fine-tune models locally with Unsloth and Docker<\/h2>\n

Can a model smaller than 1GB reliably mask personally identifiable information (PII)?<\/p>\n

Here\u2019s the test input<\/strong>:<\/p>\n

This is an example of text that contains some data.
The author of this text is Ignacio L\u00f3pez Luna, but everybody calls him Ignasi.
His ID number is 123456789.
He has a son named Arnau L\u00f3pez, who was born on 21-07-2021.<\/p>\n

Desired output<\/strong>:<\/p>\n

This is an example of text that contains some data.
The author of this text is [MASKED] [MASKED], but everybody calls him [MASKED].
His ID number is [MASKED].
He has a son named [MASKED], who was born on [MASKED].<\/p>\n

When tested with Gemma 3 270M<\/strong> using Docker Model Runner<\/a>, the output was:<\/p>\n

[PERSON]<\/p>\n

Clearly, not usable. Time to fine-tune.<\/p>\n

Step 1: Clone the example project<\/strong><\/h3>\n
\n\u200b\u200bgit clone https:\/\/github.com\/ilopezluna\/fine-tuning-examples.git
\ncd fine-tuning-examples\/pii-masking\n<\/div>\n

The project contains a ready to use python script to fine tune Gemma 3 using the pii-masking-400k dataset<\/a> from ai4privacy<\/a>.<\/p>\n

Step 2: Start Docker Offload (with GPU)<\/strong><\/h3>\n
\ndocker offload start\n<\/div>\n

Select your account.<\/p>\n

Answer Yes<\/strong> when asked about GPU support (you\u2019ll get an NVIDIA L4-backed instance).<\/p>\n

Check status:<\/p>\n

\ndocker offload status\n<\/div>\n

See the Docker Offload Quickstart guide<\/a>.<\/p>\n

Step 3: Run the Unsloth container<\/strong><\/h3>\n

The official Unsloth image includes Jupyter and some example notebooks. You can start it like this:<\/p>\n

\ndocker run -d -e JUPYTER_PORT=8000
\n -e JUPYTER_PASSWORD=”mypassword”
\n -e USER_PASSWORD=”unsloth2024″
\n -p 8000:8000
\n -v $(pwd):\/workspace\/work
\n –gpus all
\n unsloth\/unsloth\n<\/div>\n

Now, let\u2019s attach a shell to the container:\u00a0<\/p>\n

\n docker exec -it $(docker ps -q) bash\n<\/div>\n

Useful paths inside the container:<\/p>\n

\/workspace\/unsloth-notebooks\/ \u2192 example fine-tuning notebooks<\/p>\n

\/workspace\/work\/ \u2192 your mounted working directory<\/p>\n

Thanks to Docker Offload<\/strong> (with Mutagen under the hood), the folder \/workspace\/work\/ stays in sync between cloud GPU and local dev machine.<\/p>\n

Step 4: Fine-tune<\/strong><\/h3>\n

The script finetune.py<\/em> is a small training loop built around Unsloth. Its purpose is to take a base language model and adapt it to a new task using supervised fine-tuning with LoRA. In this example, the model is trained on a dataset that teaches it how to mask personally identifiable information (PII) in text.<\/p>\n

LoRA makes the process lightweight: instead of updating all of the model\u2019s parameters, it adds small adapter layers and only trains those. That means the fine-tune runs quickly, fits on a single GPU, and produces a compact set of weights you can later merge back into the base model.<\/p>\n

When you run:<\/p>\n

\nunsloth@46b6d7d46c1a:\/workspace$ cd work
\nunsloth@46b6d7d46c1a:\/workspace\/work$ python finetune.py
\nUnsloth: Will patch your computer to enable 2x faster free finetuning.
\n[…]\n<\/div>\n

The script loads the base model, prepares the dataset, runs a short supervised fine-tuning pass, and saves the resulting LoRA weights into your mounted \/workspace\/work\/ folder. Thanks to Docker Offload, those results are also synced back to your local machine automatically.<\/p>\n

The whole training run is designed to complete in under 20 minutes on a modern GPU, leaving you with a model that has \u201clearned\u201d the new masking behavior and is ready for conversion in the next step.<\/p>\n

For a deeper walkthrough of how the dataset is built, why it\u2019s important and how LoRA is configured, stay tuned for part 2 of this blog!\u00a0\u00a0<\/p>\n

Step 5: Convert to GGUF<\/strong><\/h3>\n

At this point you\u2019ll have the fine-tuned model artifacts sitting under \/workspace\/work\/.<\/p>\n

To package the model for Docker Hub and Docker Model Runner usage, it must be in GGUF format<\/strong>. (Unsloth will support this directly soon, but for now we convert manually.)<\/p>\n

\nunsloth@1b9b5b5cfd49:\/workspace\/work$ cd ..
\nunsloth@1b9b5b5cfd49:\/workspace$ git clone https:\/\/github.com\/ggml-org\/llama.cpp
\nCloning into ‘llama.cpp’…
\n[…]
\nResolving deltas: 100% (45613\/45613), done.
\nunsloth@1b9b5b5cfd49:\/workspace$ python .\/llama.cpp\/convert_hf_to_gguf.py work\/result\/ –outfile work\/result.gguf
\n[…]
\nINFO:hf-to-gguf:Model successfully exported to work\/result.gguf\n<\/div>\n

Next, check that the file already exists locally (this indicates the automatic Mutagen-powered file sync process did finish):<\/p>\n

\nunsloth@46b6d7d46c1a:\/workspace$ exit
\nexit
\n((.env3.12) ) ilopezluna@localhost pii-masking % ls -alh result.gguf
\n-rw-r–r–@ 1 ilopezluna staff 518M Sep 23 15:58 result.gguf\n<\/div>\n

At this point, you can stop Docker Offload:<\/p>\n

\ndocker offload stop\n<\/div>\n

Step 6: Package and share on Docker Hub<\/strong><\/h3>\n

Now let\u2019s package the fine-tuned model and push it to Docker Hub:<\/p>\n

\n((.env3.12) ) ilopezluna@localhost pii-masking % docker model package –gguf \/Users\/ilopezluna\/Projects\/fine-tuning-examples\/pii-masking\/result.gguf ignaciolopezluna020\/my-awesome-model:version1 –push
\nAdding GGUF file from “\/Users\/ilopezluna\/Projects\/fine-tuning-examples\/pii-masking\/result.gguf”
\nPushing model to registry…
\nUploaded: 517.69 MB
\nModel pushed successfully\n<\/div>\n

You can find more details on distributing models in the Docker blog on packaging models<\/a>.<\/p>\n

Step 7: Try the results!<\/strong><\/h3>\n

Finally, run the fine-tuned model using Docker Model Runner:<\/p>\n

\ndocker model run ignaciolopezluna020\/my-awesome-model:version1 “Mask all PII in the following text. Replace each entity with the exact UPPERCASE label in square brackets (e.g., [PERSON], [EMAIL], [PHONE], [USERNAME], [ADDRESS], [CREDIT_CARD], [TIME], etc.). Preserve all non-PII text, whitespace, ‘ ‘ and punctuation exactly. Return ONLY the redacted text. Text: This is an example of text that contains some data. The author of this text is Ignacio L\u00f3pez Luna, but everybody calls him Ignasi. His ID number is 123456789. He has a son named Arnau L\u00f3pez, who was born on 21-07-2021”
\nThis is an example of text that contains some data. The author of this text is [GIVENNAME_1] [SURNAME_1], but everybody calls him [GIVENNAME_1]. His ID number is [IDCARDNUM_1]. He has a son named [GIVENNAME_1] [SURNAME_1], who was born on [DATEOFBIRTH_1]\n<\/div>\n

Just compare with the original Gemma 3 270M output:<\/p>\n

\n((.env3.12) ) ilopezluna@F2D5QD4D6C pii-masking % docker model run ai\/gemma3:270M-F16 “Mask all PII in the following text. Replace each entity with the exact UPPERCASE label in square brackets (e.g., [PERSON], [EMAIL], [PHONE], [USERNAME], [ADDRESS], [CREDIT_CARD], [TIME], etc.). Preserve all non-PII text, whitespace, ‘ ‘ and punctuation exactly. Return ONLY the redacted text. Text: This is an example of text that contains some data. The author of this text is Ignacio L\u00f3pez Luna, but everybody calls him Ignasi. His ID number is 123456789. He has a son named Arnau L\u00f3pez, who was born on 21-07-2021”
\n[PERSON]\n<\/div>\n

The fine tuned model is far more useful<\/strong>, and now it\u2019s already published on Docker Hub for anyone to try.<\/p>\n

Why fine-tuning models with Docker matters<\/strong><\/h2>\n

This experiment shows that small local models don\u2019t have to stay as \u201ctoys\u201d or curiosities. With the right tooling, they can become practical, specialized assistants<\/strong> for real-world problems.<\/p>\n

Speed<\/strong>: Fine-tuning a sub-1GB model took under 20 minutes with Unsloth and Docker Offload. That\u2019s fast enough for iteration and experimentation.<\/p>\n

Accessibility<\/strong>: Even on a machine without a GPU, Docker Offload unlocked GPU-backed training without extra hardware.<\/p>\n

Portability<\/strong>: Once packaged, the model is easy to share, and runs anywhere thanks to Docker.<\/p>\n

Utility<\/strong>: Instead of producing vague or useless answers, the fine-tuned model reliably performs one job, masking PII, something that could be immediately valuable in many workflows.<\/p>\n

This is the power of fine-tuning models: turning small, general-purpose models into focused, reliable tools<\/strong>. And with Docker\u2019s ecosystem, you don\u2019t need to be an ML researcher with a huge workstation to make it happen. You can train, test, package, and share, all with familiar Docker workflows.<\/p>\n

So next time you think \u201csmall models aren\u2019t useful\u201d<\/em>, remember, with a bit of fine-tuning, they absolutely can be.<\/p>\n

This takes small local models from \u201cinteresting demo\u201d to practical, usable tools<\/strong>.<\/p>\n

We\u2019re building this together!<\/h2>\n

Docker Model Runner is a community-friendly project at its core, and its future is shaped by contributors like you. If you find this tool useful, please head over to our GitHub repository<\/a>. Show your support by giving us a star, fork the project to experiment with your own ideas, and contribute. Whether it\u2019s improving documentation, fixing a bug, or a new feature, every contribution helps. Let\u2019s build the future of model deployment together!<\/p>\n

Start with Docker Offload for GPU on demand<\/a> \u2192<\/p>\n

Learn more<\/h3>\n

Check out Model Runner General Availability announcement<\/a><\/p>\n

Visit our Model Runner GitHub repo<\/a>!<\/p>\n

Learn how Compose makes building AI apps and agents easier<\/a><\/p>\n

Check out Unsloth documentation<\/a> for more details on the Unsloth Docker image.<\/p>","protected":false},"excerpt":{"rendered":"

I\u2019ve been experimenting with local models for a while now, and the progress in making them accessible has been exciting. […]<\/p>\n","protected":false},"author":0,"featured_media":0,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"site-sidebar-layout":"default","site-content-layout":"","ast-site-content-layout":"default","site-content-style":"default","site-sidebar-style":"default","ast-global-header-display":"","ast-banner-title-visibility":"","ast-main-header-display":"","ast-hfb-above-header-display":"","ast-hfb-below-header-display":"","ast-hfb-mobile-header-display":"","site-post-title":"","ast-breadcrumbs-content":"","ast-featured-img":"","footer-sml-layout":"","ast-disable-related-posts":"","theme-transparent-header-meta":"","adv-header-id-meta":"","stick-header-meta":"","header-above-stick-meta":"","header-main-stick-meta":"","header-below-stick-meta":"","astra-migrate-meta-layouts":"default","ast-page-background-enabled":"default","ast-page-background-meta":{"desktop":{"background-color":"var(--ast-global-color-4)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"ast-content-background-meta":{"desktop":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"footnotes":""},"categories":[4],"tags":[],"class_list":["post-2549","post","type-post","status-publish","format-standard","hentry","category-docker"],"_links":{"self":[{"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/posts\/2549","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/types\/post"}],"replies":[{"embeddable":true,"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/comments?post=2549"}],"version-history":[{"count":0,"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/posts\/2549\/revisions"}],"wp:attachment":[{"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/media?parent=2549"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/categories?post=2549"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/tags?post=2549"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}