{"id":2506,"date":"2025-09-19T12:16:39","date_gmt":"2025-09-19T12:16:39","guid":{"rendered":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/2025\/09\/19\/beyond-containers-llama-cpp-now-pulls-gguf-models-directly-from-docker-hub\/"},"modified":"2025-09-19T12:16:39","modified_gmt":"2025-09-19T12:16:39","slug":"beyond-containers-llama-cpp-now-pulls-gguf-models-directly-from-docker-hub","status":"publish","type":"post","link":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/2025\/09\/19\/beyond-containers-llama-cpp-now-pulls-gguf-models-directly-from-docker-hub\/","title":{"rendered":"Beyond Containers: llama.cpp Now Pulls GGUF Models Directly from Docker Hub"},"content":{"rendered":"

The world of local AI is moving at an incredible pace, and at the heart of this revolution is llama.cpp\u2014the powerhouse C++ inference engine that brings Large Language Models (LLMs) to everyday hardware (and it\u2019s also the inference engine that powers Docker Model Runner<\/a>). Developers love llama.cpp for its performance and simplicity. And we at Docker are obsessed with making developer workflows simpler.<\/p>\n

That\u2019s why we\u2019re thrilled to announce a game-changing new feature in llama.cpp: native support for pulling and running GGUF models directly from Docker Hub.<\/p>\n

This isn\u2019t about running llama.cpp in<\/em> a Docker container. This is about using Docker Hub as a powerful, versioned, and centralized repository for your AI models, just like you do for your container images.<\/p>\n

Why Docker Hub for AI Models?<\/strong><\/h2>\n

Managing AI models can be cumbersome. You\u2019re often dealing with direct download links, manual version tracking, and scattered files. By integrating with Docker Hub, llama.cpp leverages a mature and robust ecosystem to solve these problems.<\/p>\n

Rock-Solid Versioning<\/strong>: The familiar repository:tag syntax you use for images now applies to models. Easily switch between gemma3 and smollm2:135M-Q4_0 with complete confidence.<\/p>\n

Centralized & Discoverable<\/strong>: Docker Hub can become the canonical source for your team\u2019s models. No more hunting for the \u201clatest\u201d version on a shared drive or in a chat history.<\/p>\n

Simplified Workflow<\/strong>: Forget curl, wget or manually downloading from web UIs. A single command-line flag now handles discovery, download, and caching.<\/p>\n

Reproducibility<\/strong>: By referencing a model with its immutable digest or tag, you ensure that your development, testing, and production environments are all using the exact same artifact, leading to more consistent and reproducible results.<\/p>\n

How It Works Under the Hood\u00a0<\/strong><\/h2>\n

This new feature cleverly uses the Open Container Initiative (OCI) specification, which is the foundation of Docker images. The GGUF model file is treated as a layer within an OCI manifest, identified by a special media type like application\/vnd.docker.ai.gguf.v3. For more details on why the OCI standard matters for models, check out our blog<\/a>.<\/p>\n

When you use the new \u2013docker-repo flag, llama.cpp performs the following steps:<\/p>\n

Authentication<\/strong>: It first requests an authentication token from the Docker registry to authorize the download.<\/p>\n

Manifest Fetch<\/strong>: It then fetches the manifest for the specified model and tag (e.g., ai\/gemma3:latest).<\/p>\n

Layer Discovery<\/strong>: It parses the manifest to find the specific layer that contains the GGUF model file by looking for the correct media type.<\/p>\n

Blob Download<\/strong>: Using the layer\u2019s unique digest (a sha256 hash), it downloads the model file directly from the registry\u2019s blob storage.<\/p>\n

Caching<\/strong>: The model is saved to a local cache, so subsequent runs are instantaneous.<\/p>\n

This entire process is seamless and happens automatically in the background.<\/p>\n

Get Started in Seconds<\/strong><\/h2>\n

Ready to try it? If you have a recent build<\/a> of llama.cpp, you can serve a model from Docker Hub with one simple command. The new flag is \u2013docker-repo (or -dr).<\/p>\n

Let\u2019s run gemma3<\/a>, a model available from Docker Hub.<\/p>\n

\n# Now, serve a model from Docker Hub!
\nllama-server -dr gemma3\n<\/div>\n

The first time you execute this, you\u2019ll see llama.cpp log the download progress. After that, it will use the cached version. It\u2019s that easy! The default organization is ai\/, so gemma3 is resolved to ai\/gemma3. The default tag is :latest, but a tag can be specified like :1B-Q4_K_M.<\/p>\n

For a complete Docker-integrated experience with OCI pushing and pulling support try out Docker Model Runner. The docker model runner equivalent for chatting is:<\/p>\n

\n# Pull, serve and chat to a model from Docker Hub!
\ndocker model run ai\/gemma3\n<\/div>\n

The Future of AI Model Distribution<\/strong><\/h2>\n

This integration represents a powerful shift in how we think about distributing and managing AI artifacts. By using OCI-compliant registries like Docker Hub, the AI community can build more robust, reproducible, and scalable MLOps pipelines.<\/p>\n

This is just the beginning. We envision a future where models, datasets, and the code that runs them are all managed through the same streamlined, developer-friendly workflow that has made Docker an essential tool for millions.<\/p>\n

Check out the latest llama.cpp to try it out, and explore the growing collection of models on Docker Hub<\/a> today!<\/p>\n

Learn more<\/h3>\n

Read our quickstart guide to Docker Model Runner<\/a>.<\/p>\n

Visit our Model Runner GitHub repo<\/a>! Docker Model Runner is open-source, and we welcome collaboration and contributions from the community!<\/p>\n

Discover curated models on Docker Hub<\/a><\/p>","protected":false},"excerpt":{"rendered":"

The world of local AI is moving at an incredible pace, and at the heart of this revolution is llama.cpp\u2014the […]<\/p>\n","protected":false},"author":0,"featured_media":0,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"site-sidebar-layout":"default","site-content-layout":"","ast-site-content-layout":"default","site-content-style":"default","site-sidebar-style":"default","ast-global-header-display":"","ast-banner-title-visibility":"","ast-main-header-display":"","ast-hfb-above-header-display":"","ast-hfb-below-header-display":"","ast-hfb-mobile-header-display":"","site-post-title":"","ast-breadcrumbs-content":"","ast-featured-img":"","footer-sml-layout":"","ast-disable-related-posts":"","theme-transparent-header-meta":"","adv-header-id-meta":"","stick-header-meta":"","header-above-stick-meta":"","header-main-stick-meta":"","header-below-stick-meta":"","astra-migrate-meta-layouts":"default","ast-page-background-enabled":"default","ast-page-background-meta":{"desktop":{"background-color":"var(--ast-global-color-4)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"ast-content-background-meta":{"desktop":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"footnotes":""},"categories":[4],"tags":[],"class_list":["post-2506","post","type-post","status-publish","format-standard","hentry","category-docker"],"_links":{"self":[{"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/posts\/2506","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/types\/post"}],"replies":[{"embeddable":true,"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/comments?post=2506"}],"version-history":[{"count":0,"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/posts\/2506\/revisions"}],"wp:attachment":[{"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/media?parent=2506"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/categories?post=2506"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/tags?post=2506"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}