As a Senior DevOps Engineer and Docker Captain, I\u2019ve helped build AI systems for everything from retail personalization to medical imaging. One truth stands out: AI capabilities are core to modern infrastructure.<\/p>\n
This guide will show you how to run and package local AI models with Docker Model Runner<\/a> \u2014 a lightweight, developer-friendly tool for working with AI models pulled from Docker Hub or Hugging Face. You\u2019ll learn how to run models in the CLI or via API, publish your own model artifacts, and do it all without setting up Python environments or web servers.<\/p>\n Artificial Intelligence (AI) refers to systems that mimic human intelligence, including:<\/p>\n Making decisions via machine learning<\/p>\n Understanding language through NLP<\/p>\n Recognizing images with computer vision<\/p>\n Learning from new data automatically<\/p>\n Machine Learning (ML)<\/strong>: Learns from structured and unstructured data<\/p>\n Deep Learning<\/strong>: Neural networks for pattern recognition<\/p>\n Natural Language Processing (NLP)<\/strong>: Understands\/generates human language<\/p>\n Computer Vision<\/strong>: Recognizes and interprets images<\/p>\n Local model packaging and execution offer full control over your AI workflows. Instead of relying on external APIs, you can run models directly on your machine \u2014 unlocking:<\/p>\n Faster inference with local compute (no latency from API calls)<\/p>\n Greater privacy by keeping data and prompts on your own hardware<\/p>\n Customization through packaging and versioning your own models<\/p>\n Seamless CI\/CD integration with tools like Docker and GitHub Actions<\/p>\n Offline capabilities for edge use cases or constrained environments<\/p>\n Platforms like Docker and Hugging Face make cutting-edge AI models instantly accessible without building from scratch. Running them locally means lower latency, better privacy, and faster iteration.<\/p>\n Chatbots & Virtual Assistants<\/strong>: Automate support (e.g., ChatGPT, Alexa)<\/p>\n Generative AI<\/strong>: Create text, art, music (e.g., Midjourney, Lensa)<\/p>\n Dev Tools<\/strong>: Autocomplete and debug code (e.g., GitHub Copilot)<\/p>\n Retail Intelligence<\/strong>: Recommend products based on behavior<\/p>\n Medical Imaging<\/strong>: Analyze scans for faster diagnosis<\/p>\n Docker Desktop<\/a> 4.40+ installed<\/p>\n Experimental features and Model Runner enabled<\/a> in Docker Desktop settings<\/p>\n (Recommended) Windows 11 with NVIDIA GPU or Mac with Apple Silicon<\/p>\n Internet access for downloading models from Docker Hub<\/a> or Hugging Face<\/p>\n Open Docker Desktop<\/p>\n Go to Settings \u2192 Features in development<\/p>\n Under the Experimental features tab, enable Access experimental features<\/p>\n Click Apply and restart<\/p>\n Quit and reopen Docker Desktop to ensure changes take effect<\/p>\n Reopen Settings \u2192 Features in development<\/p>\n Switch to the Beta tab and check Enable Docker Model Runner<\/strong><\/p>\n (Optional)<\/em> Enable host-side TCP support to access the API from localhost<\/p>\n Once enabled, you can use the docker model CLI and manage models in the Models tab.<\/strong><\/p>\n Screenshot of Docker Desktop\u2019s Features in development tab with Docker Model Runner and Dev Environments enabled.<\/em><\/p>\n\n From Docker Hub:<\/p>\n Or from Hugging Face (GGUF format):<\/p>\n Note:<\/strong> Only GGUF models<\/strong> are supported. GGUF (GPT-style General Use Format) is a lightweight binary file format designed for efficient local inference, especially with CPU-optimized runtimes like llama.cpp. It includes the model weights, tokenizer, and metadata all in one place, making it ideal for packaging and distributing LLMs in containerized environments.<\/p>\n If you want to push models to a private or local registry:<\/p>\n Tag model with your registry\u2019s address:<\/p>\n Run a local Docker registry:<\/p>\n Push the model to the local registry:<\/p>\n Check your local models with:<\/p>\n Run a prompt (one-shot)<\/p>\n Interactive chat mode<\/p>\n Note: <\/strong>Models are loaded into memory on demand and unloaded after 5 minutes of inactivity.<\/p>\n To call the model from the host:<\/p>\n Enable TCP host access<\/strong> for Model Runner (via Docker Desktop GUI or CLI)<\/p>\n Screenshot of Docker Desktop\u2019s Features in development tab showing host-side TCP support enabled for Docker Model Runner.<\/em><\/p>\n Send a prompt using the OpenAI-compatible chat endpoint:<\/p>\n Note:<\/strong> No API key required \u2014 this runs locally and securely on your machine.<\/p>\n You can package your own pre-trained GGUF model<\/strong> as a Docker-compatible artifact if you already have a .gguf file \u2014 such as one downloaded from Hugging Face or converted using tools like llama.cpp.<\/p>\n Note: <\/strong>This guide assumes you already have a .gguf model file. It does not cover how to train or convert models to GGUF.<\/em><\/p>\n This is ideal for custom-trained or private models. You can now pull it like any other model:<\/p>\n Use docker model logs to monitor model usage and debug issues<\/p>\n Set up CI\/CD to automate pulls, scans, and packaging<\/p>\n Track model lineage and training versions to ensure consistency<\/p>\n Use semantic versioning (:v1, :2025-05, etc.) instead of latest when packaging custom models<\/p>\n Only one model can be loaded at a time; requesting a new model will unload the previous one.<\/p>\n Docker Compose v2.35+ (included in Docker Desktop 4.41+) introduces support for AI model services using a new provider.type: model. You can define models directly in your compose.yml and reference them in app services using depends_on.<\/p>\n During docker compose up, Docker Model Runner automatically pulls the model and starts it on the host system, then injects connection details into dependent services using environment variables such as MY_MODEL_URL and MY_MODEL_MODEL, where MY_MODEL matches the name of the model service.<\/p>\n This enables seamless multi-container AI applications \u2014 with zero extra glue code. Learn more<\/a>.<\/p>\n Latency<\/strong>: Use quantized GGUF models<\/p>\n Security<\/strong>: Never run unknown models; validate sources and attach licenses<\/p>\n Compliance<\/strong>: Mask PII, respect data consent<\/p>\n Costs<\/strong>: Run locally to avoid cloud compute bills<\/p>\n Prefer GGUF models for optimal CPU inference<\/p>\n Use the –license flag when packaging custom models to ensure compliance<\/p>\n Use versioned tags (e.g., :v1, :2025-05) instead of latest<\/p>\n Monitor model logs using docker model logs<\/p>\n Validate model sources before pulling or packaging<\/p>\n Only pull models from trusted sources (e.g., Docker Hub\u2019s ai\/ namespace or verified Hugging Face repos).<\/p>\n Review the license and usage terms for each model before packaging or deploying.<\/p>\n Support for Retrieval-Augmented Generation (RAG)<\/p>\n Expanded multimodal support (text + images, video, audio)<\/p>\n LLMs as services in Docker Compose (Requires Docker Compose v2.35+)<\/em> More granular Model Dashboard features in Docker Desktop<\/p>\n Secure packaging and deployment pipelines for private AI models<\/p>\n Docker Model Runner lets DevOps teams treat models like any other artifact \u2014 pulled, tagged, versioned, tested, and deployed.<\/p>\n\n You don\u2019t need a GPU cluster or external API to build AI apps. Learn more and explore everything you can do with Docker Model Runner<\/a>:<\/p>\n Pull prebuilt models from Docker Hub or Hugging Face<\/p>\n Run them locally using the CLI, API, or Docker Desktop\u2019s Model tab<\/strong> Package and push your own models as OCI artifacts<\/p>\n Integrate with your CI\/CD pipelines securely<\/p>\n You can also find other helpful information to get started at:\u00a0<\/p>\n Docker Hub \u2013 AI Namespace\u00a0<\/a><\/p>\n Docker Model Runner Docs\u00a0<\/a><\/p>\n OpenAI-Compatible API Guide<\/a><\/p>\n hello-genai Sample App<\/a><\/p>\n You\u2019re not just deploying containers \u2014 you\u2019re delivering intelligence.<\/strong><\/p>\n\n Read our quickstart guide to Docker Model Runner<\/a>.<\/p>\n Find documentation for Model Runner<\/a>.<\/p>\n Subscribe to the Docker Navigator Newsletter<\/a>.<\/p>\n New to Docker? Create an account<\/a>.\u00a0<\/p>\nWhat is AI in Development?<\/h2>\n
Common Types of AI in Development:<\/h3>\n
Why Package and Run Your Own AI Model?<\/h2>\n
Real-World Use Cases for AI<\/h2>\n
How to Package and Run AI Models Locally with Docker Model Runner<\/h2>\n
Prerequisites:<\/h3>\n
Step 0 \u2014 Enable Docker Model Runner<\/h3>\n
Step 1: Pull a Model<\/h3>\n
Step 2: Tag and Push to Local Registry (Optional)<\/h3>\n
Step 3: Run the Model<\/h3>\n
Step 4: Test via OpenAI-Compatible API<\/h3>\n
\n -H “Content-Type: application\/json”
\n -d ‘{
\n “model”: “ai\/smollm2”,
\n “messages”: [
\n {“role”: “system”, “content”: “You are a helpful assistant.”},
\n {“role”: “user”, “content”: “Tell me about the fall of Rome.”}
\n ]
\n }’\n<\/div>\nStep 5: Package Your Own Model<\/h3>\n
\n –gguf “$(pwd)\/model.gguf”
\n –license “$(pwd)\/LICENSE.txt”
\n –push registry.example.com\/ai\/custom-llm:v1\n<\/div>\nStep 6: Optimize & Iterate<\/h3>\n
Compose Integration (Optional)<\/h3>\n
Navigating AI Development Challenges<\/h2>\n
Best Practices<\/h2>\n
The Road Ahead<\/h2>\n
<\/em><\/p>\nFinal Thoughts<\/h2>\n
<\/strong><\/p>\nLearn more<\/h3>\n