{"id":2590,"date":"2025-10-14T00:16:25","date_gmt":"2025-10-14T00:16:25","guid":{"rendered":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/2025\/10\/14\/docker-model-runner-on-the-new-nvidia-dgx-spark-a-new-paradigm-for-developing-ai-locally\/"},"modified":"2025-10-14T00:16:25","modified_gmt":"2025-10-14T00:16:25","slug":"docker-model-runner-on-the-new-nvidia-dgx-spark-a-new-paradigm-for-developing-ai-locally","status":"publish","type":"post","link":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/2025\/10\/14\/docker-model-runner-on-the-new-nvidia-dgx-spark-a-new-paradigm-for-developing-ai-locally\/","title":{"rendered":"Docker Model Runner on the new NVIDIA DGX Spark: a new paradigm for developing AI locally"},"content":{"rendered":"

We\u2019re thrilled to bring NVIDIA DGX\u2122 Spark support to Docker Model Runner<\/a>. The new NVIDIA DGX Spark delivers incredible performance, and Docker Model Runner makes it accessible. With Model Runner, you can easily run and iterate on larger models right on your local machine, using the same intuitive Docker experience you already trust.<\/p>\n

In this post, we\u2019ll show how DGX Spark and Docker Model Runner work together to make local model development faster and simpler, covering the unboxing experience, how to set up Model Runner, and how to use it in real-world developer workflows.<\/p>\n

What is NVIDIA DGX Spark<\/h2>\n

NVIDIA DGX Spark is the newest member of the DGX family: a compact, workstation-class AI system, powered by the Grace Blackwell GB10 Superchip\u00a0 that delivers incredible\u00a0 performance for local model development. Designed for researchers and developers, it makes prototyping, fine-tuning, and serving large models fast and effortless, all without relying on the cloud.<\/p>\n

Here at Docker, we were fortunate to get a preproduction version of\u00a0 DGX Spark. And yes, it\u2019s every bit as impressive in person as it looks in NVIDIA\u2019s launch materials.<\/p>\n\n

Why Run Local AI Models and How Docker Model Runner and NVIDIA DGX Spark Make It Easy\u00a0<\/h2>\n

Many of us at Docker and across the broader developer community are experimenting with local AI models. <\/strong>Running locally has clear advantages:<\/p>\n

Data privacy and control<\/strong>: no external API calls; everything stays on your machine<\/p>\n

Offline availability<\/strong>: work from anywhere, even when you\u2019re disconnected<\/p>\n

\u00a0Ease of customization<\/strong>: experiment with prompts, adapters, or fine-tuned variants without relying on remote infrastructure<\/p>\n

But there are also familiar tradeoffs:<\/p>\n

Local GPUs and memory can be limiting for large models<\/p>\n

Setting up CUDA, runtimes, and dependencies often eats time<\/p>\n

Managing security and isolation for AI workloads can be complex<\/p>\n

This is where DGX Spark and Docker Model Runner (DMR) shine. DMR provides an easy and secure way to run AI models in a sandboxed environment, fully integrated with Docker Desktop or Docker Engine. When combined with the DGX Spark\u2019s NVIDIA AI software stack and large 128GB unified memory, you get the best of both worlds: plug-and-play GPU acceleration and Docker-level simplicity.<\/p>\n\n

Unboxing NVIDIA DGX Spark<\/h2>\n

The device arrived well-packaged, sleek, and surprisingly small, resembling more a mini-workstation than a server.<\/p>\n

Setup was refreshingly straightforward: plug in power, network, and peripherals, then boot into NVIDIA DGX OS, which includes NVIDIA drivers, CUDA, and AI software stack pre-installed.<\/p>\n

<\/div>\n

Once on the network, enabling SSH access makes it easy to integrate the Spark into your existing workflow.<\/p>\n

This way, the DGX Spark becomes an AI co-processor for your everyday development environment, augmenting, not replacing, your primary machine.<\/p>\n

Getting Started with Docker Model Runner on NVIDIA DGX Spark<\/h2>\n

Installing Docker Model Runner on the DGX Spark is simple and can be done in a matter of minutes.<\/p>\n

1. Verify Docker CE is Installed<\/h3>\n

DGX OS comes with Docker Engine (CE) preinstalled. Confirm you have it:<\/p>\n

\ndocker version\n<\/div>\n

If it\u2019s missing or outdated, install following the regular Ubuntu installation instructions<\/a>.<\/p>\n\n

2. Install the Docker Model CLI Plugin<\/h3>\n

The Model Runner CLI is distributed as a Debian package via Docker\u2019s apt repository. Once the repository is configured (see linked instructions above) install via the following commands:<\/p>\n

\nsudo apt-get update
\nsudo apt-get apt-get install docker-model-plugin\n<\/div>\n

Or use Docker\u2019s handy installation script:<\/p>\n

\ncurl -fsSL https:\/\/get.docker.com | sudo bash\n<\/div>\n

You can confirm it\u2019s installed with:<\/p>\n

\ndocker model version\n<\/div>\n

3. Pull and Run a Model<\/h3>\n

Now that the plugin is installed, let\u2019s pull a model from the Docker Hub AI Catalog<\/a>. For example, the Qwen 3 Coder <\/strong>model:<\/p>\n

\ndocker model pull ai\/qwen3-coder\n<\/div>\n

The Model Runner container will automatically expose an OpenAI-compatible endpoint at:<\/p>\n

\nhttp:\/\/localhost:12434\/engines\/v1\n<\/div>\n

You can verify it\u2019s live with a quick test:<\/p>\n

\n# Test via API\n

curl http:\/\/localhost:12434\/engines\/v1\/chat\/completions \u00a0 -H ‘Content-Type: application\/json’ \u00a0 -d
\n‘{“model”:”ai\/qwen3-coder”,”messages”:[{“role”:”user”,”content”:”Hello!”}]}’<\/p>\n

# Or via CLI
\ndocker model run ai\/qwen3-coder\n<\/p><\/div>\n\n

GPUs are allocated to the Model Runner container via <\/strong>nvidia-container-runtime and the Model Runner will take advantage of any available GPUs automatically. To see GPU usage:<\/p>\n

\nnvidia-smi\n<\/div>\n

4. Architecture Overview<\/h3>\n

Here\u2019s what\u2019s happening under the hood:<\/p>\n

\n[ DGX Spark Hardware (GPU + Grace CPU) ]\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u2502<\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0(NVIDIA Container Runtime)<\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u2502<\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0[ Docker Engine (CE) ]<\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u2502<\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0[ Docker Model Runner Container ]<\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u2502<\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0OpenAI-compatible API :12434\n<\/p><\/div>\n\n

The NVIDIA Container Runtime bridges the NVIDIA GB10 Grace Blackwell Superchip drivers and Docker Engine, so containers can access CUDA directly. Docker Model Runner then runs inside its own container, managing the model lifecycle and providing the standard OpenAI API endpoint. (For more info on Model Runner architecture, see this blog<\/a>).<\/p>\n

From a developer\u2019s perspective, interact with models similarly to any other Dockerized service \u2014 docker model pull, list, inspect, and run all work out of the box.<\/p>\n

Using Local Models in Your Daily Workflows<\/h2>\n

If you\u2019re using a laptop or desktop as your primary machine, the DGX Spark can act as your remote model host<\/strong>. With a few SSH tunnels, you can both access the Model Runner API and monitor GPU utilization via the DGX dashboard, all from your local workstation.<\/p>\n

1. Forward the DMR Port (for Model Access)<\/h3>\n

To access the DGX Spark via SSH first set up an SSH server:<\/p>\n

Using Local Models in Your Daily Workflows
If you\u2019re using a laptop or desktop as your primary machine, the DGX Spark can act as your remote model host<\/strong>. With a few SSH tunnels, you can both access the Model Runner API and monitor GPU utilization via the DGX dashboard, all from your local workstation.<\/p>\n

\nsudo apt install openssh-server
\nsudo systemctl enable –now ssh\n<\/div>\n

Run the following command to access Model Runner via your local machine. Replace user with the username you configured when you first booted the DGX Spark and replace dgx-spark.local with the IP address of the DGX Spark on your local network or a hostname configured in \/etc\/hosts.\u00a0<\/strong><\/p>\n\n

\nssh -N -L localhost:12435:localhost:12434 user@dgx-spark.local\n<\/div>\n

This forwards the Model Runner API from the DGX Spark to your local machine.
Now, in your IDE, CLI tool, or app that expects an OpenAI-compatible API, just point it to:<\/p>\n

\nhttp:\/\/localhost:12435\/engines\/v1\/models\n<\/div>\n

Set the model name (e.g. ai\/qwen3-coder) and you\u2019re ready to use local inference seamlessly.<\/p>\n\n

2. Forward the DGX Dashboard Port (for Monitoring)<\/h3>\n

The DGX Spark exposes a lightweight browser dashboard showing real-time GPU, memory, and thermal stats, usually served locally at:<\/p>\n

\nhttp:\/\/localhost:11000\n<\/div>\n\n

You can forward it through the same SSH session or a separate one:<\/p>\n

\nssh -N -L localhost:11000:localhost:11000 user@dgx-spark.local\n<\/div>\n

Then open http:\/\/localhost:11000<\/a> in your browser on your main workstation to monitor the DGX Spark performance while running your models.<\/p>\n

<\/div>\n

This combination makes the DGX Spark feel like a remote, GPU-powered extension of your development environment. Your IDE or tools still live on your laptop, while model execution and resource-heavy workloads happen securely on the Spark.<\/p>\n

Example application: Configuring Opencode with Qwen3-Coder<\/h2>\n

Let\u2019s make this concrete.<\/p>\n

Suppose you use OpenCode<\/a>, an open-source, terminal-based AI coding agent.<\/p>\n

Once your DGX Spark is running Docker Model Runner with ai\/qwen3-coder pulled and the port is forwarded, you can configure OpenCode by adding the following to ~\/.config\/opencode\/opencode.json<\/p>\n

\n{
\n\u00a0\u00a0“$schema”: “https:\/\/opencode.ai\/config.json”,
\n\u00a0\u00a0“provider”: {
\n\u00a0\u00a0\u00a0\u00a0“dmr”: {
\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0“npm”: “@ai-sdk\/openai-compatible”,
\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0“name”: “Docker Model Runner”,
\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0“options”: {
\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0“baseURL”: “http:\/\/localhost:12435\/engines\/v1” \u00a0 \/\/ DMR\u2019s OpenAI-compatible base
\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0},
\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0“models”: {
\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0“ai\/qwen3-coder”: { “name”: “Qwen3 Coder” }
\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0}
\n\u00a0\u00a0\u00a0\u00a0}
\n\u00a0\u00a0},
\n\u00a0\u00a0“model”: “ai\/qwen3-coder”
\n}\n<\/div>\n

Now run opencode and select Qwen3 Coder with the \/models command:<\/p>\n

<\/div>\n

That\u2019s it! Completions and chat requests will be routed through Docker Model Runner on your DGX Spark, meaning Qwen3-Coder now powers your agentic development experience locally.<\/p>\n

<\/div>\n

You can verify that the model is running by opening http:\/\/localhost:11000<\/a> (the DGX dashboard) to watch GPU utilization in real time while coding.
This setup lets you:<\/p>\n

Keep your laptop light while leveraging the DGX Spark GPUs<\/p>\n

Experiment with custom or fine-tuned models through DMR<\/p>\n

Stay fully within your local environment for privacy and cost-control<\/p>\n

Summary<\/h2>\n

Running Docker Model Runner on the NVIDIA DGX Spark makes it remarkably easy to turn powerful local hardware into a seamless extension of your everyday Docker workflow.<\/p>\n

You install one plugin and use familiar Docker commands (docker model pull, docker model run).
You get full GPU acceleration through NVIDIA\u2019s container runtime.
You can forward both the model API and monitoring dashboard to your main workstation for effortless development and visibility.<\/p>\n

This setup bridges the gap between developer productivity and AI infrastructure, giving you the speed, privacy, and flexibility of local execution with the reliability and simplicity Docker provides.<\/p>\n

As local model workloads continue to grow, the DGX Spark + Docker Model Runner combo represents a practical, developer-friendly way to bring serious AI compute to your desk \u2014 no data center or cloud dependency required.<\/p>\n

Learn more<\/strong>:<\/p>\n

Read the official announcement of DGX Spark launch on NVIDIA newsroom<\/a><\/p>\n

Check out the Docker Model Runner General Availability announcement<\/a><\/p>\n

Visit our Model Runner GitHub repo<\/a>. Docker Model Runner is open-source, and we welcome collaboration and contributions from the community!\u00a0Star, fork and contribute.<\/p>","protected":false},"excerpt":{"rendered":"

We\u2019re thrilled to bring NVIDIA DGX\u2122 Spark support to Docker Model Runner. The new NVIDIA DGX Spark delivers incredible performance, […]<\/p>\n","protected":false},"author":0,"featured_media":0,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"site-sidebar-layout":"default","site-content-layout":"","ast-site-content-layout":"default","site-content-style":"default","site-sidebar-style":"default","ast-global-header-display":"","ast-banner-title-visibility":"","ast-main-header-display":"","ast-hfb-above-header-display":"","ast-hfb-below-header-display":"","ast-hfb-mobile-header-display":"","site-post-title":"","ast-breadcrumbs-content":"","ast-featured-img":"","footer-sml-layout":"","ast-disable-related-posts":"","theme-transparent-header-meta":"","adv-header-id-meta":"","stick-header-meta":"","header-above-stick-meta":"","header-main-stick-meta":"","header-below-stick-meta":"","astra-migrate-meta-layouts":"default","ast-page-background-enabled":"default","ast-page-background-meta":{"desktop":{"background-color":"var(--ast-global-color-4)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"ast-content-background-meta":{"desktop":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"footnotes":""},"categories":[4],"tags":[],"class_list":["post-2590","post","type-post","status-publish","format-standard","hentry","category-docker"],"_links":{"self":[{"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/posts\/2590","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/types\/post"}],"replies":[{"embeddable":true,"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/comments?post=2590"}],"version-history":[{"count":0,"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/posts\/2590\/revisions"}],"wp:attachment":[{"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/media?parent=2590"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/categories?post=2590"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/tags?post=2590"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}